Bilateral exchange of user attention stream for foveated 3D communication streaming

A three-dimensional (3D) communication device comprising:

• a display screen configured to present a 3D representation of a first scene;
• a communication interface;
• an array of speakers configured to present 3D audio;
• and
• a processor communicatively coupled to the display screen and the communication interface, the processor configured to perform a process comprising:
• communicating, by way of the communication interface, with an additional 3D communication device to:
• transmit a first attention stream that indicates where attention of a first user of the 3D communication device is directed,
• receive a second attention stream indicating where attention of a second user of the additional 3D communication device is directed,
• transmit, based on the second attention stream, a first foveated image stream including a 3D representation of the first user,
• receive a second foveated image stream that is transmitted based on the first attention stream and includes a 3D representation of the second user, and
• receive a second foveated audio stream that is transmitted based on the first attention stream and includes a second 3D audio representation for a second scene in which the second user is located, the second 3D audio representation being indicative of first coordinates of a first sound source within the second scene, second coordinates of a second sound source within the second scene, and a ranking of the first sound source with respect to the second sound source;
• and
• presenting the second foveated image stream on the display screen and presenting the second foveated audio stream on the array of speakers in synchronization with the second foveated image stream, the presenting of the second foveated audio stream including simplifying a head-related transfer function (HRTF) convolution of at least one of the first sound source or the second sound source based on the first coordinates, the second coordinates, and the ranking.

A non-transitory computer-readable medium storing instructions that, when executed, cause a processor of a first three-dimensional (3D) communication device used by a first user to perform a process comprising:

• transmitting, to a second 3D communication device used by a second user, a first attention stream indicating where attention of the first user is directed;
• receiving, from the second 3D communication device, a second attention stream indicating where attention of the second user is directed;
• transmitting, to the second 3D communication device and based on the second attention stream, a first foveated image stream including a 3D representation of the first user;
• receiving, from the second 3D communication device:
• a second foveated image stream that is transmitted based on the first attention stream and includes a 3D representation of the second user, and
• a second foveated audio stream that is transmitted based on the first attention stream and includes a second 3D audio representation for a second scene in which the second user is located, the second 3D audio representation being indicative of first coordinates of a first sound source within the second scene, second coordinates of a second sound source within the second scene, and a ranking of the first sound source with respect to the second sound source;
• and
• presenting the second foveated image stream for viewing by the first user and the second foveated audio stream in synchronization with the second foveated image stream, the presenting of the second foveated audio stream including simplifying a head-related transfer function (HRTF) convolution of at least one of the first sound source or the second sound source based on the first coordinates, the second coordinates, and the ranking.