Head-tracking based media selection for video communications in virtual environments

A system for video communications, comprising:

• at least one computer of a computer system with memory and at least one processor implementing a three-dimensional virtual environment configured to be accessed by a plurality of client devices, each having a corresponding user graphical representation within the three-dimensional virtual environment, each user graphical representation having a corresponding virtual camera comprising a view of the three-dimensional virtual environment presented through a display to the corresponding users of the client devices;
• a data exchange management module implemented in memory and configured to receive head tracking metadata of a first client device that comprises six degrees of freedom head tracking information generated in response to tracking movement of key facial landmarks obtained from video of a face of a first user of the first client device, wherein a first virtual camera comprises a view of the three-dimensional virtual environment presented to the first user;
• and
• a machine vision module implemented in memory and configured to identify graphical elements within a field of view of a second virtual camera comprising a view of the three-dimensional virtual environment presented to a second user of at least one second client device, wherein the identified graphical elements comprise a user graphical representation of the first user, wherein the machine vision module is further configured to send to the second client device, via the data exchange management module, the head tracking metadata of the first client device and the identified graphical elements, and wherein the system enables exchanging video and audio data between users of the three-dimensional virtual environment to provide video communications in the three-dimensional virtual environment;
• wherein the machine vision module is further configured to track movement of the key facial landmarks in six degrees of freedom based on the movement of the head of the first user, and adjust the position and orientation of the first virtual camera based on the tracked movement of the key facial landmarks.

A non-transitory computer-readable medium having stored thereon instructions configured to cause at least one server computer comprising a processor and memory to perform steps comprising:

• implementing a three-dimensional virtual environment configured to be accessed by a plurality of client devices, each having a corresponding user graphical representation within the three-dimensional virtual environment, each user graphical representation having a corresponding virtual camera comprising a view of the three-dimensional implementing a three-dimensional virtual environment configured to be accessed by a plurality of client devices, each having a corresponding user graphical representation within the three-dimensional virtual environment, each user graphical representation having a corresponding virtual camera comprising a view of the three-dimensional virtual environment presented through a display to the corresponding users of the client devices, wherein the at least one server computer enables exchanging video and audio data between users of the three-dimensional virtual environment to provide video communications in the three-dimensional virtual environment;
• receiving head tracking metadata of a first client device that comprises head tracking information generated in response to tracking movement of key facial landmarks obtained from video of a face of a first user of the first client device;
• tracking movement of the key facial landmarks in the six degrees of freedom based on the movement of the head of the first user;
• adjusting the position and orientation of the first virtual camera based on the tracked movement of the key facial landmarks;
• identifying graphical elements within a field of view of a virtual camera of a second client device, wherein the identified graphical elements comprise a user graphical representation of the first user, and
• sending, to the second client device, the head tracking metadata of the first client device and the identified graphical elements.