Multiuser asymmetric immersive teleconferencing with synthesized audio-visual feed

A system for enabling a teleconference among one or more host-side users located at a host site and one or more client-side users that are remotely located relative to the host site, wherein a host device is located at the host site, and wherein each of the one or more client-side users uses a respective client device to participate in the teleconference, the system comprising one or more processor configured to:

• receive from the host device a host site audio-visual feed that includes audio and video of the host site, captured from the host site by the host device;
• at least one of produce or update a three-dimensional (3D) model of the host site using the host site audio-visual feed;
• assign or otherwise obtain for each client-side user, of the one or more client-side users, a virtual location within the 3D model of the host site and thereby mapping each client-side user to a corresponding real-world location at the host site;
• allocate a respective virtual camera rendering engine to each of the one or more client-side users;
• use a said virtual camera rendering engine, allocated to producc for a said client-side user, to render a synthesized audio-visual feed that appears as if it were generated by a virtual camera located at the virtual location of the said client-side user within the 3D model of the host site;
• provide the synthesized audio-visual feed, or a modified version thereof, to the client device of the said client-side user to thereby enable consumption thereof by the said client-side user;
• receive, from the client device of the said client-side user, orientation information that includes an indication of a visual portion of the synthesized audio-visual feed, or the modified version thereof, that the said client-side user is viewing at any given time, which corresponds to less than a full field of view (FOV) provided by the synthesized audio- visual feed;
• and
• use the said virtual camera rendering engine, allocated to the said client-side user, to update the said synthesized audio-visual feed for the said client-side user, in response to at least one of the virtual location of the said client-side user changing or the orientation information for the said client-side user changing.

One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method for enabling a teleconference among one or more host-side users located at a host site and one or more client-side users that are remotely located relative to the host site, wherein a host device is located at the host site, and wherein each of the one or more client-side users uses a respective client device to participate in the teleconference, the method comprising:

• receiving from the host device a host site audio-visual feed that includes audio and video of the host site, captured from the host site by the host device;
• at least one of producing or updating a three-dimensional (3D) model of the host site using the host site audio-visual feed;
• assigning or otherwise obtaining for each client-side user, of the one or more client-side users, a virtual location within the 3D model of the host site and thereby mapping each client-side user to a corresponding real-world location at the host site;
• allocating a respective virtual camera rendering engine to each of the one or more client-side users;
• using a said virtual camera rendering engine, allocated to a said client-side user, to render a synthesized audio-visual feed that appears as if it were generated by a virtual camera located at the virtual location of the said client-side user within the 3D model of the host site;
• providing the synthesized audio-visual feed, or a modified version thereof, to the client device of the said client-side user to thereby enable consumption thereof by the said client-side user;
• receiving, from the client device of the said client-side user, orientation information that includes an indication of a visual portion of the synthesized audio-visual feed, or the modified version thereof, that the said client-side user is viewing at any given time, which corresponds to less than a full field of view (FOV) provided by the synthesized audio-visual feed;
• and
• using the said virtual camera rendering engine, allocated to the said client-side user, to update the said synthesized audio-visual feed for the said client-side user, in response to at least one of the virtual location of the said client-side user changing or the orientation information for the said client-side user changing.