Back to results
US 11,694,438 B2Granted
Scalable extended reality video conferencing
AssigneeCarnegie Mellon University
InventorsAnthony Rowe, Michael Farb, Ivan Liang, Edward Lu, Nuno Pereira, Eric Riebling
Life of this patent
Estimating the term…
Abstract
Some embodiments of the present inventive concept provide for improved telepresence and other virtual sessions dynamic scaling and/or assignment of computing resources. An XR telepresence platform can allow for immersive multi-user video conferencing from within a web browser or other medium. The platform can support spatial audio and/or user video. The platform can scale to hundreds or thousands of users concurrently in a single or multiple virtual environments. Disclosed herein are resource allocation techniques for dynamically allocating client connections across multiple servers.
Claims16
A method for allocating resources for an extended reality telepresence session, the method comprising:
- receiving an indication of a new client connection within an extended reality telepresence session, wherein the extended reality telepresence session comprises a plurality of client connections hosted by a plurality of servers, wherein the client connections are grouped into a plurality of conversation clusters based on virtual distances between client connections, wherein every client connection of a particular conversation cluster is hosted by a common server of the plurality of servers;
- determining, based on a virtual location of the new client connection, that the new client connection should be part of a first conversation cluster of the plurality of conversation clusters, wherein the first conversation cluster and a second conversation cluster are hosted by a first server of the plurality of servers, wherein the first conversation cluster includes a first group of client connections;
- based on a determination that the first server is unavailable to support additional client connections, generating a first server allocation map, wherein the first server allocation map indicates an assignment of the first group of client connections and the second conversation cluster to the first server and indicates an assignment of the first group of client connections and the new client connection to a second server;
- allocating the plurality of client connections to the plurality of servers based on the first server allocation map;
- generating a second server allocation map, wherein the second server allocation map indicates an assignment of the second conversation cluster to the first server and an assignment of the first group of client connections and the new client connection to the second server, wherein the second server allocation map indicates no assignments of the first group of client connections or the new client connection to the first server;
- and
- based on a map transition policy, transitioning from the first server allocation map to the second server allocation map.
4 more elements
A computing system of a telepresence management system, the computing system comprising:
- memory;
- and
- one or more processors coupled to the memory and configured to:
- receive an indication that a first client connection in combination with a set of client connections forms a first conversation cluster within an extended reality telepresence session, wherein the extended reality telepresence session comprises a plurality of client connections hosted by a plurality of servers, wherein the plurality of client connections includes the first client connection and the set of client connections;
- generate a first server allocation map, wherein the first server allocation map indicates an assignment of at least one server of the plurality of servers to each client connection of the plurality of client connections, wherein the first server allocation map indicates an assignment of more servers of the plurality of servers to other client connections of the first conversation cluster than the first client connection;
- allocate the plurality of client connections to the plurality of servers based on the first server allocation map;
- and
- based on a map transition policy, transition from the first server allocation map to a second server allocation map, wherein based on the transitioning, the plurality of client connections are allocated to the plurality of servers based on the second server allocation map, wherein the second server allocation map indicates an assignment of fewer servers of the plurality of servers to the other client connections of the first conversation cluster than the first server allocation map,
- wherein a first server generates an audio stream for a first avatar based on spatial relationships between the first avatar and other avatars in a telepresence environment of the extended reality telepresence session, generates a video stream for the first avatar based on a field-of-view of the first avatar;
- and communicates the audio stream and video stream to the first client connection, wherein the first client connection is associated with the first avatar.
7 more elements
Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
- receive an indication that a first client connection in combination with a set of client connections forms a first conversation cluster within an extended reality telepresence session, wherein the extended reality telepresence session comprises a plurality of client connections hosted by a plurality of servers, wherein the plurality of client connections includes the first client connection and the set of client connections;
- generate a first server allocation map, wherein the first server allocation map indicates an assignment of at least one server of the plurality of servers to each client connection of the plurality of client connections, wherein the first server allocation map indicates an assignment of more servers of the plurality of servers to other client connections of the first conversation cluster than the first client connection;
- allocate the plurality of client connections to the plurality of servers based on the first server allocation map;
- and
- based on a map transition policy, transition from the first server allocation map to a second server allocation map, wherein based on the transitioning, the plurality of client connections is allocated to the plurality of servers based on the second server allocation map, wherein the second server allocation map allocates a first server to the first client connection and indicates an assignment of fewer servers of the plurality of servers to the other client connections of the first conversation cluster than the first server allocation map, wherein the first server:
- determines spatial relationships between at least some virtual avatars of a collection of virtual avatars in the extended reality telepresence session, wherein the extended reality telepresence session is an immersive extended reality 3D environment allowing real-time audiovisual interactions between two or more virtual avatars of the collection of virtual avatars;
- identifies a field-of-view region in a telepresence environment that corresponds to a field-of-view region of a first virtual avatar of the collection of virtual avatars;
- dynamically generates an audio stream for the first virtual avatar based at least in part on the spatial relationships between the first virtual avatar and other virtual avatars of the collection of virtual avatars;
- dynamically generates a video stream for the first virtual avatar based at least in part on the field-of-view region and the spatial relationships between the first virtual avatar and other virtual avatars of the collection of virtual avatars;
- and
- communicates the audio stream and video stream to the first client connection, wherein the first client connection is associated with the first virtual avatar, wherein an audio device produces audio associated with the audio stream, and wherein a display device displays a video image associated with the video stream.
8 more elements
Dependencies are inferred from the claim text.