# US 12,112,435 B1 · Collaboration spaces in extended reality conference sessions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,435 B1 - Document ID: US12112435B1 - Status: Granted - Kind: B1 - Application number: 17/246,434 - Filed: 2021-04-30 - Granted: 2024-10-08 - Earliest claimed: 2021-02-18 - Assignees: Splunk Inc. - Applicants: Splunk Inc. - Inventors: Devin Bhushan; Caelin Thomas Jackson-King; Stanislav Yazhenskikh; Jim Jiaming Zhu - Web page: https://patentgrep.com/patent/US12112435B1 ## CPC - G06F 16/74: Browsing; Visualisation therefor (end-user interfaces for requesting or interacting with video content, e.g. video on demand interfaces or electronic programme guides, H04N21/472) - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 7/13: Edge detection - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - H04L 65/1069: Session establishment or de-establishment - H04L 65/403: Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15) - H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)} - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2219/024: Multi-user, collaborative environment - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Extended reality (XR) software application programs establish remote collaboration sessions in which a host device and one or more remote devices can interact. When initiating a remote collaboration session, an XR application in a host device determines a collaboration area. The collaboration area corresponds to a portion of a real-world environment that is shared by the host device with the one or more remote devices. In some embodiments, the collaboration area can be determined automatically and/or based on user input. The XR application causes sensors associated with the host device to scan the collaboration area. Then, the XR application transmits, to the one or more remote devices, a three-dimensional representation of the collaboration area for rendering in one or more remote XR environments. ## Claims ### Claim 1 A computer-implemented method comprising: initiating a remote conference session; receiving, from one or more depth sensors associated with a host device, depth data associated with a three-dimensional (3D) environment; receiving, from one or more image sensors associated with the host device, image data associated with the 3D environment; determining a collaboration area around at least one automatically detected portion of a first user in the 3D environment, wherein the collaboration area is at least one of (i) a predefined 3D shape encompassing the at least one automatically detected portion of the first user, or (ii) determined based on a boundary of the at least one automatically detected portion of the first user, and wherein the at least one automatically detected portion can be moved or re-sized responsive to user interaction to control a size or location of the collaboration area; generating a first 3D representation of the at least one automatically detected portion of the first user based on the depth data, the image data, and the collaboration area; and transmitting the first 3D representation for rendering within a first extended reality (XR) environment associated with the remote conference session. ### Claim 2 (depends on 1) The method of claim 1, further comprising: receiving a second 3D representation of at least a portion of a second user; and rendering, within a second XR environment associated with the remote conference session, the second 3D representation of the at least a portion of the second user. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the collaboration area comprises: detecting, based on a machine learning model and at least one of the depth data or the image data, the boundary of the at least one automatically detected portion of the first user in the 3D environment; and generating the collaboration area based on the boundary. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the collaboration area comprises: detecting, based on a saliency model and at least one of the depth data or the image data, the boundary of the at least one automatically detected portion of the first user in the 3D environment; and generating the collaboration area based on the boundary. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the collaboration area comprises: determining, based on the depth data, a plane associated with the 3D environment; receiving, from the first user, a selection of a location; generating the collaboration area based on the location and the plane; receiving, via one or more input devices, one or more interactions by the first user; and modifying at least one of a size or a position of the collaboration area based on the one or more interactions. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the collaboration area comprises: detecting the at least one automatically detected portion of the first user in the image data; determining a mapping between the at least one automatically detected portion of the first user in the image data and the depth data; and generating the collaboration area based on the mapping. ### Claim 7 (depends on 1) The method of claim 1, wherein the collaboration area is a 3D region around the at least one automatically detected portion of the first user and one or more objects within the 3D environment. ### Claim 8 (depends on 1) The method of claim 1, further comprising displaying, via a display device associated with the host device, an indication that image data associated with at least a portion of the collaboration area needs to be captured via the one or more image sensors. ### Claim 9 (depends on 1) The method of claim 1, wherein generating the first 3D representation comprises: generating, based on the depth data, one or more 3D meshes, wherein each 3D mesh included in the one or more 3D meshes comprises a set of vertices and a set of faces formed from edges between pairs of vertices in the set of vertices; determining one or more vertices and one or more faces included in the one or more 3D meshes that are within the collaboration area; and determining one or more textures associated with the one or more faces based on the image data. ### Claim 10 (depends on 1) The method of claim 1, further comprising: generating a 3D background associated with a portion of the 3D environment that is outside the collaboration area; and transmitting, to the remote device, the 3D background for rendering in the first XR environment. ### Claim 11 (depends on 1) The method of claim 1, further comprising: receiving, from the one or more depth sensors, additional depth data associated with the 3D environment; receiving, from the one or more image sensors, additional image data associated with the 3D environment; determining, based on the additional depth data, changes in one or more geometries within the collaboration area; and generating a second 3D representation of the first user based on the additional depth data, the additional image data, and the collaboration area; and transmitting, to the remote device, the second 3D representation for rendering in the first XR environment. ### Claim 12 (depends on 1) The method of claim 1, further comprising: detecting, based on at least one of the depth data or the image data, a third user within the 3D environment; determining another collaboration area around at least a portion of the third user in the 3D environment; generating a third 3D representation of the at least a portion of the third user based on the depth data, the image data, and the another collaboration area; and transmitting, to the remote device, the third 3D representation for rendering in the first XR environment. ### Claim 13 One or more non-transitory computer-readable media storing program instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of: initiating a remote conference session; receiving, from one or more depth sensors associated with a host device, depth data associated with a three-dimensional (3D) environment; receiving, from one or more image sensors associated with the host device, image data associated with the 3D environment; determining a collaboration area around at least one automatically detected portion of a first user in the 3D environment, wherein the collaboration area is at least one of (i) a predefined 3D shape encompassing the at least one automatically detected portion of the first user, or (ii) determined based on a boundary of the at least one automatically detected portion of the first user, and wherein the at least one automatically detected portion can be moved or re-sized responsive to user interaction to control a size or location of the collaboration area; generating a first 3D representation of the at least one automatically detected portion of the first user based on the depth data, the image data, and the collaboration area; and transmitting, to a remote device, the first 3D representation for rendering in a first extended reality (XR) environment associated with the remote conference session. ### Claim 14 (depends on 13) The one or more non-transitory computer-readable media of claim 13, the steps further comprising: receiving a second 3D representation of at least a portion of a second user; and rendering, in a second XR environment associated with the remote conference session, the second 3D representation of the at least a portion of the second user. ### Claim 15 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein determining the collaboration area comprises: detecting, based on at least one of a machine learning model or a saliency model and at least one of the depth data or the image data, the boundary of the at least one automatically detected portion of the first user in the 3D environment; and generating the collaboration area based on the boundary. ### Claim 16 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein determining the collaboration area comprises: determining, based on the depth data, a plane associated with the 3D environment; receiving, from the first user, a selection of a location; generating the collaboration area based on the location and the plane; receiving, via one or more input devices, one or more interactions by the first user; and modifying at least one of a size or a position of the collaboration area based on the one or more interactions. ### Claim 17 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein the collaboration area is a 3D region around the at least one automatically detected portion of the first user and one or more objects within the 3D environment. ### Claim 18 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein generating the first 3D representation comprises: generating, based on the depth data, one or more 3D meshes, wherein each 3D mesh included in the one or more 3D meshes comprises a set of vertices and a set of faces formed from edges between pairs of vertices in the set of vertices; determining one or more vertices and one or more faces included in the one or more 3D meshes that are within the collaboration area; and determining one or more textures associated with the one or more faces based on the image data. ### Claim 19 (depends on 13) The one or more non-transitory computer-readable media of claim 13, the steps further comprising: generating a two-dimensional (2D) or 3D background associated with a portion of the 3D environment that is outside the collaboration area; and transmitting, to the remote device, the 2D or 3D background for rendering in the first XR environment. ### Claim 20 A system, comprising: one or more memories storing instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to: initiate a remote conference session, receive, from one or more depth sensors associated with a host device, depth data associated with a three-dimensional (3D) environment, receive, from one or more image sensors associated with the host device, image data associated with the 3D environment, determine a collaboration area around at least one automatically detected portion of a first user in the 3D environment, wherein the collaboration area is at least one of (i) a predefined 3D shape encompassing the at least one automatically detected portion of the first user, or (ii) determined based on a boundary of the at least one automatically detected portion of the first user, and wherein the at least one automatically detected portion can be moved or re-sized responsive to user interaction to control a size or location of the collaboration area, generate a first 3D representation of the at least one automatically detected portion of the first user based on the depth data, the image data, and the collaboration area, and transmit, to a remote device, the first 3D representation for rendering in a first extended reality (XR) environment associated with the remote conference session.