# US 12,010,157 B2 · Systems and methods for enabling user-controlled extended reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,010,157 B2
- Document ID: US12010157B2
- Status: Granted
- Kind: B2
- Application number: 17/706,730
- Filed: 2022-03-29
- Published: 2023-10-05
- Granted: 2024-06-11
- Earliest claimed: 2022-03-29
- Assignees: Rovi Guides, Inc.
- Applicants: Rovi Guides, Inc.
- Inventors: Serhad Doken
- Web page: https://patentgrep.com/patent/US12010157B2

## CPC

- H04L 65/1089: by adding media; by removing media
- H04L 65/403: Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15)
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- H04L 65/1059: End-user terminal functionalities specially adapted for real-time communication
- H04L 65/762: {at the source (reformatting of additional data in video distribution servers H04N21/2355)}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10016: Video; Image sequence
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods are described for enabling user-controlled extended reality (XR). The provided systems and methods may determine an XR portion should be included in a video feed that comprises a depiction of a first user, and the XR portion may be configured to occlude the entity in the video feed. An entity may be detected in the video feed, and user input may be received specifying that a detected entity in the video feed should not be occluded by the XR portion. Based on the user input, the video feed may be modified to exclude the XR portion from a portion of the video feed depicting the detected entity, wherein the portion is included at a different portion of the video feed, and the depiction of the first user is included in the video feed, and transmit the modified video feed to a second user device.

## Claims

### Claim 1

A system comprising:
a first user device associated with a first user, wherein a server is configured to establish a video communication session over a network, and wherein the video communication session enables the first user device to present a depiction of a second user, associated with a second user device, and an environment surrounding the second user, and the video communication session enables the second user device to present a depiction of the first user and an environment surrounding the first user, and wherein a portion of the environment surrounding the first user comprises an entity other than the first user; and
control circuitry configured to:
determine that an extended reality (XR) portion should be included in a video feed of the video communication session, wherein the video feed comprises the depiction of the first user, and the XR portion is configured to occlude the entity in the video feed;
detect the entity in the video feed;
generate for display a user interface element, at which user input specifying that the entity should not be occluded by the XR portion in the video feed is received, the user interface element including a plurality of options comprising:
a first option to cause the video feed to comprise the XR portion surrounding the depiction of the first user and occluding any entities in a background surrounding the first user;
a second option to cause the video feed to comprise the XR portion, the first user, and one or more additional users detected in the environment surrounding the first user:
a third option to cause the video feed to comprise the XR portion, the first user, and one or more objects detected in the environment surrounding the first user; and
a fourth option to cause the video feed to comprise no XR portion surrounding the depiction of the first user;
receive the user input specifying that the entity should not be occluded by the XR portion in the video feed;
modify, based on the user input, the video feed to exclude the XR portion from a portion of the video feed depicting the detected entity, wherein the XR portion is included at a different portion of the video feed, and the depiction of the first user is included in the video feed; and
transmit the modified video feed to the second user device via the server.

### Claim 2 (depends on 1)

The system of claim 1, wherein the video communication session corresponds to:
a video conference in which the first user and the second user are participants, the XR portion is a virtual background simultaneously presented with the depiction of the first user, and the virtual background does not occlude the entity specified by the user input; or
an augmented reality scene simultaneously presented with the depiction of the first user.

### Claim 3 (depends on 1)

The system of claim 1, wherein the entity corresponds to one or more other users in the environment surrounding the first user or one or more objects in the environment surrounding the first user.

### Claim 4 (depends on 1)

The system of claim 1, wherein the server is further configured to:
receive an API request from a service provider; and
transmit, via an API, one or more software tools to enable the service provider to perform one or more of the determining, the detecting, the receiving, the modifying and the transmitting.

### Claim 5 (depends on 1)

The system of claim 1, wherein the control circuitry is configured to receive the user input specifying that the entity should not be occluded by the XR portion in the video feed during the video communication session and perform the modifying by dynamically modifying the video feed to exclude the XR portion from the portion of the video feed depicting the detected entity.

### Claim 6 (depends on 1)

The system of claim 1, wherein the video feed is a first video feed, and wherein the XR portion is a first XR portion, and the control circuitry is further configured to:
receive, via the server and from the second user device, a second video feed comprising a second XR portion modifying an appearance of the environment surrounding the second user; and
modify, based on user input received from the first user, the appearance of the environment surrounding the second user, for presentation at the first user device.

### Claim 7

A system comprising:
a first user device associated with a first user, wherein a server is configured to establish a video communication session over a network, and wherein the video communication session enables the first user device to present a depiction of a second user, associated with a second user device, and an environment surrounding the second user, and the video communication session enables the second user device to present a depiction of the first user and an environment surrounding the first user, and wherein a portion of the environment surrounding the first user comprises an entity other than the first user; and
control circuitry configured to:
determine that an extended reality (XR) portion should be included in a video feed of the video communication session, wherein the video feed comprises the depiction of the first user, and the XR portion is configured to occlude the entity in the video feed;
detect the entity in the video feed;
receive user input specifying that the entity should not be occluded by the XR portion in the video feed;
modify, based on the user input, the video feed to exclude the XR portion from a portion of the video feed depicting the detected entity, wherein the XR portion is included at a different portion of the video feed, and the depiction of the first user is included in the video feed;
transmit the modified video feed to the second user device via the server;
determine ambient conditions of the environment surrounding the first user; and
determine whether one or more entities should not be occluded from the XR portion based on the determined ambient conditions.

### Claim 8 (depends on 1)

The system of claim 1, wherein the first user device corresponds to a near-eye display device, and wherein the video feed is a first video feed, and wherein the XR portion is a first XR portion, and the control circuitry is further configured to:
receive, via the server and from the second user device, a second video feed comprising the depiction of the second user; and
cause a second XR portion to span a full field of view of the near-eye display device, while generating for display the second video feed depiction of the second user.

### Claim 9 (depends on 8)

The system of claim 8, wherein the entity is a first entity, and wherein the control circuitry is further configured to:
detect that a second entity is moving within the environment surrounding the first user; and
cause the detected second entity not to be occluded by the second XR portion otherwise spanning the full field of view of the near-eye display device.

### Claim 10

A method comprising:
determining that a first extended reality (XR) portion should be included in a first video feed of a video communication session, wherein the first video feed comprises a depiction of a first user associated with a first user device, and wherein the first user device corresponds to a near-eye display device, and the first XR portion is configured to occlude a first entity in the first video feed, and wherein the video communication session is established by a server over a network and enables the first user device to present a depiction of a second user, associated with a second user device, and an environment surrounding the second user, and the video communication session enables the second user device to present the depiction of the first user and an environment surrounding the first user;
detecting the first entity other than the first user in the first video feed;
receiving user input specifying that the first entity should not be occluded by the first XR portion in the first video feed;
modifying, based on the user input, the first video feed to exclude the first XR portion from a portion of the first video feed depicting the detected first entity, wherein the first XR portion is included at a different portion of the first video feed, and the depiction of the first user is included in the first video feed;
transmitting the modified first video feed to the second user device via the server;
receiving, via the server and from the second user device, a second video feed comprising the depiction of the second user;
causing a second XR portion to span a full field of view of the near-eye display device, while generating for display the second video feed depiction of the second user;
detecting that a second entity is moving within the environment surrounding the first user; and
causing the detected second entity not to be occluded by the second XR portion otherwise spanning the full field of view of the near-eye display device.

### Claim 11 (depends on 10)

The method of claim 10, wherein the video communication session corresponds to:
a video conference in which the first user and the second user are participants, the first XR portion is a virtual background simultaneously presented with the depiction of the first user, and the virtual background does not occlude the first entity specified by the user input; or
an augmented reality scene simultaneously presented with the depiction of the first user.

### Claim 12 (depends on 10)

The method of claim 10, wherein the first entity corresponds to one or more other users in the environment surrounding the first user or one or more objects in the environment surrounding the first user.

### Claim 13 (depends on 10)

The method of claim 10, wherein the server is further configured to:
receive an API request from a service provider; and
transmit, via an API, one or more software tools to enable the service provider to perform one or more of the determining, the detecting, the receiving, the modifying and the transmitting.

### Claim 14 (depends on 10)

The method of claim 10, further comprising generating for display a user interface element, at which the user input specifying that the first entity should not be occluded by the first XR portion in the first video feed is received, the user interface element including a plurality of options comprising:
a first option to cause the first video feed to comprise the first XR portion surrounding the depiction of the first user and occluding any entities in a background surrounding the first user;
a second option to cause the first video feed to comprise the first XR portion, the first user, and one or more additional users detected in the environment surrounding the first user;
a third option to cause the first video feed to comprise the first XR portion, the first user, and one or more objects detected in the environment surrounding the first user; and
a fourth option to cause the first video feed to comprise no first XR portion surrounding the depiction of the first user.

### Claim 15 (depends on 10)

The method of claim 10, wherein the user input specifying that the first entity should not be occluded by the first XR portion in the first video feed is received during the video communication session and the modifying comprises dynamically modifying the first video feed to exclude the first XR portion from the portion of the first video feed depicting the detected first entity.

### Claim 16 (depends on 10)

The method of claim 10, wherein the second video feed comprises the second XR portion modifying an appearance of the environment surrounding the second user, and the method further comprises:
modifying, based on user input received from the first user, the appearance of the environment surrounding the second user, for presentation at the first user device.

### Claim 17 (depends on 10)

The method of claim 10, wherein the method further comprises:
determining ambient conditions of the environment surrounding the first user; and
determining whether one or more entities should not be occluded from the first XR portion based on the determined ambient conditions.

### Claim 18 (depends on 1)

The system of claim 1, wherein the video communication session corresponds to a video conference in which the first user and the second user are participants, the XR portion is a virtual background simultaneously presented with the depiction of the first user, and the virtual background does not occlude the entity specified by the user input.

### Claim 19 (depends on 7)

The system of claim 7, wherein the video communication session corresponds to a video conference in which the first user and the second user are participants, the XR portion is a virtual background simultaneously presented with the depiction of the first user, and the virtual background does not occlude the entity specified by the user input.

### Claim 20 (depends on 10)

The method of claim 10, wherein the video communication session corresponds to a video conference in which the first user and the second user are participants, the first XR portion is a virtual background simultaneously presented with the depiction of the first user, and the virtual background does not occlude the first entity specified by the user input.