# US 2021/0084259 A1 · Placement of Virtual Content in Environments with a Plurality of Physical Participants

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0084259 A1
- Document ID: US20210084259A1
- Status: Published application
- Kind: A1
- Application number: 16/572,030
- Filed: 2019-09-16
- Published: 2021-03-18
- Earliest claimed: 2019-09-16
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Jonathan KIES; Robert TARTZ; Daniel James GUEST
- Web page: https://patentgrep.com/patent/US20210084259A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- A63F 13/332: using wireless networks, e.g. cellular phone networks
- A63F 13/5255: according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
- A63F 13/655: by importing photos, e.g. of the player
- G02B 27/017: {Head mounted}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06T 7/73: using feature-based methods
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- H04N 7/142: {Constructional details of the terminal equipment, e.g. arrangements of the camera and the display}
- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A device may be configured to determining display properties for virtual content in an environment with a plurality of physical participants by capturing an image of the environment, analyzing the captured image to identify at least one object in the environment, determining a parameter for the identified object, and determining a display property of a digital representation of virtual content based on the determined parameter. Embodiments may include negotiating display properties with other devices to generate coordinated display properties, and rendering the digital representation of the virtual content so that the remote participant appears to be in the same fixed position to all co-located participants and sized consistent with the co-located participants.

## Claims

### Claim 1

A method of determining display properties for virtual content in an environment with a plurality of participants, comprising:
capturing, by a processor, an image of the environment;
analyzing the captured image to identify an object in the environment;
determining a parameter for the identified object;
determining a location of a digital representation of the virtual content relative to the identified object based on the determined parameter; and
sending the determined location to at least one other device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising assigning a priority to the determined parameter, wherein determining the location of the digital representation of the virtual content relative to the identified object based on the determined parameter comprises determining the location of the digital representation of the virtual content relative to the identified object based on the priority assigned to the determined parameter.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the parameter for the identified object comprises:
determining at least one or more of a fixed position parameter, a size parameter, or a line-of-sight parameter; and
determining at least one or more of a field-of-view parameter, a spacing parameter, an inter-person distance parameter, a height parameter, and a visibility parameter.

### Claim 4 (depends on 3)

The method of claim 3, wherein assigning a priority to the determined parameter comprises:
assigning a first priority to the fixed position parameter, the size parameter, or the line-of-sight parameter; and
assigning a second priority to the field-of-view parameter, the spacing parameter, the inter-person distance parameter, the height parameter, or the visibility parameter, and
wherein the first priority is a higher priority than the second priority.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
identifying the one or more other devices in the environment, wherein sending the determined location to at least one other device comprises sending the determined location to the identified one or more other devices.

### Claim 6 (depends on 1)

The method of claim 1, further comprising receiving an additional parameter from at least one other device in the environment, wherein determining the location of the digital representation of the virtual content relative to the identified object based on the determined parameter comprises determining one or more locations of the digital representation of the virtual content relative to the identified object based on the determined parameter and the received additional parameter.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the location of the digital representation of the virtual content relative to the identified object based on the determined parameter comprises:
negotiating one or more locations for rendering the digital representation of the virtual content relative to the identified object with at least one other device in the environment to generate one or more coordinated display properties.

### Claim 8 (depends on 7)

The method of claim 7, further comprising using the one or more coordinated display properties to render the digital representation of the virtual content so that all co-located participants physically present in the environment perceive the virtual content to be in the same fixed position in the environment.

### Claim 9 (depends on 7)

The method of claim 7, further comprising using the one or more coordinated display properties to size the digital representation of the virtual content based on a size of a participant physically present in the environment.

### Claim 10 (depends on 7)

The method of claim 7, further comprising using the one or more coordinated display properties to render the digital representation of the virtual content so that the virtual content appears to be positioned between a first participant physically present in the environment and a second participant physically present in the environment.

### Claim 11 (depends on 10)

The method of claim 10, wherein using the one or more coordinated display properties to render the digital representation of the virtual content so that the virtual content appears to be positioned between the first participant physically present in the environment and the second participant physically present in the environment comprises using the one or more coordinated display properties to render the digital representation of the virtual content so that the virtual content appears to be approximately equidistant to the first participant physically present in the environment and the second participant physically present in the environment.

### Claim 12 (depends on 10)

The method of claim 10, wherein the virtual content comprises a remote participant.

### Claim 13 (depends on 12)

The method of claim 12, further comprising using the one or more coordinated display properties to render the digital representation of the remote participant so that the remote participant appears to be anchored to an empty seat.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
capturing an updated image of the environment;
analyzing the captured updated image to determine whether a number, location, position, or size of the identified object has changed;
determining an updated parameter for the identified object in response to determining that the location, position, or size of the identified object has changed;
determining whether a difference between the determined parameter and the determined updated parameter exceeds a threshold value; and
determining an updated location for the digital representation of the virtual content relative to the identified object in response to determining that the difference between the determined parameter and the determined updated parameter exceeds the threshold value.

### Claim 15 (depends on 1)

The method of claim 1, wherein determining the location of the digital representation of the virtual content relative to the identified object based on the determined parameter comprises determining at least one location for each digital representation of each of a plurality of remote participants.

### Claim 16 (depends on 1)

The method of claim 1, wherein determining the parameter for the identified object comprises determining at least one or more of:
a fixed position parameter;
a size parameter;
a line-of-sight parameter;
a field-of-view parameter;
a spacing parameter;
an inter-person distance parameter;
a height parameter; or
a visibility parameter.

### Claim 17 (depends on 1)

The method of claim 1, wherein the virtual content is related to a game.

### Claim 18

A computing device, comprising:
a memory; and
a processor coupled to the memory and configured to:
capture an image of an environment;
analyze the captured image to identify an object in the environment;
determine a parameter for the identified object; and
determine a location of a digital representation of virtual content relative to the identified object based on the determined parameter; and
send the determined location to at least one other device.

### Claim 19 (depends on 18)

The computing device of claim 18, wherein the processor is configured to:
assign a priority to the determined parameter; and
determine the location of the digital representation of the virtual content relative to the identified object based on the determined parameter by determining the location of the digital representation of the virtual content relative to the identified object based on the priority assigned to the determined parameter.

### Claim 20 (depends on 19)

The computing device of claim 19, wherein the processor is configured to:
determine the parameter for the identified object by:
determining at least one or more of a fixed position parameter, a size parameter, or a line-of-sight parameter; and
determining at least one or more of a field-of-view parameter, a spacing parameter, an inter-person distance parameter, a height parameter, and a visibility parameter.

### Claim 21 (depends on 20)

The computing device of claim 20, wherein the processor is configured to:
assign the priority to the determined parameter by:
assigning a first priority to the fixed position parameter, the size parameter, or the line-of-sight parameter; and
assigning a second priority to the field-of-view parameter, the spacing parameter, the inter-person distance parameter, the height parameter, or the visibility parameter, wherein the first priority is a higher priority than the second priority.

### Claim 22 (depends on 18)

The computing device of claim 18, wherein the processor is configured to:
identify one or more other devices in the environment,
wherein sending the determined location to at least one other device comprises sending the determined location to the identified one or more other devices.

### Claim 23 (depends on 18)

The computing device of claim 18, further comprising receiving an additional parameter from at least one other device in the environment, wherein determining the location of the digital representation of the virtual content relative to the identified object based on the determined parameter comprises determining one or more locations of the digital representation of the virtual content relative to the identified object based on the determined parameter and the received additional parameter.

### Claim 24 (depends on 18)

The computing device of claim 18, wherein the processor is configured to determine the location of the digital representation of the virtual content relative to the identified object based on the determined parameter comprises negotiating one or more locations for rendering the digital representation of the virtual content relative to the identified object with at least one other device in the environment to generate one or more coordinated display properties.

### Claim 25 (depends on 24)

The computing device of claim 24, wherein the processor is configured to use the one or more coordinated display properties to render the digital representation of the virtual content so that all co-located participants physically present in the environment perceive the virtual content to be in the same fixed position in the environment.

### Claim 26 (depends on 24)

The computing device of claim 24, wherein the processor is configured to use the one or more coordinated display properties to size the digital representation of the virtual content based on a size of a participant physically present in the environment.

### Claim 27 (depends on 24)

The computing device of claim 24, wherein the processor is configured to use the one or more coordinated display properties to render the digital representation of the virtual content so that the virtual content appears to be positioned between a first participant physically present in the environment and a second participant physically present in the environment.

### Claim 28 (depends on 24)

The computing device of claim 24, wherein the processor is configured such that the virtual content is a remote participant.

### Claim 29 (depends on 28)

The computing device of claim 28, wherein the processor is configured to use the one or more coordinated display properties to render the digital representation of the remote participant so that the remote participant appears to be anchored to an empty seat.

### Claim 30 (depends on 18)

The computing device of claim 18, wherein the processor is configured to:
capture an updated image of the environment;
analyze the captured updated image to determine whether a number, location, position, or size of the identified object has changed;
determine an updated parameter for the identified object in response to determining that the location, position, or size of the identified object has changed;
determine whether a difference between the determined parameter and the determined updated parameter exceeds a threshold value; and
determine an updated location for the digital representation of the virtual content relative to the identified object in response to determining that the difference between the determined parameter and the determined updated parameter exceeds the threshold value.

### Claim 31 (depends on 18)

The computing device of claim 18, wherein the processor is configured to determine the location of the digital representation of the virtual content relative to the identified object based on the determined parameter by determining at least one location for each digital representation of each of a plurality of remote participants.

### Claim 32 (depends on 18)

The computing device of claim 18, further comprising:
an image sensor; and
a projector configured to project images onto optical lenses,
wherein the processor is coupled to the image sensor and to the projector.

### Claim 33 (depends on 18)

The computing device of claim 18, wherein the computing device is a smartphone.

### Claim 34 (depends on 18)

The computing device of claim 18, wherein the virtual content is related to a game.
