# US 2024/0205370 A1 · Extended reality servers preforming actions directed to virtual objects based on overlapping field of views of participants

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0205370 A1
- Document ID: US20240205370A1
- Status: Published application
- Kind: A1
- Application number: 18/287,662
- Filed: 2021-04-21
- Published: 2024-06-20
- Earliest claimed: 2021-04-21
- Assignees: Telefonaktiebolaget LM Ericsson (publ)
- Applicants: Telefonaktiebolaget LM Ericsson (publ)
- Inventors: Peter ÖKVIST; Tommy ARNGREN
- Web page: https://patentgrep.com/patent/US20240205370A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- H04N 7/152: {Multipoint control units therefor}
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T 2219/2016: Rotation, translation, scaling
- G06T 2219/2021: Shape modification
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An XR environment server is configured to represent an immersive XR environment on a display device viewable by a participant among a group of participants who have associated avatars representing the participants and which are rendered for viewing in the immersive XR environment. The XR environment server is further configured to determine direction of a FOV of the participant from a perspective of a location of the associated avatar in the immersive XR environment relative to a coordinate system of the immersive virtual environment. Based on the direction of the FOV of the participant, the XR environment server is further configured to identify a virtual object that is rendered for display at a location in the immersive XR environment which is within the FOV of the participant.

## Claims

### Claim 1

An extended reality, XR, environment server comprising:
at least one processor; and
at least one memory storing instructions executable by the at least one processor to perform operations to:
represent an immersive XR environment on a display device viewable by a participant among a group of participants who have associated avatars representing the participants and which are rendered for viewing in the immersive XR environment;
determine direction of a field of view, FOV, of the participant from a perspective of a location of the associated avatar in the immersive XR environment relative to a coordinate system of the immersive virtual environment;
based on the direction of the FOV of the participant, identify a virtual object that is rendered for display at a location in the immersive XR environment which is within the FOV of the participant; and
based on when a measurement of the FOV with respect to the virtual object satisfies a virtual object action rule, perform a defined action directed to the virtual object.

### Claim 2 (depends on 1)

The XR environment server of claim 1, wherein an operation to determine when the measurement of the FOV with respect to the virtual object satisfies the virtual object action rule, comprises:
determine a FOV hold time that the participant has maintained the FOV within a threshold angular range of the location of the virtual object; and
determine that the virtual object action rule is satisfied based on the FOV hold time satisfying a hold time threshold number.

### Claim 3 (depends on 1)

The XR environment server of claim 1, wherein an operation to determine when the measurement of the FOV with respect to the virtual object satisfies the virtual object action rule, comprises:
measure a FOV hold stability value indicating an amount of angular movement occurring between a center of the FOV of the participant from the location of the virtual object during a defined duration; and
determine that the virtual object action rule is satisfied based on the FOV hold stability value satisfying a FOV hold stability threshold number.

### Claim 4 (depends on 1)

The XR environment server of claim 1, wherein an operation to determine when the measurement of the FOV with respect to the virtual object satisfies the virtual object action rule, comprises:
determining whether a virtual object access rule is satisfied based on at least one of:
determining whether the participant has a defined privilege that satisfies the virtual object access rule for being allowed to perform an action directed to the virtual object; and determining whether the participant has a defined role in the immersive XR environment that satisfies the virtual object access rule for being allowed to perform an action directed to the virtual object.

### Claim 5 (depends on 1)

The XR environment server of claim 1, wherein the instructions executable by the at least one processor further perform operations to:
determine directions of FOVs of the participants from perspectives of locations of the associated avatars in the immersive XR environment relative to the coordinate system of the immersive virtual environment;
based on the directions of the FOVs of the participants, identify that the virtual object which is rendered for display at the location in the immersive XR environment is within an overlap of the FOVs of a number of the participants; and
based on when the number of the participants with the FOVs which overlap the virtual object satisfies the virtual object action rule, perform the defined action directed to the virtual object.

### Claim 6 (depends on 5)

The XR environment server of claim 5, wherein an operation to determine when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, comprises:
for each of the number of the participants with the FOVs that overlap, determine a FOV hold time that the participant has maintained the FOV within a threshold angular range of the location of the virtual object; and
determine that the virtual object action rule is satisfied based on the number of the participants with the FOVs that overlap satisfying a group size threshold number and based on the FOV hold time satisfying a hold time threshold number for the participants with the FOVs that overlap.

### Claim 7 (depends on 5)

The XR environment server of claim 5, wherein an operation to determine when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, comprises:
for each of the number of the participants with the FOVs that overlap, measure a FOV hold stability value indicating an amount of angular movement occurring between a center of the FOV of the participant from the location of the virtual object during a defined duration; and
determine that the virtual object action rule is satisfied based on the number of the participants with the FOVs that overlap satisfying a group size threshold number and based on the FOV hold stability value satisfying a FOV hold stability threshold number for the participants with the FOVs that overlap.

### Claim 8 (depends on 5)

The XR environment server of claim 5, wherein an operation to determine when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, comprises at least one of:
set a threshold number defined by the virtual object action rule based on a total number of participants who are present in the immersive XR environment, wherein the operation to determine when the number of the participants with the FOVs that overlap satisfies the virtual object action rule comprising comparing the threshold number to the number of the participants with the FOVs that overlap; and
set the threshold number defined by the virtual object action rule based on an indication of how quickly the avatars associated with the group of participants are moving within the immersive XR environment.

### Claim 9 (depends on 5)

The XR environment server of claim 5, based on when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, the operation to perform the defined action directed to the virtual object comprises at least one of:
trigger playout of an audio stream, which is programmatically linked to the virtual object, to the participants with the FOVs that overlap;
trigger playout of a video stream, which is programmatically linked to the virtual object, to the participants with the FOVs that overlap; and
trigger generation of haptic feedback and/or other notification, which is programmatically linked to the virtual object, to the participants with the FOVs that overlap.

### Claim 10 (depends on 5)

The XR environment server of claim 5, wherein:
the virtual object corresponds to one of the avatars associated with one of the participants in the group, and
based on when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, the operation to perform the defined action directed to the virtual object that is associated with the one of the participants in the group, comprises at least one of:
designating the one of the participants in the group as a presenter for a group meeting and initiating sharing of content from the presenter with the other participants in the group meeting;
designating the one of the participants in the group as the presenter for the group meeting and unmuting the presenter's microphone to initiate streaming of audio from a microphone of the presenter to the other participants in the group meeting; and
designating the one of the participants in the group as the presenter for the group meeting and initiating display of streaming video from a camera of the presenter within a graphical representation of a virtual display rendered in the immersive XR environment for viewing by the other participants in the group meeting.

### Claim 11 (depends on 5)

The XR environment server of claim 5, wherein:
the virtual object corresponds to one of the avatars associated with one of the participants in the group, and
based on when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, the operation to perform the defined action directed to the virtual object that is associated with the one of the participants in the group, comprises at least one of:
designating the one of the participants in the group as a presenter for the group meeting and moving the avatar associated with the presenter to a defined location in the immersive XR environment for viewing by the other participants in the group meeting; and
designating the one of the participants in the group as the presenter for the group meeting and changing a size, color, and/or shape of the avatar associated with the presenter in the immersive XR environment to indicate the presenter's status for viewing by the other participants in the group meeting.

### Claim 12 (depends on 5)

The XR environment server of claim 5, wherein:
the virtual object corresponds to one of the avatars associated with one of the participants in the group, and
based on when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, the operation to perform the defined action directed to the virtual object comprises at least one of:
increasing rendered resolution of the avatar associated with the one of the participants;
increasing resolution and/or size of streaming video from a camera of the one of the participants displayed within a graphical representation of a virtual display rendered in the immersive XR environment for viewing by the other participants in the group; and
increasing loudness of audio from a microphone of the one of the participants streamed to the other participants in the group.

### Claim 13 (depends on 5)

The XR environment server of claim 5, wherein:
based on when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, the operation to perform the defined action directed to the virtual object comprises at least one of:
controlling location of where the virtual object is rendered in the immersive XR environment to track virtual object movement input received from one or more of the participants with the FOVs that overlap the virtual object; and
controlling shape, color, and/or size of the virtual object based on virtual object modification input received from one or more of the participants with the FOVs that overlap the virtual object.

### Claim 14 (depends on 5)

The XR environment server of claim 5, based on when the number of the participants with the FOVs that overlap does not satisfy the virtual object action rule, performing at least one of:
display an indication that the virtual object action rule for the virtual object is not satisfied, for view by the participants with the FOVs that overlap;
display an indication of how many other participants need to join the participants with overlapping FOVs of the virtual object in order for the virtual object action rule to become satisfied; and
display an indication of a particular person's name who needs to join the participants with overlapping FOVs of the virtual object in order for the virtual object action rule to become satisfied.

### Claim 15 (depends on 1)

The XR environment server of claim 1, based on when the measurement of the FOV with respect to the virtual object satisfies the virtual object action rule, the operation to perform the defined action directed to the virtual object comprises at least one of:
provide access for the participant to view textual content programmatically linked to the virtual object;
provide access for the participant to view video playout of video content programmatically linked to the virtual object;
provide access for the participant to listen to audio playout of audio content programmatically linked to the virtual object;
provide access for the participant to edit a characteristic of the virtual object;
provide access for the participant to programmatically link a picture, video, audio, and/or text to the virtual object; and
programmatically link a digital signature of the participant to the virtual object.

### Claim 16 (depends on 1)

The XR environment server of claim 1, based on when the measurement of the FOV with respect to the virtual object satisfies the virtual object action rule, the operation to perform the defined action directed to the virtual object comprises at least one of:
control supply of power to a wall plug socket communicatively connected to the XR environment server;
control supply of power to a light fixture communicatively connected to the XR environment server;
control turning on and/or turning off a television communicatively connected to the XR environment server;
control tuning of channels by a television communicatively connected to the XR environment server;
control locking and unlocking of an entrance door lock communicatively connected to the XR environment server;
launch execution of an application by a computer device communicatively connected to the XR environment server; and
provide input to an application being executed by a computing device communicatively connected to the XR environment server.

### Claim 17

A method by an extended reality, XR, environment server comprising:
representing an immersive XR environment on a display device viewable by a participant among a group of participants who have associated avatars representing the participants and which are rendered for viewing in the immersive XR environment;
determining direction of a field of view, FOV, of the participant from a perspective of a location of the associated avatar in the immersive XR environment relative to a coordinate system of the immersive virtual environment;
based on the direction of the FOV of the participant, identifying a virtual object that is rendered for display at a location in the immersive XR environment which is within the FOV of the participant; and
based on when a measurement of the FOV with respect to the virtual object satisfies a virtual object action rule, performing a defined action directed to the virtual object.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
determining directions of FOVs of the participants from perspectives of locations of the associated avatars in the immersive XR environment relative to the coordinate system of the immersive virtual environment;
based on the directions of the FOVs of the participants, identifying that the virtual object which is rendered for display at the location in the immersive XR environment is within an overlap of the FOVs of a number of the participants; and
based on when the number of the participants with the FOVs which overlap the virtual object satisfies the virtual object action rule, performing the defined action directed to the virtual object.

### Claim 19 (depends on 18)

The method of claim 18, based on when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, the method to perform the defined action directed to the virtual object comprises at least one of:
triggering playout of an audio stream, which is programmatically linked to the virtual object, to the participants with the FOVs that overlap;
triggering playout of a video stream, which is programmatically linked to the virtual object, to the participants with the FOVs that overlap; and
triggering generation of haptic feedback, which is programmatically linked to the virtual object, to the participants with the FOVs that overlap.

### Claim 20 (depends on 18)

The method of claim 18, wherein:
the virtual object corresponds to one of the avatars associated with one of the participants in the group, and
based on when the number of the participants with the FOVs that overlap satisfies the virtual object action rule, the method to perform the defined action directed to the virtual object that is associated with the one of the participants in the group, comprises at least one of:
designating the one of the participants in the group as a presenter for a group meeting and initiating sharing of content from the presenter with the other participants in the group meeting;
designating the one of the participants in the group as the presenter for the group meeting and unmuting the presenter's microphone to initiate streaming of audio from a microphone of the presenter to the other participants in the group meeting; and
designating the one of the participants in the group as the presenter for the group meeting and initiate display of streaming video from a camera of the presenter within a graphical representation of a virtual display rendered in the immersive XR environment for viewing by the other participants in the group meeting.

### Claim 21 (depends on 17)

The method of claim 17, based on when the measurement of the FOV with respect to the virtual object satisfies the virtual object action rule, the method to perform the defined action directed to the virtual object comprises at least one of:
providing access for the participant to view textual content programmatically linked to the virtual object;
providing access for the participant to view video playout of video content programmatically linked to the virtual object;
providing access for the participant to listen to audio playout of audio content programmatically linked to the virtual object;
providing access for the participant to edit a characteristic of the virtual object;
providing access for the participant to programmatically link a picture, video, audio, and/or text to the virtual object; and
programmatically linking a digital signature of the participant to the virtual object.

### Claim 22 (depends on 17)

The method of claim 17, based on when the measurement of the FOV with respect to the virtual object satisfies the virtual object action rule, the method to perform the defined action directed to the virtual object comprises at least one of:
controlling supply of power to a wall plug socket communicatively connected to the XR environment server;
controlling supply of power to a light fixture communicatively connected to the XR environment server;
controlling turning on and/or turning off a television communicatively connected to the XR environment server;
controlling tuning of channels by a television communicatively connected to the XR environment server;
control locking and unlocking of an entrance door lock communicatively connected to the XR environment server;
launching execution of an application by a computer device communicatively connected to the XR environment server; and
providing input to an application being executed by a computing device communicatively connected to the XR environment server.

### Claim at position 23

-26. (canceled)
