# US 2022/0277166 A1 · Methods and systems for rendering virtual objects in user-defined spatial boundary in extended reality environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277166 A1
- Document ID: US20220277166A1
- Status: Published application
- Kind: A1
- Application number: 17/187,663
- Filed: 2021-02-26
- Published: 2022-09-01
- Earliest claimed: 2021-02-26
- Assignees: Huawei Technologies Co., Ltd.
- Inventors: Changqing ZOU; Yumna Wassaf AKHTAR; Szu Wen FAN; Xu JIANPENG; Wei LI
- Web page: https://patentgrep.com/patent/US20220277166A1

## CPC

- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods and systems for rendering a virtual overlay in an extended reality (XR) environment are described. Image data is received, representing a current field of view (FOV) in the XR environment. A spatial boundary is defined in the current FOV, based on user input. An image label representing a region of interest (ROI) within the defined spatial boundary, and one or more object labels representing one or more objects within the defined spatial boundary are generated. At least one relevant virtual object is identified. The virtual object is relevant to a semantic meaning of the ROI based on the image label and/or the one or more object labels. Identification of the at least one relevant virtual object is outputted, to be renderable in the XR environment.

## Claims

### Claim 1

A method for rendering a virtual overlay in an extended reality (XR) environment, the method comprising:
obtaining image data representing a current field of view (FOV) in the XR environment;
defining a spatial boundary in the current FOV, based on user input;
generating an image label representing a region of interest (ROI) within the defined spatial boundary;
generating one or more object labels identifying respective one or more objects within the defined spatial boundary;
identifying at least one relevant virtual object that is relevant to a semantic meaning of the ROI based on at least one of: the image label and the one or more object labels; and
outputting identification of the at least one relevant virtual object to be renderable in the XR environment.

### Claim 2 (depends on 1)

The method of claim 1, wherein defining the spatial boundary in the current FOV comprises:
displaying a virtual viewfinder in the current FOV, the virtual viewfinder representing a spatial boundary shape enclosing a portion of the current FOV; and
in response to the user input, defining the spatial boundary in the current FOV using the virtual viewfinder.

### Claim 3 (depends on 1)

The method of claim 1, wherein defining the spatial boundary in the current FOV comprises:
performing gesture recognition on the image data to identify a gesture input;
identifying a spatial boundary shape corresponding to the identified gesture input; and
defining the spatial boundary in the current FOV using the identified spatial boundary shape.

### Claim 4 (depends on 3)

The method of claim 3, wherein the image data comprises a sequence of frames, and wherein the gesture recognition is performed to identify a dynamic gesture input over a plurality of frames.

### Claim 5 (depends on 3)

The method of claim 3, wherein the image data comprises one or more frames, and wherein the gesture recognition is performed to identify a static gesture input in at least one frame.

### Claim 6 (depends on 1)

The method of claim 1, wherein defining the spatial boundary in the current FOV comprises:
receiving touch input;
identifying a spatial boundary shape corresponding to the touch input; and
defining the spatial boundary in the current FOV using the identified spatial boundary shape.

### Claim 7 (depends on 6)

The method of claim 6, wherein the spatial boundary shape is identified corresponding to a path travelled by the touch input.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
rendering the at least one relevant virtual object in the ROI within the defined spatial boundary.

### Claim 9 (depends on 1)

The method of claim 1, wherein a plurality of relevant virtual objects is identified, the method further comprising:
receiving a selection of one relevant virtual object from the plurality of relevant virtual objects; and
rendering the one relevant virtual object in the ROI within the defined spatial boundary.

### Claim 10 (depends on 9)

The method of claim 9, wherein outputting the identification of the plurality of relevant virtual objects comprises ranking the plurality of relevant virtual objects based on relevance to the semantic meaning of the ROI.

### Claim 11

An electronic device comprising:
a processing unit coupled to a memory storing machine-executable instructions thereon, wherein the instructions, when executed by the processing unit, cause the electronic device to:
obtain image data representing a current field of view (FOV) in the XR environment;
define a spatial boundary in the current FOV, based on user input;
generate an image label representing a region of interest (ROI) within the defined spatial boundary;
generate one or more object labels identifying respective one or more objects within the defined spatial boundary;
identify at least one relevant virtual object that is relevant to a semantic meaning of the ROI based on at least one of: the image label and the one or more object labels; and
output identification of the at least one relevant virtual object to be renderable in the XR environment.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the instructions cause the electronic device to define the spatial boundary in the current FOV by:
displaying a virtual viewfinder in the current FOV, the virtual viewfinder representing a spatial boundary shape enclosing a portion of the current FOV; and
in response to the user input, defining the spatial boundary in the current FOV using the virtual viewfinder.

### Claim 13 (depends on 11)

The electronic device of claim 11, wherein the instructions cause the electronic device to define the spatial boundary in the current FOV by:
performing gesture recognition on the image data to identify a gesture input;
identifying a spatial boundary shape corresponding to the identified gesture input; and
defining the spatial boundary in the current FOV using the identified spatial boundary shape.

### Claim 14 (depends on 13)

The electronic device of claim 13, wherein the image data comprises a sequence of frames, and wherein the gesture recognition is performed to identify a dynamic gesture input over a plurality of frames.

### Claim 15 (depends on 13)

The electronic device of claim 13, wherein the image data comprises one or more frames, and wherein the gesture recognition is performed to identify a static gesture input in at least one frame.

### Claim 16 (depends on 11)

The electronic device of claim 11, wherein the instructions cause the electronic device to define the spatial boundary in the current FOV by:
receiving touch input;
identifying a spatial boundary shape corresponding to the touch input; and
defining the spatial boundary in the current FOV using the identified spatial boundary shape.

### Claim 17 (depends on 16)

The electronic device of claim 16, wherein the spatial boundary shape is identified corresponding to a path travelled by the touch input.

### Claim 18 (depends on 11)

The electronic device of claim 11, wherein the instructions further cause the electronic device to:
render the at least one relevant virtual object in the ROI within the defined spatial boundary.

### Claim 19 (depends on 11)

The electronic device of claim 11, wherein the electronic device is one of:
a head-mounted display (HMD) device;
augmented reality (AR) glasses;
a wearable device; or
a handheld device.

### Claim 20

A non-transitory computer-readable medium having machine-executable instructions stored thereon, the instructions, when executed by a processing unit of an electronic device, cause the electronic device to:
obtain image data representing a current field of view (FOV) in the XR environment;
define a spatial boundary in the current FOV, based on detected user input;
generate an image label representing a region of interest (ROI) within the defined spatial boundary;
generate one or more object labels identifying respective one or more objects within the defined spatial boundary;
identify at least one relevant virtual object that is relevant to a semantic meaning of the ROI based on at least one of: the image label and the one or more object labels; and
output identification of the at least one relevant virtual object to be renderable in the XR environment.
