# US 2020/0005539 A1 · Visual flairs for emphasizing gestures in artificial-reality environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0005539 A1
- Document ID: US20200005539A1
- Status: Published application
- Kind: A1
- Application number: 16/020,391
- Filed: 2018-06-27
- Published: 2020-01-02
- Earliest claimed: 2018-06-27
- Assignees: Facebook Technologies, LLC
- Applicants: Facebook Technologies, LLC
- Inventors: Albert Peter Hwang; Kenrick Cheng-Kuo Kin
- Web page: https://patentgrep.com/patent/US20200005539A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0172: {characterised by optical features}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A near eye display (NED) system is configured to present an augmented reality environment to the user. In addition, the NED system uses an imaging device to detect individuals within a local area, and identify positions of hands of the individuals from which the NED system may be able to detect one or more performed gestures. In response to detecting a predetermined gesture, the NED system displays a user of the NED one or more virtual objects corresponding to the identified first gesture at a location associated with a predetermined portion of the individual's body. The displayed virtual objects may be referred to as “visual flair,” and are used to accentuate, stylize, or provide emphasis to gestures performed by individuals within the local area.

## Claims

### Claim 1

A system comprising:
a near eye display (NED) comprising:
an optical assembly having an electronic display that is configured to display images corresponding to virtual objects in accordance with display instructions to a first individual corresponding to a user of the NED;
an imaging device configured to capture one or more images of a portions of a local area of the NED; and
a controller configured to:
analyze the one or more captured images using a trained model to identify a plurality of hand poses associated with a hand of a second individual located within the local area different from the first individual;
identify a first gesture performed by the hand of the second individual based upon the plurality of identified hand poses, the first gesture corresponding to a predetermined sequence of hand poses over time;
identify a portion of the second individual's body, based upon the identified first gesture; and
update the display instructions to cause the electronic display to display, in an augmented reality environment as presented to the first individual via the electronic display, at least a first virtual object indicating the identified first gesture at a location corresponding to the identified portion of the second individual's body.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The system of claim 1, wherein the first gesture comprises a movement by a first hand of the second individual in a path that intersects a surface of an object within the local area.

### Claim 4 (depends on 1)

The system of claim 1, wherein the first gesture comprises a hand position in which the hands of the second individual are located in predetermined positions relative to each other and maintain their positions for at least a threshold period of time.

### Claim 5 (depends on 1)

The system of claim 1, further comprising an audio sensor configured to detect audio information corresponding to a sound in the local environment, the audio information detected within a threshold period of time as the first gesture, and wherein the controller is further configured to update the display instructions to cause the electronic display to display the first virtual object based upon detected audio information and the first gesture.

### Claim 6 (depends on 5)

The system of claim 5, wherein the first virtual object comprises content based upon a transcription of detected audio information.

### Claim 7 (depends on 1)

The system of claim 1, further comprising an audio device configured to play audio content to the user, and the controller is further configured to cause the audio device to play audio content to the user based upon the first gesture contemporaneously with the electronic display displaying the first virtual object.

### Claim 8 (depends on 1)

The system of claim 1, wherein the controller is further configured to:
analyze the one or more captured images using the trained model to identify one or more joint positions associated with the hand of the second individual; and
convert the identified one or more joint positions into a hand pose.

### Claim 9 (depends on 1)

The system of claim 1, wherein the first gesture corresponds to a predetermined sequence of hand poses over time in which a movement of the hand exceeds a threshold speed.

### Claim 10 (depends on 1)

The system of claim 1, wherein the controller is further configured to identify the first gesture based upon a position of the hand of the second individual relative to the second individual's body as indicated by the plurality of hand poses.

### Claim 11 (depends on 1)

The system of claim 1, wherein the first virtual object is selected based upon a movement speed of the hand as determined based upon the plurality of hand poses.

### Claim 12 (depends on 1)

The system of claim 1, wherein a rate at which the first virtual object is displayed or a rate of movement of the first virtual object is based upon a movement speed of the hand as determined based upon the plurality of hand poses.

### Claim 13

(canceled)

### Claim 14

A system comprising:
a near eye display (NED) comprising:
an optical assembly having an electronic display that is configured to display images corresponding to virtual objects in accordance with display instructions;
an imaging device configured to capture one or more images of a portions of a local area of the NED;
an audio sensor configured to detect audio information corresponding to sounds in the local environment; and
a controller configured to:
analyze the one or more captured images using a trained model to identify a plurality of hand poses associated with a hand of an individual located within the local area;
identify a first gesture performed by the hand of the individual based upon the plurality of identified hand poses, the first gesture corresponding to a predetermined sequence of hand poses over time;
using the audio sensor, identify a first sound in the local environment detected within a threshold period of time as the identified first gesture; and
update the display instructions to cause the electronic display to display, in an augmented reality environment as presented to a user of the NED via the electronic display, at least a first virtual object selected based upon the first gesture and the first sound, the first virtual object displayed at a location associated with a predetermined portion of the individual's body associated with the first gesture.
