# US 2017/0287194 A1 · Tracking portions of a user's face uncovered by a head mounted display worn by the user

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0287194 A1
- Document ID: US20170287194A1
- Status: Published application
- Kind: A1
- Application number: 15/089,447
- Filed: 2016-04-01
- Published: 2017-10-05
- Earliest claimed: 2016-04-01
- Assignees: Oculus VR, LLC
- Applicants: Oculus VR, LLC
- Inventors: Dov Katz; Michael John Toksvig; Ziheng Wang; Timothy Paul Omernick; Torin Ross Herndon
- Web page: https://patentgrep.com/patent/US20170287194A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G02B 27/0176: {characterised by mechanical features}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06T 7/33: using feature-based methods
- G06V 40/166: {using acquisition arrangements}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- H04N 23/20: for generating image signals from infrared radiation only
- H04N 23/50: Constructional details
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2207/30201: Face
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A virtual reality system includes a head-mounted display (HMD) having one or more facial sensors and illumination sources mounted to a surface of the HMD. For example, the facial sensors are image capture devices coupled to a bottom side of the HMD. The illumination sources illuminate portions of a user's face outside of the HMD, while the facial sensors capture images of the illuminated portions of the user's face. A controller receives the captured images and generates a representation of the portions of the user's face by identifying landmarks of the user's face in the captured images and performing other suitable image processing methods. Based on the representation, the controller or another component of the virtual reality system generates content for presentation to the user.

## Claims

### Claim 1

A head mounted display (HMD) comprising:
a rigid body including display element configured to display content to a user wearing the HMD and an optics block configured to direct light from the display element to an exit pupil of the HMD;
one or more illumination sources coupled to a surface of the rigid body and configured illuminate portions of a face of the user that are external to the rigid body;
one or more image capture devices coupled to the surface of the rigid body and configured to capture one or more images of one or more portions of the face of the user external to the rigid body that are illuminated by the one or more illumination sources; and
a controller configured to receive one or more captured images from the one or more image capture devices and to generate a reconstruction of the portion of the face of the user from the one or more captured images.

### Claim 2 (depends on 1)

The HMD of claim 1, wherein the controller is further configured to:
provide instructions to the user to perform one or more facial expressions at different time intervals;
receive one or more images from the one or more image capture devices captured during the different time intervals, images captured during a time interval corresponding to a facial expression the user was instructed to perform during the time interval;
identify one or more landmarks of the face of the user based on images captured during time intervals corresponding to different facial expressions the user was instructed to perform; and
generate the reconstruction of the portion of the face of the user based on the one or more landmarks and the one or more captured images.

### Claim 3 (depends on 1)

The HMD of claim 1, wherein generate the reconstruction of the portion of the face of the user from the one or more captured images comprises:
identify a pixel having a maximum brightness in each of the captured images;
generate mappings between the brightest pixel in each of the captured images and a surface of the portion of the face of the user; and
generate the reconstruction of the portion of the face of the user based on the mappings.

### Claim 4 (depends on 1)

The HMD of claim 1, wherein an image capture device comprises an infrared camera.

### Claim 5 (depends on 4)

The HMD of claim 4, wherein the one or more illumination sources are configured to emit infrared light.

### Claim 6 (depends on 1)

The HMD of claim 1, wherein the surface of the rigid body comprises a bottom side of the HMD.

### Claim 7 (depends on 1)

The HMD of claim 1, wherein the controller is further configured to generate content based on the reconstruction and provide the content to the electronic display, such as instructions for displaying the reconstruction, and provides the content to the display element for presentation to the user

### Claim 8 (depends on 1)

The HMD of claim 1, wherein the controller is further configured to communicate the reconstruction to a virtual reality console that generates content based on the reconstruction and the display element is configured to receive the generated content from the virtual reality console and to present the generated content to the user.

### Claim 9 (depends on 1)

The HMD of claim 1, wherein the controller is further configured to store the reconstruction in association with an identifier of the user.

### Claim 10

A head mounted display (HMD) comprising:
a rigid body including display element configured to display content to a user wearing the HMD and an optics block configured to direct light from the display element to an exit pupil of the HMD;
one or more illumination sources coupled to a surface of the rigid body and configured illuminate portions of a face of the user that are external to the rigid body;
one or more facial sensors coupled to the surface of the rigid body and configured to capture characteristics of one or more portions of the face of the user external to the rigid body that are illuminated by the one or more illumination sources; and
a controller configured to receive the captured characteristics from the one or more image capture devices and to generate a reconstruction of the portion of the face of the user from the one or more captured characteristics.

### Claim 11 (depends on 11)

The HMD of claim 11, wherein the one or more facial sensors include one or more non-optical sensors.

### Claim 12 (depends on 11)

The HMD of claim 11, wherein a non-optical sensor is selected from a group consisting of: an audio sensor, a strain gauge, an electric sensor, a magnetic sensor, a proximity sensor, and any combination thereof.

### Claim 13 (depends on 10)

The HMD of claim 10, wherein the controller is further configured to:
provide instructions to the user to perform one or more facial expressions at different time intervals;
receive characteristics captured by the one or more facial sensors during the different time intervals, characteristics captured during a time interval corresponding to a facial expression the user was instructed to perform during the time interval;
identify one or more landmarks of the face of the user based at least in part on one or more characteristics corresponding to facial expression the user was instructed to perform during different time intervals; and
generate the reconstruction of the portion of the face of the user based on the one or more landmarks and the captured characteristics.

### Claim 14 (depends on 10)

The HMD of claim 10, wherein one or more of the facial sensors comprise image capture devices.

### Claim 15 (depends on 14)

The HMD of claim 14, wherein the one or more captured characteristics comprise images of one or more portions of the face of the user external to the rigid body that are illuminated by the one or more illumination sources and wherein generate the reconstruction of the portion of the face of the user from the one or more captured characteristics comprises:
identify a pixel having a maximum brightness in each of the captured images;
generate mappings between the brightest pixel in each of the captured images and a surface of the portion of the face of the user; and
generate the reconstruction of the portion of the face of the user based on the mappings.

### Claim 16 (depends on 14)

The HMD of claim 14, wherein an image capture device comprises an infrared camera.

### Claim 17 (depends on 16)

The HMD of claim 16, wherein the one or more illumination sources are configured to emit infrared light.

### Claim 18 (depends on 1)

The HMD of claim 1, wherein the controller is further configured to generate content based on the reconstruction and provide the content to the electronic display, such as instructions for displaying the reconstruction, and provides the content to the display element for presentation to the user

### Claim 19 (depends on 1)

The HMD of claim 1, wherein the controller is further configured to communicate the reconstruction to a virtual reality console that generates content based on the reconstruction and the display element is configured to receive the generated content from the virtual reality console and to present the generated content to the user.

### Claim 20

A method comprising:
illuminating a portion of a face of a user wearing a head mounted display (HMD) external to the HMD using one or more illumination sources coupled to one or more surfaces of the HMD;
capturing images of the portion of the face illuminated by the one or more illumination sources;
identifying a pixel having a maximum brightness in each of the captured images;
generating mappings between the identified pixel each of the captured images and a surface of the portion of the face of the user;
generating the reconstruction of the portion of the face of the user based on the mappings; and
providing content based on the generated reconstruction for presentation to the user via the HMD.
