# US 11,308,686 B1 · Captured image data in a computer-generated reality environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,686 B1
- Document ID: US11308686B1
- Status: Granted
- Kind: B1
- Application number: 17/100,685
- Filed: 2020-11-20
- Granted: 2022-04-19
- Earliest claimed: 2017-09-29
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Clément Pierre Nicolas Boissière; Samuel Lee Iglesias; Edwin Iskandar; Timothy R. Oriol
- Web page: https://patentgrep.com/patent/US11308686B1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 15/20: Perspective computation
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure relates generally to for displaying a set of images based on captured image data. In some embodiments, a system displays a first set of images corresponding to a first perspective in a virtual reality (VR) (or mixed reality (MR)) environment. The system receives a request while displaying the first set of images. The system captures image data corresponding to a second perspective in the VR (or MR) environment in response to receiving the request. The system displays a second set of images based on the captured image data.

## Claims

### Claim 1

A system, comprising:
a display;
an orientation sensor;
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, on a display, to a user, a first set of images corresponding to a first perspective in a virtual reality (VR) environment, wherein the first set of images is based on a first head position of the user;
while displaying the first set of images, receiving a request;
in response to receiving the request, capturing image data corresponding to a second perspective in the VR environment,
wherein the captured image data is based on a position of the user; and
wherein the first perspective is different from the second perspective;
displaying, on the display, a second set of images, wherein the second set of images is based on the captured image data corresponding to the second perspective and is based on a second head position of the user;
while displaying the second set of images, detecting, using an orientation sensor, a change in head position of the user from the second head position to a third head position; and
displaying, on the display, a third set of images, wherein the third set of images is based on the captured image data corresponding to the second perspective and is based on the third head position of the user.

### Claim 2 (depends on 1)

The system of claim 1, wherein the display is head-mounted.

### Claim 3 (depends on 1)

The system of claim 1, wherein the captured image data corresponds to a point in time.

### Claim 4 (depends on 1)

The system of claim 1, wherein the captured image data corresponds to a period of time.

### Claim 5 (depends on 1)

The system of claim 1, the one or more programs further including instructions for:
positioning a virtual camera at a location corresponding to the second perspective in the VR environment.

### Claim 6 (depends on 5)

The system of claim 5, wherein the location of the virtual camera is based on user input.

### Claim 7 (depends on 5)

The system of claim 5, wherein the positioning of the virtual camera occurs automatically without user input.

### Claim 8 (depends on 1)

The system of claim 1, wherein the captured image data further corresponds to the first perspective in the VR environment, the one or more programs further including instructions for:
while displaying the second set of images, concurrently displaying, on the display, a related set of images, wherein the related set of images is based on the captured image data corresponding to the first perspective.

### Claim 9 (depends on 1)

The system of claim 1, the one or more programs further including instructions for:
prior to receiving the request, not capturing image data.

### Claim 10

A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a system with a display and an orientation sensor, the one or more programs including instructions for:
displaying, on a display, to a user, a first set of images corresponding to a first perspective in a virtual reality (VR) environment, wherein the first set of images is based on a first head position of the user;
while displaying the first set of images, receiving a request;
in response to receiving the request, capturing image data corresponding to a second perspective in the VR environment,
wherein the captured image data is based on a position of the user; and
wherein the first perspective is different from the second perspective;
displaying, on the display, a second set of images, wherein the second set of images is based on the captured image data corresponding to the second perspective and is based on a second head position of the user;
while displaying the second set of images, detecting, using an orientation sensor, a change in head position of the user from the second head position to a third head position; and
displaying, on the display, a third set of images, wherein the third set of images is based on the captured image data corresponding to the second perspective and is based on the third head position of the user.

### Claim 11 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the display is head-mounted.

### Claim 12 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the captured image data corresponds to a point in time.

### Claim 13 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the captured image data corresponds to a period of time.

### Claim 14 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, the one or more programs further including instructions for:
positioning a virtual camera at a location corresponding to the second perspective in the VR environment.

### Claim 15 (depends on 14)

The non-transitory computer-readable storage medium of claim 14, wherein the location of the virtual camera is based on user input.

### Claim 16 (depends on 14)

The non-transitory computer-readable storage medium of claim 14, wherein the positioning of the virtual camera occurs automatically without user input.

### Claim 17 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the captured image data further corresponds to the first perspective in the VR environment, the one or more programs further including instructions for:
while displaying the second set of images, concurrently displaying, on the display, a related set of images, wherein the related set of images is based on the captured image data corresponding to the first perspective.

### Claim 18 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, the one or more programs further including instructions for:
prior to receiving the request, not capturing image data.

### Claim 19

A method for capturing image data in a virtual reality (VR) environment, comprising:
displaying, on a display, to a user, a first set of images corresponding to a first perspective in the VR environment, wherein the first set of images is based on a first head position of the user;
while displaying the first set of images, receiving a request;
in response to receiving the request, capturing image data corresponding to a second perspective in the VR environment,
wherein the captured image data is based on a position of the user; and
wherein the first perspective is different from the second perspective;
displaying, on the display, a second set of images, wherein the second set of images is based on the captured image data corresponding to the second perspective and is based on a second head position of the user;
while displaying the second set of images, detecting, using an orientation sensor, a change in head position of the user from the second head position to a third head position; and
displaying, on the display, a third set of images, wherein the third set of images is based on the captured image data corresponding to the second perspective and is based on the third head position of the user.

### Claim 20 (depends on 19)

The method of claim 19, wherein the display is head-mounted.

### Claim 21 (depends on 19)

The method of claim 19, further comprising:
positioning a virtual camera at a location corresponding to the second perspective in the VR environment.

### Claim 22 (depends on 21)

The method of claim 21, wherein the location of the virtual camera is based on user input.

### Claim 23 (depends on 21)

The method of claim 21, wherein the positioning of the virtual camera occurs automatically without user input.

### Claim 24 (depends on 19)

The method of claim 19, wherein the captured image data further corresponds to the first perspective in the VR environment, further comprising:
while displaying the second set of images, concurrently displaying, on the display, a related set of images, wherein the related set of images is based on the captured image data corresponding to the first perspective.

### Claim 25 (depends on 19)

The method of claim 19, further comprising:
prior to receiving the request, not capturing image data.