# US 2024/0404230 A1 · Late-Stage Warp of Dynamic-Anchored Content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0404230 A1
- Document ID: US20240404230A1
- Status: Published application
- Kind: A1
- Application number: 18/672,992
- Filed: 2024-05-23
- Published: 2024-12-05
- Earliest claimed: 2023-06-02
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Conner J. Brooks; Seyedkoosha Mirhosseini
- Web page: https://patentgrep.com/patent/US20240404230A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G02B 27/017: {Head mounted}
- G06F 3/012: {Head tracking input arrangements}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 2207/30201: Face
- G06T 2219/2016: Rotation, translation, scaling
- G06V 20/20: in augmented reality scenes
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one implementation, a method of displaying an image is performed by a device including an image sensor, a display, one or more processors, and non-transitory memory. The method includes capturing, using the image sensor, an image of an object in a physical environment. The method includes obtaining, based on the image, a first predicted object pose of the object in the physical environment at a display time. The method includes rendering virtual content based on the first predicted object pose. The method includes obtaining a second predicted object pose of the object in the physical environment at the display time. The method includes warping the virtual content based on the second predicted object pose. The method includes displaying, on the display at the display time, the warped virtual content.

## Claims

### Claim 1

A method comprising:
at a device including an image sensor, a display, one or more processors, and non-transitory memory:
capturing, using the image sensor, an image of an object in a physical environment;
obtaining, based on the image, a first predicted object pose of the object in the physical environment at a display time;
rendering virtual content based on the first predicted object pose;
obtaining a second predicted object pose of the object in the physical environment at the display time;
warping the virtual content based on the second predicted object pose; and
displaying, on the display at the display time, the warped virtual content.

### Claim 2 (depends on 1)

The method of claim 1, wherein the object is a dynamic object in the physical environment.

### Claim 3 (depends on 2)

The method of claim 2, wherein the object is a hand of a user of the device.

### Claim 4 (depends on 2)

The method of claim 2, wherein the object is the face of a person.

### Claim 5 (depends on 1)

The method of claim 1, wherein obtaining the second predicted object pose is based on an additional image of the object.

### Claim 6 (depends on 1)

The method of claim 1, wherein obtaining the second predicted object pose is based on data received from an inertial measurement unit.

### Claim 7 (depends on 1)

The method of claim 1, wherein warping the virtual content is based on a difference between the first predicted object pose and the second predicted object pose.

### Claim 8 (depends on 1)

The method of claim 1, wherein warping the virtual content includes performing a homographic transformation of the virtual content.

### Claim 9 (depends on 1)

The method of claim 1, wherein warping the virtual content includes shifting the virtual content.

### Claim 10 (depends on 1)

The method of claim 1, further comprising generating a stencil indicating portions of the virtual content to be warped based on the second predicted object pose, wherein warping the virtual content is further based on the stencil.

### Claim 11 (depends on 10)

The method of claim 10, wherein the stencil further indicates portions of the virtual content that are not to be warped.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
obtaining a first predicted device pose of the device at the display time, wherein rendering the virtual content is based on a first predicted device pose, and
after rendering the virtual content, obtaining a second predicted device pose of the device at the display time, wherein warping the virtual content is based on the second predicted device pose.

### Claim 13 (depends on 12)

The method of claim 12, wherein obtaining the second predicted device pose is based on data received from an inertial measurement unit of the device.

### Claim 14

A device comprising:
an image sensor;
a display;
a non-transitory memory; and
one or more processors to:
capture, using the image sensor, an image of an object in a physical environment;
obtain, based on the image, a first predicted object pose of the object in the physical environment at a display time;
render virtual content based on the first predicted object pose;
obtain a second predicted object pose of the object in the physical environment at the display time;
warp the virtual content based on the second predicted object pose; and
display, on the display at the display time, the warped virtual content.

### Claim 15 (depends on 14)

The device of claim 14, wherein the object is a dynamic object in the physical environment.

### Claim 16 (depends on 15)

The device of claim 15, wherein the object is a hand of a user of the device.

### Claim 17 (depends on 15)

The device of claim 15, wherein the object is the face of a person.

### Claim 18 (depends on 14)

The device of claim 14, wherein the one or more processors are to warp the virtual content based on a difference between the first predicted object pose and the second predicted object pose.

### Claim 19 (depends on 14)

The device of claim 14, wherein the one or more processors are further to:
obtain a first predicted device pose of the device at the display time, wherein rendering the virtual content is based on a first predicted device pose, and
after rendering the virtual content, obtain a second predicted device pose of the device at the display time, wherein warping the virtual content is based on the second predicted device pose.

### Claim 20

A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device including an image sensor and a display, cause the device to:
capture, using the image sensor, an image of an object in a physical environment;
obtain, based on the image, a first predicted object pose of the object in the physical environment at a display time;
render virtual content based on the first predicted object pose;
obtain a second predicted object pose of the object in the physical environment at the display time;
warp the virtual content based on the second predicted object pose; and
display, on the display at the display time, the warped virtual content.
