# US 2023/0419593 A1 · Context-based object viewing within 3d environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419593 A1
- Document ID: US20230419593A1
- Status: Published application
- Kind: A1
- Application number: 18/244,355
- Filed: 2023-09-11
- Published: 2023-12-28
- Earliest claimed: 2021-03-18
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Alexandre Da Veiga; Tobias Rick; Timothy R. Pease
- Web page: https://patentgrep.com/patent/US20230419593A1

## CPC

- G06T 15/20: Perspective computation
- H04N 13/156: Mixing image signals
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06T 7/593: from stereo images
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/361: Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
- H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Various implementations disclosed herein include devices, systems, and methods that present views of media objects using different viewing states determined based on context. In some implementations, a view of a 3D environment is presented. Then, a context associated with viewing one or more media objects within the 3D environment is determined, the media objects associated with data for providing an appearance of depth within the one or more media objects. Based on the context, a viewing state is determined for viewing a media object of the one or more media objects within the 3D environment, the viewing state defining whether the media object will be presented as a planar object or with depth within the media object. In accordance with a determination that the viewing state is a first viewing state, the media object is presented within the 3D environment using its associated data for providing the appearance of depth.

## Claims

### Claim 1

A method comprising:
at a processor:
presenting a view of a three-dimensional (3D) environment;
determining a context associated with viewing one or more media objects within the 3D environment, the one or more media objects associated with data for providing an appearance of depth within the one or more media objects;
based on the context, determining a viewing state for viewing a media object of the one or more media objects within the 3D environment, the viewing state defining whether the media object will be presented as a planar object or with depth within the media object; and
in accordance with a determination that the viewing state is a first viewing state, presenting the media object within the 3D environment using its associated data for providing an appearance of depth.

### Claim 2 (depends on 1)

The method of claim 1, the method further comprising in accordance with a determination that the viewing state is a second viewing state, presenting the media object within the 3D environment without using its associated data for providing an appearance of depth.

### Claim 3 (depends on 1)

The method of claim 1, wherein the second viewing state comprises a monoscopic viewing state in which a left eye view and a right eye view receive a pair of the same view of the media object.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first viewing state comprises a stereo viewing state in which a left eye view and a right eye view receive a stereoscopic pair of viewing locations within the 3D environment generated using image-pairs captured at different locations within a capturing environment.

### Claim 5 (depends on 4)

The method of claim 4, wherein the stereo viewing state of the media object is presented for different viewing positions of the media object in the 3D environment.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first viewing state comprises a stereo viewing state in which a left eye view and a right eye view provide different views of the media object by:
separating a foreground portion and a background portion of the media object using the associated data for providing an appearance of depth, wherein the associated data for providing an appearance of depth comprises depth data associated with the media object; and
adjusting the foreground portion and the background portion to provide the different views for the left eye view and the right eye view.

### Claim 7 (depends on 1)

The method of claim 1, wherein the associated data for providing an appearance of depth is obtained from a depth sensor.

### Claim 8 (depends on 1)

The method of claim 1, wherein the associated data for providing an appearance of depth is obtained from stereo pair depth estimation.

### Claim 9 (depends on 1)

The method of claim 1, wherein the associated data for providing an appearance of depth is obtained from computer vision techniques.

### Claim 10 (depends on 1)

The method of claim 1, wherein the view depicts depth of objects of the 3D environment using first data corresponding to the 3D positions of the objects within the 3D environment.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
based on the context, determining a viewing state for viewing a second media object of the one or more media objects within the 3D environment; and
in accordance with a determination that the viewing state for viewing the second media object is a second viewing state different than the first viewing state, presenting, concurrently with the media object, the second media object within the 3D environment without using its associated data for providing an appearance of depth.

### Claim 12 (depends on 1)

The method of claim 1, wherein determining the context comprises determining whether a gaze direction is directed towards one of the one or more media objects within the 3D environment.

### Claim 13 (depends on 1)

The method of claim 1, wherein determining the context comprises determining whether there is an interaction with one of the one or more media objects within the 3D environment.

### Claim 14 (depends on 1)

The method of claim 1, wherein determining the context comprises determining whether an orientation is directed to within a distance of one of the one or more media objects in the 3D environment.

### Claim 15 (depends on 1)

The method of claim 1, wherein determining the context comprises determining whether a user interface component overlays one of the one or more media objects in the view of the 3D environment.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
determining an updated context associated with viewing the one or more media objects within the 3D environment;
based on the updated context, determining an updated viewing state for viewing the media object within the 3D environment; and
in accordance with a determination that the viewing state is the first viewing state and the updated viewing state is a second viewing state, transitioning the presentation of the media object from presenting the media object using its associated data for providing an appearance of depth to presenting the media object without using its associated data for providing an appearance of depth over a time period.

### Claim 17 (depends on 16)

The method of claim 16, further comprising transitioning the viewing state from the first viewing state to a second viewing state by adjusting one eye's view of the media object to match the other eye's view of the media object over a time period.

### Claim 18 (depends on 16)

The method of claim 16, wherein the transitioning comprises performing a reprojection for each of multiple frames of a transition using depth data or a 3D representation of the 3D environment.

### Claim 19 (depends on 16)

The method of claim 16, wherein the transitioning is based on the associated data for providing an appearance of depth, wherein the associated data for providing an appearance of depth comprises a 3D geometry of content of the media object.

### Claim 20 (depends on 16)

The method of claim 16, wherein at least some content of the media object is hidden or obscured during the transitioning.

### Claim 21 (depends on 1)

The method of claim 1, wherein the media object comprises a 3D photo.

### Claim 22 (depends on 1)

The method of claim 1, wherein the media object comprises a 3D video.

### Claim 23 (depends on 1)

The method of claim 1, wherein the media object comprises a 3D image of a virtual environment.

### Claim 24

A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
presenting a view of a three-dimensional (3D) environment;
determining a context associated with viewing one or more media objects within the 3D environment, the one or more media objects associated with data for providing an appearance of depth within the one or more media objects;
based on the context, determining a viewing state for viewing a media object of the one or more media objects within the 3D environment, the viewing state defining whether the media object will be presented as a planar object or with depth within the media object; and
in accordance with a determination that the viewing state is a first viewing state, presenting the media object within the 3D environment using its associated data for providing an appearance of depth.

### Claim 25

A non-transitory computer-readable storage medium, storing program instructions executable via one or more processors to perform operations comprising:
presenting a view of a three-dimensional (3D) environment;
determining a context associated with viewing one or more media objects within the 3D environment, the one or more media objects associated with data for providing an appearance of depth within the one or more media objects;
based on the context, determining a viewing state for viewing a media object of the one or more media objects within the 3D environment, the viewing state defining whether the media object will be presented as a planar object or with depth within the media object; and
in accordance with a determination that the viewing state is a first viewing state, presenting the media object within the 3D environment using its associated data for providing an appearance of depth.
