# US 2024/0004513 A1 · Devices, Methods, and Graphical User Interfaces for Providing Computer-Generated Experiences

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004513 A1
- Document ID: US20240004513A1
- Status: Published application
- Kind: A1
- Application number: 18/368,532
- Filed: 2023-09-14
- Published: 2024-01-04
- Earliest claimed: 2020-09-15
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Philipp Rockel; William A. Sorrentino, III; Stephen O. Lemay; Peter D. Anton
- Web page: https://patentgrep.com/patent/US20240004513A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

At a computer system, in response to detecting a respective user input: if the respective user input meets first criteria requiring the one or both hands of the user maintain a first grip while a first type of movement of the one or both hands is detected, the computer system moves a virtual object in a three-dimensional environment relative to the three-dimensional environment in accordance with the movement of the one or both hands in the three-dimensional environment; and if the respective user input meets second criteria requiring the one or both hands of the user maintain a second grip while a second type of movement of the one or both hands is detected, the computer system replaces the view of the three-dimensional environment corresponding to a first viewpoint with a view of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint.

## Claims

### Claim 1

A method, comprising:
at a computer system including a display generation component and one or more input devices:
displaying, via the display generation component, a view of at least a portion of a three-dimensional environment corresponding to a first viewpoint;
while displaying the view of the at least the portion of the three-dimensional environment corresponding to the first viewpoint, detecting a respective user input by a user of the computer system, including detecting movement of one or both hands of a user in a physical environment; and
in response to detecting the respective user input:
in accordance with a determination that the respective user input meets first criteria, wherein the first criteria include a first criterion that is met when the one or both hands of the user maintain a first grip while a first type of movement of the one or both hands is detected, moving a virtual object in the three-dimensional environment relative to the three-dimensional environment in accordance with the movement of the one or both hands in the physical environment; and
in accordance with a determination that the respective user input meets second criteria different from the first criteria, wherein the second criteria include a second criterion that is met when the one or both hands of the user maintain a second grip that is different from the first grip while a second type of movement of the one or both hands is detected, replacing the view of at least the portion of the three-dimensional environment corresponding to the first viewpoint with a view of at least the portion of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint.

### Claim 2 (depends on 1)

The method of claim 1, wherein detecting that the one or both hands of the user are maintaining the first grip includes detecting that a first set of fingers of a respective hand of the one or both hands of the user is maintaining contact with a palm of the respective hand; and wherein detecting that the one or both hands of the user are maintaining the second grip includes detecting a third set of fingers of the respective hand of the one or both hands are maintaining contact with a fourth set of fingers of the respective hand of the one or both hands.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the first criterion is met when a single hand of the user maintains the first grip while the first type of movement of the single hand is detected, and
the second criterion is met when the single hand of the user maintains the second grip while the second type of movement of the single hand is detected.

### Claim 4 (depends on 3)

The method of claim 3, wherein:
the first type of movement is the same as the second type of movement,
moving the virtual object in the three-dimensional environment in accordance with the movement of the one or both hands in the physical environment includes moving the virtual object in the three-dimensional environment in accordance with the movement of the single hand of the user in the physical environment while the single hand maintains the first grip, and
replacing the view corresponding to the first viewpoint with the view corresponding to the second viewpoint includes displaying an animated transition corresponding to moving a viewpoint of a currently displayed view of the three-dimensional environment from the first viewpoint to the second viewpoint in accordance with the movement of the single hand of the user in the physical environment while the single hand maintains the second grip.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
moving the virtual object in the three-dimensional environment includes translating the virtual object in the three-dimensional environment relative to a position corresponding to the first viewpoint in accordance with the movement of the single hand while the single hand maintains the first grip; and
moving the viewpoint of the currently displayed view of the three-dimensional environment from the first viewpoint to the second viewpoint includes translating the viewpoint in accordance with the movement of the single hand while the single hand maintains the second grip.

### Claim 6 (depends on 4)

The method of claim 4, wherein:
moving the virtual object in the three-dimensional environment includes rotating the virtual object in the three-dimensional environment around a position corresponding to the first viewpoint; and
moving the viewpoint of the currently displayed view of the three-dimensional environment from the first viewpoint to the second viewpoint includes rotating the viewpoint around an anchor position with a predetermined spatial relationship to the first viewpoint.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the first criterion is met when both hands of the user maintain the first grip while the first type of movement of the both hands of the user is detected, and
the second criterion is met when both hand of the user maintain the second grip while the second type of movement of the both hands of the user is detected.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the first type of movement is the same as the second type of movement;
moving the virtual object in the three-dimensional environment in accordance with the movement of the one or both hands in the physical environment includes rotating the virtual object around an axis of the virtual object in accordance with the movement of both hands of the user in the physical environment while the both hands maintain the first grip; and
replacing the view corresponding to the first viewpoint with the view corresponding to the second viewpoint includes displaying an animated transition showing rotation of a virtual position of the user in the three-dimensional environment around the axis of the virtual object in accordance with the movement of both hands of the user in the physical environment while the both hands maintain the second grip.

### Claim 9 (depends on 1)

The method of claim 1, further including:
in response to detecting the respective user input:
in accordance with a determination that the respective user input meets third criteria, wherein the third criteria include a third criterion that is met when the both hands of the user maintain a third grip while detecting a third type of movement of the both hands, resizing the virtual object in the three-dimensional environment relative to the three-dimensional environment in accordance with the movement of the both hands in the physical environment.

### Claim 10 (depends on 1)

The method of claim 1, wherein:
detecting the respective user input includes detecting a gaze input directed to a portion of the three-dimensional environment in conjunction with detecting the movement of both hands of the user in the three-dimensional environment; and
in response to detecting the respective user input:
in accordance with a determination that the respective user input meets fourth criteria, wherein the fourth criteria include a fourth criterion that is met when both hands of the user maintain a fourth grip while a fourth type of movement of the both hands of the user is detected and the portion of the three-dimensional environment corresponds to a position of a movable virtual object, rotating the movable virtual object in the three-dimensional environment; and
in accordance with a determination that the respective user input meets fifth criteria, wherein the fifth criteria include a fifth criterion that is met when both hands of the user maintain a fifth grip while a fifth type of movement of the both hands of the user is detected and the portion of the three-dimensional environment does not correspond to a movable virtual object, rotating the three-dimensional environment around a virtual position of the user in the three-dimensional environment.

### Claim 11 (depends on 1)

The method of claim 1, further including:
in response to detecting the respective user input:
in accordance with a determination that the respective user input meets sixth criteria, wherein the sixth criteria include a sixth criterion that is met when both hands of the user maintain a sixth grip while a flexing movement of the both hands is detected, adjusting a height of the virtual object in the three-dimensional environment in accordance with the flexing movement of the both hands.

### Claim 12 (depends on 1)

The method of claim 1, further including:
in response to detecting the respective user input:
in accordance with a determination that the respective user input meets seventh criteria, wherein the seventh criteria include a seventh criterion that is met when a single hand of the user maintains a seventh grip while detecting flexing movement of the single hand, moving the virtual object in the three-dimensional environment towards a virtual position of the user in the three-dimensional environment.

### Claim 13

A computer system, comprising:
a display generation component;
one or more input devices;
one or more processors; and
memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, via the display generation component, a view of at least a portion of a three-dimensional environment corresponding to a first viewpoint;
while displaying the view of the at least the portion of the three-dimensional environment corresponding to the first viewpoint, detecting a respective user input, including detecting movement of one or both hands of a user in a physical environment; and
in response to detecting the respective user input:
in accordance with a determination that the respective user input meets first criteria, wherein the first criteria include a first criterion that is met when the one or both hands of the user maintain a first grip while a first type of movement of the one or both hands is detected, moving a virtual object in the three-dimensional environment relative to the three-dimensional environment in accordance with the movement of the one or both hands in the physical environment; and
in accordance with a determination that the respective user input meets second criteria different from the first criteria, wherein the second criteria include a second criterion that is met when the one or both hands of the user maintain a second grip that is different from the first grip while a second type of movement of the one or both hands is detected, replacing the view of at least the portion of the three-dimensional environment corresponding to the first viewpoint with a view of at least the portion of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint.

### Claim 14 (depends on 13)

The computer system of claim 13, wherein detecting that the one or both hands of the user are maintaining the first grip includes detecting that a first set of fingers of a respective hand of the one or both hands of the user is maintaining contact with a palm of the respective hand; and wherein detecting that the one or both hands of the user are maintaining the second grip includes detecting a third set of fingers of the respective hand of the one or both hands are maintaining contact with a fourth set of fingers of the respective hand of the one or both hands.

### Claim 15 (depends on 13)

The computer system of claim 13, wherein:
the first criterion is met when a single hand of the user maintains the first grip while the first type of movement of the single hand is detected, and
the second criterion is met when the single hand of the user maintains the second grip while the second type of movement of the single hand is detected.

### Claim 16 (depends on 15)

The computer system of claim 15, wherein:
the first type of movement is the same as the second type of movement,
moving the virtual object in the three-dimensional environment in accordance with the movement of the one or both hands in the physical environment includes moving the virtual object in the three-dimensional environment in accordance with the movement of the single hand of the user in the physical environment while the single hand maintains the first grip, and
replacing the view corresponding to the first viewpoint with the view corresponding to the second viewpoint includes displaying an animated transition corresponding to moving a viewpoint of a currently displayed view of the three-dimensional environment from the first viewpoint to the second viewpoint in accordance with the movement of the single hand of the user in the physical environment while the single hand maintains the second grip.

### Claim 17 (depends on 16)

The computer system of claim 16, wherein:
moving the virtual object in the three-dimensional environment includes translating the virtual object in the three-dimensional environment relative to a position corresponding to the first viewpoint in accordance with the movement of the single hand while the single hand maintains the first grip; and
moving the viewpoint of the currently displayed view of the three-dimensional environment from the first viewpoint to the second viewpoint includes translating the viewpoint in accordance with the movement of the single hand while the single hand maintains the second grip.

### Claim 18 (depends on 16)

The computer system of claim 16, wherein:
moving the virtual object in the three-dimensional environment includes rotating the virtual object in the three-dimensional environment around a position corresponding to the first viewpoint; and
moving the viewpoint of the currently displayed view of the three-dimensional environment from the first viewpoint to the second viewpoint includes rotating the viewpoint around an anchor position with a predetermined spatial relationship to the first viewpoint.

### Claim 19 (depends on 13)

The computer system of claim 13, wherein:
the first criterion is met when both hands of the user maintain the first grip while the first type of movement of the both hands of the user is detected, and
the second criterion is met when both hand of the user maintain the second grip while the second type of movement of the both hands of the user is detected.

### Claim 20

A computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system that includes a display generation component and one or more input devices, cause the computer system to:
display, via the display generation component, a view of at least a portion of a three-dimensional environment corresponding to a first viewpoint;
while displaying the view of the at least the portion of the three-dimensional environment corresponding to the first viewpoint, detect a respective user input, including detecting movement of one or both hands of a user in a physical environment; and
in response to detecting the respective user input:
in accordance with a determination that the respective user input meets first criteria, wherein the first criteria include a first criterion that is met when the one or both hands of the user maintain a first grip while a first type of movement of the one or both hands is detected, move a virtual object in the three-dimensional environment relative to the three-dimensional environment in accordance with the movement of the one or both hands in the physical environment; and
in accordance with a determination that the respective user input meets second criteria different from the first criteria, wherein the second criteria include a second criterion that is met when the one or both hands of the user maintain a second grip that is different from the first grip while a second type of movement of the one or both hands is detected, replace the view of at least the portion of the three-dimensional environment corresponding to the first viewpoint with a view of at least the portion of the three-dimensional environment corresponding to a second viewpoint different from the first viewpoint.
