# US 10,762,716 B1 · Devices, methods, and graphical user interfaces for displaying objects in 3D contexts

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,762,716 B1
- Document ID: US10762716B1
- Status: Granted
- Kind: B1
- Application number: 16/581,685
- Filed: 2019-09-24
- Granted: 2020-09-01
- Earliest claimed: 2019-05-06
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Grant R. Paul; Nicolas V. Scapel; David Lui; Xiao Jin Yu; Hon-ming Chen
- Web page: https://patentgrep.com/patent/US10762716B1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An electronic device detects a first manipulation input directed to a respective virtual object of a plurality of virtual objects. In response to detecting the first manipulation input, in accordance with a determination that the plurality of virtual objects are displayed with at least a portion of a field of view of the one or more cameras, the device changes a display property of the respective virtual object based on the first manipulation input, and, in accordance with a determination that the plurality of virtual objects are displayed in a user interface that does not include the field of view of the one or more cameras, the electronic device changes a display property of the plurality of virtual objects based on the first manipulation input.

## Claims

### Claim 1

A method, comprising:
at an electronic device including a display generation component, one or more input devices, and one or more cameras:
displaying, by the display generation component, a plurality of virtual objects that include a first virtual object and a second virtual object;
while displaying the plurality of virtual objects, detecting a first manipulation input directed to the first virtual object of the plurality of virtual objects; and
in response to detecting the first manipulation input directed to the first virtual object of the plurality of virtual objects:
in accordance with a determination that the plurality of virtual objects are displayed in a user interface region that includes at least a portion of a field of view of the one or more cameras, changing a display property of the first virtual object based on the first manipulation input without changing a display property of the second virtual object based on the first manipulation input; and
in accordance with a determination that the plurality of virtual objects are displayed in a user interface region that does not include the field of view of the one or more cameras:
changing the display property of the first virtual object based on the first manipulation input; and
changing the display property of the second virtual object based on the first manipulation input.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
while the plurality of virtual objects are displayed in the user interface region that does not include the field of view of the one or more cameras, the first virtual object and the second virtual object of the plurality of virtual objects are each displayed with a first set of display characteristics.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
the first set of display characteristics includes a height ordering characteristic; and
displaying the plurality of virtual objects in accordance with the height ordering characteristic includes ordering the first virtual object and the second virtual object of the plurality of virtual objects in accordance with respective heights of the first virtual object and the second virtual object.

### Claim 4 (depends on 2)

The method of claim 2, wherein:
the first set of display characteristics includes a spacing characteristic; and
displaying the plurality of virtual objects in accordance with the spacing characteristic includes positioning the first virtual object and the second virtual object of the plurality of virtual objects with a predefined distance between respective boundaries of the first virtual object and the second virtual object.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
the first virtual object of the plurality of virtual objects has a first boundary that corresponds to a static state of the first virtual object;
the first virtual object is associated with a first action that causes at least a portion of the first virtual object to extend beyond the first boundary;
the first virtual object has a second boundary that corresponds to an extended range of the first virtual object during the first action;
the second virtual object of the plurality of virtual objects is adjacent to the first virtual object and has a third boundary that corresponds to a static state of the second virtual object; and
the third boundary of the second virtual object includes the second boundary of the first virtual object.

### Claim 6 (depends on 2)

The method of claim 2, including:
while displaying the plurality of virtual objects in the user interface region that includes at least a portion of a field of view of the one or more cameras, after changing the display property of the first virtual object based on the first manipulation input, detecting a request to display the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras; and
in response to detecting the request to display the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras, displaying the user interface region that does not include the field of view of the one or more cameras, including displaying the first virtual object and the second virtual object with the first set of display characteristics.

### Claim 7 (depends on 6)

The method of claim 6, including:
after changing the display property of the first virtual object and changing the display property of the second virtual object based on the first manipulation input, while displaying the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras, detecting a request to re-display the plurality of virtual objects in the user interface region that includes at least a portion of a field of view of the one or more cameras; and
in response to detecting the request to re-display the plurality of virtual objects in the user interface region that includes at least a portion of a field of view of the one or more cameras, re-displaying the plurality of virtual objects in the user interface region that includes at least a portion of a field of view of the one or more cameras, including re-displaying the first virtual object and the second virtual object with the display property of the first virtual object and the display property of the second virtual object as changed based on the first manipulation input.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the first manipulation input is a rotation input.

### Claim 9 (depends on 1)

The method of claim 1, including:
prior to displaying the plurality of virtual objects, displaying a selection user interface;
while the selection user interface is displayed, detecting selection of the plurality of virtual objects that include the first virtual object and the second virtual object;
while the plurality of virtual objects are selected, receiving a request to display the plurality of virtual objects; and
in response to receiving the request to display the plurality of virtual objects, displaying the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras.

### Claim 10 (depends on 9)

The method of claim 9, wherein:
the first virtual object of the plurality of virtual objects has a first defined plane type;
the second virtual object of the plurality of virtual objects has a second defined plane type that is distinct from the first defined plane type; and
in response to receiving the request to display the plurality of virtual objects while displaying the first virtual object and the second virtual object in the user interface region that includes the field of view of the one or more cameras:
the first virtual object of the plurality of virtual objects is displayed at a fixed position relative to a first plane that corresponds to the first defined plane type; and
the second virtual object of the plurality of virtual objects is displayed at a fixed position relative to a second plane that corresponds to the second defined plane type that is distinct from the first defined plane type.

### Claim 11 (depends on 10)

The method of claim 10, including:
while displaying the first virtual object and the second virtual object in the user interface region that includes the field of view of the one or more cameras, displaying the first virtual object at the fixed position relative to the first plane, and displaying the second virtual object at the fixed position relative to the second plane, detecting a rotation input; and
in response to detecting the rotation input:
in accordance with a determination that the rotation input is directed to the first virtual object, rotating the first virtual object about a first axis; and
in accordance with a determination that the rotation input is directed to the second virtual object, rotating the second virtual object about a second axis that is distinct from the first axis.

### Claim 12

A computer system, comprising:
a display generation component;
one or more input devices;
one or more cameras;
one or more processors; and
memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, by the display generation component, a plurality of virtual objects that include a first virtual object and a second virtual object;
while displaying the plurality of virtual objects, detecting a first manipulation input directed to the first virtual object of the plurality of virtual objects; and
in response to detecting the first manipulation input directed to the first virtual object of the plurality of virtual objects:
in accordance with a determination that the plurality of virtual objects are displayed in a user interface region that includes at least a portion of a field of view of the one or more cameras, changing a display property of the first virtual object based on the first manipulation input without changing a display property of the second virtual object based on the first manipulation input; and
in accordance with a determination that the plurality of virtual objects are displayed in a user interface region that does not include the field of view of the one or more cameras;
changing the display property of the first virtual object based on the first manipulation input; and
changing the display property of the second virtual object based on the first manipulation input.

### Claim 13 (depends on 12)

The computer system of claim 12, wherein:
while the plurality of virtual objects are displayed in the user interface region that does not include the field of view of the one or more cameras, the first virtual object and the second virtual object of the plurality of virtual objects are each displayed with a first set of display characteristics.

### Claim 14 (depends on 13)

The computer system of claim 13, wherein:
the first set of display characteristics includes a height ordering characteristic; and
displaying the plurality of virtual objects in accordance with the height ordering characteristic includes ordering the first virtual object and the second virtual object of the plurality of virtual objects in accordance with respective heights of the first virtual object and the second virtual object.

### Claim 15 (depends on 13)

The computer system of claim 13, wherein:
the first set of display characteristics includes a spacing characteristic; and
displaying the plurality of virtual objects in accordance with the spacing characteristic includes positioning the first virtual object and the second virtual object of the plurality of virtual objects with a predefined distance between respective boundaries of the first virtual object and the second virtual object.

### Claim 16 (depends on 15)

The computer system of claim 15, wherein:
the first virtual object of the plurality of virtual objects has a first boundary that corresponds to a static state of the first virtual object;
the first virtual object is associated with a first action that causes at least a portion of the first virtual object to extend beyond the first boundary;
the first virtual object has a second boundary that corresponds to an extended range of the first virtual object during the first action;
the second virtual object of the plurality of virtual objects is adjacent to the first virtual object and has a third boundary that corresponds to a static state of the second virtual object; and
the third boundary of the second virtual object includes the second boundary of the first virtual object.

### Claim 17 (depends on 13)

The computer system of claim 13, wherein the one or more programs include instructions for:
while displaying the plurality of virtual objects in the user interface region that includes at least a portion of a field of view of the one or more cameras, after changing the display property of the first virtual object based on the first manipulation input, detecting a request to display the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras; and
in response to detecting the request to display the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras, displaying the user interface region that does not include the field of view of the one or more cameras, including displaying the first virtual object and the second virtual object with the first set of display characteristics.

### Claim 18 (depends on 17)

The computer system of claim 17, wherein the one or more programs include instructions for:
after changing the display property of the first virtual object and changing the display property of the second virtual object based on the first manipulation input, while displaying the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras, detecting a request to re-display the plurality of virtual objects in a user interface region that includes at least a portion of a field of view of the one or more cameras; and
in response to detecting the request to re-display the plurality of virtual objects with at least a portion of a field of view of the one or more cameras, re-displaying the plurality of virtual objects in a user interface region that includes at least a portion of a field of view of the one or more cameras, including re-displaying the first virtual object and the second virtual object with the display property of the first virtual object and the display property of the second virtual object as changed based on the first manipulation input.

### Claim 19 (depends on 12)

The computer system of claim 12, wherein:
the first manipulation input is a rotation input.

### Claim 20 (depends on 12)

The computer system of claim 12, wherein the one or more programs include instructions for:
prior to displaying the plurality of virtual objects, displaying a selection user interface;
while the selection user interface is displayed, detecting selection of the plurality of virtual objects including the first virtual object and the second virtual object;
while the plurality of virtual objects are selected, receiving a request to display the plurality of virtual objects; and
in response to receiving the request to display the plurality of virtual objects, displaying the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras.

### Claim 21 (depends on 20)

The computer system of claim 20, wherein:
the first virtual object of the plurality of virtual objects has a first defined plane type;
the second virtual object of the plurality of virtual objects has a second defined plane type that is distinct from the first defined plane type; and
in response to receiving the request to display the plurality of virtual objects:
the first virtual object of the plurality of virtual objects is displayed at a fixed position relative to a first plane that corresponds to the first defined plane type; and
the second virtual object of the plurality of virtual objects is displayed at a fixed position relative to a second plane that corresponds to the second defined plane type that is distinct from the first defined plane type.

### Claim 22 (depends on 21)

The computer system of claim 21, wherein the one or more programs include instructions for:
while the first virtual object is displayed at the fixed position relative to the first plane and the second virtual object is displayed at the fixed position relative to the second plane, detecting a rotation input; and
in response to detecting the rotation input:
in accordance with a determination that the rotation input is directed to the first virtual object, rotating the first virtual object about a first axis; and
in accordance with a determination that the rotation input is directed to the second virtual object, rotating the second virtual object about a second axis that is distinct from the first axis.

### Claim 23

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by a computer system with a display generation component, one or more input devices, and one or more cameras, cause the computer system to:
display, by the display generation component, a plurality of virtual objects that include a first virtual object and a second virtual object;
while displaying the plurality of virtual objects, detect a first manipulation input directed to the first virtual object of the plurality of virtual objects; and
in response to detecting the first manipulation input directed to the first virtual object of the plurality of virtual objects:
in accordance with a determination that the plurality of virtual objects are displayed in a user interface region that includes at least a portion of a field of view of the one or more cameras, change a display property of the first virtual object based on the first manipulation input without changing a display property of the second virtual object based on the first manipulation input; and
in accordance with a determination that the plurality of virtual objects are displayed in a user interface region that does not include the field of view of the one or more cameras:
change the display property of the first virtual object based on the first manipulation input; and
change the display property of the second virtual object based on the first manipulation input.

### Claim 24 (depends on 23)

The computer readable storage medium of claim 23, wherein:
while the plurality of virtual objects are displayed in the user interface region that does not include the field of view of the one or more cameras, the first virtual object and the second virtual object of the plurality of virtual objects are each displayed with a first set of display characteristics.

### Claim 25 (depends on 24)

The computer readable storage medium of claim 24, wherein:
the first set of display characteristics includes a height ordering characteristic; and
displaying the plurality of virtual objects in accordance with the height ordering characteristic includes ordering the first virtual object and the second virtual object of the plurality of virtual objects in accordance with respective heights of the first virtual object and the second virtual object.

### Claim 26 (depends on 24)

The computer readable storage medium of claim 24, wherein:
the first set of display characteristics includes a spacing characteristic; and
displaying the plurality of virtual objects in accordance with the spacing characteristic includes positioning the first virtual object and the second virtual object of the plurality of virtual objects with a predefined distance between respective boundaries of the first virtual object and the second virtual object.

### Claim 27 (depends on 26)

The computer readable storage medium of claim 26, wherein:
the first virtual object of the plurality of virtual objects has a first boundary that corresponds to a static state of the first virtual object;
the first virtual object is associated with a first action that causes at least a portion of the first virtual object to extend beyond the first boundary;
the first virtual object has a second boundary that corresponds to an extended range of the first virtual object during the first action;
the second virtual object of the plurality of virtual objects is adjacent to the first virtual object and has a third boundary that corresponds to a static state of the second virtual object; and
the third boundary of the second virtual object includes the second boundary of the first virtual object.

### Claim 28 (depends on 24)

The computer readable storage medium of claim 24, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
while displaying the plurality of virtual objects in the user interface region that includes at least a portion of a field of view of the one or more cameras, after changing the display property of the first virtual object based on the first manipulation input, detect a request to display the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras; and
in response to detecting the request to display the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras, display the user interface region that does not include the field of view of the one or more cameras, including displaying the first virtual object and the second virtual object with the first set of display characteristics.

### Claim 29 (depends on 28)

The computer readable storage medium of claim 28, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
after changing the display property of the first virtual object and changing the display property of the second virtual object based on the first manipulation input, while displaying the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras, detect a request to re-display the plurality of virtual objects in the user interface region that includes at least a portion of a field of view of the one or more cameras; and
in response to detecting the request to re-display the plurality of virtual objects with at least a portion of a field of view of the one or more cameras, re-display the plurality of virtual objects in the user interface region that includes at least a portion of a field of view of the one or more cameras, including re-displaying the first virtual object and the second virtual object with the display property of the first virtual object and the display property of the second virtual object as changed based on the first manipulation input.

### Claim 30 (depends on 23)

The computer readable storage medium of claim 23, wherein:
the first manipulation input is a rotation input.

### Claim 31 (depends on 23)

The computer readable storage medium of claim 23, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
prior to displaying the plurality of virtual objects, display a selection user interface;
while the selection user interface is displayed, detect selection of the plurality of virtual objects including the first virtual object and the second virtual object;
while the plurality of virtual objects are selected, receive a request to display the plurality of virtual objects; and
in response to receiving the request to display the plurality of virtual objects, display the plurality of virtual objects in the user interface region that does not include the field of view of the one or more cameras.

### Claim 32 (depends on 31)

The computer readable storage medium of claim 31, wherein:
the first virtual object of the plurality of virtual objects has a first defined plane type;
the second virtual object of the plurality of virtual objects has a second defined plane type that is distinct from the first defined plane type; and
in response to receiving the request to display the plurality of virtual objects:
the first virtual object of the plurality of virtual objects is displayed at a fixed position relative to a first plane that corresponds to the first defined plane type; and
the second virtual object of the plurality of virtual objects is displayed at a fixed position relative to a second plane that corresponds to the second defined plane type that is distinct from the first defined plane type.

### Claim 33 (depends on 32)

The computer readable storage medium of claim 32, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
while the first virtual object is displayed at the fixed position relative to the first plane and the second virtual object is displayed at the fixed position relative to the second plane, detect a rotation input; and
in response to detecting the rotation input:
in accordance with a determination that the rotation input is directed to the first virtual object, rotate the first virtual object about a first axis; and
in accordance with a determination that the rotation input is directed to the second virtual object, rotate the second virtual object about a second axis that is distinct from the first axis.