# US 2021/0333979 A1 · Devices, Methods, and Graphical User Interfaces for System-Wide Behavior for 3D Models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0333979 A1
- Document ID: US20210333979A1
- Status: Published application
- Kind: A1
- Application number: 17/372,400
- Filed: 2021-07-09
- Published: 2021-10-28
- Earliest claimed: 2018-01-24
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Philipp Rockel; Jonathan R. Dascola; Stephen O. Lemay; Jeffrey M. Faulkner; Giancarlo Yerkes
- Web page: https://patentgrep.com/patent/US20210333979A1

## CPC

- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 1/1643: {the display being associated to a digitizer, e.g. laptops that can be used as penpads (details related to the relative motion of the display enclosure with respect to the body enclosure, e.g. to move between laptop and tablet PC configuration G06F1/1615)}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0485: Scrolling or panning
- G06F 3/0486: Drag-and-drop
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 13/20: Three-dimensional [3D] animation
- G06T 19/003: {Navigation within 3D models or images}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06F 2203/014: Force feedback applied to GUI
- G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
- G06F 2203/04808: Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
- G06T 2207/30244: Camera pose
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A computer system with a display and a touch-sensitive surface receives a request to display a first user interface that includes a first item, and, in response to the request, displays the first user interface with a representation of the first item. Depending on whether the first item corresponds to a respective virtual three-dimensional object, the first item is displayed either with a visual indication to indicate that the first item corresponds to a first respective virtual three-dimensional object or without the visual indication. In response to a request to display a second user interface, the second user interface is displayed with a representation of the second item. Depending on whether the second item corresponds to a respective virtual three-dimensional object, the representation of the second item is displayed with or without the visual indication.

## Claims

### Claim 1

A method, comprising:
at a device having a display and a touch-sensitive surface:
receiving a request to display a first user interface that includes a first item;
in response to the request to display the first user interface, displaying the first user interface with a representation of the first item, including:
in accordance with a determination that the first item corresponds to a respective virtual three-dimensional object, displaying the representation of the first item with a visual indication to indicate that the first item corresponds to a first respective virtual three-dimensional object; and,
in accordance with a determination that the first item does not correspond to a respective virtual three-dimensional object, displaying the representation of the first item without the visual indication;
after displaying the representation of the first item, receiving a request to display a second user interface that includes a second item; and,
in response to the request to display the second user interface, displaying the second user interface with a representation of the second item, including:
in accordance with a determination that the second item corresponds to a respective virtual three-dimensional object, displaying the representation of the second item with the visual indication to indicate that the second item corresponds to a second respective virtual three-dimensional object; and,
in accordance with a determination that the second item does not correspond to a second respective virtual three-dimensional object, displaying the representation of the second item without the visual indication.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the device includes one or more device orientation sensors; and
displaying the representation of the first item with the visual indication to indicate that the first item corresponds to a first respective virtual three-dimensional object includes:
in response to detecting a movement of the device that results in a change from a first device orientation to a second device orientation, displaying movement of the first item that corresponds to the change from the first device orientation to the second device orientation.

### Claim 3 (depends on 1)

The method of claim 1, wherein displaying the representation of the first item with the visual indication to indicate that the first item corresponds to a first respective virtual three-dimensional object includes:
in response to detecting a first input by a first contact that scrolls the first user interface while the representation of the first item is displayed in the first user interface:
translating the representation of the first item on the display in accordance with scrolling of the first user interface; and
rotating the representation of the first item relative to a plane defined by the first user interface in accordance with a direction in which the first user interface is scrolled.

### Claim 4 (depends on 1)

The method of claim 1, including, while displaying the representation of the first item with the visual indication in the first user interface, displaying a representation of a third item, wherein the representation of the third item is displayed without the visual indication in order to indicate that the third item does not correspond to a virtual three-dimensional object.

### Claim 5 (depends on 1)

The method of claim 1, including, while displaying the representation of the second item with the visual indication in the second user interface, displaying a representation of a fourth item, wherein the representation of the fourth item is displayed without the visual indication in order to indicate that the fourth item does not correspond to a respective virtual three-dimensional object.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the first user interface corresponds to a first application;
the second user interface corresponds to a second application that is distinct from the first application; and
the representation of the first item displayed with the visual indication and the representation of the second item displayed with the visual indication share a predefined set of visual characteristics and/or behavioral characteristics.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first user interface is an Internet browser application user interface and the first item is an element of a web page.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first user interface is an e-mail application user interface and the first item is an attachment to an e-mail.

### Claim 9 (depends on 1)

The method of claim 1, wherein the first user interface is a messaging application user interface and the first item is an attachment or an element in a message.

### Claim 10 (depends on 1)

The method of claim 1, wherein the first user interface is a file management application user interface and the first item is a file preview object.

### Claim 11 (depends on 1)

The method of claim 1, wherein the first user interface is a map application user interface and the first item is a representation of a point of interest in a map.

### Claim 12 (depends on 1)

The method of claim 1, wherein the visual indication that the first item corresponds to the first respective virtual three-dimensional object includes an animation of the first item that occurs without requiring an input directed to the representation of the first item.

### Claim 13 (depends on 1)

The method of claim 1, wherein:
the device includes one or more cameras; and
the method includes:
while displaying the representation of the second item with the visual indication to indicate that the second item corresponds to a second respective virtual three-dimensional object, detecting a second input by a second contact at a location on the touch-sensitive surface that corresponds to the representation of the second item; and,
in response to detecting the second input by the second contact and in accordance with a determination that the second input by the second contact meets first criteria:
displaying a third user interface region on the display, including replacing display of at least a portion of the second user interface with a representation of a field of view of the one or more cameras; and
continuously displaying the second respective virtual three-dimensional object while switching from displaying the second user interface to displaying the third user interface region.

### Claim 14 (depends on 1)

The method of claim 1, wherein:
the device includes one or more cameras; and
the method includes:
while displaying the second item with the visual indication to indicate that the second item corresponds to a second respective virtual three-dimensional object, detecting a third input by a third contact at a location on the touch-sensitive surface that corresponds to the representation of the second item;
in response to detecting the third input by the third contact and in accordance with a determination that the third input by the third contact meets first criteria, displaying the second virtual three-dimensional object in a fourth user interface that is different from the second user interface;
while displaying the second virtual three-dimensional object in the fourth user interface, detecting a fourth input; and,
in response to detecting the fourth input:
in accordance with a determination that the fourth input corresponds to a request to manipulate the second virtual three-dimensional object in the fourth user interface, changing a display property of the second virtual three-dimensional object within the fourth user interface based on the fourth input; and
in accordance with a determination that the fourth input corresponds to a request to display the second virtual object in an augmented reality environment, displaying the second virtual three-dimensional object with a representation of a field of view of the one or more cameras.

### Claim 15

A computer system, comprising:
a display;
a touch-sensitive surface;
one or more processors; and
memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a request to display a first user interface that includes a first item;
in response to the request to display the first user interface, displaying the first user interface with a representation of the first item:
in accordance with a determination that the first item corresponds to a respective virtual three-dimensional object, displaying a representation of the first item with a visual indication to indicate that the first item corresponds to a first respective virtual three-dimensional object; and,
in accordance with a determination that the first item does not correspond to a respective virtual three-dimensional object, displaying the representation of the first item without the visual indication;
after displaying the representation of the first item, receiving a request to display a second user interface that includes a second item; and,
in response to the request to display the second user interface, displaying the second user interface with a representation of the second item:
in accordance with a determination that the second item corresponds to a respective virtual three-dimensional object, displaying a representation of the second item with the visual indication to indicate that the second item corresponds to a second respective virtual three-dimensional object; and,
in accordance with a determination that the second item does not correspond to a respective virtual three-dimensional object, displaying the representation of the second item without the visual indication.

### Claim 16 (depends on 15)

The computer system of claim 15, wherein:
the device includes one or more device orientation sensors; and
displaying the representation of the first item with the visual indication to indicate that the first item corresponds to a first respective virtual three-dimensional object includes:
in response to detecting a movement of the device that results in a change from a first device orientation to a second device orientation, displaying movement of the first item that corresponds to the change from the first device orientation to the second device orientation.

### Claim 17 (depends on 15)

The computer system of claim 15, wherein displaying the representation of the first item with the visual indication to indicate that the first item corresponds to a first respective virtual three-dimensional object includes:
in response to detecting a first input by a first contact that scrolls the first user interface while the representation of the first item is displayed in the first user interface:
translating the representation of the first item on the display in accordance with scrolling of the first user interface; and
rotating the representation of the first item relative to a plane defined by the first user interface in accordance with a direction in which the first user interface is scrolled.

### Claim 18 (depends on 15)

The computer system of claim 15, wherein the one or more programs include instructions for: displaying a representation of a third item while displaying the representation of the first item with the visual indication in the first user interface, wherein the representation of the third item is displayed without the visual indication in order to indicate that the third item does not correspond to a virtual three-dimensional object.

### Claim 19 (depends on 15)

The computer system of claim 15, wherein the one or more programs include instructions for: displaying a representation of a fourth item while displaying the representation of the second item with the visual indication in the second user interface, wherein the representation of the fourth item is displayed without the visual indication in order to indicate that the fourth item does not correspond to a respective virtual three-dimensional object.

### Claim 20

A computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by a computer system with a display, and a touch-sensitive surface, cause the computer system to:
receive a request to display a first user interface that includes a first item;
in response to the request to display the first user interface, display the first user interface with a representation of the first item:
in accordance with a determination that the first item corresponds to a respective virtual three-dimensional object, display a representation of the first item with a visual indication to indicate that the first item corresponds to a first respective virtual three-dimensional object; and,
in accordance with a determination that the first item does not correspond to a respective virtual three-dimensional object, display the representation of the first item without the visual indication;
after displaying the representation of the first item, receive a request to display a second user interface that includes a second item; and,
in response to the request to display the second user interface, display the second user interface with a representation of the second item:
in accordance with a determination that the second item corresponds to a respective virtual three-dimensional object, display a representation of the second item with the visual indication to indicate that the second item corresponds to a second respective virtual three-dimensional object; and,
in accordance with a determination that the second item does not correspond to a respective virtual three-dimensional object, display the representation of the second item without the visual indication.
