# US 2021/0241505 A1 · Systems, Methods, and Graphical User Interfaces for Annotating, Measuring, and Modeling Environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241505 A1
- Document ID: US20210241505A1
- Status: Published application
- Kind: A1
- Application number: 17/018,980
- Filed: 2020-09-11
- Published: 2021-08-05
- Earliest claimed: 2020-02-03
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Allison W. Dryer; Giancarlo Yerkes; Grant R. Paul; Lisa K. Forssell
- Web page: https://patentgrep.com/patent/US20210241505A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06T 7/13: Edge detection
- G06T 7/55: from multiple images
- G06T 7/62: of area, perimeter, diameter or volume
- G06T 11/23: using straight lines or curves
- G06T 13/00: Animation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/20: in augmented reality scenes
- G01S 5/16: using electromagnetic waves other than radio waves
- G01S 11/12: using electromagnetic waves other than radio waves
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2219/004: Annotating, labelling
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

A computer system displays an annotation placement user interface that includes a representation of a field of view of one or more cameras that is updated over time based on changes in the field of view, and a placement user interface element indicating a location. If the placement user interface element is over a representation of a first type of feature in the physical environment, the appearance of the placement user interface element changes to indicate an anchor point corresponding to the first type of feature, and the system displays a first set of guides. If the placement user interface element is over a representation of a second, different type of feature in the physical environment, the appearance of the placement user interface element changes to indicate an anchor point corresponding to the second type of feature, and the system displays a second, different set of guides.

## Claims

### Claim 1

A method, comprising:
at a computer system with a display device and one or more cameras:
displaying, via the display device, an annotation placement user interface, the annotation placement user interface including:
a representation of a field of view of the one or more cameras, including a representation of a portion of a three-dimensional physical environment that is in the field of view of the one or more cameras, wherein the representation of the field of view is updated over time based on changes in the field of view of the one or more cameras; and
a placement user interface element that indicates a location at which a virtual annotation would be placed in the representation of the field of view in response to receiving an annotation placement input;
while displaying the annotation placement user interface, detecting movement of the one or more cameras relative to the physical environment; and
in response to detecting the movement of the one or more cameras relative to the physical environment:
updating the representation of the field of view based on the movement of the one or more cameras;
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a first type of feature in the physical environment:
changing an appearance of the placement user interface element to indicate that an annotation would be placed at an anchor point corresponding to the first type of feature in response to an annotation placement input; and
automatically displaying, without receiving an annotation placement input, a first set of one or more guides corresponding to at least a portion of the representation of the first type of feature; and
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a second type of feature in the physical environment, wherein the second type of feature is different from the first type of feature:
changing the appearance of the placement user interface element to indicate that an annotation would be placed at an anchor point corresponding to the second type of feature in response to an annotation placement input; and
automatically displaying, without receiving an annotation placement input, a second set of one or more guides, different from the first set of one or more guides, corresponding to at least a portion of the representation of the second type of feature.

### Claim 2 (depends on 1)

The method of claim 1, wherein, in accordance with a determination that the anchor point is located on one or more edges in the physical environment, the displayed respective set of one or more guides includes, for each edge of the one or more edges, a respective guide that extends from the anchor point along the respective edge.

### Claim 3 (depends on 1)

The method of claim 1, wherein, in accordance with a determination that the anchor point is located where at least a first detected surface in the physical environment and a second detected surface in the physical environment meet, the displayed respective set of one or more guides includes at least a first guide that extends from the anchor point in a direction perpendicular to the first detected surface and a second guide that extends from the anchor point in a direction perpendicular to the second detected surface.

### Claim 4 (depends on 1)

The method of claim 1, including:
in response to detecting the movement of the one or more cameras relative to the physical environment:
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a third type of feature in the physical environment that is different from the first type of feature and the second type of feature, wherein the placement user interface element indicates a respective location in the representation of the field of view at which a virtual annotation would be placed in response to receiving an annotation placement input:
displaying a guide extending from the respective location in a direction in the representation of the field of view that corresponds to a vertical direction from the third type of feature.

### Claim 5 (depends on 1)

The method of claim 1, wherein displaying a respective set of one or more guides includes, for each guide in the respective set, visually emphasizing a first portion of the respective guide relative to a second portion of the respective guide, wherein the first portion corresponds to locations in physical space that are occupied by a physical object in the physical environment, and the second portion corresponds to locations in physical space that are not occupied by a physical object in the physical environment.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
the second portion of a respective guide includes a first point that is a first distance from the first portion of the respective guide and a second point that is a second distance, greater than the first distance, from the first portion of the respective guide; and
the second point is visually deemphasized relative to the first point.

### Claim 7 (depends on 1)

The method of claim 1, wherein displaying a respective set of one or more guides includes displaying an animation showing the one or more guides progressively extending from the anchor point over time.

### Claim 8

A computer system, comprising:
a display device;
one or more cameras;
one or more processors; and
memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, via the display device, an annotation placement user interface, the annotation placement user interface including:
a representation of a field of view of the one or more cameras, including a representation of a portion of a three-dimensional physical environment that is in the field of view of the one or more cameras, wherein the representation of the field of view is updated over time based on changes in the field of view of the one or more cameras; and
a placement user interface element that indicates a location at which a virtual annotation would be placed in the representation of the field of view in response to receiving an annotation placement input;
while displaying the annotation placement user interface, detecting movement of the one or more cameras relative to the physical environment; and
in response to detecting the movement of the one or more cameras relative to the physical environment:
updating the representation of the field of view based on the movement of the one or more cameras;
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a first type of feature in the physical environment:
changing an appearance of the placement user interface element to indicate that an annotation would be placed at an anchor point corresponding to the first type of feature in response to an annotation placement input; and
displaying, without receiving an annotation placement input, a first set of one or more guides corresponding to at least a portion of the representation of the first type of feature; and
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a second type of feature in the physical environment, wherein the second type of feature is different from the first type of feature:
changing the appearance of the placement user interface element to indicate that an annotation would be placed at an anchor point corresponding to the second type of feature in response to an annotation placement input; and
displaying, without receiving an annotation placement input, a second set of one or more guides, different from the first set of one or more guides, corresponding to at least a portion of the representation of the second type of feature.

### Claim 9 (depends on 8)

The computer system of claim 8, wherein, in accordance with a determination that the anchor point is located on one or more edges in the physical environment, the displayed respective set of one or more guides includes, for each edge of the one or more edges, a respective guide that extends from the anchor point along the respective edge.

### Claim 10 (depends on 8)

The computer system of claim 8, wherein, in accordance with a determination that the anchor point is located where at least a first detected surface in the physical environment and a second detected surface in the physical environment meet, the displayed respective set of one or more guides includes at least a first guide that extends from the anchor point in a direction perpendicular to the first detected surface and a second guide that extends from the anchor point in a direction perpendicular to the second detected surface.

### Claim 11 (depends on 8)

The computer system of claim 8, wherein the one or more programs include instructions for:
in response to detecting the movement of the one or more cameras relative to the physical environment:
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a third type of feature in the physical environment that is different from the first type of feature and the second type of feature, wherein the placement user interface element indicates a respective location in the representation of the field of view at which a virtual annotation would be placed in response to receiving an annotation placement input:
displaying a guide extending from the respective location in a direction in the representation of the field of view that corresponds to a vertical direction from the third type of feature.

### Claim 12 (depends on 8)

The computer system of claim 8, wherein displaying a respective set of one or more guides includes, for each guide in the respective set, visually emphasizing a first portion of the respective guide relative to a second portion of the respective guide, wherein the first portion corresponds to locations in physical space that are occupied by a physical object in the physical environment, and the second portion corresponds to locations in physical space that are not occupied by a physical object in the physical environment.

### Claim 13 (depends on 12)

The computer system of claim 12, wherein:
the second portion of a respective guide includes a first point that is a first distance from the first portion of the respective guide and a second point that is a second distance, greater than the first distance, from the first portion of the respective guide; and
the second point is visually deemphasized relative to the first point.

### Claim 14 (depends on 8)

The computer system of claim 8, wherein displaying a respective set of one or more guides includes displaying an animation showing the one or more guides progressively extending from the anchor point over time.

### Claim 15

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system that includes a display device and one or more cameras, cause the computer system to:
display, via the display device, an annotation placement user interface, the annotation placement user interface including:
a representation of a field of view of the one or more cameras, including a representation of a portion of a three-dimensional physical environment that is in the field of view of the one or more cameras, wherein the representation of the field of view is updated over time based on changes in the field of view of the one or more cameras; and
a placement user interface element that indicates a location at which a virtual annotation would be placed in the representation of the field of view in response to receiving an annotation placement input;
while displaying the annotation placement user interface, detect movement of the one or more cameras relative to the physical environment; and
in response to detecting the movement of the one or more cameras relative to the physical environment:
update the representation of the field of view based on the movement of the one or more cameras;
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a first type of feature in the physical environment:
change an appearance of the placement user interface element to indicate that an annotation would be placed at an anchor point corresponding to the first type of feature in response to an annotation placement input; and
display, without receiving an annotation placement input, a first set of one or more guides corresponding to at least a portion of the representation of the first type of feature; and
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a second type of feature in the physical environment, wherein the second type of feature is different from the first type of feature:
change the appearance of the placement user interface element to indicate that an annotation would be placed at an anchor point corresponding to the second type of feature in response to an annotation placement input; and
display, without receiving an annotation placement input, a second set of one or more guides, different from the first set of one or more guides, corresponding to at least a portion of the representation of the second type of feature.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein, in accordance with a determination that the anchor point is located on one or more edges in the physical environment, the displayed respective set of one or more guides includes, for each edge of the one or more edges, a respective guide that extends from the anchor point along the respective edge.

### Claim 17 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein, in accordance with a determination that the anchor point is located where at least a first detected surface in the physical environment and a second detected surface in the physical environment meet, the displayed respective set of one or more guides includes at least a first guide that extends from the anchor point in a direction perpendicular to the first detected surface and a second guide that extends from the anchor point in a direction perpendicular to the second detected surface.

### Claim 18 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
in response to detecting the movement of the one or more cameras relative to the physical environment:
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a third type of feature in the physical environment that is different from the first type of feature and the second type of feature, wherein the placement user interface element indicates a respective location in the representation of the field of view at which a virtual annotation would be placed in response to receiving an annotation placement input:
display a guide extending from the respective location in a direction in the representation of the field of view that corresponds to a vertical direction from the third type of feature.

### Claim 19 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein displaying a respective set of one or more guides includes, for each guide in the respective set, visually emphasizing a first portion of the respective guide relative to a second portion of the respective guide, wherein the first portion corresponds to locations in physical space that are occupied by a physical object in the physical environment, and the second portion corresponds to locations in physical space that are not occupied by a physical object in the physical environment.

### Claim 20 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein:
the second portion of a respective guide includes a first point that is a first distance from the first portion of the respective guide and a second point that is a second distance, greater than the first distance, from the first portion of the respective guide; and
the second point is visually deemphasized relative to the first point.

### Claim 21 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein displaying a respective set of one or more guides includes displaying an animation showing the one or more guides progressively extending from the anchor point over time.