# US 2021/0241483 A1 · Systems, Methods, and Graphical User Interfaces for Annotating, Measuring, and Modeling Environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241483 A1
- Document ID: US20210241483A1
- Status: Published application
- Kind: A1
- Application number: 17/018,958
- Filed: 2020-09-11
- Published: 2021-08-05
- Earliest claimed: 2020-02-03
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Allison W. Dryer; Giancarlo Yerkes; Grant R. Paul; Lisa K. Forssell
- Web page: https://patentgrep.com/patent/US20210241483A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06T 7/13: Edge detection
- G06T 7/55: from multiple images
- G06T 7/62: of area, perimeter, diameter or volume
- G06T 11/23: using straight lines or curves
- G06T 13/00: Animation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/20: in augmented reality scenes
- G01S 5/16: using electromagnetic waves other than radio waves
- G01S 11/12: using electromagnetic waves other than radio waves
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2219/004: Annotating, labelling
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

A computer system displays an annotation placement user interface that includes a representation of a field of view of one or more cameras that is updated over time based on changes in the field of view, a placement user interface element indicating a virtual annotation placement location. If the placement user interface element is over a representation of a physical feature in the physical environment that can be measured, the appearance of the placement user interface element changes in accordance with one or more aspects of the representation of the physical feature, and, in response to an input to perform one or more measurements of the physical feature: if the physical feature is a first type of feature, measurements of a first measurement type are made; and, if a second, different type of physical feature, measurements of a second, different measurement type are made.

## Claims

### Claim 1

A method, comprising:
at a computer system with a display device and one or more cameras:
displaying, via the display device, an annotation placement user interface, the annotation placement user interface including:
a representation of a field of view of the one or more cameras, including a representation of a portion of a three-dimensional physical environment that is in the field of view of the one or more cameras, wherein the representation of the field of view is updated over time based on changes in the field of view of the one or more cameras; and
a placement user interface element that indicates a location at which a virtual annotation would be placed in the representation of the field of view in response to receiving an annotation placement input;
while displaying the annotation placement user interface, detecting a first movement of the one or more cameras relative to the physical environment; and
in response to detecting the first movement of the one or more cameras relative to the physical environment:
updating the representation of the field of view based on the first movement of the one or more cameras; and
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a physical feature in the physical environment that can be measured, changing an appearance of the placement user interface element in accordance with one or more aspects of the representation of the physical feature;
while displaying the annotation placement user interface, receiving an annotation placement input comprising a request to perform one or more measurements of the physical feature; and
in response to receiving the annotation placement input comprising a request to perform one or more measurements of the physical feature:
in accordance with a determination that the physical feature is a first type of physical feature, displaying, over the representation of the physical feature, a first set of one or more representations of measurements of a first measurement type; and
in accordance with a determination that the physical feature is a second type of physical feature, different from the first type of physical feature, displaying, over the representation of the physical feature, a second set of one or more representations of measurements of a second measurement type different from the first measurement type.

### Claim 2 (depends on 1)

The method of claim 1, wherein the physical feature is a first respective type of physical feature, the physical feature is a first portion of a physical object in the physical environment that is in the field of view of the one or more cameras, and a second portion of the physical object is at most partially in the field of view of the one or more cameras, and the method includes:
after displaying, over the representation of the physical feature, a first respective set of one or more representations of measurements of a first respective measurement type:
detecting a second movement of the one or more cameras relative to the physical environment such that the second portion of the physical object is in the field of view of the one or more cameras;
in response to detecting the second movement of the one or more cameras:
updating the representation of the field of view based on the second movement of the one or more cameras, including displaying, in the representation of the field of view, a representation of the physical object that includes a representation of the second portion of the physical object;
in accordance with a determination that the placement user interface element is over at least a portion of the representation of the physical object, changing the appearance of the placement user interface element in accordance with one or more aspects of the representation of the physical object including the second portion of the physical object;
while displaying the annotation placement user interface including the representation of the physical object, receiving a second annotation placement input comprising a request to perform one or more measurements of the physical object; and
in response to receiving the second annotation placement input comprising a request to perform one or more measurements of the physical object:
displaying, over the representation of the physical object, a second respective set of one or more representations of measurements of a second respective measurement type that is based on the second portion of the physical object.

### Claim 3 (depends on 1)

The method of claim 1, wherein a first portion of a physical object in the physical environment is in the field of view of the one or more cameras, the first portion of the physical object includes an edge region that is adjacent to a second portion of the physical object that is not in the field of view of the one or more cameras, and the method includes:
displaying, in the representation of the field of view of the one or more cameras, a representation of the first portion of the physical object, including visually deemphasizing the representation of the edge region that is adjacent to the second portion of the physical object.

### Claim 4 (depends on 1)

The method of claim 1, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a piece of furniture, and the measurements of the first measurement type include one or more of: a height, a width, a depth, and a volume of the physical feature.

### Claim 5 (depends on 1)

The method of claim 1, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a person, and the measurements of the first measurement type include a distance between a base of the person and a top of the person.

### Claim 6 (depends on 1)

The method of claim 1, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is an animal, and the measurements of the first measurement type include one or more of: a neck circumference of the animal, a chest circumference of the animal, and a length of the animal.

### Claim 7 (depends on 1)

The method of claim 1, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a face of a person, and the measurements of the first measurement type include a distance between a first pupil of a first eye of the person and a second pupil of a second eye of the person.

### Claim 8 (depends on 1)

The method of claim 1, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a foot of a person, and the measurements of the first measurement type include a length of the foot.

### Claim 9 (depends on 1)

The method of claim 1, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature includes a first hand and a second hand separated in space, and the measurements of the first measurement type include a distance between the first hand and the second hand.

### Claim 10 (depends on 1)

The method of claim 1, including:
while displaying a respective set of one or more representations of measurements over the representation of the physical feature, wherein the respective set includes a first representation of a measurement, the first representation including a first measurement label and a first measurement segment that is displayed using a first level of detail while the one or more cameras are located a first distance from the physical feature, detecting movement of the one or more cameras that places the one or more cameras at a second distance, less than the first distance, from the physical feature; and
while the one or more cameras are located at the second distance from the physical feature:
forgoing displaying the first measurement label; and
displaying the first measurement segment using a second level of detail that is different from the first level of detail.

### Claim 11 (depends on 10)

The method of claim 10, wherein displaying the first measurement segment using the second level of detail includes displaying a set of scale markers at intervals along the first measurement segment, and the method includes:
while displaying the first measurement segment using the second level of detail, in accordance with a determination that the placement user interface element is over a respective scale marker, changing an appearance of the placement user interface element to indicate that an annotation would be placed at an anchor point on the respective scale marker in response to receiving an annotation placement input.

### Claim 12 (depends on 10)

The method of claim 10, including:
while displaying the first measurement segment using the second level of detail, receiving an input corresponding to a request to add an annotation at a respective location in the representation of the field of view that is indicated by the second level of detail and that corresponds to a respective physical location in the physical environment;
in response to the input corresponding to the request to add an annotation, adding an annotation at the respective location indicated by the second level of detail;
after adding the annotation, detecting movement of the one or more cameras that places the one or more cameras at the first distance from the physical feature;
in response to detecting the movement of the one or more cameras that places the one or more cameras at the first distance from the physical feature:
updating the representation of the field of view based on the movement of the one or more cameras; and
while the one or more cameras are located at the first distance from the physical feature:
displaying the first measurement segment using the first level of detail;
displaying the first measurement label; and
displaying the annotation at a respective location in the updated representation of the field of view that corresponds to the respective physical location in the physical environment without regard to whether the respective location is indicated by the first level of detail.

### Claim 13

A computer system, comprising:
a display device;
one or more cameras;
one or more processors; and
memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, via the display device, an annotation placement user interface, the annotation placement user interface including:
a representation of a field of view of the one or more cameras, including a representation of a portion of a three-dimensional physical environment that is in the field of view of the one or more cameras, wherein the representation of the field of view is updated over time based on changes in the field of view of the one or more cameras; and
a placement user interface element that indicates a location at which a virtual annotation would be placed in the representation of the field of view in response to receiving an annotation placement input;
while displaying the annotation placement user interface, detecting a first movement of the one or more cameras relative to the physical environment; and
in response to detecting the first movement of the one or more cameras relative to the physical environment:
updating the representation of the field of view based on the first movement of the one or more cameras; and
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a physical feature in the physical environment that can be measured, changing an appearance of the placement user interface element in accordance with one or more aspects of the representation of the physical feature;
while displaying the annotation placement user interface, receiving an annotation placement input comprising a request to perform one or more measurements of the physical feature; and
in response to receiving the annotation placement input comprising a request to perform one or more measurements of the physical feature:
in accordance with a determination that the physical feature is a first type of physical feature, displaying, over the representation of the physical feature, a first set of one or more representations of measurements of a first measurement type; and
in accordance with a determination that the physical feature is a second type of physical feature, different from the first type of physical feature, displaying, over the representation of the physical feature, a second set of one or more representations of measurements of a second measurement type different from the first measurement type.

### Claim 14 (depends on 13)

The computer system of claim 13, wherein
the physical feature is a first respective type of physical feature, the physical feature is a first portion of a physical object in the physical environment that is in the field of view of the one or more cameras, and a second portion of the physical object is at most partially in the field of view of the one or more cameras; and
the one or more programs include instructions for:
after displaying, over the representation of the physical feature, a first respective set of one or more representations of measurements of a first respective measurement type:
detecting a second movement of the one or more cameras relative to the physical environment such that the second portion of the physical object is in the field of view of the one or more cameras;
in response to detecting the second movement of the one or more cameras:
updating the representation of the field of view based on the second movement of the one or more cameras, including displaying, in the representation of the field of view, a representation of the physical object that includes a representation of the second portion of the physical object;
in accordance with a determination that the placement user interface element is over at least a portion of the representation of the physical object, changing the appearance of the placement user interface element in accordance with one or more aspects of the representation of the physical object including the second portion of the physical object;
while displaying the annotation placement user interface including the representation of the physical object, receiving a second annotation placement input comprising a request to perform one or more measurements of the physical object; and
in response to receiving the second annotation placement input comprising a request to perform one or more measurements of the physical object:
displaying, over the representation of the physical object, a second respective set of one or more representations of measurements of a second respective measurement type that is based on the second portion of the physical object.

### Claim 15 (depends on 13)

The computer system of claim 13, wherein
a first portion of a physical object in the physical environment is in the field of view of the one or more cameras, the first portion of the physical object includes an edge region that is adjacent to a second portion of the physical object that is not in the field of view of the one or more cameras; and
the one or more programs include instructions for displaying, in the representation of the field of view of the one or more cameras, a representation of the first portion of the physical object, including visually deemphasizing the representation of the edge region that is adjacent to the second portion of the physical object.

### Claim 16 (depends on 13)

The computer system of claim 13, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a piece of furniture, and the measurements of the first measurement type include one or more of: a height, a width, a depth, and a volume of the physical feature.

### Claim 17 (depends on 13)

The computer system of claim 13, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a person, and the measurements of the first measurement type include a distance between a base of the person and a top of the person.

### Claim 18 (depends on 13)

The computer system of claim 13, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is an animal, and the measurements of the first measurement type include one or more of: a neck circumference of the animal, a chest circumference of the animal, and a length of the animal.

### Claim 19 (depends on 13)

The computer system of claim 13, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a face of a person, and the measurements of the first measurement type include a distance between a first pupil of a first eye of the person and a second pupil of a second eye of the person.

### Claim 20

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system that includes a display device and one or more cameras, cause the computer system to:
display, via the display device, an annotation placement user interface, the annotation placement user interface including:
a representation of a field of view of the one or more cameras, including a representation of a portion of a three-dimensional physical environment that is in the field of view of the one or more cameras, wherein the representation of the field of view is updated over time based on changes in the field of view of the one or more cameras; and
a placement user interface element that indicates a location at which a virtual annotation would be placed in the representation of the field of view in response to receiving an annotation placement input;
while displaying the annotation placement user interface, detect a first movement of the one or more cameras relative to the physical environment; and
in response to detecting the first movement of the one or more cameras relative to the physical environment:
update the representation of the field of view based on the first movement of the one or more cameras; and
in accordance with a determination that the placement user interface element is over at least a portion of a representation of a physical feature in the physical environment that can be measured, change an appearance of the placement user interface element in accordance with one or more aspects of the representation of the physical feature;
while displaying the annotation placement user interface, receive an annotation placement input comprising a request to perform one or more measurements of the physical feature; and
in response to receiving the annotation placement input comprising a request to perform one or more measurements of the physical feature:
in accordance with a determination that the physical feature is a first type of physical feature, display, over the representation of the physical feature, a first set of one or more representations of measurements of a first measurement type; and
in accordance with a determination that the physical feature is a second type of physical feature, different from the first type of physical feature, display, over the representation of the physical feature, a second set of one or more representations of measurements of a second measurement type different from the first measurement type.

### Claim 21 (depends on 20)

The non-transitory computer readable storage medium of claim 20, wherein the physical feature is a first respective type of physical feature, the physical feature is a first portion of a physical object in the physical environment that is in the field of view of the one or more cameras, and a second portion of the physical object is at most partially in the field of view of the one or more cameras, and the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
after displaying, over the representation of the physical feature, a first respective set of one or more representations of measurements of a first respective measurement type:
detect a second movement of the one or more cameras relative to the physical environment such that the second portion of the physical object is in the field of view of the one or more cameras;
in response to detecting the second movement of the one or more cameras:
update the representation of the field of view based on the second movement of the one or more cameras, including displaying, in the representation of the field of view, a representation of the physical object that includes a representation of the second portion of the physical object;
in accordance with a determination that the placement user interface element is over at least a portion of the representation of the physical object, change the appearance of the placement user interface element in accordance with one or more aspects of the representation of the physical object including the second portion of the physical object;
while displaying the annotation placement user interface including the representation of the physical object, receive a second annotation placement input comprising a request to perform one or more measurements of the physical object; and
in response to receiving the second annotation placement input comprising a request to perform one or more measurements of the physical object:
display, over the representation of the physical object, a second respective set of one or more representations of measurements of a second respective measurement type that is based on the second portion of the physical object.

### Claim 22 (depends on 20)

The non-transitory computer readable storage medium of claim 20, wherein a first portion of a physical object in the physical environment is in the field of view of the one or more cameras, the first portion of the physical object includes an edge region that is adjacent to a second portion of the physical object that is not in the field of view of the one or more cameras, and the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
display, in the representation of the field of view of the one or more cameras, a representation of the first portion of the physical object, including visually deemphasizing the representation of the edge region that is adjacent to the second portion of the physical object.

### Claim 23 (depends on 20)

The non-transitory computer readable storage medium of claim 20, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a piece of furniture, and the measurements of the first measurement type include one or more of: a height, a width, a depth, and a volume of the physical feature.

### Claim 24 (depends on 20)

The non-transitory computer readable storage medium of claim 20, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a person, and the measurements of the first measurement type include a distance between a base of the person and a top of the person.

### Claim 25 (depends on 20)

The non-transitory computer readable storage medium of claim 20, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is an animal, and the measurements of the first measurement type include one or more of: a neck circumference of the animal, a chest circumference of the animal, and a length of the animal.

### Claim 26 (depends on 20)

The non-transitory computer readable storage medium of claim 20, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a face of a person, and the measurements of the first measurement type include a distance between a first pupil of a first eye of the person and a second pupil of a second eye of the person.

### Claim 27 (depends on 20)

The non-transitory computer readable storage medium of claim 20, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a foot of a person, and the measurements of the first measurement type include a length of the foot.

### Claim 28 (depends on 20)

The non-transitory computer readable storage medium of claim 20, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature includes a first hand and a second hand separated in space, and the measurements of the first measurement type include a distance between the first hand and the second hand.

### Claim 29 (depends on 20)

The non-transitory computer readable storage medium of claim 20, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
while displaying a respective set of one or more representations of measurements over the representation of the physical feature, wherein the respective set includes a first representation of a measurement, the first representation including a first measurement label and a first measurement segment that is displayed using a first level of detail while the one or more cameras are located a first distance from the physical feature, detect movement of the one or more cameras that places the one or more cameras at a second distance, less than the first distance, from the physical feature; and
while the one or more cameras are located at the second distance from the physical feature:
forgo displaying the first measurement label; and
display the first measurement segment using a second level of detail that is different from the first level of detail.

### Claim 30 (depends on 29)

The non-transitory computer readable storage medium of claim 29, wherein displaying the first measurement segment using the second level of detail includes displaying a set of scale markers at intervals along the first measurement segment, and the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
while displaying the first measurement segment using the second level of detail, in accordance with a determination that the placement user interface element is over a respective scale marker, change an appearance of the placement user interface element to indicate that an annotation would be placed at an anchor point on the respective scale marker in response to receiving an annotation placement input.

### Claim 31 (depends on 29)

The non-transitory computer readable storage medium of claim 29, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to:
while displaying the first measurement segment using the second level of detail, receive an input corresponding to a request to add an annotation at a respective location in the representation of the field of view that is indicated by the second level of detail and that corresponds to a respective physical location in the physical environment;
in response to the input corresponding to the request to add an annotation, add an annotation at the respective location indicated by the second level of detail;
after adding the annotation, detect movement of the one or more cameras that places the one or more cameras at the first distance from the physical feature;
in response to detecting the movement of the one or more cameras that places the one or more cameras at the first distance from the physical feature:
update the representation of the field of view based on the movement of the one or more cameras; and
while the one or more cameras are located at the first distance from the physical feature:
display the first measurement segment using the first level of detail;
display the first measurement label; and
display the annotation at a respective location in the updated representation of the field of view that corresponds to the respective physical location in the physical environment without regard to whether the respective location is indicated by the first level of detail.

### Claim 32 (depends on 13)

The computer system of claim 13, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature is a foot of a person, and the measurements of the first measurement type include a length of the foot.

### Claim 33 (depends on 13)

The computer system of claim 13, wherein the determination that the physical feature is the first type of physical feature includes a determination that the physical feature includes a first hand and a second hand separated in space, and the measurements of the first measurement type include a distance between the first hand and the second hand.

### Claim 34 (depends on 13)

The computer system of claim 13, wherein the one or more programs include instructions for:
while displaying a respective set of one or more representations of measurements over the representation of the physical feature, wherein the respective set includes a first representation of a measurement, the first representation including a first measurement label and a first measurement segment that is displayed using a first level of detail while the one or more cameras are located a first distance from the physical feature, detecting movement of the one or more cameras that places the one or more cameras at a second distance, less than the first distance, from the physical feature; and
while the one or more cameras are located at the second distance from the physical feature:
forgoing displaying the first measurement label; and
displaying the first measurement segment using a second level of detail that is different from the first level of detail.

### Claim 35 (depends on 34)

The computer system of claim 34, wherein displaying the first measurement segment using the second level of detail includes displaying a set of scale markers at intervals along the first measurement segment, and the one or more programs include instructions for:
while displaying the first measurement segment using the second level of detail, in accordance with a determination that the placement user interface element is over a respective scale marker, changing an appearance of the placement user interface element to indicate that an annotation would be placed at an anchor point on the respective scale marker in response to receiving an annotation placement input.

### Claim 36 (depends on 34)

The computer system of claim 34, wherein the one or more programs include instructions for:
while displaying the first measurement segment using the second level of detail, receiving an input corresponding to a request to add an annotation at a respective location in the representation of the field of view that is indicated by the second level of detail and that corresponds to a respective physical location in the physical environment;
in response to the input corresponding to the request to add an annotation, adding an annotation at the respective location indicated by the second level of detail;
after adding the annotation, detecting movement of the one or more cameras that places the one or more cameras at the first distance from the physical feature;
in response to detecting the movement of the one or more cameras that places the one or more cameras at the first distance from the physical feature:
updating the representation of the field of view based on the movement of the one or more cameras; and
while the one or more cameras are located at the first distance from the physical feature:
displaying the first measurement segment using the first level of detail;
displaying the first measurement label; and
displaying the annotation at a respective location in the updated representation of the field of view that corresponds to the respective physical location in the physical environment without regard to whether the respective location is indicated by the first level of detail.