# US 2023/0410332 A1 · Structuring visual data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410332 A1
- Document ID: US20230410332A1
- Status: Published application
- Kind: A1
- Application number: 18/457,166
- Filed: 2023-08-28
- Published: 2023-12-21
- Earliest claimed: 2020-01-16
- Assignees: Fyusion, Inc.
- Applicants: Fyusion, Inc.
- Inventors: Stefan Johannes Josef Holzer; Matteo Munaro; Radu Bogdan Rusu
- Web page: https://patentgrep.com/patent/US20230410332A1

## CPC

- G06T 7/38: Registration of image sequences
- G06T 15/20: Perspective computation
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/33: using feature-based methods
- G06T 7/75: {involving models}
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 15/205: {Image-based rendering}
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 19/003: {Navigation within 3D models or images}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image}
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30244: Camera pose
- G06T 2219/004: Annotating, labelling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Mappings are determined between viewpoints of an object and an object model representing the object. Each mapping identifies a location on the object model corresponding with a portion of the object captured in one of the viewpoints. Tags for the object model are created based on the mappings, where each tag links one of the viewpoints to one of the locations on the object model. A user interface that includes the object model and the tags is provided for presentation on a display screen in a user interface. One of the viewpoints is presented in the user interface when the corresponding tag is selected in the object model.

## Claims

### Claim 1

A method comprising:
determining a respective mapping between each of a plurality of viewpoints of a vehicle and a vehicle model representing the vehicle, each mapping identifying a location on the vehicle model corresponding with a portion of the vehicle captured in the respective viewpoint;
automatically detecting, using artificial intelligence, defects associated with the vehicle;
creating a plurality of tags for the vehicle model based on the mappings, each tag linking one or more of the viewpoints to a respective one of the locations on the vehicle model, one or more of the tags corresponding to the defects; and
providing for presentation on a display screen a user interface that includes the vehicle model and the tags, the user interface presenting a designated one of the plurality of viewpoints when a designated tag corresponding with the designated viewpoint is selected in the vehicle model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the tags are represented in the user interface with green circles.

### Claim 3 (depends on 1)

The method of claim 1 further comprising:
responsive to detecting the defects automatically producing high resolution images of locations associated with the defects.

### Claim 4 (depends on 1)

The method recited in claim 1, wherein the designated tag is mapped to a set of the viewpoints including the designated viewpoint, and wherein each of the set of viewpoints is presented when the designated viewpoint is selected.

### Claim 5 (depends on 1)

The method recited in claim 1, wherein determining the respective mapping comprises identifying an object component represented in the respective viewpoint.

### Claim 6 (depends on 1)

The method recited in claim 1, the method further comprising:
determining whether a designated object component is represented in the plurality of viewpoints; and
when it is determined that the designated object component is not represented in the plurality of viewpoints, providing recording guidance via a user interface at a computing device to facilitate capture of the designated object component in an additional viewpoint of the object.

### Claim 7 (depends on 6)

The method recited in claim 6, wherein the recording guidance is provided via an overlay presented on a live camera feed at a smartphone.

### Claim 8

A cloud-based system comprising:
a remote server system comprising one or more processors configurable to cause:
determining a respective mapping between each of a plurality of viewpoints of a vehicle and a vehicle model representing the vehicle, each mapping identifying a location on the vehicle model corresponding with a portion of the vehicle captured in the respective viewpoint;
automatically detecting, using artificial intelligence, defects associated with the vehicle;
creating a plurality of tags for the vehicle model based on the mappings, each tag linking one or more of the viewpoints to a respective one of the locations on the vehicle model, one or more of the tags corresponding to the defects; and
providing for presentation on a display screen a user interface that includes the vehicle model and the tags, the user interface presenting a designated one of the plurality of viewpoints when a designated tag corresponding with the designated viewpoint is selected in the vehicle model.

### Claim 9 (depends on 8)

The cloud-based system of claim 8, wherein the tags are represented in the user interface with green circles.

### Claim 10 (depends on 8)

The cloud-based system of claim 8, the one or more processors further configurable to cause:
responsive to detecting the defects automatically producing high resolution images of locations associated with the defects.

### Claim 11 (depends on 8)

The cloud-based system of claim 8 wherein the designated tag is mapped to a set of the viewpoints including the designated viewpoint, and wherein each of the set of viewpoints is presented when the designated viewpoint is selected.

### Claim 12 (depends on 8)

The cloud-based system of claim 8, wherein determining the respective mapping comprises identifying an object component represented in the respective viewpoint.

### Claim 13 (depends on 8)

The cloud-based system of claim 8, the one or more processors further configurable to cause:
determining whether a designated object component is represented in the plurality of viewpoints; and
when it is determined that the designated object component is not represented in the plurality of viewpoints, providing recording guidance via a user interface at a computing device to facilitate capture of the designated object component in an additional viewpoint of the object.

### Claim 14 (depends on 13)

The cloud-based system of claim 13, wherein the recording guidance is provided via an overlay presented on a live camera feed at a smartphone.

### Claim 15

One or more non-transitory machine-readable media having instructions stored thereon for performing a method, the method comprising:
determining a respective mapping between each of a plurality of viewpoints of a vehicle and a vehicle model representing the vehicle, each mapping identifying a location on the vehicle model corresponding with a portion of the vehicle captured in the respective viewpoint;
automatically detecting, using artificial intelligence, defects associated with the vehicle;
creating a plurality of tags for the vehicle model based on the mappings, each tag linking one or more of the viewpoints to a respective one of the locations on the vehicle model, one or more of the tags corresponding to the defects; and
providing for presentation on a display screen a user interface that includes the vehicle model and the tags, the user interface presenting a designated one of the plurality of viewpoints when a designated tag corresponding with the designated viewpoint is selected in the vehicle model.

### Claim 16 (depends on 15)

The one or more non-transitory machine-readable media of claim 15, wherein the tags are represented in the user interface with green circles.

### Claim 17 (depends on 15)

The one or more non-transitory machine-readable media of claim 15, the method further comprising:
responsive to detecting the defects automatically producing high resolution images of locations associated with the defects.

### Claim 18 (depends on 15)

The one or more non-transitory machine-readable media of claim 15, wherein the designated tag is mapped to a set of the viewpoints including the designated viewpoint, and wherein each of the set of viewpoints is presented when the designated viewpoint is selected.

### Claim 19 (depends on 15)

The one or more non-transitory machine-readable media of claim 15, wherein determining the respective mapping comprises identifying an object component represented in the respective viewpoint.

### Claim 20 (depends on 15)

The one or more non-transitory machine-readable media of claim 15, the method further comprising:
determining whether a designated object component is represented in the plurality of viewpoints; and
when it is determined that the designated object component is not represented in the plurality of viewpoints, providing recording guidance via a user interface at a computing device to facilitate capture of the designated object component in an additional viewpoint of the object.
