# US 2021/0241522 A1 · Three-dimensional object reconstruction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241522 A1
- Document ID: US20210241522A1
- Status: Published application
- Kind: A1
- Application number: 17/301,926
- Filed: 2021-04-19
- Published: 2021-08-05
- Earliest claimed: 2018-11-16
- Assignees: Ariel Al Ltd
- Applicants: Ariel Al Ltd.
- Inventors: Riza Alp Guler; Iason Kokkinos
- Web page: https://patentgrep.com/patent/US20210241522A1

## CPC

- G06T 7/50: Depth or shape recovery
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/02: Neural networks
- G06T 7/75: {involving models}
- G06T 17/005: {Tree description, e.g. octree, quadtree}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30196: Human being; Person
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

This disclosure relates to reconstructing three-dimensional models of objects from two-dimensional images. According to a first aspect, this specification describes a computer implemented method for creating a three-dimensional reconstruction from a two-dimensional image, the method comprising: receiving a two-dimensional image; identifying an object in the image to be reconstructed and identifying a type of said object; spatially anchoring a pre-determined set of object landmarks within the image; extracting a two-dimensional image representation from each object landmark; estimating a respective three-dimensional representation for the respective two-dimensional image representations; and combining the respective three-dimensional representations resulting in a fused three-dimensional representation of the object.

## Claims

### Claim 1

A computer implemented method for creating a three-dimensional reconstruction from a two-dimensional image, the method comprising:
receiving a two-dimensional image;
identifying an object in the image to be reconstructed and identifying a type of said object;
spatially anchoring a pre-determined set of object landmarks within the image;
extracting a two-dimensional image representation from each object landmark;
estimating a respective three-dimensional representation for the respective two-dimensional image representations; and
combining the respective three-dimensional representations resulting in a fused three-dimensional representation of the object.

### Claim 2 (depends on 1)

The method of claim 1, wherein spatially anchoring, extracting, estimating and combining are performed in an encoder neural network.

### Claim 3 (depends on 2)

The method of claim 2, wherein the encoder neural network is one of a plurality of encoder networks, the method further comprising choosing the encoder neural network based on the identified object type.

### Claim 4 (depends on 1)

The method of claim 1, wherein the receiving further comprises receiving a plurality of images, wherein spatially anchoring, extracting, estimating and combining are done for each received image, for example in real-time.

### Claim 5 (depends on 1)

The method of claim 1, wherein the combining further comprises:
decoupling respective estimated three-dimensional representations of object landmarks which are kinematically independent; and
wherein each respective estimated three-dimensional representation comprises a corresponding set of possible orientation angles of the respective object landmark, the method further comprising applying a weighting to each orientation angle in the set of possible orientation angles based on a kinematic association of the respective object landmark to the respective possible orientation and visibility of the landmark in the received image.

### Claim 6 (depends on 5)

The method of claim 5, wherein prior to the weighting, the method further comprising identifying object landmarks, within a predetermined set of object landmarks, which are not visible in the two-dimensional image.

### Claim 7 (depends on 1)

The method of claim 1, wherein the method further comprises applying the fused three-dimensional representation of the object to a part-based three-dimensional shape representation model.

### Claim 8 (depends on 7)

The method of claim 7, wherein the part-based three-dimensional shape representation model is a kinematic tree.

### Claim 9

A computation unit for creating a three-dimensional reconstruction from a single two-dimensional image, the computation unit comprising:
a memory; and
at least one processor, wherein at least one processor is configured to perform operations comprising:
receiving a two-dimensional image;
identifying an object in the image to be reconstructed and identifying a type of said object;
spatially anchoring a pre-determined set of object landmarks within the image;
extracting a two-dimensional image representation from each object landmark;
estimating a respective three-dimensional representation for the respective two-dimensional image representations; and
combining the respective three-dimensional representations resulting in a fused three-dimensional representation of the object.

### Claim 10

A computer readable medium that stores a set of instructions that is executable by at least one processor of a computation unit for creating a three-dimensional reconstruction from a single two-dimensional image to cause the computation unit to perform operations comprising:
receiving a two-dimensional image;
identifying an object in the image to be reconstructed and identifying a type of said object;
spatially anchoring a pre-determined set of object landmarks within the image;
extracting a two-dimensional image representation from each object landmark;
estimating a respective three-dimensional representation for the respective two-dimensional image representations; and
combining the respective three-dimensional representations resulting in a fused three-dimensional representation of the object.

### Claim 11

A method for updating a fused three-dimensional representation of an object comprised in a two-dimensional image, the method comprising:
projecting the fused three-dimensional representation on to a two-dimensional image plane resulting in a projected representation;
comparing respective positions of the projected representation with the object in the two-dimensional image;
determining an error value based on the comparing; and
adjusting parameters of the fused three-dimensional representation based on the error value, wherein the comparing, measuring and adjusting are iterated until the measured error value is below a predetermined threshold value or a threshold number of iterations is surpassed.

### Claim 12 (depends on 11)

The method of claim 11, wherein comparing further comprises: comparing respective positions of object landmarks and surface-based points of the projected representation and the two-dimensional image.

### Claim 13 (depends on 12)

The method of claim 12, further comprising obtaining a Cartesian sampling or charting of a known surface model based on a detected object type of the object in the two-dimensional image.

### Claim 14

A computation unit for creating a three-dimensional reconstruction from a two-dimensional image for updating a fused three-dimensional representation of an object comprised in a two-dimensional image, the computation unit comprising:
a memory; and
at least one processor, wherein the at least one processor is configured to perform operations comprising:
projecting the fused three-dimensional representation on to a two-dimensional image plane resulting in a projected representation;
comparing respective positions of the projected representation with the object in the two-dimensional image;
determining an error value based on the comparing; and
adjusting parameters of the fused three-dimensional representation based on the error value, wherein the comparing, measuring and adjusting are iterated until the measured error value is below a predetermined threshold value or a threshold number of iterations is surpassed.

### Claim 15 (depends on 14)

The computation unit of claim 14, wherein comparing further comprises: comparing respective positions of object landmarks and surface-based points of the projected representation and the two-dimensional image.

### Claim 16 (depends on 15)

The computation unit of claim 15, further comprising operations for obtaining a Cartesian sampling or charting of a known surface model based on a detected object type of the object in the two-dimensional image.

### Claim 17

A computer readable medium that stores a set of instructions that is executable by at least one processor of a computation unit for updating a fused three-dimensional representation of an object comprised in a two-dimensional image to cause the computation unit to perform operations comprising:
projecting the fused three-dimensional representation on to a two-dimensional image plane resulting in a projected representation;
comparing respective positions of the projected representation with the object in the two-dimensional image;
determining an error value based on the comparing; and
adjusting parameters of the fused three-dimensional representation based on the error value, wherein the comparing, measuring and adjusting are iterated until the measured error value is below a predetermined threshold value or a threshold number of iterations is surpassed.

### Claim 18 (depends on 17)

The computer readable medium of claim 17, wherein comparing further comprises: comparing respective positions of object landmarks and surface-based points of the projected representation and the two-dimensional image.

### Claim 19 (depends on 18)

The computer readable medium of claim 18, further comprising operations for obtaining a Cartesian sampling or charting of a known surface model based on a detected object type of the object in the two-dimensional image.
