# US 10,769,411 B2 · Pose estimation and model retrieval for objects in images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,411 B2
- Document ID: US10769411B2
- Status: Granted
- Kind: B2
- Application number: 15/946,669
- Filed: 2018-04-05
- Published: 2019-05-16
- Granted: 2020-09-08
- Earliest claimed: 2017-11-15
- Assignees: QUALCOMM Technologies, Inc.
- Applicants: Qualcomm Technologies,Inc.
- Inventors: Alexander Grabner; Peter Michael Roth; Vincent Lepetit
- Web page: https://patentgrep.com/patent/US10769411B2

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 7/75: {involving models}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/82: using neural networks
- G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2210/12: Bounding box
- G06T 2219/2016: Rotation, translation, scaling
- G06V 2201/10: Recognition assisted with metadata
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques are provided for selecting a three-dimensional model. An input image including an object can be obtained, and a pose of the object in the input image can be determined. One or more candidate three-dimensional models representing one or more objects in the determined pose can be obtained. From the one or more candidate three-dimensional models, a candidate three-dimensional model can be determined to represent the object in the input image.

## Claims

### Claim 1

A method of selecting a three-dimensional model, the method comprising:
obtaining a first input image including an object;
determining a pose of the object in the first input image;
obtaining a plurality of candidate three-dimensional models representing one or more objects;
obtaining a descriptor for the object in the first input image and a plurality of descriptors for the plurality of candidate three-dimensional models;
determining, from the plurality of candidate three-dimensional models, a candidate three-dimensional model to represent the object in one or more images based on the descriptor for the object in the first input image and the plurality of descriptors for the plurality of candidate three-dimensional models;
obtaining a second input image including the object in a different pose than the pose of the object in the first input image; and
adjusting a pose of the candidate three-dimensional model in an output image based on a difference between the pose of the object in the second input image and the pose of the object in the first input image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising generating the output image based on the candidate three-dimensional model and the second input image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving a user input to manipulate the candidate three-dimensional model; and
adjusting one or more of a pose or a location of the candidate three-dimensional model in an output image based on the user input.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of candidate three-dimensional models represent the one or more objects having a same pose as the determined pose of the object in the first input image.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
obtaining a plurality of three-dimensional models representing a plurality of objects; and
selecting a subset of the plurality of three-dimensional models as the plurality of candidate three-dimensional models.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
determining a category of the object in the first input image; and
determining one or more categories associated with the plurality of three-dimensional models;
wherein the subset of the plurality of three-dimensional models are selected from the plurality of three-dimensional models based on the plurality of candidate three-dimensional models having the category of the object in the first input image.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
obtaining a plurality of images for the plurality of candidate three-dimensional models, wherein the plurality of images are generated to include the one or more objects in a same pose as the determined pose, the plurality of descriptors obtained for the plurality of candidate three-dimensional models being generated from the plurality of images.

### Claim 8 (depends on 7)

The method of claim 7, wherein the plurality of images generated for the plurality of candidate three-dimensional models include a plurality of depth maps.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
comparing the plurality of descriptors obtained for the plurality of candidate three-dimensional models to the descriptor obtained for the object in the first input image; and
determining a descriptor from the plurality of descriptors that has a closest match to the descriptor obtained for the object in the first input image, wherein the candidate three-dimensional model is determined based on the descriptor from the plurality of descriptors having the closest match to the descriptor obtained for the object in the first input image.

### Claim 10 (depends on 9)

The method of claim 9, wherein comparing the plurality of descriptors to the descriptor obtained for the object in the first input image includes: performing a nearest-neighbor search using the descriptor obtained for the object in the first input image as input.

### Claim 11 (depends on 1)

The method of claim 1, wherein the plurality of candidate three-dimensional models include a plurality of three-dimensional meshes representing the one or more objects.

### Claim 12 (depends on 1)

The method of claim 1, wherein determining the pose of the object in the first input image includes:
determining a plurality of two-dimensional projections of a three-dimensional bounding box of the object in the first input image; and
estimating the pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box.

### Claim 13 (depends on 12)

The method of claim 12, wherein the plurality of two-dimensional projections of the three-dimensional bounding box are determined by applying a trained convolutional network to the first input image, wherein the trained convolutional network is trained to predict two-dimensional projections of the three-dimensional bounding box of the object in a plurality of poses.

### Claim 14 (depends on 12)

The method of claim 12, wherein estimating the pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box includes applying a perspective-n-point (PnP) problem using correspondences between the plurality of two-dimensional projections and three-dimensional points of the three-dimensional bounding box that correspond to the plurality of two-dimensional projections.

### Claim 15 (depends on 1)

The method of claim 1, further comprising:
obtaining an additional input image, the additional input image including the object in a different location than a location of the object in the first input image or the second input image; and
adjusting a location of the candidate three-dimensional model in at least one output image based on a difference between the location of the object in the additional input image and the location of the object in the first input image or the second input image.

### Claim 16

An apparatus for selecting a three-dimensional model, comprising:
a memory configured to store one or more images; and
a processor configured to:
obtain a first input image including an object;
determine a pose of the object in the first input image;
obtain a plurality of candidate three-dimensional models representing one or more objects;
obtain a descriptor for the object in the first input image and a plurality of descriptors for the plurality of candidate three-dimensional models;
determine, from the plurality of candidate three-dimensional models, a candidate three-dimensional model to represent the object in one or more images based on the descriptor for the object in the first input image and the plurality of descriptors for the plurality of candidate three-dimensional models;
obtain a second input image including the object in a different pose than the pose of the object in the first input image; and
adjust a pose of the candidate three-dimensional model in an output image based on a difference between the pose of the object in the second input image and the pose of the object in the first input image.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the processor is configured to generate the output image based on the candidate three-dimensional model and the second input image.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein the processor is configured to:
receive a user input to manipulate the candidate three-dimensional model; and
adjust one or more of a pose or a location of the candidate three-dimensional model in an output image based on the user input.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein the plurality of candidate three-dimensional models represent the one or more objects having a same pose as the determined pose of the object in the first input image.

### Claim 20 (depends on 16)

The apparatus of claim 16, wherein the processor is configured to:
determine a category of the object in the first input image; and
determine one or more categories associated with a plurality of three-dimensional models;
selecting a subset of the plurality of three-dimensional models from the plurality of three-dimensional models based on the plurality of candidate three-dimensional models having the category of the object in the first input image.

### Claim 21 (depends on 16)

The apparatus of claim 16, wherein the processor is configured to:
obtain a plurality of images for the plurality of candidate three-dimensional models, wherein the plurality of images are generated to include the one or more objects in a same pose as the determined pose, the plurality of descriptors obtained for the plurality of candidate three-dimensional models being generated from the plurality of images.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein the plurality of images generated for the plurality of candidate three-dimensional models include a plurality of depth maps.

### Claim 23 (depends on 16)

The apparatus of claim 16, wherein the processor is configured to:
compare the plurality of descriptors obtained for the plurality of candidate three-dimensional models to the descriptor obtained for the object in the first input image; and determine a descriptor from the plurality of descriptors that has a closest match to the descriptor obtained for the object in the first input image, wherein the candidate three-dimensional model is determined based on the descriptor from the plurality of descriptors having the closest match to the descriptor obtained for the object in the first input image.

### Claim 24 (depends on 16)

The apparatus of claim 16, wherein determining the pose of the object in the first input image includes:
determining a plurality of two-dimensional projections of a three-dimensional bounding box of the object in the first input image; and
estimating the pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box.

### Claim 25 (depends on 24)

The apparatus of claim 24, wherein estimating the pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box includes applying a perspective-n-point (PnP) problem using correspondences between the plurality of two-dimensional projections and three-dimensional points of the three-dimensional bounding box that correspond to the plurality of two-dimensional projections.

### Claim 26 (depends on 16)

The apparatus of claim 16, further comprising a camera for capturing the one or more images.

### Claim 27 (depends on 16)

The apparatus of claim 16, wherein the apparatus comprises a mobile device with a camera for capturing the one or more images.

### Claim 28 (depends on 16)

The apparatus of claim 16, further comprising a display for displaying the one or more images.

### Claim 29 (depends on 16)

The apparatus of claim 16, wherein the processor is configured to:
obtain an additional input image, the additional input image including the object in a different location than a location of the object in the first input image or the second input image; and
adjust a location of the candidate three-dimensional model in at least one output image based on a difference between the location of the object in the additional input image and the location of the object in the first input image or the second input image.

### Claim 30

A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:
obtain a first input image including an object;
determine a pose of the object in the first input image;
obtain a plurality of candidate three-dimensional models representing one or more objects;
obtain a descriptor for the object in the first input image and a plurality of descriptors for the plurality of candidate three-dimensional models;
determine, from the plurality of candidate three-dimensional models, a candidate three-dimensional model to represent the object in one or more images based on the descriptor for the object in the first input image and the plurality of descriptors for the plurality of candidate three-dimensional models;
obtain a second input image including the object in a different pose than the pose of the object in the first input image; and
adjust a pose of the candidate three-dimensional model in an output image based on a difference between the pose of the object in the second input image and the pose of the object in the first input image.