# US 2022/0277489 A1 · Modeling objects from monocular camera outputs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277489 A1
- Document ID: US20220277489A1
- Status: Published application
- Kind: A1
- Application number: 17/189,105
- Filed: 2021-03-01
- Published: 2022-09-01
- Earliest claimed: 2021-03-01
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Ashar ALI; Gokce DANE; Upal MAHBUB; Samuel SUNARJO; Gerhard REITMAYR
- Web page: https://patentgrep.com/patent/US20220277489A1

## CPC

- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and techniques are provided for modeling three-dimensional (3D) meshes using images. An example method can include receiving, via a neural network system, an image of a target and metadata associated with the image and/or a device that captured the image; determining, based on the image and metadata, first 3D mesh parameters of a first 3D mesh of the target, the first 3D mesh parameters and first 3D mesh corresponding to a first reference frame associated with the image and/or the device; and determining, based on the first 3D mesh parameters, second 3D mesh parameters for a second 3D mesh of the target, the second 3D mesh parameters and second 3D mesh corresponding to a second reference frame, the second reference frame including a 3D coordinate system of a real-world scene where the target is located.

## Claims

### Claim 1

A apparatus comprising:
memory; and
one or more processors coupled to the memory, the one or more processors being configured to:
receive an image of a target and metadata, the metadata associated with at least one of the image and an image capture device that captured the image;
determine, based on the image and the metadata, first three-dimensional (3D) mesh parameters of a first 3D mesh of the target, the first 3D mesh parameters and the first 3D mesh corresponding to a first reference frame associated with at least one of the image and the image capture device; and
determine, based on the first 3D mesh parameters, second 3D mesh parameters for a second 3D mesh of the target, the second 3D mesh parameters and the second 3D mesh corresponding to a second reference frame, the second reference frame comprising a 3D coordinate system of a real-world scene in which the target is located.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are configured to:
update one or more parameters associated with a neural network system based on one or more losses associated with a third 3D mesh of the target in a real-world reference frame; and
determine, by the neural network system using the updated one or more parameters, the second 3D mesh parameters for the second 3D mesh of the target.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the one or more processors are configured to:
determine the third 3D mesh of the target based on at least one of a non-parametric mesh of the target, a parametric mesh of the target, rotation parameters, translation parameters, image features and metadata features.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the one or more processors are configured to:
determine, based on an additional image of the target and additional metadata, the image features and the metadata features;
determine, based on the image features and the metadata features, the non-parametric mesh of the target and mesh parameters associated with the target; and
determine, based on pose parameters and shape parameters associated with the target, the parametric mesh of the target.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the mesh parameters comprise at least one of pose parameters, shape parameters, rotation parameters, and translation parameters.

### Claim 6 (depends on 4)

The apparatus of claim 4, wherein the parametric mesh, the non-parametric mesh and the mesh parameters correspond to a particular reference frame of a device that captured the additional image, and wherein the one or more losses comprise at least one of a loss of the third 3D mesh, a loss of 3D keypoints in the third 3D mesh, a pose and shape regularization loss, and a loss of two-dimensional (2D) keypoint projections from at least one of the third 3D mesh and the 3D keypoints.

### Claim 7 (depends on 4)

The apparatus of claim 4, wherein the metadata features comprise intrinsic parameters associated with a device that captured the additional image, the intrinsic parameters comprising at least one of a focal length of the device, an optical center of the device, a skew of the device, a size of the additional image, a lens distortion associated with the device, and a particular reference frame of the device.

### Claim 8 (depends on 3)

The apparatus of claim 3, wherein the one or more processors are configured to:
train the neural network system using the non-parametric mesh of the target and the parametric mesh of the target.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are configured to generate the second 3D mesh of the target based on the second 3D mesh parameters, wherein the second 3D mesh is in the 3D coordinate system of the real-world scene.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the second 3D mesh comprises at least one of a skinned mesh model and a parametric mesh model, and wherein the target comprises an articulated object.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein, to determine the second 3D mesh parameters, the one or more processors are configured to:
transform one or more of the first 3D mesh parameters from the first reference frame to the second reference frame.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein, to determine the second 3D mesh parameters, the one or more processors are configured to:
determine a rotation and translation of the first 3D mesh from the first reference frame to the second reference frame.

### Claim 13 (depends on 1)

The apparatus of claim 1, wherein, to determine the second 3D mesh parameters, the one or more processors are configured to:
identify 3D keypoints associated with the first 3D mesh of the target; and
based on the image and the metadata, determine locations and orientations of the 3D keypoints in the 3D coordinate system of the real-world scene.

### Claim 14 (depends on 1)

The apparatus of claim 1, wherein the first 3D mesh parameters comprise pose parameters, shape parameters, translation parameters, location parameters, and visibility parameters, and wherein the visibility parameters indicate at least one of an estimated visibility of one or more points in the first 3D mesh and an estimated occlusion of the one or more points in the first 3D mesh.

### Claim 15 (depends on 1)

The apparatus of claim 1, wherein the metadata comprises intrinsic parameters associated with the image capture device, the intrinsic parameters comprising at least one of a focal length of the image capture device, an optical center of the image capture device, a skew of the image capture device, a size of the image, a lens distortion associated with the image capture device, a respective reference frame of the image capture device, the respective reference frame comprising the first reference frame.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the image comprises a cropped monocular image, and wherein the metadata comprises a location of a bounding box around the target in the cropped monocular image.

### Claim 17 (depends on 1)

The apparatus of claim 1, wherein the apparatus comprises the image capture device, and wherein the image capture device comprises a camera device.

### Claim 18 (depends on 1)

The apparatus of claim 1, wherein the apparatus comprises a mobile device.

### Claim 19 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are configured to:
determine image features in the image of the target;
determine metadata features and image features based the image and the metadata; and
determine the first 3D mesh parameters based on a combination of the metadata features and the image features, the first 3D mesh parameters comprising at least one of pose parameters, shape parameters, translation parameters, location parameters, and visibility parameters, wherein the visibility parameters indicate at least one of an estimated visibility of one or more points in the first 3D mesh and an estimated occlusion of the one or more points in the first 3D mesh.

### Claim 20

A method comprising:
receiving an image of a target and metadata, the metadata associated with at least one of the image and an image capture device that captured the image;
determining, based on the image and the metadata, first three-dimensional (3D) mesh parameters of a first 3D mesh of the target, the first 3D mesh parameters and the first 3D mesh corresponding to a first reference frame associated with at least one of the image and the image capture device; and
determining, based on the first 3D mesh parameters, second 3D mesh parameters for a second 3D mesh of the target, the second 3D mesh parameters and the second 3D mesh corresponding to a second reference frame, the second reference frame comprising a 3D coordinate system of a real-world scene in which the target is located.

### Claim 21 (depends on 20)

The method of claim 20, further comprising:
updating one or more parameters associated with a neural network system based on one or more losses associated with a third 3D mesh of the target in a real-world reference frame; and
determining, by the neural network system using the updated one or more parameters, the second 3D mesh parameters for the second 3D mesh of the target.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
determining the third 3D mesh of the target based on at least one of a non-parametric mesh of the target, a parametric mesh of the target, rotation parameters, translation parameters, image features and metadata features.

### Claim 23 (depends on 22)

The method of claim 22, further comprising:
determining, based on an additional image of the target and additional metadata, the image features and the metadata features;
determining, based on the image features and the metadata features, the non-parametric mesh of the target and mesh parameters associated with the target; and
determining, based on pose parameters and shape parameters associated with the target, the parametric mesh of the target, wherein the mesh parameters comprise at least one of pose parameters, shape parameters, rotation parameters, and translation parameters.

### Claim 24 (depends on 23)

The method of claim 23, wherein the parametric mesh, the non-parametric mesh and the mesh parameters correspond to a particular reference frame of a device that captured the additional image, and wherein the one or more losses comprise at least one of a loss of the third 3D mesh, a loss of 3D keypoints in the third 3D mesh, a pose and shape regularization loss, and a loss of two-dimensional (2D) keypoint projections from at least one of the third 3D mesh and the 3D keypoints.

### Claim 25 (depends on 23)

The method of claim 23, wherein the metadata features comprise intrinsic parameters associated with a device that captured the additional image, the intrinsic parameters comprising at least one of a focal length of the device, an optical center of the device, a skew of the device, a size of the additional image, a lens distortion associated with the device, and a particular reference frame of the device, the method further comprising training the neural network system using the non-parametric mesh of the target and the parametric mesh of the target.

### Claim 26 (depends on 20)

The method of claim 20, further comprising: generating the second 3D mesh of the target based on the second 3D mesh parameters, wherein the second 3D mesh is in the 3D coordinate system of the real-world scene, wherein the second 3D mesh comprises at least one of a skinned mesh model and a parametric mesh model, wherein the target comprises an articulated object, and wherein the first 3D mesh parameters comprise pose parameters, shape parameters, translation parameters, location parameters, and visibility parameters, the visibility parameters indicating at least one of an estimated visibility of one or more points in the first 3D mesh and an estimated occlusion of the one or more points in the first 3D mesh.

### Claim 27 (depends on 20)

The method of claim 20, wherein determining the second 3D mesh parameters comprises at least one of:
transforming one or more of the first 3D mesh parameters from the first reference frame to the second reference frame; and
determining a rotation and translation of the first 3D mesh from the first reference frame to the second reference frame.

### Claim 28 (depends on 20)

The method of claim 20, wherein the metadata comprises intrinsic parameters associated with the image capture device, wherein the image comprises a monocular image, wherein the metadata comprises a location of a bounding box around the target in the monocular image, the intrinsic parameters comprising at least one of a focal length of the image capture device, an optical center of the image capture device, a skew of the image capture device, a size of the image, a lens distortion associated with the image capture device, a respective reference frame of the image capture device comprising the first reference frame.

### Claim 29 (depends on 20)

The method of claim 20, further comprising:
determining image features in the image of the target;
determining metadata features and image features based the image and the metadata; and
determining the first 3D mesh parameters based on a combination of the metadata features and the image features, the first 3D mesh parameters comprising at least one of pose parameters, shape parameters, translation parameters, location parameters, and visibility parameters, wherein the visibility parameters indicate at least one of an estimated visibility of one or more points in the first 3D mesh and an estimated occlusion of the one or more points in the first 3D mesh.

### Claim 30

A non-transitory computer-readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to:
receive an image of a target and metadata, the metadata associated with at least one of the image and an image capture device that captured the image;
determine, based on the image and the metadata, first three-dimensional (3D) mesh parameters of a first 3D mesh of the target, the first 3D mesh parameters and the first 3D mesh corresponding to a first reference frame associated with at least one of the image and the image capture device; and
determine, based on the first 3D mesh parameters, second 3D mesh parameters for a second 3D mesh of the target, the second 3D mesh parameters and the second 3D mesh corresponding to a second reference frame, the second reference frame comprising a 3D coordinate system of a real-world scene in which the target is located.
