# US 2023/0410338 A1 · Method for optimizing depth estimation model, computer device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410338 A1
- Document ID: US20230410338A1
- Status: Published application
- Kind: A1
- Application number: 17/896,835
- Filed: 2022-08-26
- Published: 2023-12-21
- Earliest claimed: 2022-05-31
- Assignees: Hon Hai Precision Industry Co., Ltd.
- Applicants: Hon Hai Precision Industry Co., Ltd.
- Inventors: TSUNG-WEI LIU; CHIN-PIN KUO
- Web page: https://patentgrep.com/patent/US20230410338A1

## CPC

- G06T 7/55: from multiple images
- G06V 10/757: {Matching configurations of points or features}
- G01S 13/867: {Combination of radar systems with cameras}
- G06T 7/73: using feature-based methods
- G06T 7/74: {involving reference images or patches}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- Y02T 10/40: Engine management systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

This application provides a method for optimizing a depth estimation model. The method includes obtaining a video of an object and capturing a first image and a second image from the video. An initial depth estimation model is obtained. An updated depth estimation model is obtained by performing an optimization process on the initial depth estimation model, and the optimization process is repeatedly performed on the updated depth estimation model. Once the updated depth estimation model meets predetermined requirements, the updated depth estimation model meeting predetermined requirements is determined as a target depth estimation model.

## Claims

### Claim 1

A method for optimizing a depth estimation model applied to a computer device, the method comprising:
obtaining a video of an object and capturing a first image and a second image from the video;
obtaining an initial depth estimation model, and obtaining an updated depth estimation model by performing an optimization process on the initial depth estimation model; and
repeatedly performing the optimization process on the updated depth estimation model until the updated depth estimation model meets predetermined requirements, and determining the updated depth estimation model that meets the predetermined requirements as a target depth estimation model;
wherein the optimization process on the initial depth estimation model comprises:
obtaining a first depth image of the first image by using the initial depth estimation model;
obtaining a correspondence between each pixel point in the first image and each pixel point in the second image;
obtaining a third image by performing a back projection on the first depth image;
obtaining a fourth image by updating a pixel value of each pixel point in the third image to be a pixel value of the corresponding pixel point in the second image according to the correspondence between each pixel point in the first image and each pixel point in the second image;
obtaining an initial loss function between the first image and the fourth image based on the initial depth estimation model;
obtaining a depth value of each pixel point in the first image and the fourth image by using a radar device, and obtaining an updated loss function by optimizing the initial loss function based on the depth value; and
obtaining the updated depth estimation model by optimizing the initial depth estimation model using the updated loss function.

### Claim 2 (depends on 1)

The method according to claim 1, wherein obtaining the video of the object and capturing the first image and the second image from the video comprises:
taking the video of the object using a camera device;
obtaining the video from the camera device;
obtaining two image frames from the video;
setting the two image frames as the first image and the second image, the two image frames corresponding to different viewing angles of the object.

### Claim 3 (depends on 1)

The method according to claim 1, wherein obtaining of the correspondence between each pixel point in the first image and each pixel point in the second image comprises:
calculating a pose transformation relationship between the first image and the second image; and
calculating the correspondence between each pixel point in the first mage and each pixel point in the second image according to the pose transformation relationship.

### Claim 4 (depends on 3)

The method according to claim 3, wherein calculating of the pose transformation relationship between the first image and the second image comprises:
obtaining a preset number of pairs of matching points by using an image feature point detection algorithm and an image feature matching algorithm, each pair of matching points comprising a feature point in the first image and a corresponding feature point in the second image;
obtaining three-dimensional space coordinates of each feature point in the first image and obtaining three-dimensional space coordinates of each feature point in the second image, by using a triangulation method based on each of the preset number of pairs of matching points;
calculating a first pose when the camera device captures the first image by using a perspective-n-point algorithm based on the three-dimensional space coordinates of each feature point in the first image; and calculating a second pose when the camera device captures the second image by using the perspective-n-point algorithm based on the three-dimensional space coordinates of each feature point in the second image; and
calculating the pose transformation relationship between the first image and the second image according to the first pose and the second pose.

### Claim 5 (depends on 1)

The method according to claim 1, wherein obtaining of the initial loss function between the first image and the fourth image based on the initial depth estimation model comprises:
obtaining a fourth depth image of the fourth image by inputting the fourth image into the initial depth estimation model;
calculating an initial depth difference value between corresponding pixel points in the first depth image and the fourth depth image;
establishing the initial loss function based on the initial depth difference value, the initial loss function comprising structural similarity of the first depth image and the fourth depth image.

### Claim 6 (depends on 5)

The method according to claim 5, wherein obtaining of the depth value of each pixel point in the first image and the fourth image by using the radar device, and the obtaining of the updated loss function by optimizing the initial loss function based on the depth value comprises:
calculating a depth difference value between each pixel point in the first image and the corresponding pixel point in the fourth image, based on the depth value of each pixel point in the first image and the fourth image obtained by using the radar device, and taking the calculated depth difference value as a calibrated depth difference value; and
optimizing the initial loss function by setting the calibrated depth difference value as an optimization target of the initial loss function.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising:
in response that the updated loss function converges to a preset threshold range, determining that the updated depth estimation model meets the requirements.

### Claim 8

A computer device comprising:
a storage device;
at least one processor; and
the storage device storing one or more programs, which when executed by the at least one processor, cause the at least one processor to:
obtain a video of an object and capturing a first image and a second image from the video;
obtain an initial depth estimation model, and obtain an updated depth estimation model by perforating an optimization process on the initial depth estimation model; and
repeatedly perform the optimization process on the updated depth estimation model until the updated depth estimation model meets predetermined requirements, and determine the updated depth estimation model that meets the predetermined requirements as a target depth estimation model;
wherein the optimization process on the initial depth estimation model comprises:
obtaining a first depth image of the first image by using the initial depth estimation model;
obtaining a correspondence between each pixel point in the first image and each pixel point in the second image;
obtaining a third image by performing a back projection on the first depth image;
obtaining a fourth image by updating a pixel value of each pixel point in the third image to be a pixel value of the corresponding pixel point in the second image according to the correspondence between each pixel point in the first image and each pixel point in the second image;
obtaining an initial loss function between the first image and the fourth image based on the initial depth estimation model;
obtaining a depth value of each pixel point in the first image and the fourth image by using a radar device, and obtaining an updated loss function by optimizing the initial loss function based on the depth value; and
obtaining the updated depth estimation model by optimizing the initial depth estimation model using the updated loss function.

### Claim 9 (depends on 8)

The computer device according to claim 8. wherein the at least one processor obtains the video of the object and captures the first image and the second image from the video by:
taking the video of the object using a camera device;
obtaining the video from the camera device;
obtaining two image frames from the video;
setting the two image frames as the first image and the second image, the two image frames corresponding to different viewing angles of the object.

### Claim 10 (depends on 8)

The computer device according to claim 8, wherein the at least one processor obtains the correspondence between each pixel point in the first image and each pixel point in the second image by:
calculating a pose transformation relationship between the first image and the second image; and
calculating the correspondence between each pixel point in the first image and each pixel point in the second image according to the pose transformation relationship.

### Claim 11 (depends on 10)

The computer device according to claim 10, wherein the at least one processor calculates the pose transformation relationship between the first image and the second image by:
obtaining a preset number of pairs of matching points by using an image feature point detection algorithm and an image feature matching algorithm, each pair of matching points comprising a feature point in the first image and a corresponding feature point in the second image;
obtaining three-dimensional space coordinates of each feature point in the first image and obtaining three-dimensional space coordinates of each feature point in the second image, by using a triangulation method based on each of the preset number of pairs of matching points;
calculating a first pose when the camera device captures the first image by using a perspective-n-point algorithm based on the three-dimensional space coordinates of each feature point in the first image; and calculating a second pose when the camera device captures the second image by using the perspective-n-point algorithm based on the three-dimensional space coordinates of each feature point in the second image; and
calculating the pose transformation relationship between the first image and the second image according to the first pose and the second pose.

### Claim 12 (depends on 8)

The computer device according to claim 8, wherein the at least one processor obtains the initial loss function between the first image and the fourth image based on the initial depth estimation model by:
obtaining a fourth depth image of the fourth image by inputting the fourth image into the initial depth estimation model;
calculating an initial depth difference value between corresponding pixel points in the first depth image and the fourth depth image;
establishing the initial loss function based on the initial depth difference value, the initial loss function comprising structural similarity of the first depth image and the fourth depth image.

### Claim 13 (depends on 12)

The computer device according to claim 12, wherein the at least one processor obtains the depth value of each pixel point in the first image and the fourth image by using the radar device, and obtains the updated loss function by optimizing the initial loss function based on the depth value by:
calculating a depth difference value between each pixel point in the first image and the corresponding pixel point in the fourth image, based on the depth value of each pixel point in the first image and the fourth image obtained by using the radar device, and taking the calculated depth difference value as a calibrated depth difference value; and
optimizing the initial loss function by setting the calibrated depth difference value as an optimization target of the initial loss function.

### Claim 14 (depends on 8)

The computer device according to claim 8, wherein the at least one processor is further caused to:
in response that the updated loss function converges to a preset threshold range, determine that the updated depth estimation model meets the requirements.

### Claim 15

A non-transitory storage medium having stored thereon at least one computer-readable instructions, which when executed by a processor of a computer device, causes the processor to perform a method for optimizing a depth estimation model, wherein the method comprises:
obtaining a video of an object and capturing a first image and a second image from the video;
obtaining an initial depth estimation model, and obtaining an updated depth estimation model by performing an optimization process on the initial depth estimation model; and
repeatedly performing the optimization process on the updated depth estimation model until the updated depth estimation model meets predetermined requirements, and determining the updated depth estimation model that meets the predetermined requirements as a target depth estimation model;
wherein optimization process on the initial depth estimation model comprises:
obtaining a first depth image of the first image by using the initial depth estimation model;
obtaining a correspondence between each pixel point in the first image and each pixel point in the second image;
obtaining a third image by performing back projection on the first depth image;
obtaining a fourth image by updating a pixel value of each pixel point in the third image to be a pixel value of the corresponding pixel point in the second image according to the correspondence between each pixel point in the first image and each pixel point in the second image;
obtaining an initial loss function between the first image and the fourth image based on the initial depth estimation model;
obtaining a depth value of each pixel point in the first image and the fourth image by using a radar device, and obtaining an updated loss function by optimizing the initial loss function based on the depth value; and
obtaining the updated depth estimation model by optimizing the initial depth estimation model using the updated loss function.

### Claim 16 (depends on 15)

The non-transitory storage medium according to claim 15, wherein obtaining the video of the object and capturing the first image and the second image from the video comprises:
taking the video of the object using a camera device;
obtaining the video from the camera device;
obtaining two image frames from the video;
setting the two image frames as the first image and the second image, the two image frames corresponding to different viewing angles of the object.

### Claim 17 (depends on 15)

The non-transitory storage medium according to claim 15, wherein obtaining of the correspondence between each pixel point in the first image and each pixel point in the second image comprises:
calculating a pose transformation relationship between the first image and the second image; and
calculating the correspondence between each pixel point in the first image and each pixel point in the second image according to the pose transformation relationship.

### Claim 18 (depends on 17)

The non-transitory storage medium according to claim 17, wherein calculating of the pose transformation relationship between the first image and the second image comprises:
obtaining a preset number of pairs of matching points by using an image feature point detection algorithm and an image feature matching algorithm, each pair of matching points comprising a feature point in the first image and a corresponding feature point in the second image;
obtaining three-dimensional space coordinates of each feature point in the first image and obtaining three-dimensional space coordinates of each feature point in the second image, by using a triangulation method based on each of the preset number of pairs of matching points;
calculating a first pose when the camera device captures the first image by using a perspective-n-point algorithm based on the three-dimensional space coordinates of each feature point in the first image; and calculating a second pose when the camera device captures the second image by using the perspective-n-point algorithm based on the three-dimensional space coordinates of each feature point in the second image; and
calculating the pose transformation relationship between the first image and the second image according to the first pose and the second pose.

### Claim 19 (depends on 15)

The non-transitory storage medium according to claim 15, wherein obtaining of the initial loss function between the first image and the fourth image based on the initial depth estimation model comprises:
obtaining a fourth depth image of the fourth image by inputting the fourth image into the initial depth estimation model;
calculating an initial depth difference value between corresponding pixel points in the first depth image and the fourth depth image;
establishing the initial loss function based on the initial depth difference value, the initial loss function comprising structural similarity of the first depth image and the fourth depth image.

### Claim 20 (depends on 19)

The non-transitory storage medium according to claim 19, wherein obtaining of the depth value of each pixel point in the first image and the fourth image by using the radar device, and the obtaining of the updated loss function by optimizing the initial loss function based on the depth value comprises:
calculating a depth difference value between each pixel point in the first image and the corresponding pixel point in the fourth image, based on the depth value of each pixel point in the first image and the fourth image obtained by using the radar device, and taking the calculated depth difference value as a calibrated depth difference value; and
optimizing the initial loss function by setting the calibrated depth difference value as an optimization target of the initial loss function.
