# US 11,122,209 B2 · Three-dimensional shape estimation method, three-dimensional shape estimation system, flying object, program and recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,122,209 B2
- Document ID: US11122209B2
- Status: Granted
- Kind: B2
- Application number: 16/387,103
- Filed: 2019-04-17
- Published: 2019-08-15
- Granted: 2021-09-14
- Earliest claimed: 2016-10-17
- Assignees: Sz Dji Technology Co., Ltd.
- Applicants: Sz Dji Technology Co., Ltd.
- Inventors: Lei Gu; Yukiyoshi Sasao; Zongyao Qu; Tao Wu
- Web page: https://patentgrep.com/patent/US11122209B2

## CPC

- G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence)
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- G01C 11/00: Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C 11/06: by comparison of two or more pictures of the same area
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G05D 1/106: {Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones}
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/55: from multiple images
- G06T 7/593: from stereo images
- H04N 7/185: {from a mobile camera, e.g. for remote control}
- B64U 2101/30: for imaging, photography or videography
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2207/30168: Image quality inspection
- G06T 2207/30181: Earth observation
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR

## Abstract

A three-dimensional shape estimation method includes generating a set of three-dimensional shape data for a flight range of a movable object based on a plurality of first images captured by a movable object, detecting a loss area based on the set of three-dimensional shape data for the flight range, generating a flight route for imaging the loss area, acquiring a plurality of second images that are captured in a horizontal direction toward the loss area with imaging ranges partially overlapped while the movable object moves along the flight route, and generating three-dimensional shape data for the loss area based on the plurality of second images.

## Claims

### Claim 1

A three-dimensional shape estimation method comprising:
generating, based on a plurality of first images captured by a movable object, a first set of three-dimensional shape data for a flight range of the movable object, the first images being captured while the movable object moves along a second flight route;
detecting, based on the first set of three-dimensional shape data for the flight range, a loss area;
generating a first flight route for imaging the loss area at different altitudes;
acquiring a plurality of second images, the plurality of second images being captured in a horizontal direction toward the loss area with imaging ranges partially overlapped while the movable object moves along the first flight route;
generating, based on the plurality of second images, a second set of three-dimensional shape data for the loss area;
generating, based on position information of the loss area, a third flight route by inserting the first flight route into the second flight route;
acquiring the plurality of first images in the vertical direction and the plurality of second images in the horizontal direction of the loss area in the third flight route;
generating, based on the first images and the second images, a third set of three-dimensional shape data for the flight range;
controlling the movable object to fly through the third flight route after flying through the second flight route;
imaging at a first imaging position interval on the second flight route; and
imaging at a second imaging position interval on the third flight route, the second imaging position interval being shorter than the first imaging position interval.

### Claim 2 (depends on 1)

The three-dimensional shape estimation method of claim 1, wherein the plurality of first images are captured with imaging ranges overlapped partially while the movable object moves along the second flight route.

### Claim 3 (depends on 2)

The three-dimensional shape estimation method of claim 2, wherein the second flight route is along a horizontal direction.

### Claim 4 (depends on 1)

The three-dimensional shape estimation method of claim 1, further comprising:
synthesizing the first set of three-dimensional shape data for the flight range and the second set of three-dimensional shape data for the loss area to generate a fourth set of three-dimensional shape data.

### Claim 5

A three-dimensional shape estimation method comprising:
generating, during flight of a movable object and based on a plurality of first images, a first set of three-dimensional shape data for a flight range of the movable object, the first images being captured while the movable object moves along a second flight route;
determining, during the flight, whether a loss area exists based on the first set of three-dimensional shape data for the flight range;
controlling the movable object to move toward the loss area in response to a determination that the loss area exists;
generating, during the flight, a first flight route for imaging the loss area at different altitudes;
controlling the movable object to capture a plurality of second images in a horizontal direction toward the loss area with imaging ranges partially overlapped, while controlling the movable object to move along the first flight route;
generating, based on the plurality of second images, a second set of three-dimensional shape data for the loss area;
returning the movable object to the flight following the second flight route after acquisition of the second images; and
acquiring the first images at imaging positions in the second flight route where no image has been previously captured.

### Claim 6 (depends on 5)

The three-dimensional shape estimation method of claim 5, wherein the plurality of first images are captured with imaging ranges overlapped partially, while the movable object is being controlled to move along the second flight route.

### Claim 7 (depends on 6)

The three-dimensional shape estimation method of claim 6, wherein the second flight route is along a horizontal direction.

### Claim 8 (depends on 5)

The three-dimensional shape estimation method of claim 5, further comprising:
synthesizing, during the flight, the first set of three-dimensional shape data for the flight range and the second set of three-dimensional shape data for the loss area to generate a third set of three-dimensional shape data for the flight range.

### Claim 9

A movable object comprising:
a main body; and
a processor coupled to the main body and configured to:
generate, based on a plurality of first images captured by the movable object, a first set of three-dimensional shape data for a flight range of the movable object, the first images being captured while the movable object moves along a second flight route;
detect, based on the first set of three-dimensional shape data for the flight range, a loss area;
generate a first flight route for imaging the loss area at different altitudes;
acquiring a plurality of second images, in a horizontal direction toward the loss area with imaging ranges partially overlapped, while the movable object moves along the first flight route;
generate, based on the plurality of second images, a second set of three-dimensional shape data for the loss area;
return the movable object to the flight following the second flight route after acquisition of the second images; and
acquire the first images at imaging positions in the second flight route where no image has been previously captured.

### Claim 10

A non-transitory computer-readable medium recording a program for causing a computer to:
generate, based on a plurality of first images captured by a movable object, a first set of three-dimensional shape data for a flight range of the movable object, the first images being captured while the movable object moves along a second flight route;
detect, based on the set of three-dimensional shape data for the flight range, a loss area;
generate a first flight route for imaging the loss area at different altitudes;
acquire a plurality of second images, the plurality of second images being captured in a horizontal direction toward the loss area with imaging ranges partially overlapped while the movable object moves along the first flight route;
generate, based on the plurality of second images, a second set of three-dimensional shape data for the loss area;
return the movable object to the flight following the second flight route after acquisition of the second images; and
acquire the first images at imaging positions in the second flight route where no image has been previously captured.