# US 12,118,698 B1 · Method for inpainting highlight region of vegetation image captured by unmanned aerial vehicle, device, medium, and product

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,698 B1
- Document ID: US12118698B1
- Status: Granted
- Kind: B1
- Application number: 18/631,002
- Filed: 2024-04-09
- Granted: 2024-10-15
- Earliest claimed: 2024-03-05
- Assignees: Chengdu University of Technology
- Applicants: Chengdu University of Technology
- Inventors: Jing He; Yifu Chen; Gang Liu; Weile Li
- Web page: https://patentgrep.com/patent/US12118698B1

## CPC

- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/82: using neural networks
- G06V 20/17: taken from planes or by drones
- G06V 20/188: {Vegetation}
- G06T 2207/20132: Image cropping
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Provided are a method for inpainting a highlight region of a vegetation image captured by an unmanned aerial vehicle, a device, a medium, and a product. The method includes: acquiring an image to be inpainted, and a historic image; inputting the image to be inpainted to a trained target detection network to obtain a waterbody highlight region image block; determining a template image block of the image to be inpainted based on the waterbody highlight region image block; cropping the historical image to obtain a plurality of candidate image blocks of the image to be inpainted; determining similarity between each candidate image block and the template image block by using a deep learning image coarse matching method; and screening candidate image blocks with the similarity greater than a predetermined threshold, and determining an optimal candidate image block in the candidate image blocks by using a pixel-by-pixel matching method.

## Claims

### Claim 1

A method for inpainting a highlight region of a vegetation image captured by an unmanned aerial vehicle, comprising:
acquiring an image to be inpainted and a historic image, wherein the image to be inpainted is an image with a highlight region, the historic image is an image obtained by capturing a target region at a previous moment, and the target region is a region shown in the image to be inpainted;
inputting the image to be inpainted to a trained target detection network to obtain a waterbody highlight region image block in the image to be inpainted;
determining a template image block of the image to be inpainted based on the waterbody highlight region image block, wherein the template image block is an adjacent image block of the waterbody highlight region image block, and the template image block is used for image block inpainting matching;
cropping the historic image based on a coarse tuning-fine tuning method to obtain a plurality of candidate image blocks of the image to be inpainted;
determining similarity between each candidate image block and the template image block by using a deep learning image coarse matching method based on the template image block;
determining candidate image blocks with the similarity greater than a predetermined threshold as target candidate image blocks, and determining an optimal candidate image block in the target candidate image blocks by a pixel-by-pixel matching method; and
inpainting the waterbody highlight region image block according to the optimal candidate image block.

### Claim 2 (depends on 1)

The method according to claim 1, wherein prior to the inputting the image to be inpainted to a trained target detection network to obtain a waterbody highlight region image block, the method further comprises:
constructing a target detection network, wherein the target detection network is a detection network determined based on an improved YoloV5 target detection algorithm, wherein the improved YoloV5 target detection algorithm is an algorithm obtained by replacing an efficient channel attention (ECA) module in Backbone with a C3 module on the basis of an original YoloV5 target detection algorithm.

### Claim 3 (depends on 2)

The method according to claim 2, wherein a training process of the target detection network is as follows:
acquiring a sample image to be inpainted, wherein the sample image to be inpainted is an image with a waterbody highlight region image block; and
based on the sample image to be inpainted, training the improved YoloV5 target detection algorithm by using an Adam model optimizer, thus obtaining the trained target detection network.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the determining a template image block of the image to be inpainted based on the waterbody highlight region image block comprises:
acquiring coordinates of a center point of the waterbody highlight region image block and a length of the waterbody highlight region image block;
determining an abscissa of a center point of the template image block according to Equation centerx2=centerx1+l+cache; and
determining an ordinate of the center point of the template image block according to Equation centery2=centery1;
wherein centerx1 is an abscissa of the center point of the waterbody highlight region image block; centery1 is an ordinate of the center point of the waterbody highlight region image block; centerx2 is the abscissa of the center point of the template image block; centery2 is the ordinate of the center point of the template image block; l is the length of the waterbody highlight region image block; and cache is a predetermined image matching buffer distance.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the determining similarity between each candidate image block and the template image block by using a deep learning image coarse matching method based on the template image block comprises:
extracting feature vectors from the template image block and each candidate image block by using a ViT deep learning model; and
determining the similarity between each candidate image block and the template image block by using a cosine similarity calculation method according to the feature vectors in the template image block and each candidate image block.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the determining an optimal candidate image block in the target candidate image blocks by a pixel-by-pixel matching method comprises:
determining the optimal candidate image block in the target candidate image blocks according to Equation
R
⁡
(
x
,
y
)
=
∑
i
=
1
m
⁢
∑
j
=
1
n
⁢
(
I
⁡
(
i
+
x
,
j
+
y
)
-
T
⁡
(
i
,
j
)
)
2
∑
i
=
1
m
⁢
∑
j
=
1
n
⁢
I
⁡
(
i
+
x
,
j
+
y
)
2
*
∑
i
=
1
m
⁢
∑
j
=
1
n
⁢
T
⁡
(
i
,
j
)
2
;
wherein in the Equation, (x,y) denotes coordinates of a search target in a source image; a pixel value on coordinates (i,j) of a template image is denoted by T(i,j); and R(x,y) is a coefficient characterized in that as R(x,y) decreases, the similarity between the target candidate image block and a template image block increases.

### Claim 7 (depends on 1)

The method according to claim 1, wherein after determining candidate image blocks with the similarity greater than a predetermined threshold as target candidate image blocks, and determining an optimal candidate image block in the target candidate image blocks by a pixel-by-pixel matching method, the method comprises:
performing brightness equalization on the optimal candidate image block and the waterbody highlight region image block.

### Claim 8 (depends on 1)

A computer device, comprising: a memory, a processor, and a computer program stored on the memory and capable of being operated on the processor, wherein the processor, when executing the computer program, is configured to achieve steps of the method for inpainting a highlight region of a vegetation image captured by an unmanned aerial vehicle according to claim 1.

### Claim 9 (depends on 8)

The computer device according to claim 8, wherein the processor, when executing the computer program, is configured to:
construct a target detection network, wherein the target detection network is a detection network determined based on an improved YoloV5 target detection algorithm, wherein the improved YoloV5 target detection algorithm is an algorithm obtained by replacing an efficient channel attention (ECA) module in Backbone with a C3 module on the basis of an original YoloV5 target detection algorithm.

### Claim 10 (depends on 9)

The computer device according to claim 9, wherein the processor, when executing the computer program, is configured to:
acquire a sample image to be inpainted, wherein the sample image to be inpainted is an image with a waterbody highlight region image block; and
based on the sample image to be inpainted, train the improved YoloV5 target detection algorithm by using an Adam model optimizer, thus obtaining the trained target detection network.

### Claim 11 (depends on 8)

The computer device according to claim 8, wherein the processor, when executing the computer program, is configured to:
acquire coordinates of a center point of the waterbody highlight region image block and a length of the waterbody highlight region image block;
determine an abscissa of a center point of the template image block according to Equation centerx2=centerx1+l+cache; and
determine an ordinate of the center point of the template image block according to Equation centery2=centery1;
wherein centerx1 is an abscissa of the center point of the waterbody highlight region image block; centery1 is an ordinate of the center point of the waterbody highlight region image block; centerx2 is the abscissa of the center point of the template image block; centery2 is the ordinate of the center point of the template image block; l is the length of the waterbody highlight region image block; and cache is a predetermined image matching buffer distance.

### Claim 12 (depends on 8)

The computer device according to claim 8, wherein the processor, when executing the computer program, is configured to:
extract feature vectors from the template image block and each candidate image block by using a ViT deep learning model; and
determine the similarity between each candidate image block and the template image block by using a cosine similarity calculation method according to the feature vectors in the template image block and each candidate image block.

### Claim 13 (depends on 8)

The computer device according to claim 8, wherein the processor, when executing the computer program, is configured to:
determine the optimal candidate image block in the target candidate image blocks according to Equation
R
⁡
(
x
,
y
)
=
∑
i
=
1
m
⁢
∑
j
=
1
n
⁢
(
I
⁡
(
i
+
x
,
j
+
y
)
-
T
⁡
(
i
,
j
)
)
2
∑
i
=
1
m
⁢
∑
j
=
1
n
⁢
I
⁡
(
i
+
x
,
j
+
y
)
2
*
∑
i
=
1
m
⁢
∑
j
=
1
n
⁢
T
⁡
(
i
,
j
)
2
;
wherein in the Equation, (x,y) denotes coordinates of a search target in a source image; a pixel value on coordinates (i,j) of a template image is denoted by T(i,j); and R(x,y) is a coefficient characterized in that as R(x,y) decreases, the similarity between the target candidate image block and a template image block increases.

### Claim 14 (depends on 8)

The computer device according to claim 8, wherein the processor, when executing the computer program, is configured to:
perform brightness equalization on the optimal candidate image block and the waterbody highlight region image block.