# US 2021/0278836 A1 · Method and device for dual-light image integration, and unmanned aerial vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0278836 A1
- Document ID: US20210278836A1
- Status: Published application
- Kind: A1
- Application number: 17/313,365
- Filed: 2021-05-06
- Published: 2021-09-09
- Earliest claimed: 2018-11-07
- Assignees: Autel Robotics Co., Ltd.
- Applicants: Autel Robotics Co., Ltd.
- Inventors: Zhaozao LI
- Web page: https://patentgrep.com/patent/US20210278836A1

## CPC

- G06V 20/13: Satellite images
- G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)}
- B64U 2101/30: for imaging, photography or videography
- B64U 2201/20: Remote controls
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR

## Abstract

Embodiments of the present invention relate to a dual-light image integration method, a dual-light image integration device and an unmanned aerial vehicle (UAV). The method includes: receiving first image data from a first image device and second image data from a second image device; separately storing the first image data and the second image data; combining the first image data and the second image data to composite third image data; and transmitting the third image data. According to the present invention, dual-light images are processed by using different data processing methods depending on subsequent operations, which can maintain synchronization of the dual-light images during image transmission while avoiding information loss during image storage, thereby well satisfying user needs.

## Claims

### Claim 1

A dual-light image integration method, comprising:
receiving first image data from a first image device and second image data from a second image device;
separately storing the first image data and the second image data;
combining the first image data and the second image data to composite third image data; and
providing the third image data.

### Claim 2 (depends on 1)

The dual-light image integration method according to claim 1, wherein the third image data is a picture-in-picture image comprising a primary display picture and a secondary display picture superimposed on the primary display picture.

### Claim 3 (depends on 2)

The dual-light image integration method according to claim 2, wherein the step of combining the first image data and the second image data to composite the third image data comprises:
scaling the first image data to a size corresponding to a first video picture and scaling the second image data to a size corresponding to a second video picture;
with the first video picture as the primary display picture, superimposing the second video picture as the secondary display picture on the primary display picture; or
with the second video picture as the primary display picture, superimposing the first video picture as the secondary display picture on the primary display picture; and
generating the picture-in-picture image comprising the primary display picture and the secondary display picture.

### Claim 4 (depends on 1)

The dual-light image integration method according to claim 1, wherein the third image data is a fused image obtained by fusing, pixel by pixel, the first image data and the second image data.

### Claim 5 (depends on 4)

The dual-light image integration method according to claim 4, wherein the step of combining the first image data and the second image data to composite the third image data comprises:
determining a first image property of the first image data at a kth pixel position and a second image property of the second image data at the kth pixel position according to gray values of the first image data and the second image data at each pixel position;
comparing the first image property with the second image property to determine the gray value at the kth pixel position, wherein k is an integer from 1 to K, wherein K is a number of pixels of the first image data and the second image data; and
obtaining the fused image according to the gray values at all pixel positions.

### Claim 6 (depends on 5)

The dual-light image integration method according to claim 5, wherein the step of determining the first image property of the first image data at the kth pixel position and the second image property of the second image data at the kth pixel position according to the gray values of the first image data and the second image data at each pixel position comprises:
determining, through a preset detection window, whether smoothness at the kth pixel position exceeds a preset smoothness threshold, wherein
if so, the image properties at the kth pixel position are determined to be smooth, and
if not, the image properties at the kth pixel position are determined to be non-smooth.

### Claim 7 (depends on 6)

The dual-light image integration method according to claim 6, wherein the step of comparing the first image property with the second image property to determine the gray value at the kth pixel position comprises:
when the first image property and the second image property are both smooth, determining that the gray value at the kth pixel position is a mean value of the gray values of the first image data and the second image data at the kth pixel position;
when the first image property is smooth and the second image property is non-smooth, determining that the gray value at the kth pixel position is the gray value of the second image data at the kth pixel position; and
when the first image property is non-smooth and the second image property is smooth, determining that the gray value at the kth pixel position is the gray value of the first image data at the kth pixel position.

### Claim 8 (depends on 6)

The dual-light image integration method according to claim 6, wherein the step of comparing the first image property with the second image property to determine the gray value at the kth pixel position comprises:
when the first image property and the second image property are both non-smooth, comparing the gray values of the first image data and the second image data at the kth pixel position; and
selecting a larger one of the gray values as the gray value at the kth pixel position.

### Claim 9 (depends on 6)

The dual-light image integration method according to claim 6, wherein the step of determining, through the preset detection window, whether the smoothness at the kth pixel position exceeds the preset smoothness threshold comprises:
calculating differences between the gray value of the first image data at the kth pixel position and gray values at a plurality of adjacent pixel positions, wherein the adjacent pixel positions are selected by using the detection window; and
determining whether the differences from the gray values at the plurality of adjacent pixel positions are all less than a first preset smoothness threshold, wherein
if so, the first image property at the kth pixel position is determined to be smooth, and
if not, the first image property at the kth pixel position is determined to be non-smooth.

### Claim 10 (depends on 6)

The dual-light image integration method according to claim 6, wherein the step of determining, through the preset detection window, whether the smoothness at the kth pixel position exceeds the preset smoothness threshold comprises:
calculating differences between the gray value of the second image data at the kth pixel position and gray values at a plurality of adjacent pixel positions, wherein the adjacent pixel positions are selected by using the detection window; and
determining whether the differences from the gray values at the plurality of adjacent pixel positions are all less than a second preset smoothness threshold, wherein
if so, the second image property at the kth pixel position is determined to be smooth, and
if not, the second image property at the kth pixel position is determined to be non-smooth.

### Claim 11 (depends on 4)

The dual-light image integration method according to claim 4, further comprising: performing one or more visual processing operations on the fused image.

### Claim 12 (depends on 1)

The dual-light image integration method according to claim 1, wherein the first image data is infrared thermal imaging image data, and the second image data is visible light image data.

### Claim 13 (depends on 1)

The dual-light image integration method according to claim 1, further comprising: encoding the first image data, the second image data and the third image data by using a preset encoding algorithm.

### Claim 14

A dual-light image integration device, comprising:
a memory storing computer executable instructions; and
a processor configured to execute the instructions to:
receive first image data from a first image device and second image data from a second image device;
separately store the first image data and the second image data;
combine the first image data and the second image data to composite third image data; and
provide the third image data.

### Claim 15 (depends on 14)

The dual-light image integration device according to claim 14, wherein the processor is further configured to:
separately store the encoded first image data and the encoded second image data.

### Claim 16 (depends on 14)

The dual-light image integration device according to claim 14, wherein
when the first image data and the second image data are pictures, the first image data, the second image data and the third image data are encoded through JPEG encoding, DNG encoding or TIFF encoding; and
when the first image data and the second image data are videos, the first image data, the second image data and the third image data are encoded through H265 encoding or H264 encoding.

### Claim 17

An unmanned aerial vehicle (UAV), comprising:
a UAV body carrying a first image device and a second image device arranged side by side;
a dual-light image integration device, wherein the dual-light image integration device is connected to the first image device and the second image device and is configured to receive the first image data and the second image data; and
an image transmission apparatus connected to the dual-light image integration device and configured to transmit the third image data composited by the dual-light image integration device to a ground terminal;
wherein the dual-light image integration device comprising:
a memory storing computer executable instructions; and
a processor configured to execute the instructions to:
receive first image data from the first image device and second image data from the second image device;
separately store the first image data and the second image data;
combine the first image data and the second image data to composite third image data; and
provide the third image data.

### Claim 18 (depends on 17)

The UAV according to claim 17, wherein the first image device is an infrared camera, and the second image device is a high-definition camera.
