# US 2021/0334580 A1 · Image processing device, image processing method and image processing system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334580 A1
- Document ID: US20210334580A1
- Status: Published application
- Kind: A1
- Application number: 17/203,785
- Filed: 2021-03-17
- Published: 2021-10-28
- Earliest claimed: 2020-04-23
- Assignees: Hitachi, Ltd.
- Applicants: Hitachi, Ltd.
- Inventors: Viviana CRESCITELLI; Takashi OSHIMA
- Web page: https://patentgrep.com/patent/US20210334580A1

## CPC

- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/253: {of extracted features}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/806: {of extracted features}
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image processing device including a storage unit configured to store an object detection neural network trained using a first K-channel image generated from a first M-channel image and a first N-channel image generated from the first M-channel image, a reception unit configured to receive, from a sensor, a second M-channel image and a second N-channel image that include an identical subject, and an image analysis unit configured to generate, using the object detection neural network trained using the first K-channel image, object detection result information with respect to a second K-channel image generated from the second M-channel image and the second N-channel image, and output the object detection result information.

## Claims

### Claim 1

An image processing device comprising:
a storage unit configured to store an object detection neural network trained using a first K-channel image generated from a first M-channel image and a first N-channel image generated from the first M-channel image;
a reception unit configured to receive, from a sensor, a second M-channel image and a second N-channel image that include an identical subject; and
an image analysis unit configured to generate, using the object detection neural network trained using the first K-channel image, object detection result information with respect to a second K-channel image generated from the second M-channel image and the second N-channel image, and output the object detection result information.

### Claim 2 (depends on 1)

The image processing device according to claim 1, further comprising:
an image generation unit for converting the first M-channel image into the first N-channel image.

### Claim 3 (depends on 2)

The image processing device according to claim 2, wherein:
the image generation unit is a generative adversarial network including:
a generator network configured to generate a candidate image for the first N-channel image based on the first M-channel image, and
a discriminator network configured to:
compare the candidate image with ground truth for the first N-channel image in order to determine a divergence of the candidate image with respect to the ground truth, and
output, as the first N-channel image, a candidate image that satisfies a predetermined divergence criterion.

### Claim 4 (depends on 1)

The image processing device according to claim 1, further comprising:
an image fusion unit configured to fuse an M-channel image and an N-channel image that include an identical subject to generate a K-channel image.

### Claim 5 (depends on 4)

The image processing device according to claim 4, wherein:
the image fusion unit generates the K-channel image by fusing:
a first feature set extracted from the M-channel image; and
a second feature set extracted from the N-channel image.

### Claim 6 (depends on 4)

The image processing device according to claim 4, wherein:
the M-channel image and the N-channel image are selected from the group consisting of RGB images, RGB-D images, infrared images, X-ray images, and thermal images.

### Claim 7

An image processing method comprising:
a step of acquiring a first M-channel image;
a step of generating a first N-channel image using the first M-channel image;
a step of fusing the first M-channel image and the first N-channel image to generate a first K-channel image;
a step of training an object detection neural network using the first K-channel image;
a step of acquiring a second M-channel image and a second N-channel image that include an identical subject;
a step of fusing the second M-channel image and the second N-channel image to generate a second K-channel image; and
a step of generating, using the object detection neural network trained using the first K-channel image, object detection result information with respect to the second K-channel image, and outputting the object detection result information.

### Claim 8 (depends on 7)

The image processing method according to claim 7, wherein the step of generating the first N-channel image using the first M-channel image includes:
a step of generating a candidate image for the first N-channel image based on the first M-channel image;
a step of comparing the candidate image with ground truth for the first N-channel image in order to determine a divergence of the candidate image with respect to the ground truth; and
a step of outputting, as the first N-channel image, a candidate image that satisfies a predetermined divergence criterion.

### Claim 9 (depends on 8)

The image processing method according to claim 8, wherein the step of fusing the first M-channel image and the first N-channel image to generate the first K-channel image includes:
a step of extracting a first feature set from the second M-channel image;
a step of extracting a second feature set from the second N-channel image; and
a step of fusing the first feature set and the second feature set to generate the second K-channel image.

### Claim 10

An image processing system comprising:
a client terminal;
a sensor; and
an image processing device connected via a communication network;
wherein the image processing device includes:
a storage unit configured to store an object detection neural network trained using a first K-channel image generated from a first M-channel image and a first N-channel image generated from the first M-channel image;
a reception unit configured to receive, from the sensor, a second M-channel image and a second N-channel image that include an identical subject; and
an image analysis unit configured to generate, using the object detection neural network trained using the first K-channel image, object detection result information with respect to a second K-channel image generated from the second M-channel image and the second N-channel image, and output the object detection result information to the client terminal.

### Claim 11 (depends on 10)

The image processing system according to claim 10, wherein the image processing device further comprises:
an image generation unit for converting the first M-channel image into the first N-channel image.

### Claim 12 (depends on 11)

The image processing system according to claim 11, wherein:
the image generation unit is a generative adversarial network including:
a generator network configured to generate a candidate image for the first N-channel image based on the first M-channel image, and
a discriminator network configured to:
compare the candidate image with ground truth for the first N-channel image in order to determine a divergence of the candidate image with respect to the ground truth, and
output, as the first N-channel image, a candidate image that satisfies a predetermined divergence criterion.
