# US 2018/0322361 A1 · Image processing device, semiconductor device, image recognition device, mobile device, and image processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322361 A1
- Document ID: US20180322361A1
- Status: Published application
- Kind: A1
- Application number: 15/771,840
- Filed: 2017-03-30
- Published: 2018-11-08
- Earliest claimed: 2016-03-30
- Assignees: Kabushikikaisha Equos Research; Kyushu Institute of Technology
- Applicants: Equos Research Co., Ltd.; Kyushu Institute of Technology
- Inventors: Hideo YAMADA; Kazuhiro KUNO; Hakaru TAMUKOH; Shuichi ENOKIDA; Shiryu OOE
- Web page: https://patentgrep.com/patent/US20180322361A1

## CPC

- G06V 10/955: {using specific electronic processors}
- G06V 10/507: {Summing image-intensity values; Histogram projection analysis}
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A hardware configuration is constructed for calculating at high speed the co-occurrence of luminance gradient directions between differing resolutions for a subject image. In an image processing device, a processing line for high-resolution images, a processing line for medium-resolution images, and a processing line for low-resolution images are arranged in parallel, and the luminance gradient directions are extracted for each pixel simultaneously in parallel from images having the three resolutions. Co-occurrence matrix preparation units prepare co-occurrence matrices by using the luminance gradient directions extracted from these images having the three resolutions, and a histogram preparation unit outputs a histogram as an MRCoHOG feature amount by using these matrices. To concurrently processing the images having the three resolutions, high-speed processing can be performed, and moving pictures output from a camera can be processed in real time.

## Claims

### Claim 1

An image processing device comprising:
luminance outputting means for sequentially outputting luminances of pixels constituting an image based on an order of pixels;
gradient direction outputting means for sequentially outputting gradient directions of luminances of the respective pixels having a plurality of resolutions in parallel in accordance with each resolution with the use of the sequentially output luminances;
co-occurrence matrix preparing means for preparing a co-occurrence matrix including co-occurrences of the gradient directions at the different resolutions by sequentially combining the sequentially output gradient directions for each of the resolutions; and
co-occurrence matrix outputting means for outputting the prepared co-occurrence matrix as an image feature amount of the image.

### Claim 2 (depends on 1)

The image processing device according to claim 1,
wherein the gradient direction outputting means comprises a plurality of resolution-specific gradient direction outputting means which are provided in parallel in accordance with the plurality of resolutions respectively, and output gradient directions of luminances of the pixels having a corresponding resolution from the sequentially output luminances; and
the resolution-specific gradient direction outputting means are concurrently operated to sequentially output the gradient directions for each of the resolutions in parallel.

### Claim 3 (depends on 2)

The image processing device according to claim 2,
wherein each of the resolution-specific gradient direction outputting means selects the luminances sequentially output from the luminance outputting means with the use of a frequency based on a corresponding resolution to sequentially output the luminances at the corresponding resolution, and
it also outputs gradient directions at the corresponding resolution by using the output luminances.

### Claim 4 (depends on 2)

The image processing device according to claim 2,
wherein each of the resolution-specific gradient direction outputting means identifies luminances of adjacent pixels which are horizontally and vertically adjacent to a pixel of interest based on an order of outputting the luminances at the corresponding resolution, and
it also outputs a gradient direction of the pixel of interest by using the identified luminances of the adjacent pixels.

### Claim 5 (depends on 4)

The image processing device according to claim 4,
wherein each of the resolution-specific gradient direction outputting means acquires horizontal luminance gradient intensity and a vertical luminance gradient intensity of the pixel of interest by using the identified luminances of the adjacent pixels, and
it outputs quantized gradient directions by making reference to a correspondence table, in which positive and negative and magnitudes of horizontal luminance gradient intensity and vertical luminance gradient intensity are associated with the quantized gradient directions, as to the acquired horizontal luminance gradient intensity and vertical luminance gradient intensity.

### Claim 6 (depends on 4)

The image processing device according to claim 4,
wherein each of the resolution-specific gradient direction outputting means identifies the luminances of the adjacent pixels by arranging the luminances in an array where positions of the adjacent pixels are associated based on the order of outputting the luminances at the corresponding resolution.

### Claim 7 (depends on 6)

The image processing device according to claim 6,
wherein the array is constituted of three arrays corresponding to a pixel row to which the pixel of interest belongs and two pixel rows which are vertically adjacent to the pixel row, and
each of the resolution-specific gradient direction outputting means arranges luminances of the three pixel rows in the corresponding three arrays, and identifies the luminances of the adjacent pixels based on positions at which the luminances are arranged.

### Claim 8 (depends on 1)

The image processing device according to claim 1,
wherein the co-occurrence matrix preparing means sequentially identifies a gradient direction of the pixel of interest and gradient directions of pixels which are combined with the pixel of interest based on an order of outputting the gradient directions at each resolution which are sequentially output from the gradient direction outputting means, and
it sequentially performs voting in the co-occurrence matrix based on combinations of the identified gradient directions to prepare the co-occurrence matrix.

### Claim 9 (depends on 8)

The image processing device according to claim 8,
wherein the co-occurrence matrix preparing means arranges the gradient directions in arrays, in which the pixel of interest is associated with positions of the pixels combined with the pixel of interest and which are provided for each of the resolutions, in accordance with each of the resolutions based on the outputting order for each of the resolutions to identify the gradient directions of the combined pixels.

### Claim 10 (depends on 9)

The image processing device according to claim 9,
wherein the arrays are constituted of six arrays corresponding to respective pairs of pixel rows which are vertically adjacent to each other for each of the resolutions, and
the co-occurrence matrix preparing means arranges the gradient directions in each pair of pixel rows for each resolution in the corresponding pair of arrays, and identifies the gradient directions of the combined pixels based on positions at which the gradient directions are arranged.

### Claim 11 (depends on 1)

A semiconductor device comprising:
the image processing device according to claim 1 mounted therein.

### Claim 12 (depends on 1)

An image recognition device comprising:
the image processing device according to claim 1; and
recognizing means for comparing a histogram based on a co-occurrence matrix output from the image processing device with a histogram based on a co-occurrence matrix of a reference image to recognize an object shown in the image.

### Claim 13 (depends on 12)

A mobile device comprising:
detecting means for detecting an object by using the image recognition device according to claim 12; and
moving means for moving to a predetermined position to the detected object.

### Claim 14

An image processing method comprising:
a luminance output step of sequentially outputting luminances of pixels constituting an image based on an order of the pixels;
a gradient direction output step of sequentially outputting gradient directions of the luminances of the respective pixels at a plurality of resolutions in parallel in accordance with each of the resolutions by using the sequentially output luminances;
a co-occurrence matrix preparation step of preparing a co-occurrence matrix including co-occurrences of the gradient directions at the different resolutions by sequentially combining the sequentially output gradient directions for the respective resolutions; and
a co-occurrence matrix output step of outputting the prepared co-occurrence matrix as an image feature amount of the image.
