# US 2001/0043720 A1 · Image processing apparatus, image processing method and transmission medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2001/0043720 A1
- Document ID: US20010043720A1
- Status: Published application
- Kind: A1
- Application number: 09/123,089
- Filed: 1998-07-27
- Published: 2001-11-22
- Earliest claimed: 1997-08-01
- Assignees: Sony Corporation
- Inventors: KAZUYOSHI HAYASHI
- Web page: https://patentgrep.com/patent/US20010043720A1

## CPC

- G06T 7/593: from stereo images
- G06T 7/596: {from three or more stereo images}
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06T 2207/10012: Stereo images
- H04N 13/189: Recording image signals; Reproducing recorded image signals
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The invention intends to achieve higher accuracy and higher speeds in image processing based on a stereo method. Images picked up by a plurality cameras are applied to respective image input portions and stored in built-in memories. The stored images are then read out in such sequence as to cancel lens aberrations of the cameras, and output to respective SAD circuits after being compensated for differences of intensity values due to variations in characteristics of the cameras. The SAD circuit calculates an absolute value of difference for each pixel value between a reference image output from a reference camera and two compared images output from other two cameras, and outputs the calculated value to an SSAD circuit. The SSAD circuit performs a block matching process upon outputs from the corresponding SAD circuit, and outputs obtained results to a second-order portion. The second-order portion performs second-order upon data output from the corresponding SSAD circuit and outputs an obtained result. A minimum value selecting portion and a result memory cooperate to select a minimum value from among data output from the second-order portions.

## Claims

### Claim 1

An image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, said apparatus comprising:
first input means for receiving, as a reference image, an image supplied from one of said plurality of image pickup devices,
second input means for receiving, as a compared image, an image supplied from another image pickup device,
matching means for performing a matching process between the reference image and the compared image applied from said first and second input means,
interpolating means for performing interpolation for data obtained by said matching means, and
detecting means for detecting a minimum value from among data resulted from the interpolation performed by said interpolating means.

### Claim 2 (depends on 1)

An image processing apparatus according to claim 1, wherein said interpolating means performs interpolation based on approximation using a second-order curve.

### Claim 3 (depends on 1)

An image processing apparatus according to claim 1, wherein said second input means receives the compared images from two or more image pickup devices,
said matching means performs the matching process between the reference image and each of the compared images applied from said second input means,
said interpolating means performs interpolation for data obtained for each of the compared images, and
said detecting means detects a minimum value from among data resulted from the interpolation for all the compared images.

### Claim 4 (depends on 1)

An image processing apparatus according to claim 1, wherein said second input means receives the compared images from two or more image pickup devices,
said matching means combines predetermined two of the compared images received from said second input means to create a pair of compared images, and performs the matching process between the reference image and each of the compared images of each pair,
said interpolating means performs interpolation for data obtained for each pair of the compared images, and
said detecting means detects a minimum value from among data resulted from the interpolation for all the pairs of compared images.

### Claim 5

An image processing method for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, said method comprising:
a first input step of receiving, as a reference image, an image supplied from one of said plurality of image pickup devices,
a second input step of receiving, as a compared image, an image supplied from another image pickup device,
a matching step of performing a matching process between the reference image and the compared image applied in said first and second input steps,
an interpolating step of performing interpolation for data obtained by said matching step, and
a detecting step of detecting a minimum value from among data resulted from the interpolation performed by said interpolating step.

### Claim 6 (depends on 5)

An image processing method according to claim 5, wherein said interpolating step performs interpolation based on approximation using a second-order curve.

### Claim 7 (depends on 5)

An image processing method according to claim 5, wherein said second input step receives the compared images from two or more image pickup devices,
said matching step performs the matching process between the reference image and each of the compared images applied from said second input step,
said interpolating step performs interpolation for data obtained for each of the compared images, and
said detecting step detects a minimum value from among data resulted from the interpolation for all the compared images.

### Claim 8 (depends on 5)

An image processing apparatus according to claim 5, wherein said second input step receives the compared images from two or more image pickup devices,
said matching step combines predetermined two of the compared images received from said second input step to create a pair of compared images, and performs the matching process between the reference image and each of the compared images of each pair,
said interpolating step performs interpolation for data obtained for each pair of the compared images, and
said detecting step detects a minimum value from among data resulted from the interpolation for all the pairs of compared images.

### Claim 9

A transmission medium for transmitting a computer program used in an image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, said medium transmitting a program comprising:
a first input step of receiving, as a reference image, an image supplied from one of said plurality of image pickup devices,
a second input step of receiving, as a compared image, an image supplied from another image pickup device,
a matching step of performing a matching process between the reference image and the compared image applied in said first and second input steps,
an interpolating step of performing interpolation for data obtained by said matching step, and
a detecting step of detecting a minimum value from among data resulted from the interpolation performed by said interpolating step.

### Claim 10

An image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, said apparatus comprising:
first storage means for storing an image output from each of said plurality of image pickup devices,
second storage means for storing a sequence of reading out pixel data constituting the image stored in said first storage means,
reading means for reading out the pixel data stored in said first storage means in accordance with the sequence stored in said second storage means, and
converting means for converting gradation of the pixel data read out by said reading means.

### Claim 11 (depends on 10)

An image processing apparatus according to claim 10, wherein sequence data stored in said second storage means are created depending on lens aberration of each of said image pickup devices.

### Claim 12 (depends on 10)

An image processing apparatus according to claim 10, wherein said converting means is set so that gradations of said plurality of image pickup devices are all equal to one another.

### Claim 13

An image processing method for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, said method comprising:
a storing step of storing an image output from each of said plurality of image pickup devices,
a reading step of reading out pixel data, which constitute the image stored in said storing step, in accordance with a reading sequence stored beforehand, and
a converting step of converting gradation of the pixel data read out in said reading step.

### Claim 14 (depends on 13)

An image processing method according to claim 13, wherein the reading sequence stored beforehand are created depending on lens aberration of each of said image pickup devices.

### Claim 15 (depends on 13)

An image processing method according to claim 13, wherein said converting step is set so that gradations of said plurality of image pickup devices are all equal to one another.

### Claim 16

A transmission medium for transmitting a computer program used in an image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, said medium transmitting a program comprising:
a storing step of storing an image output from each of said plurality of image pickup devices,
a reading step of reading out pixel data, which constitute the image stored in said storing step, in accordance with a reading sequence stored beforehand, and
a converting step of converting gradation of the pixel data read out in said reading step.

### Claim 17

An image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, said apparatus comprising:
first input means for receiving, as a reference image, an image supplied from one of said plurality of image pickup devices,
second input means for receiving, as a compared image, an image supplied from another image pickup device,
searching means for searching an area, which corresponds to a predetermined area of the reference image applied from said first input means, from the compared image applied from said second input means by a matching process,
detecting means for detecting if the area searched by said searching means is outside a region of the compared image, and
control means for controlling the matching process performed by said searching means in accordance with a result detected by said detecting means.

### Claim 18 (depends on 17)

An image processing apparatus according to claim 17, further comprising storage means for storing data related to the correlation between the reference image and the compared image,
wherein said detecting means detects if the search area is outside the region of the compared image, by referring to an effective bit added to the data stored in said storage means.

### Claim 19 (depends on 17)

An image processing apparatus according to claim 17, wherein said searching means performs the matching process between the reference image and each of first and second compared images, and
said control means controls said searching means such that when said detecting means detects that the search area is within both regions of said first and second compared images, said control means controls said searching means to execute the matching process for both the compared images,
when said detecting means detects that the search area is outside one of the regions of said first and second compared images, said control means controls said searching means to execute the matching process for only the compared image for which the search area is positioned within the image region, and
when said detecting means detects that the search area is outside both the regions of said first and second compared images, said control means controls said searching means to output a predetermined value as a result of the matching process.

### Claim 20 (depends on 17)

An image processing apparatus according to claim 17, wherein said matching process calculates absolute values of differences in respective pixel values between the reference image and the compared image, and determines a minimum value of the calculated absolute values.

### Claim 21

An image processing method for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, said method comprising:
a first input step of receiving, as a reference image, an image supplied from one of said plurality of image pickup devices,
a second input step of receiving, as a compared image, an image supplied from another image pickup device,
a searching step of searching an area, which corresponds to a predetermined area of the reference image applied in said first input step, from the compared image applied in said second input step by a matching process,
a detecting step of detecting if the area searched in said searching step is outside a region of the compared image, and
a control step of controlling the matching process performed in said searching step in accordance with a result detected in said detecting step.

### Claim 22 (depends on 21)

An image processing method according to claim 21, wherein said detecting step detects if the search area is outside the region of the compared image, by referring to an effective bit added to the data that are stored beforehand and related to the correlation between the reference image and the compared image.

### Claim 23 (depends on 21)

An image processing method according to claim 21, wherein said searching step performs the matching process between the reference image and each of first and second compared images, and
said control step controls said searching step such that when said detecting step detects that the search area is within both regions of said first and second compared images, said control step controls said searching step to execute the matching process for both the compared images,
when said detecting step detects that the search area is outside one of the regions of said first and second compared images, said control step controls said searching step to execute the matching process for only the compared image for which the search area is positioned within the image region, and
when said detecting step detects that the search area is outside both the regions of said first and second compared images, said control step controls said searching step to output a predetermined value as a result of the matching process.

### Claim 24 (depends on 21)

An image processing method according to claim 21, wherein said matching process calculates absolute values of differences in respective pixel values between the reference image and the compared image, and determines a minimum value of the calculated absolute values.

### Claim 25

A transmission medium for transmitting a computer program used in an image processing apparatus for calculating a distance to a target point with image processing based on images picked up by a plurality of image pickup devices, said medium transmitting a program comprising:
a first input step of receiving, as a reference image, an image supplied from one of said plurality of image pickup devices,
a second input step of receiving, as a compared image, an image supplied from another image pickup device,
a searching step of searching an area, which corresponds to a predetermined area of the reference image applied in said first input step, in the compared image applied in said second input step by a matching process,
a detecting step of detecting if the area searched in said searching step is outside a region of the compared image, and
a control step of controlling the matching process performed in said searching step in accordance with a result detected in said detecting step.
