# US 2015/0103145 A1 · Image processing apparatus, imaging apparatus, and image processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0103145 A1
- Document ID: US20150103145A1
- Status: Published application
- Kind: A1
- Application number: 14/573,040
- Filed: 2014-12-17
- Published: 2015-04-16
- Earliest claimed: 2012-06-20
- Assignees: Olympus Corporation
- Applicants: Olympus Corporation
- Inventors: Seiichiro SAKATA
- Web page: https://patentgrep.com/patent/US20150103145A1

## CPC

- G06T 7/55: from multiple images
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/579: from motion
- G06T 7/73: using feature-based methods
- H04N 13/128: Adjusting depth or disparity
- H04N 23/6811: {based on the image signal}
- H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory}
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G06T 2207/10012: Stereo images
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30244: Camera pose
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing apparatus includes an image acquisition unit that acquires a plurality of images, a corresponding point acquisition unit, a first fundamental matrix calculation unit, a depth calculation unit, a corresponding point extraction unit, and a fundamental matrix determination unit. The corresponding point acquisition unit acquires a plurality of first corresponding points of the images. The first fundamental matrix calculation unit calculates a first fundamental matrix based on the first corresponding points. The depth calculation unit calculates depths for the first corresponding points based on the first fundamental matrix. The corresponding point extraction unit extracts a plurality of second corresponding points from the first corresponding points based on the depths. The fundamental matrix determination unit calculates a second fundamental matrix based on the second corresponding points.

## Claims

### Claim 1

An image processing apparatus comprising:
an image acquisition unit that acquires a plurality of images;
a corresponding point acquisition unit that acquires a plurality of first corresponding points corresponding to each other between the images;
a first fundamental matrix calculation unit that calculates a first fundamental matrix based on the first corresponding points;
a depth calculation unit that calculates depths for the first corresponding points based on the first fundamental matrix;
a corresponding point extraction unit that extracts a plurality of second corresponding points from the first corresponding points based on the depths; and
a fundamental matrix determination unit that calculates a second fundamental matrix based on the second corresponding points, wherein
the corresponding point extraction unit extracts the second corresponding points from the first corresponding points based on the depths such that the second corresponding points includes a nearest one of the first corresponding points and a farthest one of the first corresponding points.

### Claim 2 (depends on 1)

The image processing apparatus according to claim 1, wherein
the corresponding point extraction unit successively extracts the second corresponding points from the first corresponding points based on the depths in order from the nearest corresponding point, and in order from the farthest corresponding point.

### Claim 3 (depends on 1)

The image processing apparatus according to claim 1, wherein
the corresponding point extraction unit evaluates a reliability of the first corresponding points based on the first fundamental matrix, and extracts the second corresponding points based on the reliability.

### Claim 4 (depends on 1)

The image processing apparatus according to claim 1, wherein
the images are images of a same subject captured at different time, and
the image processing apparatus further comprises:
a correction unit that corrects a shake present between the images based on the second fundamental matrix.

### Claim 5 (depends on 1)

The image processing apparatus according to claim 1, further comprising:
a three-dimensional reconstruction unit that performs three-dimensional reconstruction of the images based on the second fundamental matrix.

### Claim 6 (depends on 1)

The image processing apparatus according to claim 1, wherein
the first fundamental matrix calculation unit calculates the first fundamental matrix by an iterative operation performed by randomly selecting a predetermined number of corresponding points from the first corresponding points.

### Claim 7

An imaging apparatus comprising:
an imaging unit that captures an image of a subject;
an image acquisition unit that acquires a plurality of the images of the same subject captured at different time;
a corresponding point acquisition unit that acquires a plurality of first corresponding points corresponding to each other between the images;
a first fundamental matrix calculation unit that calculates a first fundamental matrix based on the first corresponding points;
a depth calculation unit that calculates depths for the first corresponding points based on the first fundamental matrix;
a corresponding point extraction unit that extracts a plurality of second corresponding points from the first corresponding points based on the depths;
a fundamental matrix determination unit that calculates a second fundamental matrix based on the second corresponding points; and
a correction unit that corrects a shake present between the images based on the second fundamental matrix, wherein
the corresponding point extraction unit extracts the second corresponding points from the first corresponding points based on the depths such that the second corresponding points includes a nearest one of the first corresponding points and a farthest one of the first corresponding points.

### Claim 8 (depends on 7)

The imaging apparatus according to claim 7, wherein
the corresponding point extraction unit successively extracts the second corresponding points from the first corresponding points based on the depths in order from the nearest corresponding point, and in order from the farthest corresponding point.

### Claim 9 (depends on 7)

The imaging apparatus according to claim 7, wherein
the corresponding point extraction unit evaluates a reliability of the first corresponding points based on the first fundamental matrix, and extracts the second corresponding points based on the reliability.

### Claim 10 (depends on 7)

The imaging apparatus according to claim 7, further comprising:
a three-dimensional reconstruction unit that performs three-dimensional reconstruction of the images based on the second fundamental matrix.

### Claim 11 (depends on 7)

The imaging apparatus according to claim 7, wherein
the first fundamental matrix calculation unit calculates the first fundamental matrix by an iterative operation performed by randomly selecting a predetermined number of corresponding points from the first corresponding points.

### Claim 12

An image processing method comprising:
acquiring a plurality of images;
acquiring a plurality of first corresponding points corresponding to each other between the images;
calculating a first fundamental matrix based on the first corresponding points;
calculating depths for the first corresponding points based on the first fundamental matrix;
extracting a plurality of second corresponding points from the first corresponding points based on the depths; and
calculating a second fundamental matrix based on the second corresponding points, wherein
the extracted second corresponding points are extracted from the first corresponding points based on the depths such that the second corresponding points includes a nearest one of the first corresponding points and a farthest one of the first corresponding points.

### Claim 13 (depends on 12)

The image processing method according to claim 12, wherein
the extracted second corresponding points are successively extracted from the first corresponding points based on the depths in order from the nearest corresponding point and further, in order from the farthest corresponding point.

### Claim 14 (depends on 12)

The image processing method according to claim 12, wherein
extracting the second corresponding points comprises
evaluating a reliability of the first corresponding points based on the first fundamental matrix and
extracting the second corresponding points based on the reliability.

### Claim 15 (depends on 12)

The image processing method according to claim 12, wherein
the images are images of a same subject captured at different time, and
the method further comprises:
correcting a shake present between the images based on the second fundamental matrix.

### Claim 16 (depends on 12)

The image processing method according to claim 12, further comprising:
performing three-dimensional reconstruction of the images based on the second fundamental matrix.

### Claim 17 (depends on 12)

The image processing method according to claim 12, wherein
calculating the first fundamental matrix comprises calculating the first fundamental matrix by an iterative operation performed by randomly selecting a predetermined number of corresponding points from the first corresponding points.
