# US 2021/0174527 A1 · Signal processing apparatus, signal processing method, and imaging apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174527 A1
- Document ID: US20210174527A1
- Status: Published application
- Kind: A1
- Application number: 17/046,386
- Filed: 2019-01-30
- Published: 2021-06-10
- Earliest claimed: 2018-04-17
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Tomohiro NISHI; Hideyuki ICHIHASHI; Yiwen ZHU; Masatoshi YOKOKAWA
- Web page: https://patentgrep.com/patent/US20210174527A1

## CPC

- G06T 7/579: from motion
- G06T 7/593: from stereo images
- G01B 11/00: Measuring arrangements characterised by the use of optical techniques
- G01C 3/06: Use of electric means to obtain final indication
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N 23/957: Light-field or plenoptic cameras or camera modules
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30168: Image quality inspection
- G06T 2207/30244: Camera pose
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- H04N 23/6812: {based on additional sensors, e.g. acceleration sensors}
- H04N 23/685: {performed by mechanical compensation}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Provided are a signal processing apparatus, a signal processing method, and an imaging apparatus for estimating depth with a high degree of accuracy from video signals of a plurality of cameras. The signal processing apparatus includes a detection unit that detects depth from video captured by a plurality of cameras, an estimation unit that estimates camera motion by detecting a camera position, a prediction unit that predicts present depth from camera motion and depth obtained previously, and a synthesis unit that synthesizes depth detected by the detection unit and depth predicted by the prediction unit on the basis of a result of estimating the amount of blur that occurs in each image captured by cameras. The amount of blur can be estimated using camera motion, previous depth information, and shutter time.

## Claims

### Claim 1

A signal processing apparatus comprising:
a detection unit that detects depth from video captured by a plurality of cameras;
an estimation unit that estimates camera motion by detecting a camera position;
a prediction unit that predicts present depth from camera motion and depth obtained previously; and
a synthesis unit that synthesizes depth detected by the detection unit and depth predicted by the prediction unit on a basis of a result of estimating an amount of blur that occurs in each image captured by cameras from camera motion.

### Claim 2 (depends on 1)

The signal processing apparatus according to claim 1,
wherein when the amount of blur is small, the synthesis unit increases a ratio of depth detected by the detection unit to perform a synthesis or selection, and when the amount of blur is large, the synthesis unit increases a ratio of depth predicted by the prediction unit to perform a synthesis or selection.

### Claim 3 (depends on 1)

The signal processing apparatus according to claim 1,
wherein the estimation unit estimates camera motion on a basis of a measurement result of an inertial measurement unit.

### Claim 4 (depends on 1)

The signal processing apparatus according to claim 1,
wherein the estimation unit estimates camera motion from video captured by a camera on a basis of visual simultaneous localization and mapping (SLAM) technology.

### Claim 5 (depends on 1)

The signal processing apparatus according to claim 1,
wherein the detection unit detects depth by block matching between images captured by respective cameras.

### Claim 6 (depends on 1)

The signal processing apparatus according to claim 1,
wherein the prediction unit performs a viewing transformation on depth previously output from the synthesis unit, and predicts present depth from a present camera position estimated by the estimation unit.

### Claim 7 (depends on 1)

The signal processing apparatus according to claim 1,
wherein the estimation unit estimates motion of the plurality of cameras in a baseline direction, and
the synthesis unit performs the synthesis on a basis of a result of estimating an amount of blur that occurs in a direction vertical to a baseline.

### Claim 8 (depends on 1)

The signal processing apparatus according to claim 1,
wherein the estimation unit selects cameras with less motion blur or cameras with similar motion blur from three or more cameras, and
the detection unit detects depth using images captured by the cameras selected by the estimation unit.

### Claim 9 (depends on 8)

The signal processing apparatus according to claim 8,
wherein the estimation unit selects cameras with less motion blur or cameras with similar motion blur on a basis of a measurement result of an inertial measurement unit mounted on each camera.

### Claim 10 (depends on 8)

The signal processing apparatus according to claim 8,
wherein the estimation unit selects cameras to be used for depth detection on a basis of motion of each camera estimated from a measurement result of an inertial measurement unit mounted on an information terminal including the plurality of cameras mounted thereon.

### Claim 11 (depends on 1)

The signal processing apparatus according to claim 1,
wherein the estimation unit estimates a blur uncorrected by an optical image stabilizer (OIS) and selects cameras with similar uncorrected blur.

### Claim 12

A signal processing method comprising:
a detection step of detecting depth from video captured by a plurality of cameras;
an estimation step of estimating camera motion by detecting a camera position;
a prediction step of predicting present depth from camera motion and depth obtained previously; and
a synthesis step of synthesizing depth detected in the detection step and depth predicted by the prediction unit on a basis of a result of estimating an amount of blur that occurs in each image captured by cameras from camera motion.

### Claim 13

An imaging apparatus comprising:
a plurality of cameras;
a detection unit that detects depth from video captured by the plurality of cameras;
an estimation unit that estimates camera motion by detecting a camera position;
a prediction unit that predicts present depth from camera motion and depth obtained previously; and
a synthesis unit that synthesizes depth detected by the detection unit and depth predicted by the prediction unit on a basis of a result of estimating an amount of blur that occurs in each image captured by cameras from camera motion.
