# US 10,455,219 B2 · Stereo correspondence and depth sensors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,455,219 B2
- Document ID: US10455219B2
- Status: Granted
- Kind: B2
- Application number: 13/690,724
- Filed: 2012-11-30
- Published: 2014-06-05
- Granted: 2019-10-22
- Earliest claimed: 2012-11-30
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Scott D. Cohen; Brian L. Price; Gowri Somanath
- Web page: https://patentgrep.com/patent/US10455219B2

## CPC

- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 7/593: from stereo images
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- H04N 13/128: Adjusting depth or disparity
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 2207/10012: Stereo images
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Stereo correspondence and depth sensor techniques are described. In one or more implementations, a depth map generated by a depth sensor is leveraged as part of processing of stereo images to assist in identifying which parts of stereo images correspond to each other. The depth map, for instance, may be utilized to assist in identifying depth discontinuities in the stereo images. Additionally, techniques may be employed to align the depth discontinuities identified from the depth map to image edges identified from the stereo images. Techniques may also be employed to suppress image edges that do not correspond to the depth discontinuities of the depth map in comparison with image edges that do correspond to the depth discontinuities as part of the identification.

## Claims

### Claim 1

A system comprising:
first and second image capture devices having respective first and second image sensors configured, at least partially in hardware, operative to capture stereoscopic images of an image scene;
one or more depth sensors, separate from the first and second image capture devices, configured to detect depth values at locations of the image scene; and
a computing device configured to:
generate a depth map identifying depth discontinuities, from the detected depth values;
identify image edges in the stereoscopic images based on characteristics of pixels from the stereoscopic images;
determine image edge correspondence by comparing image edges identified from the stereoscopic images with depth discontinuities identified using the depth map;
suppress image edges identified from the stereoscopic images that do not correspond to depth discontinuities identified using the depth map, by removing image edges that do not correspond to depth discontinuities from a stereo correspondence calculation;
calculate the stereo correspondence by using the depth map generated from the detected depth values, detected by the one or more separate depth sensors, in conjunction with the stereoscopic images captured by the first and second image capture devices; and
generate, using the calculated stereo correspondence, a version of the image scene that includes the image edges that do correspond to depth discontinuities and are not suppressed.

### Claim 2 (depends on 1)

A system as described in claim 1, wherein the first and second images capture devices are part of a stereoscopic camera.

### Claim 3 (depends on 1)

A system as described in claim 1, wherein the characteristics of pixels from the stereoscopic images are texture and color.

### Claim 4 (depends on 1)

A system as described in claim 1, wherein the depth sensor is disposed between first and second image capture devices used to capture stereoscopic images.

### Claim 5

A system comprising:
first and second image capture devices having respective first and second image sensors configured, at least partially in hardware, operative to capture stereoscopic images of an image scene;
one or more depth sensors, separate from the first and second image capture devices, configured to detect depths at locations of the image scene without using the captured stereoscopic images; and
a computing device configured to:
generate a depth map identifying depth discontinuities, from the detected depth values;
identify image edges in the stereoscopic images based on characteristics of pixels from the stereoscopic images;
determine image edge correspondence by projecting the depth map onto the stereoscopic images and comparing image edges identified from the stereoscopic images with depth discontinuities identified using the depth map;
classify image edges as corresponding to the depth discontinuities or as not corresponding to the depth discontinuities;
suppress image edges that do not correspond to the depth discontinuities by removing them from a stereo correspondence calculation;
calculate the stereo correspondence using the image edges that do correspond to the depth discontinuities; and
generate, using the calculated stereo correspondence, a version of the image scene that includes the image edges that do correspond to depth discontinuities and are not suppressed.

### Claim 6 (depends on 5)

A system as described in claim 5, wherein the first and second images capture devices are part of a stereoscopic camera.

### Claim 7 (depends on 5)

A system as described in claim 5, wherein the characteristics of pixels from the stereoscopic images are texture and color.

### Claim 8 (depends on 5)

A system as described in claim 5, wherein the depth sensor is disposed between first and second image capture devices used to capture stereoscopic images.

### Claim 9

A system comprising:
first and second image capture devices having respective first and second image sensors configured, at least partially in hardware, operative to capture stereoscopic images of an image scene;
one or more depth sensors, separate from the first and second image capture devices, configured in addition to the first and second image capture devices to detect depths at locations of the image scene; and
a computing device configured to use the detected depths, detected by the one or more separate depth sensors, to generate a depth map identifying depth discontinuities, and to use the depth map in conjunction with the stereoscopic images captured by the first and second image capture devices identifying image edges based on characteristics of pixels, to identify depth discontinuities in the stereoscopic images, to generate a version of the image scene that does not suppress image edges that do correspond to depth discontinuities and that does suppresses the image edges that do not correspond to depth discontinuities by removing them from a stereo correspondence calculation used to generate the version of the image scene.

### Claim 10 (depends on 9)

A system as described in claim 9, wherein the first and second images capture devices are part of a stereoscopic camera.

### Claim 11 (depends on 9)

A system as described in claim 9, wherein the depth map is usable to determine a search range used in conjunction with the stereoscopic images to calculate stereo correspondence.

### Claim 12 (depends on 11)

A system as described in claim 11, wherein the depth map is usable to determine correspondence of image edges identified from the stereoscopic images with the identified depth discontinuities in the stereoscopic images.

### Claim 13 (depends on 12)

A system as described in claim 12, wherein the characteristics of pixels from the stereoscopic images are texture and color.

### Claim 14 (depends on 12)

A system as described in claim 12, wherein the depth sensor is disposed between first and second image capture devices used to capture stereoscopic images.