# US 10,375,378 B2 · Dual camera system for real-time depth map generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,375,378 B2
- Document ID: US10375378B2
- Status: Granted
- Kind: B2
- Application number: 16/010,179
- Filed: 2018-06-15
- Published: 2019-06-13
- Granted: 2019-08-06
- Earliest claimed: 2017-12-12
- Assignees: Black Sesame International Holding Limited
- Applicants: Black Sesame International Holding Limited
- Inventors: Donghui Wu
- Web page: https://patentgrep.com/patent/US10375378B2

## CPC

- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 7/557: from light fields, e.g. from plenoptic cameras
- G06T 7/593: from stereo images
- H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
- H04N 23/67: Focus control based on electronic image sensor signals
- G06T 5/70: Denoising; Smoothing
- G06T 2207/10012: Stereo images
- G06T 2207/10024: Color image
- G06T 2207/10052: Images from lightfield camera
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20216: Image averaging
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method of real-time depth mapping in a dual camera system, including, receiving an auto focus image, rectifying the auto focus image, receiving a plenoptic image having lens that cover 2×2 pixels, extracting four plenoptic sub-images, averaging and rectifying the plenoptic sub-images, block matching the rectified auto focus image and the rectified plenoptic sub-images along epipolar lines, block matching the rectified plenoptic sub-images along a vertical and a horizontal direction, determining a composite block matching error of the plenoptic block match, determining a plenoptic disparity map of the plenoptic sub-images based on the composite block matching error, converting the plenoptic disparity map to a plenoptic depth map, converting the plenoptic depth map to an initial stereo disparity map, searching for a guided disparity map based on the initial stereo disparity map and the epipolar block match and converting the guided disparity map to a guided depth map.

## Claims

### Claim 1

A method of real-time depth mapping in a dual camera system, comprising:
receiving an auto focus image from an auto focus camera module;
converting the auto focus image to monochrome;
rectifying the converted auto focus image;
receiving a fixed focus image from a plenoptic camera module having lens that cover two by two pixels and outputting a plurality of plenoptic sub-images;
extracting four plenoptic sub-images from the plurality of plenoptic sub images;
averaging the extracted plenoptic sub-images;
rectifying the averaged plenoptic sub-images;
block matching the rectified auto focus image and the rectified plenoptic sub-images along epipolar lines, outputting an epipolar block match;
block matching the rectified plenoptic sub-images along a vertical direction, outputting vertical plenoptic block match;
block matching the rectified plenoptic sub-images along a horizontal direction, outputting horizontal plenoptic block match;
determining a composite block matching error of the vertical plenoptic block match and the horizontal plenoptic block match;
determining a plenoptic disparity map of the extracted plenoptic sub-images based on the composite block matching error,
converting the plenoptic disparity map to a plenoptic depth map;
converting the plenoptic depth map to an initial stereo disparity map;
searching for a guided disparity map based on the initial stereo disparity map and the epipolar block match; and
converting the guided disparity map to a guided depth map.

### Claim 2 (depends on 1)

The method of real-time depth mapping in the dual camera system of claim 1, further comprising determining a plenoptic confidence map of the extracted plenoptic sub-images based on the composite block matching error.

### Claim 3 (depends on 1)

The method of real-time depth mapping in the dual camera system of claim 1, further comprising receiving a color image from the auto focus camera module.

### Claim 4 (depends on 3)

The method of real-time depth mapping in the dual camera system of claim 3, further comprising smoothing the guided disparity map.

### Claim 5 (depends on 4)

The method of real-time depth mapping in the dual camera system of claim 4, further comprising converting the smoothed guided disparity map to a smoothed depth map.

### Claim 6 (depends on 1)

The method of real-time depth mapping in the dual camera system of claim 1, wherein the block matching is performed in the horizontal direction.

### Claim 7 (depends on 6)

The method of real-time depth mapping in the dual camera system of claim 6, wherein the block matching is performed in the vertical direction.

### Claim 8 (depends on 7)

The method of real-time depth mapping in the dual camera system of claim 7, wherein the block matching error is a composite block matching error based on the block matching performed in the vertical direction and the horizontal direction.

### Claim 9 (depends on 8)

The method of real-time depth mapping in the dual camera system of claim 8, further comprising:
determining a disparity map of the rectified plenoptic sub-images based on the horizontal block matching error and the vertical block matching error; and
converting the disparity map to a depth map.

### Claim 10

A dual camera system having real-time depth map generation, comprising:
an auto focus camera module outputting an auto focus image;
a monochromatic plenoptic camera module, wherein a plenoptic lens cover two by two pixels, outputting a plurality of plenoptic sub-images, wherein the plenoptic camera module is fixed focus;
a processor coupled to the auto focus camera module and the plenoptic camera module, wherein the processor;
converts the auto focus image to monochrome;
rectifies the converted auto focus image;
extracts four plenoptic sub-images from the plenoptic camera module;
averages the four plenoptic sub-images from the plurality of plenoptic sub-images;
rectifies the averaged four plenoptic sub-images; and
block matches the rectified auto focus image and the rectified four plenoptic sub-images along epipolar lines.

### Claim 11 (depends on 10)

The dual camera system having real-time depth map generation of claim 10, wherein the processor further
extracts four plenoptic sub-images from the plurality of plenoptic sub-images;
block matches the four plenoptic sub-images along a vertical direction, outputting a vertical plenoptic block match;
block matches the four plenoptic sub-images along a horizontal direction, outputting a horizontal plenoptic block match;
determines a composite block matching error of the vertical plenoptic block match and the horizontal plenoptic block match;
determines a plenoptic disparity map of the four plenoptic sub-images based on the composite block matching error;
determines a plenoptic confidence map of the four plenoptic sub-images based on the composite block matching error, and
converts the plenoptic disparity map to a plenoptic depth map.

### Claim 12 (depends on 11)

The dual camera system having real-time depth map generation of claim 11, wherein the processor further
converts the plenoptic depth map to an initial stereo disparity map;
searches for a guided disparity map based on the initial stereo disparity map;
smoothes the guided disparity map; and
converts the guided disparity map to a guided depth map.

### Claim 13 (depends on 12)

The dual camera system having real-time depth map generation of claim 12, wherein the processor further smoothes the guided depth map.

### Claim 14 (depends on 13)

The dual camera system having real-time depth map generation of claim 13, wherein the guided disparity map is determined from a displacement having a smallest matching error based on the block matches.

### Claim 15

A dual camera stereo system having real-time depth map generation, comprising:
an auto focus camera module outputting an auto focus image;
a monochromatic plenoptic camera module, wherein a plenoptic lens cover two by two pixels and outputs a plurality of plenoptic sub-images and wherein the plenoptic camera module is fixed focus;
a processor coupled to the auto focus camera module and the plenoptic camera module, wherein the processor;
extracts four plenoptic sub-images from the plurality of plenoptic sub-images;
block matches the four plenoptic sub-images along a vertical direction, outputting a vertical plenoptic block match;
block matches the four plenoptic sub-images along a horizontal direction, outputting a horizontal plenoptic block match;
determines a composite block matching error of the vertical plenoptic block match and the horizontal plenoptic block match;
determines a plenoptic disparity map of the four plenoptic sub-images based on the composite block matching error;
determines a plenoptic confidence map of the four plenoptic sub-images based on the composite block matching error; and
converts the plenoptic disparity map to a plenoptic depth map.

### Claim 16 (depends on 15)

The dual camera stereo system having real-time depth map generation of claim 15, wherein the processor further:
converts the auto focus image to monochrome;
rectifies the converted auto focus image;
averages the four plenoptic sub-images;
rectifies the averaged four plenoptic sub-images;
block matches the rectified auto focus image and the rectified four plenoptic along epipolar lines;
converts the plenoptic depth map to an initial stereo disparity map;
searches the auto focus image based on the initial stereo disparity map in a guided disparity search outputting a guided disparity map; and
converts the guided disparity map to a guided depth map.

### Claim 17 (depends on 16)

The dual camera stereo system having real-time depth map generation of claim 16, wherein the processor further smoothes the guided disparity map.

### Claim 18 (depends on 17)

The dual camera stereo system having real-time depth map generation of claim 17, wherein the processor further converts the smoothed guided disparity map to a smoothed depth map.

### Claim 19 (depends on 16)

The dual camera stereo system having real-time depth map generation of claim 16, wherein the guided disparity map is determined from a displacement having a smallest matching error based on the block matches.