# US 10,313,657 B2 · Depth map generation apparatus, method and non-transitory computer-readable medium therefor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,313,657 B2
- Document ID: US10313657B2
- Status: Granted
- Kind: B2
- Application number: 15/324,814
- Filed: 2015-12-25
- Published: 2018-07-19
- Granted: 2019-06-04
- Earliest claimed: 2015-12-25
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd
- Inventors: Jibo Zhao; Jing Yu
- Web page: https://patentgrep.com/patent/US10313657B2

## CPC

- H04N 13/20: Image signal generators
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 7/596: {from three or more stereo images}
- H04N 13/10: Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- H04N 13/128: Adjusting depth or disparity
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure provides a depth map generation apparatus, including a camera assembly with at least three cameras, an operation mode determination module and a depth map generation module. The camera assembly with at least three cameras may a first camera, a second camera and a third camera that are sequentially aligned on a same axis. The operation mode determination module may be configured to determine an operation mode of the camera assembly. The operation mode includes at least: a first mode using images of non-adjacent cameras, and a second mode using images of adjacent cameras. Further, the depth map generation module may be configured to generate depth maps according to the determined operation mode.

## Claims

### Claim 1

A depth map generation apparatus, comprising:
a camera assembly with at least three cameras including a first camera, a second camera and a third camera, wherein the first camera, the second camera and the third camera are sequentially aligned on a same axis;
an operation mode determination module including a pre-depth map generation unit and a recognition unit, and configured to determine an operation mode of the camera assembly, wherein:
the operation mode includes at least: a first mode using images of non-adjacent cameras, and a second mode using images of adjacent cameras,
the pre-depth map generation unit is configured to generate a first depth map in the first mode and a second depth map in the second mode, and
the recognition unit is configured to extract histogram information from the first depth map and the second depth map, to identify a valid object appearing at a depth range based on the histogram information, and to select one of the first mode and the second mode based on the depth range corresponding to the valid object; and
a depth map generation module configured to generate depth maps according to the determined operation mode.

### Claim 2 (depends on 1)

The depth map generation apparatus according to claim 1, wherein the operation mode determination module further comprises a sensor configured for detecting a current orientation of the camera assembly.

### Claim 3 (depends on 2)

The depth map generation apparatus according to claim 2, wherein the sensor is a motion sensor, an accelerometer, an orientation sensor, or a gyroscope.

### Claim 4 (depends on 1)

The depth map generation apparatus according to claim 1, wherein:
the histogram information presents distributions of number of pixels for each depth in the first depth map and in the second depth map.

### Claim 5 (depends on 4)

The depth map generation apparatus according to claim 4, wherein the recognition unit is further configured to:
convert intensity levels of the first depth map and intensity levels of the second depth map to depth values;
obtain number of pixels corresponding to the converted depth values of the first depth map and the second depth map; and
combine number of pixels corresponding to the converted depth values of first depth map and the second depth map to obtain the histogram which covers depth ranges of both the first depth map and the second depth map, wherein an x-axis of the histogram represents depth values.

### Claim 6 (depends on 4)

The depth map generation apparatus according to claim 4, wherein the recognition unit is further configured to:
convert intensity levels of the second depth map to a set of depth values;
obtain intensity levels of the first depth map corresponding to the set of depth values;
calculate a mapping relationship between the intensity levels of the second depth map and the intensity levels of the first depth map that correspond to the set of the depth values; and
combine number of pixels in the second depth map corresponding to the intensity levels of the first depth map based on the mapping relationship to obtain the histogram covering depth ranges of both the first depth map and the second depth map, wherein an x-axis of the histogram represents the intensity levels corresponding to the first depth map.

### Claim 7 (depends on 4)

The depth map generation apparatus according to claim 4, wherein the recognition unit is further configured to:
when an area percentage of a bump in the first depth range in the histogram is greater than a first threshold, identify that the valid object appears in the first depth range; and
when an area percentage of a bump in the second depth range in the histogram is greater than a second threshold, identify that the valid object appears in the second depth range.

### Claim 8

A depth map generation method, comprising:
providing a camera assembly with at least three cameras including a first camera, a second camera and a third camera, wherein the first camera, the second camera and the third camera are sequentially aligned on a same axis;
generating a first depth map in a first mode, and generating a second depth map in a second mode;
extracting histogram information from the first depth map and the second depth map;
identifying a valid object appearing at a depth range based on the histogram information; and
selecting one of the first mode and the second mode based on the depth range corresponding to the valid object, wherein an operation mode includes at least: the first mode using images of non-adjacent cameras, and the second mode using images of adjacent cameras; and
generating a depth map based on the images obtained from the camera assembly according to the determined operation mode.

### Claim 9 (depends on 8)

The method for depth map generation according to claim 8, further comprising:
providing a sensor to detect a current orientation of the camera assembly.

### Claim 10 (depends on 9)

The method for depth map generation according to claim 9, further comprising:
when the sensor detects the current orientation of the camera assembly is substantially vertical, determining that the camera assembly operates in the first mode; and
when the sensor detects the current orientation of the camera assembly is substantially horizontal, determining that the camera assembly operates in the second mode.

### Claim 11 (depends on 9)

The method for depth map generation according to claim 9, wherein the sensor is a motion sensor, an accelerometer, an orientation sensor, or a gyroscope.

### Claim 12 (depends on 8)

The depth map generation method according to claim 8, wherein:
the histogram information presents distributions of number of pixels for each depth in the first depth map and in the second depth map.

### Claim 13 (depends on 12)

The method for depth map generation according to claim 12, wherein extracting histogram information from the first depth map and the second depth map further comprises:
converting intensity levels of the first depth map and intensity levels of the second depth map to depth values;
obtaining number of pixels corresponding to the converted depth values of the first depth map and the second depth map; and
combining number of pixels corresponding to the converted depth values of first depth map and the second depth map to obtain the histogram which covers depth ranges of both the first depth map and the second depth map, wherein an x-axis of the histogram represents depth values.

### Claim 14 (depends on 12)

The method for depth map generation according to claim 12, wherein extracting histogram information from the first depth map and the second depth map further comprises:
converting intensity levels of the second depth map to a set of depth values;
obtaining intensity levels of the first depth map corresponding to the set of depth values;
calculating a mapping relationship between the intensity levels of the second depth map and the intensity levels of the first depth map that correspond to the set of the depth values; and
combining number of pixels in the second depth map corresponding to the intensity levels of the first depth map based on the mapping relationship to obtain the histogram covering depth ranges of both the first depth map and the second depth map, wherein an x-axis of the histogram represents the intensity levels corresponding to the first depth map.

### Claim 15 (depends on 12)

The method for depth map generation according to claim 12, further comprising:
when an area percentage of a bump in the first depth range in the histogram is greater than a first threshold, identifying that the valid object appears in the first depth range; and
when an area percentage of a bump in the second depth range in the histogram is greater than a second threshold, identifying that the valid object appears in the second depth.

### Claim 16 (depends on 8)

The method for depth map generation according to claim 8, further comprising:
in the first mode, recognizing body movements based on the generated depth map; and
in the second mode, recognizing hand gestures and finger gestures based on the generated depth map.

### Claim 17

A non-transitory computer-readable medium having computer programs for, when being executed by a processor, performing a depth map generation method, the method comprising:
providing a camera assembly with at least three cameras including a first camera, a second camera and a third camera, wherein the first camera, the second camera and the third camera are sequentially aligned on a same axis;
generating a first depth map in a first mode, and generating a second depth map in a second mode;
extracting histogram information from the first depth map and the second depth map;
identifying a valid object appearing at a depth range based on the histogram information; and
selecting one of the first mode and the second mode based on the depth range corresponding to the valid object, wherein an operation mode includes at least: the first mode using images of non-adjacent cameras, and the second mode using images of adjacent cameras; and
generating a depth map based on the images obtained from the camera assembly according to the determined operation mode.