# US 2019/0156499 A1 · Detection of humans in images using depth information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0156499 A1
- Document ID: US20190156499A1
- Status: Published application
- Kind: A1
- Application number: 16/094,997
- Filed: 2016-05-19
- Published: 2019-05-23
- Earliest claimed: 2016-05-19
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: HAIBING REN; YIMIN ZHANG; FEI DUAN
- Web page: https://patentgrep.com/patent/US20190156499A1

## CPC

- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/243: relating to the number of classes
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/50: Depth or shape recovery
- G06V 20/64: Three-dimensional [3D] objects
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Techniques are provided for detection of humans in images that include depth information. A methodology embodying the techniques includes segmenting an image into multiple windows and estimating the distance to a subject in each window based on depth pixel values in that window, and filtering to reject windows with sizes that are outside of a desired window size range. The desired window size range is based on the estimated subject distance and the focal length of the depth camera that produced the image. The method further includes generating classifier features for each remaining windows (post-filtering) for use by a cascade classifier. The cascade classifier creates candidate windows for further consideration based on a preliminary detection of a human in any of the remaining windows. The method further includes merging neighboring candidate windows and executing a linear classifier on the merged candidate windows to verify the detection of a human.

## Claims

### Claim 1

A processor-implemented method for detection of humans in images generated by a depth camera, the method comprising:
segmenting, by a processor, an image into a plurality of windows;
estimating, by the processor, distance to a subject in each of the plurality of windows, based on depth pixels of the image in the window;
filtering, by the processor, the plurality of windows to reject windows outside of a desired window size range thereby defining a target set of windows, the desired window size range based on the estimated subject distance and on a focal length of the depth camera;
generating, by the processor, classifier features for each of the windows in the target set;
creating, by the processor, candidate windows through execution of a cascade classifier, the cascade classifier to perform preliminary detection of a human in each of the windows in the target set, based on the generated features;
merging, by the processor, neighboring candidate windows; and
executing, by the processor, a linear classifier on the merged candidate windows to verify the detection of a human.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The method of claim 1, further comprising one or both of:
validating the depth pixels used in the distance estimation, and
precomputing a plurality of desired window size ranges for each of a plurality of allowable subject distances.

### Claim 4

(canceled)

### Claim 5 (depends on 1)

The method of claim 1, wherein the classifier features comprise one or more of: luminance, hue, saturation, gradient, gradient orientation, and depth values, and wherein the cascade classifier is one of a Haar classifier, a Local Binary Pattern (LBP) classifier, a Histogram of Oriented Gradients (HOG) classifier, or a Speeded-Up Robust Features (SURF) classifier.

### Claim 6

(canceled)

### Claim 7

(canceled)

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
training the cascade classifier using adaptive boosting;
selecting regions adjacent to at least one of the merged candidate windows;
calculating a difference between the average of the values of the depth pixels in the adjacent regions and the average of the values of the depth pixels of the merged candidate window; and
providing the difference to the linear classifier as an additional feature associated with context of the merged candidate window.

### Claim 9

A system for detection of humans in images generated by a depth camera, the system comprising:
a window generation circuit to segment an image into a plurality of windows;
a window filtering circuit to:
estimate distance to a subject in each of the plurality of windows, based on depth pixels of the image in the window; and
filter the plurality of windows to reject windows outside of a desired window size range thereby defining a target set of windows, the desired window size range based on the estimated subject distance and on a focal length of the depth camera;
a candidate detection circuit to:
generate classifier features for each of the windows in the target set; and
create candidate windows through execution of a cascade classifier, the cascade classifier to perform preliminary detection of a human in each of the windows in the target set, based on the generated features;
a candidate merging circuit to merge neighboring candidate windows; and
a candidate verification circuit to execute a linear classifier on the merged candidate windows to verify the detection of a human.

### Claim 10 (depends on 9)

The system of claim 9, wherein the windows are square and vary through a range of sizes and positions within the image.

### Claim 11 (depends on 9)

The system of claim 9, wherein the window filtering circuit further comprises a depth pixel validation circuit to validate the depth pixels used in the distance estimation.

### Claim 12 (depends on 9)

The system of claim 9, wherein the window filtering circuit further comprises an image size range calculation circuit to precompute a plurality of desired window size ranges for each of a plurality of allowable subject distances.

### Claim 13 (depends on 9)

The system of claim 9, wherein the classifier features comprise one or more of: luminance, hue, saturation, gradient, gradient orientation, and depth values.

### Claim 14 (depends on 9)

The system of claim 9, wherein the cascade classifier is one of a Haar classifier, a Local Binary Pattern (LBP) classifier, a Histogram of Oriented Gradients (HOG) classifier, or a Speeded-Up Robust Features (SURF) classifier.

### Claim 15 (depends on 9)

The system of claim 9, wherein the candidate detection circuit further comprises an adaptive boosting feature selection circuit to train the cascade classifier using adaptive boosting.

### Claim 16 (depends on 9)

The system of claim 9, wherein the candidate verification circuit further comprises:
an adjacent region selection circuit to select regions adjacent to at least one of the merged candidate windows;
a difference averaging circuit to calculate a difference between the average of the values of the depth pixels in the adjacent regions and the average of the values of the depth pixels of the merged candidate window; and to provide the difference to the linear classifier as an additional feature associated with context of the merged candidate window.

### Claim 17

At least one non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in the following operations for detection of humans in images generated by a depth camera, the operations comprising:
segmenting an image into a plurality of windows;
estimating distance to a subject in each of the plurality of windows, based on depth pixels of the image in the window;
filtering the plurality of windows to reject windows outside of a desired window size range thereby defining a target set of windows, the desired window size range based on the estimated subject distance and on a focal length of the depth camera;
generating classifier features for each of the windows in the target set;
creating candidate windows through execution of a cascade classifier, the cascade classifier to perform preliminary detection of a human in each of the windows in the target set, based on the generated features;
merging neighboring candidate windows; and
executing a linear classifier on the merged candidate windows to verify the detection of a human.

### Claim 18 (depends on 17)

The computer readable storage medium of claim 17, wherein the windows are square and vary through a range of sizes and positions within the image.

### Claim 19 (depends on 17)

The computer readable storage medium of claim 17, the operations further comprising validating the depth pixels used in the distance estimation.

### Claim 20 (depends on 17)

The computer readable storage medium of claim 17, the operations further comprising precomputing a plurality of desired window size ranges for each of a plurality of allowable subject distances.

### Claim 21 (depends on 17)

The computer readable storage medium of claim 17, wherein the classifier features comprise one or more of: luminance, hue, saturation, gradient, gradient orientation, and depth values.

### Claim 22 (depends on 17)

The computer readable storage medium of claim 17, wherein the cascade classifier is one of a Haar classifier, a Local Binary Pattern (LBP) classifier, a Histogram of Oriented Gradients (HOG) classifier, or a Speeded-Up Robust Features (SURF) classifier.

### Claim 23 (depends on 17)

The computer readable storage medium of claim 17, the operations further comprising training the cascade classifier using adaptive boosting.

### Claim 24 (depends on 17)

The computer readable storage medium of claim 17, the operations further comprising:
selecting regions adjacent to at least one of the merged candidate windows;
calculating a difference between the average of the values of the depth pixels in the adjacent regions and the average of the values of the depth pixels of the merged candidate window; and
providing the difference to the linear classifier as an additional feature associated with context of the merged candidate window.
