# US 10,503,966 B1 · Binocular pedestrian detection system having dual-stream deep learning neural network and the methods of using the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,503,966 B1
- Document ID: US10503966B1
- Status: Granted
- Kind: B1
- Application number: 16/157,173
- Filed: 2018-10-11
- Granted: 2019-12-10
- Earliest claimed: 2018-10-11
- Assignees: Tindei Network Technology (Shanghai) Co., Ltd.
- Applicants: Tindei Network Technology (Shanghai) Co., Ltd.
- Inventors: Zixiao Pan
- Web page: https://patentgrep.com/patent/US10503966B1

## CPC

- G06T 7/593: from stereo images
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/253: {of extracted features}
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/56: relating to colour
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/246: Calibration of cameras
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 2207/10012: Stereo images
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Aspects of present disclosure relates to a binocular pedestrian detection system (BPDS). BPDS includes: a binocular camera to capture certain binocular images of pedestrians passing through a predetermined area, an image/video processing ASIC to process binocular images captured, and a binocular pedestrian detection system controller having a processor, a network interface, and a memory storing computer executable instructions. When executed by processor, computer executable instructions cause processor to perform: capturing, by binocular camera, binocular images of pedestrians, binocularly rectifying binocular images, calculating disparity maps of binocular images rectified, training a dual-stream deep learning neural network, and detecting pedestrians passing through predetermined area using dual-stream deep learning neural network trained. Dual-stream deep learning neural network includes a neural network for extracting disparity features from disparity maps of binocular images, and a neural network for learning and fusing features from rectified left images and disparity maps of binocular images.

## Claims

### Claim 1

A binocular pedestrian detection system, comprising:
a binocular camera having a first camera, and a second camera to capture a plurality of binocular images of a plurality of pedestrians passing through a predetermined area over a predetermined period of time, wherein each binocular image comprises a left image and a right image;
an image/video processing application specific integrated circuit (ASIC) to preprocess the plurality of binocular images of the pedestrians captured by the binocular camera; and
a binocular pedestrian detection system controller, having a processor, a network interface, and a memory storing an operating system, and computer executable instructions, wherein when executed in the processor, the computer executable instructions cause the processor to perform one or more of following:
capturing, by the binocular camera, the plurality of binocular images of the pedestrians passing through the predetermined area over the predetermined period of time;
binocularly rectifying, by the image/video processing ASIC, the plurality of binocular images of the pedestrians captured to obtain a plurality of rectified binocular images, wherein each of the plurality of rectified binocular images comprises a rectified left image and a rectified right image;
calculating, by the image/video processing ASIC, disparity maps of the plurality of rectified binocular images;
training a dual-stream deep learning neural network, wherein the dual-stream deep learning neural network comprises a neural network for extracting disparity features from the disparity maps of the plurality of binocular images of the pedestrians, and a neural network for learning and fusing features from the plurality of rectified left images and disparity maps of the plurality of binocular images of the pedestrians; and
detecting, via the dual-stream deep learning neural network trained, the plurality of pedestrians passing through the predetermined area over the predetermined period of time.

### Claim 2 (depends on 1)

The binocular pedestrian detection system of claim 1, wherein the first camera comprises a first lens, and a first CMOS sensor for capturing the left image through the first lens, the second camera comprises a second lens, and a second CMOS sensor for capturing the right image through the second lens, and the left image and the right image form a binocular image.

### Claim 3 (depends on 1)

The binocular pedestrian detection system of claim 1, wherein the image/video processing ASIC is programmed to perform one or more of following operations:
performing calibration of the binocular camera;
binocularly rectifying the plurality of binocular images of the pedestrians, wherein the plurality of binocular images of the pedestrians comprises a plurality of training binocular images of the pedestrians and a plurality of real-time binocular images of the pedestrians captured by the binocular camera for pedestrian detection;
calculating the disparity maps of the plurality of training binocular images of the pedestrians during a training phase; and
calculating the disparity maps of the plurality of real-time binocular images of the pedestrians during an application phase.

### Claim 4 (depends on 3)

The binocular pedestrian detection system of claim 3, wherein the training phase comprises:
training of the neural network for extracting disparity features using the disparity maps of the plurality of binocular images of the pedestrians;
training of the neural network for learning and fusing the features of RGB and disparity using the plurality of left images of pedestrians; and
stacking the trained neural network for extracting disparity features and the neural network for learning and fusing features to form the dual-stream deep learning neural network.

### Claim 5 (depends on 4)

The binocular pedestrian detection system of claim 4, wherein the application phase comprises:
capturing, by the binocular camera, the plurality of real-time binocular images of the pedestrians;
binocularly rectifying, by the image/video processing ASIC, the plurality of real-time binocular images of the pedestrians captured;
calculating, by the image/video processing ASIC, disparity maps of the plurality of real-time binocular images of the pedestrians binocularly rectified;
detecting the plurality of the pedestrians from the plurality of rectified left images and disparity maps of the plurality of real-time binocular images of the pedestrians using the dual-stream deep learning neural network formed during the training phase; and
performing non-maximum suppression to detection results to obtain final pedestrian detection results.

### Claim 6 (depends on 5)

The binocular pedestrian detection system of claim 5, wherein the detecting the plurality of the pedestrians using the dual-stream deep learning neural network comprises:
extracting disparity features from the disparity maps of the plurality of real-time binocular images of the pedestrians using the neural network for extracting disparity features;
learning RGB features from the plurality of rectified left images of pedestrians using the first N layers of the neural network for learning and fusing the features of RGB and disparity, wherein N is a positive integer;
stacking the disparity features extracted and the RGB features learned through a plurality of channels;
fusing features of disparity and RGB using the last M−N layers of the neural network for learning and fusing features to obtain the final pedestrian detection results, wherein M is a positive integer greater than N and is a total number of layers of the neural network for learning and fusing features.

### Claim 7 (depends on 6)

The binocular pedestrian detection system of claim 6, wherein N is 7, and M is 15.

### Claim 8 (depends on 1)

The binocular pedestrian detection system of claim 1, wherein the binocular pedestrian detection system is installed over a doorway having binocular camera facing down to the plurality of pedestrians passing through the doorway.

### Claim 9 (depends on 1)

The binocular pedestrian detection system of claim 1, wherein the binocular pedestrian detection system is installed over a doorway having binocular camera facing the plurality of pedestrians passing through the doorway in a predetermined angle.

### Claim 10 (depends on 1)

The binocular pedestrian detection system of claim 1, wherein the network interface comprises a power-on-ethernet (POE) network interface, wherein the power supplying to the binocular pedestrian detection system is provided by the POE network interface, and the final pedestrian detection results are transmitted to a server that collects the final pedestrian detection results through the network interface and a communication network.

### Claim 11

A method of detecting pedestrians using a binocular pedestrian detection system, comprising:
capturing, using a binocular camera of the binocular pedestrian detection system, a plurality of binocular images of a plurality of pedestrians passing through a predetermined area over a predetermined period of time, wherein each of the plurality of binocular images comprises a left image and a right image;
binocularly rectifying, via an image/video processing application specific integrated circuit (ASIC) of the binocular pedestrian detection system, the plurality of binocular images of the pedestrians captured to obtain a plurality of rectified binocular images, wherein each of the plurality of rectified binocular images comprises a rectified left image and a rectified right image;
calculating, via the image/video processing ASIC, disparity maps of the plurality of rectified binocular images;
training a dual-stream deep learning neural network using the plurality of rectified left images and disparity maps of the plurality of binocular images of the pedestrians calculated, wherein the dual-stream deep learning neural network comprises a neural network for extracting disparity features from the disparity maps of the plurality of binocular images of the pedestrians, and a neural network for learning and fusing features from the plurality of rectified left images and disparity maps of the plurality of binocular images of the pedestrians; and
detecting, via the dual-stream deep learning neural network trained, the plurality of pedestrians passing through the predetermined area over the predetermined period of time.

### Claim 12 (depends on 11)

The method of claim 11, wherein the binocular pedestrian detection system comprises:
the binocular camera having a first camera, and a second camera to capture a plurality of binocular images of the plurality of pedestrians passing through the predetermined area over a predetermined period of time, wherein each of the plurality of binocular images comprises a left image captured by the first camera, and a right image captured by the second camera;
the image/video processing ASIC to preprocess the plurality of binocular images of the pedestrians captured by the binocular camera; and
a binocular pedestrian detection system controller, having a processor, a network interface, and a memory storing an operating system, and computer executable instructions, wherein when executed in the processor, the computer executable instructions cause the processor to perform one or more operations of the method.

### Claim 13 (depends on 12)

The method of claim 12, wherein the network interface comprises a power-on-ethernet (POE) network interface, wherein the power supply to the binocular pedestrian detection system is provided by the POE network interface, and the final pedestrian detection results are transmitted to a server that collects the final pedestrian detection results through the network interface and a communication network.

### Claim 14 (depends on 11)

The method of claim 11, wherein the image/video processing ASIC is programmed to perform one or more of following operations:
performing calibration of the binocular camera;
binocularly rectifying the plurality of binocular images of the pedestrians, wherein the plurality of binocular images of the pedestrians comprises a plurality of training binocular images of the pedestrians and a plurality of real-time binocular images of the pedestrians captured by the binocular camera for pedestrian detection;
calculating the disparity maps of the plurality of training binocular images of the pedestrians during a training phase; and
calculating the disparity maps of the plurality of real-time binocular images of the pedestrians during an application phase.

### Claim 15 (depends on 14)

The method of claim 14, wherein the training phase comprises:
training of the neural network for extracting disparity features using the disparity maps of the plurality of binocular images of the pedestrians;
training of the neural network for learning and fusing the features of RGB and disparity using the plurality of left images of pedestrians; and
stacking the trained neural network for extracting disparity features and the neural network for learning and fusing features to form the dual-stream deep learning neural network.

### Claim 16 (depends on 15)

The method of claim 15, wherein the application phase comprises:
capturing, by the binocular camera, the plurality of real-time binocular images of the pedestrians;
binocularly rectifying, by the image/video processing ASIC, the plurality of real-time binocular images of the pedestrians captured;
calculating, by the image/video processing ASIC, disparity maps of the plurality of real-time binocular images of the pedestrians binocularly rectified;
detecting the plurality of the pedestrians from the plurality of rectified left images and disparity maps of the plurality of real-time binocular images of the pedestrians using the dual-stream deep learning neural network formed during the training phase; and
performing non-maximum suppression to detection results to obtain final pedestrian detection results.

### Claim 17 (depends on 16)

The method of claim 16, wherein the detecting the plurality of the pedestrians using the dual-stream deep learning neural network comprises:
extracting disparity features from the disparity maps of the plurality of real-time binocular images of the pedestrians using the neural network for extracting disparity features;
learning RGB features from the plurality of rectified left images of pedestrians using the first N layers of the neural network for learning and fusing the features of RGB and disparity, wherein N is a positive integer;
stacking the disparity features extracted and the RGB features learned through a plurality of channels;
fusing features of disparity and RGB using the last M−N layers of the neural network for learning and fusing features to obtain the final pedestrian detection results, wherein M is a positive integer greater than N and is a total number of layers of the neural network for learning and fusing features.

### Claim 18 (depends on 17)

The method of claim 17, wherein N is 7, and M is 15.

### Claim 19 (depends on 11)

The method of claim 11, wherein the binocular pedestrian detection system is installed over a doorway having binocular camera facing down to the plurality of pedestrians passing through the doorway.

### Claim 20 (depends on 11)

The method of claim 11, wherein the binocular pedestrian detection system is installed over a doorway having binocular camera facing the plurality of pedestrians passing through the doorway in a predetermined angle.