# US 12,008,743 B2 · Hazard detection ensemble architecture system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,743 B2
- Document ID: US12008743B2
- Status: Granted
- Kind: B2
- Application number: 16/881,581
- Filed: 2020-05-22
- Published: 2021-11-25
- Granted: 2024-06-11
- Earliest claimed: 2020-05-22
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Carlos Cunha; Simon Markus Geisler; Ravi Kumar Satzoda
- Web page: https://patentgrep.com/patent/US12008743B2

## CPC

- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 20/20: Ensemble learning
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06N 3/048: Activation functions
- G06T 2207/10012: Stereo images
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system and method is disclosed that may employ a set of ensemble methods that comprise a plurality of machine learning algorithms or statistical algorithms for hazard detection. The system and method may combine images, stereo and context information using a plurality of deep learning algorithms to accurately detection hazards. The system and method may incorporate a depth channel to the red, green, and blue (RGB) channels of an image to create a four-channel RGBD image. The system and method may also overlay an RGB image with a color-map of the depth channel. The system and method may further concatenate regions of interest (ROI) to the RGB channels of an image. Lastly, the system and method may incorporate an auxiliary semantic segmentation decoder for a multi-task learning event on a drivable space.

## Claims

### Claim 1

A method comprising:
receiving a stereo image having a red channel, a blue channel, and a green channel, wherein the red channel, the blue channel, and the green channel produce a 3-channel RGB image;
receiving a depth image;
concatenating the depth image to the red, green, and blue channels of the 3-channel RGB image to produce a 4-channel RGBD image, and providing the 4-channel RGBD image as a single input into a same layer of a neural network, wherein the neural network includes:
a first plurality of layers, wherein each layer in the first plurality of layers includes:
a convolutional layer;
a batch normal layer;
a leaky ReLU activation function; and
a max pooling layer;
a second plurality of layers, wherein each layer in the second plurality of layers includes:
a convolutional layer;
a batch normal layer; and
a leaky ReLU activation function, and wherein the second plurality of layers does not include a max pooling layer;
a third plurality of layers, wherein each layer in the third plurality of layers includes:
a convolutional layer;
a batch normal layer;
a leaky ReLU activation function; and
a you-only-look-once (“YOLO”) layer; and
one or more skip architectures;
determining one or more hazards within the stereo image using the neural network;
concatenating an output of the neural network with an output of an auxiliary semantic segmentation decoder to force the neural network to learn one or more features for a drivable space to determine the one or more hazards within the stereo image; and
driving a vehicle through the drivable space while avoiding the one or more hazards.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more skip architectures include an output from a layer in the first plurality of layers that provides input to a layer including an upsample layer.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more skip architectures include an output from a layer in the second plurality of layers that provides input to the layer including an upsample layer.

### Claim 4

One or more non-transitory computer-readable media having computer-executable instructions stored thereon that when executed cause the following operations to be performed:
receiving a stereo image having a red channel, a blue channel, and a green channel, wherein the red channel, the blue channel, and the green channel produce a 3-channel RGB image;
receiving a depth image;
concatenating the depth image to the red, green, and blue channels of the 3-channel RGB image to produce a 4-channel RGBD image, and providing the 4-channel RGBD image as a single input into a same layer of a neural network, wherein the neural network includes:
a first plurality of layers, wherein each layer in the first plurality of layers includes:
a convolutional layer;
a batch normal layer;
a leaky ReLU activation function; and
a max pooling layer;
a second plurality of layers, wherein each layer in the second plurality of layers includes:
a convolutional layer;
a batch normal layer; and
a leaky ReLU activation function, and wherein the second plurality of layers does not include a max pooling layer;
a third plurality of layers, wherein each layer in the third plurality of layers includes:
a convolutional layer;
a batch normal layer;
a leaky ReLU activation function; and
a you-only-look-once (“YOLO”) layer; and
one or more skip architectures;
determining one or more hazards within the stereo image using the neural network;
concatenating an output of the neural network with an output of an auxiliary semantic segmentation decoder to force the neural network to learn one or more features for a drivable space to determine the one or more hazards within the stereo image; and
driving a vehicle through the drivable space while avoiding the one or more hazards.

### Claim 5 (depends on 4)

The media of claim 4, wherein the one or more skip architectures include an output from a layer in the first plurality of layers that provides input to a layer including an upsample layer.

### Claim 6 (depends on 4)

The media of claim 4, wherein the one or more skip architectures include an output from a layer in the second plurality of layers that provides input to the layer including an upsample layer.

### Claim 7

A system comprising:
one or more processors; and
one or more non-transitory computer-readable media having computer-executable instructions stored thereon that when executed cause the system to perform the following operations:
receiving a stereo image having a red channel, a blue channel, and a green channel, wherein the red channel, the blue channel, and the green channel produce a 3-channel RGB image;
receiving a depth image;
concatenating the depth image to the red, green, and blue channels of the 3-channel RGB image to produce a 4-channel RGBD image, and providing the 4-channel RGBD image as a single input into a same layer of a neural network, wherein the neural network includes:
a first plurality of layers, wherein each layer in the first plurality of layers includes:
a convolutional layer;
a batch normal layer;
a leaky ReLU activation function; and
a max pooling layer;
a second plurality of layers, wherein each layer in the second plurality of layers includes:
a convolutional layer;
a batch normal layer; and
a leaky ReLU activation function, and wherein the second plurality of layers does not include a max pooling layer;
a third plurality of layers, wherein each layer in the third plurality of layers includes:
a convolutional layer;
a batch normal layer;
a leaky ReLU activation function; and
a you-only-look-once (“YOLO”) layer; and
one or more skip architectures;
determining one or more hazards within the stereo image using the neural network;
concatenating an output of the neural network with an output of an auxiliary semantic segmentation decoder to force the neural network to learn one or more features for a drivable space to determine the one or more hazards within the stereo image; and
driving a vehicle through the drivable space while avoiding the one or more hazards.

### Claim 8 (depends on 7)

The system of claim 7, wherein the one or more skip architectures include an output from a layer in the first plurality of layers that provides input to a layer including an upsample layer.

### Claim 9 (depends on 7)

The system of claim 7, wherein the one or more skip architectures include an output from a layer in the second plurality of layers that provides input to the layer including an upsample layer.