# US 10,482,572 B2 · Fusion of motion and appearance features for object detection and trajectory prediction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,572 B2
- Document ID: US10482572B2
- Status: Granted
- Kind: B2
- Application number: 15/727,325
- Filed: 2017-10-06
- Published: 2019-04-11
- Granted: 2019-11-19
- Earliest claimed: 2017-10-06
- Assignees: Ford Global Technologies, LLC
- Applicants: Ford Global Technologies, LLC
- Inventors: Guy Hotson; Gintaras Vincent Puskorius; Vidya Nariyambut Murali; Gaurav Kumar Singh; Pol Llado
- Web page: https://patentgrep.com/patent/US10482572B2

## CPC

- G06V 10/82: using neural networks
- G06F 18/253: {of extracted features}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06T 3/16: Spatio-temporal transformations, e.g. video cubism
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 7/0008: {checking presence/absence}
- G06T 7/248: {involving reference images or patches}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/806: {of extracted features}
- G06V 20/20: in augmented reality scenes
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30236: Traffic on road, railway or crossing
- G06T 2207/30241: Trajectory
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Techniques and examples pertaining to objection detection and trajectory prediction for autonomous vehicles are described. A processor receives an input stream of image frames and fuses a spatiotemporal input stream of the image frames and an appearance-based stream of the image frames using a deep neural network (DNN) to generate an augmented stream of the image frames. The processor performs object detection and trajectory prediction of one or more objects in the image frames based on the augmented stream.

## Claims

### Claim 1

A method, comprising:
receiving, by a processor, an input stream of image frames;
fusing, by the processor, a spatiotemporal input stream of the image frames and an appearance-based stream of the image frames using a deep neural network (DNN) to generate an augmented stream of the image frames; and
performing, by the processor, object detection and trajectory prediction of an object in the image frames based on the augmented stream,
wherein the fusing of the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames comprising fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames with either of a late fusion architecture and a slow fusion architecture,
wherein the late fusion architecture involves:
processing the spatiotemporal input stream and the appearance-based stream through two separate stacks each comprising alternating convolutional layers and pooling layers to create two separate sets of feature maps; and
combining the two sets of feature maps, and
wherein the slow fusion architecture involves:
processing each of the spatiotemporal input stream and the appearance-based stream individually through a convolutional layer and a pooling layer; and
further processing the spatiotemporal input stream and the appearance-based stream together through a single stack comprising alternating convolutional layers and pooling layers.

### Claim 2 (depends on 1)

The method of claim 1, wherein the receiving of the input stream of the image frames comprises receiving the input stream of the image frames from one or more image sensors on a vehicle.

### Claim 3 (depends on 1)

The method of claim 1, wherein the fusing of the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN comprises augmenting the appearance-based stream of the image frames with the spatiotemporal input stream of the image frames to generate the augmented stream of the image frames.

### Claim 4 (depends on 1)

The method of claim 1, wherein the fusing of the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN comprises:
generating, by the processor, the spatiotemporal input stream of the image frames from the received stream of the image frames; and
generating, by the processor, the appearance-based stream of the image frames from the received stream of the image frames.

### Claim 5 (depends on 4)

The method of claim 4, wherein the generating of the spatiotemporal input stream of the image frames comprises performing either of:
generating the spatiotemporal input stream of the image frames using optical flow calculations and spatiotemporal filters; or
generating the spatiotemporal input stream of the image frames using a network stream which is adapted to extract motion information from the input stream of the image frames.

### Claim 6 (depends on 1)

The method of claim 1, wherein the fusing of the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN comprises fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using a convolutional neural network (CNN) that concatenates the spatiotemporal input stream and the appearance-based stream.

### Claim 7 (depends on 1)

The method of claim 1, wherein the fusing of the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN comprises concatenating the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with an early-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are separately stacked, without individual processing, as inputs for object detection and trajectory prediction.

### Claim 8 (depends on 1)

The method of claim 1, wherein the fusing of the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN comprises concatenating the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with a late-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are processed in two separate streams to create two separate sets of feature maps that are combined to form an input for object detection and trajectory prediction.

### Claim 9 (depends on 1)

The method of claim 1, wherein the fusing of the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN comprises concatenating the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with a slow-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are processed separately for one or more layers before being combined to form an input for object detection and trajectory prediction.

### Claim 10 (depends on 1)

The method of claim 1, wherein the fusing of the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN comprises fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using recurrent neural networks (RNNs) with recurrent connections to utilize outputs from a previous video frame of the image frames as inputs to a current video frame of the image frames.

### Claim 11

An apparatus implementable in a vehicle, comprising:
a processor capable of performing operations comprising:
receiving an input stream of image frames;
fusing a spatiotemporal input stream of the image frames and an appearance-based stream of the image frames using a deep neural network (DNN) to generate an augmented stream of the image frames; and
performing object detection and trajectory prediction of an object in the image frames based on the augmented stream,
wherein, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames, the processor fuses the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames with either of a late fusion architecture and a slow fusion architecture,
wherein the late fusion architecture involves:
processing the spatiotemporal input stream and the appearance-based stream through two separate stacks each comprising alternating convolutional layers and pooling layers to create two separate sets of feature maps; and
combining the two sets of feature maps, and
wherein the slow fusion architecture involves:
processing each of the spatiotemporal input stream and the appearance-based stream individually through a convolutional layer and a pooling layer; and
further processing the spatiotemporal input stream and the appearance-based stream together through a single stack comprising alternating convolutional layers and pooling layers.

### Claim 12 (depends on 11)

The apparatus of claim 11, further comprising:
one or more image sensors communicatively coupled to the processor, the one or more image sensors capable of capturing the image frames and providing the image frames as the input stream of image frames to the processor.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, the processor augments the appearance-based stream of the image frames with the spatiotemporal input stream of the image frames to generate the augmented stream of the image frames.

### Claim 14 (depends on 11)

The apparatus of claim 11, wherein, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, the processor performs operations comprising:
generating, by the processor, the spatiotemporal input stream of the image frames from the received stream of the image frames; and
generating, by the processor, the appearance-based stream of the image frames from the received stream of the image frames.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein, in generating the spatiotemporal input stream of the image frames, the processor performs either of:
generating the spatiotemporal input stream of the image frames using optical flow calculations and spatiotemporal filters; or
generating the spatiotemporal input stream of the image frames using a network stream which is adapted to extract motion information from the input stream of the image frames.

### Claim 16 (depends on 11)

The apparatus of claim 11, wherein, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, the processor fuses the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using a convolutional neural network (CNN) that concatenates the spatiotemporal input stream and the appearance-based stream.

### Claim 17 (depends on 11)

The apparatus of claim 11, wherein, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, the processor concatenates the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with an early-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are separately stacked, without individual processing, as inputs for object detection and trajectory prediction.

### Claim 18 (depends on 11)

The apparatus of claim 11, wherein, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, the processor concatenates the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with a late-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are processed in two separate streams to create two separate sets of feature maps that are combined to form an input for object detection and trajectory prediction.

### Claim 19 (depends on 11)

The apparatus of claim 11, wherein, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, the processor concatenates the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with a slow-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are processed separately for one or more layers before being combined to form an input for object detection and trajectory prediction.

### Claim 20 (depends on 11)

The apparatus of claim 11, wherein, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, the processor fuses the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using recurrent neural networks (RNNs) with recurrent connections to utilize outputs from a previous video frame of the image frames as inputs to a current video frame of the image frames.