# US 2021/0232812 A1 · Training a neural network to determine pedestrians

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232812 A1
- Document ID: US20210232812A1
- Status: Published application
- Kind: A1
- Application number: 16/773,339
- Filed: 2020-01-27
- Published: 2021-07-29
- Earliest claimed: 2020-01-27
- Assignees: Ford Global Technologies, LLC
- Applicants: Ford Global Technologies, LLC
- Inventors: Nikita Jaipuria; Aniruddh Ravindran; Hitha Revalla; Vijay Nagasamy
- Web page: https://patentgrep.com/patent/US20210232812A1

## CPC

- G06N 3/08: Learning methods
- G05D 1/0221: {involving a learning process}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/18: Eye characteristics, e.g. of the iris
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A training system for a neural network system and method of training is disclosed. The method may comprise: receiving, from a sensor, an image frame captured while an operator is controlling a vehicle; using an eye-tracking system associated with the sensor, monitoring the eyes of the operator to determine eyeball gaze data; determining, from the image frame, a plurality of pedestrians; and iteratively training the neural network system to determine, from among the plurality of pedestrians, the one or more target pedestrians using the eyeball gaze data and an answer dataset that is based on the eyeball gaze data, wherein the determined one or more target pedestrians have a relatively-higher probability of collision with the vehicle than a remainder of the plurality of pedestrians.

## Claims

### Claim 1

A method of training a neural network system to determine one or more target pedestrians, comprising:
receiving, from a sensor, an image frame captured while an operator is controlling a vehicle;
using an eye-tracking system associated with the sensor, monitoring the eyes of the operator to determine eyeball gaze data;
determining, from the image frame, a plurality of pedestrians; and
iteratively training the neural network system to determine, from among the plurality of pedestrians, the one or more target pedestrians using the eyeball gaze data and an answer dataset that is based on the eyeball gaze data,
wherein the determined one or more target pedestrians have a relatively-higher probability of collision with the vehicle than a remainder of the plurality of pedestrians.

### Claim 2 (depends on 1)

The method of claim 1, wherein the eye-tracking system is a wearable eye-tracking system and is worn by the operator while the image frame is captured by the sensor.

### Claim 3 (depends on 1)

The method of claim 1, wherein the neural network system comprises a convolutional neural network (CNN) and a fully connected neural network, wherein an input to the CNN is the image frame, wherein an output of the CNN is provided to the fully connected neural network, wherein an output of the fully connected neural network is an indication of the one or more target pedestrians.

### Claim 4 (depends on 1)

The method of claim 1, wherein the neural network system comprises a pedestrian detection algorithm and a first deep neural network (first DNN), wherein an input to the pedestrian detection algorithm is the image frame, wherein an output of the first DNN is an indication of the one or more target pedestrians.

### Claim 5 (depends on 4)

The method of claim 4, wherein the neural network system further comprises a second deep neural network (second DNN) and a third deep neural network (third DNN), wherein an input to the second DNN is the image frame, wherein an output of the second DNN and an output of the first DNN are inputs to the third DNN, wherein an output of the third DNN is an indication of the one or more target pedestrians.

### Claim 6 (depends on 5)

The method of claim 5, wherein iteratively training the neural network system further comprises providing eyeball gaze data to one or more of the first, second, and third DNNs.

### Claim 7 (depends on 5)

The method of claim 5, wherein the first and third DNNs are each fully connected neural networks, wherein the second DNN is a convolutional neural network.

### Claim 8 (depends on 7)

The method of claim 7, wherein the first DNN has a different quantity of hidden layers, a different quantity of neurons, or both than the second DNN.

### Claim 9 (depends on 5)

The method of claim 5, wherein iteratively training the neural network system includes evaluating subsequent image frames with each of the pedestrian detection algorithm, the first DNN, the second DNN, and the third DNN.

### Claim 10 (depends on 4)

The method of claim 4, wherein the pedestrian detection algorithm is a single shot multibox detector algorithm.

### Claim 11 (depends on 1)

The method of claim 1, further comprising: repeating the receiving, monitoring, and determining steps; and for each respective image frame, determining the answer dataset based on the eyeball gaze data.

### Claim 12 (depends on 1)

The method of claim 1, further comprising calculating a difference using a loss function, wherein the difference is based on the answer dataset and an output of the neural networks system, wherein iteratively training the neural network system is repeated until the difference is less than a threshold.

### Claim 13 (depends on 12)

The method of claim 12, further comprising: when the difference is not less than the threshold, adjusting a plurality of weights associated with neurons of neural network system.

### Claim 14 (depends on 12)

The method of claim 12, wherein the iteratively training comprises supervised learning.

### Claim 15 (depends on 1)

The method of claim 1, wherein the image frame is one of a plurality of video frames received from the sensor.

### Claim 16 (depends on 1)

The method of claim 1, wherein the eye-tracking system comprises the sensor, one or more illuminators directed toward the operator's eyes, one or more cameras directed toward the operator's eyes, and a computer programmed to use the eyeball gaze data and the image frame to determine a gaze point of the operator, wherein the sensor comprises a red-green-blue (RGB) camera.

### Claim 17 (depends on 1)

The method of claim 1, further comprising: installing the trained neural network system in a vehicle; and using the trained neural network system to navigate the vehicle in an autonomous mode.

### Claim 18 (depends on 1)

The method of claim 1, wherein the eyeball gaze data comprises one or more of the following information: a timestamp (of the image frame), a left eye gaze point, a right eye gaze point, a left eye three-dimensional (3D) gaze position, a right eye 3D gaze position, a combined-eye 3D gaze position, a left eye pupil position, a right eye pupil position, a left eye pupil diameter, a right eye pupil diameter, a left eye confidence parameter, a right eye confidence parameter, a galvanic skin response parameter, a tri-axis accelerometer value, a tri-axis gyroscopic value, or a mapped gaze point.

### Claim 19

A training system, comprising:
a sensor;
an eye-tracking system;
a test computer communicatively coupled to the sensor and eye-tracking system, comprising one or more processors and memory storing instructions executable by the one or more processors, the instructions comprising to:
receive, from the sensor, an image frame captured while an operator is controlling a vehicle;
using the eye-tracking system, monitor the eyes of the operator to determine eyeball gaze data;
determine, from the image frame, a plurality of pedestrians; and
iteratively train a neural network system to determine, from among the plurality of pedestrians, the one or more target pedestrians using the eyeball gaze data and an answer dataset that is based on the eyeball gaze data,
wherein the determined one or more target pedestrians have a relatively-higher probability of collision with the vehicle than a remainder of the plurality of pedestrians.

### Claim 20 (depends on 19)

The system of claim 19, wherein the neural network system comprises a pedestrian detection algorithm, a first deep neural network (first DNN), a second deep neural network (second DNN), and a third deep neural network (third DNN), wherein an input to each of the pedestrian detection algorithm and the second DNN is the image frame, wherein an output of the first and second DNNs are inputs to the third DNN, wherein an output of the third DNN is an indication of the one or more target pedestrians.
