# US 2022/0036100 A1 · Method for object recognition using queue-based model selection and optical flow in autonomous driving environment, recording medium and device for performing the method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036100 A1
- Document ID: US20220036100A1
- Status: Published application
- Kind: A1
- Application number: 17/315,554
- Filed: 2021-05-10
- Published: 2022-02-03
- Earliest claimed: 2020-07-29
- Assignees: Korea University Research and Business Foundation
- Applicants: Korea University Research and Business Foundation
- Inventors: Joongheon KIM; Won Joon YUN; SooHyun PARK
- Web page: https://patentgrep.com/patent/US20220036100A1

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 60/001: {Planning or execution of driving tasks}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2554/4049: Relationship among other objects, e.g. converging dynamic objects
- G06N 3/045: Combinations of networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An object recognition method using queue-based model selection and optical flow in an autonomous driving environment includes preprocessing data through a dense flow in a matrix form by calculating an optical flow of images captured consecutively in time by a sensor for an autonomous vehicle, generating a confidence mask by generating a vectorized confidence threshold representing a probability that there is a moving object for each cell of the preprocessed matrix, determining whether there is a moving object on the images by mapping the images captured consecutively in time to the confidence mask, and selecting an object recognition model using a tradeoff constant between object recognition accuracy and queue stability in each time unit. Accordingly, it is possible to improve the performance of object recognition in an autonomous driving environment by applying the optical flow to the confidence threshold of the object recognition system.

## Claims

### Claim 1

An object recognition method using queue-based model selection and optical flow in an autonomous driving environment, comprising:
preprocessing data through a dense flow in a matrix form by calculating an optical flow of images captured consecutively in time by a sensor for an autonomous vehicle;
generating a confidence mask by generating a vectorized confidence threshold representing a probability that there is a moving object for each cell of the preprocessed matrix;
determining whether there is a moving object on the images by mapping the images captured consecutively in time to the confidence mask; and
selecting an object recognition model using a tradeoff constant between object recognition accuracy and queue stability in each time unit.

### Claim 2 (depends on 1)

The object recognition method using queue-based model selection and optical flow in an autonomous driving environment according to claim 1, wherein preprocessing the data comprises:
min-max normalizing the matrix for the dense flow representing a magnitude;
removing static pixel information from the normalized matrix;
downsizing the matrix through interpolation;
vectorizing each element of the downsized matrix; and
replicating the vectorized elements at least one time.

### Claim 3 (depends on 2)

The object recognition method using queue-based model selection and optical flow in an autonomous driving environment according to claim 2, wherein removing the static pixel information from the normalized matrix comprises removing pixel information within a preset distance from a median value of the normalized matrix.

### Claim 4 (depends on 1)

The object recognition method using queue-based model selection and optical flow in an autonomous driving environment according to claim 1, wherein determining whether there is a moving object comprises:
determining that there is no object when the confidence of each cell mapped to the confidence mask is higher than the corresponding confidence threshold, and
determining that there is an object when the confidence of each cell is lower than the corresponding confidence threshold.

### Claim 5 (depends on 1)

The object recognition method using queue-based model selection and optical flow in an autonomous driving environment according to claim 1, wherein selecting the object recognition model comprises selecting an optimal object recognition model using a size of a stored queue based on Lyapunov optimization.

### Claim 6 (depends on 5)

The object recognition method using queue-based model selection and optical flow in an autonomous driving environment according to claim 5, wherein selecting the object recognition model comprises:
selecting a fastest object recognition model as the size of the queue is larger, and
selecting a highest accuracy object recognition model as the size of the queue is smaller.

### Claim 7 (depends on 1)

A computer-readable storage medium having recorded thereon a computer program for performing the object recognition method using queue-based model selection and optical flow in an autonomous driving environment according to claim 1.

### Claim 8

An object recognition device using queue-based model selection and optical flow in an autonomous driving environment, comprising:
a preprocessing unit to preprocess data through a dense flow in a matrix form by calculating an optical flow of images captured consecutively in time by a sensor for an autonomous vehicle;
a confidence mask generation unit to generate a confidence mask by generating a vectorized confidence threshold representing a probability that there is a moving object for each cell of the preprocessed matrix;
an object recognition unit to determine whether there is a moving object on the images by mapping the images captured consecutively in time to the confidence mask; and
a tradeoff unit to select an object recognition model using a tradeoff constant between object recognition accuracy and queue stability in each time unit.

### Claim 9 (depends on 8)

The object recognition device using queue-based model selection and optical flow in an autonomous driving environment according to claim 8, wherein the preprocessing unit preprocesses the data by min-max normalizing the matrix for the dense flow representing a magnitude, removing static pixel information from the normalized matrix, downsizing the matrix through interpolation, vectorizing each element of the downsized matrix, and replicating the vectorized elements at least one time.

### Claim 10 (depends on 8)

The object recognition device using queue-based model selection and optical flow in an autonomous driving environment according to claim 8, wherein the object recognition unit is configured to:
determine that there is no object when the confidence of each cell mapped to the confidence mask is higher than the corresponding confidence threshold, and
determine that there is an object when the confidence of each cell is lower than the corresponding confidence threshold.

### Claim 11 (depends on 8)

The object recognition device using queue-based model selection and optical flow in an autonomous driving environment according to claim 8, wherein the tradeoff unit selects an optimal object recognition model using a size of a stored queue based on Lyapunov optimization.

### Claim 12 (depends on 11)

The object recognition device using queue-based model selection and optical flow in an autonomous driving environment according to claim 11, wherein the tradeoff unit is configured to:
select a fastest object recognition model as the size of the queue is larger, and
select a highest accuracy object recognition model as the size of the queue is smaller.
