# US 12,223,737 B2 · Method for object recognition using queue-based model selection and optical flow in autonomous driving environment, recording medium and device for performing the method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,737 B2
- Document ID: US12223737B2
- Status: Granted
- Kind: B2
- Application number: 17/315,554
- Filed: 2021-05-10
- Published: 2022-02-03
- Granted: 2025-02-11
- Earliest claimed: 2020-07-29
- Assignees: Korea University Research and Business Foundation
- Applicants: Korea University Research and Business Foundation
- Inventors: Joongheon Kim; Won Joon Yun; SooHyun Park
- Web page: https://patentgrep.com/patent/US12223737B2

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- B60W 60/001: {Planning or execution of driving tasks}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2554/4049: Relationship among other objects, e.g. converging dynamic objects
- G06N 3/045: Combinations of networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An object recognition method using queue-based model selection and optical flow in an autonomous driving environment includes preprocessing data through a dense flow in a matrix form by calculating an optical flow of images captured consecutively in time by a sensor for an autonomous vehicle, generating a confidence mask by generating a vectorized confidence threshold representing a probability that there is a moving object for each cell of the preprocessed matrix, determining whether there is a moving object on the images by mapping the images captured consecutively in time to the confidence mask, and selecting an object recognition model using a tradeoff constant between object recognition accuracy and queue stability in each time unit. Accordingly, it is possible to improve the performance of object recognition in an autonomous driving environment by applying the optical flow to the confidence threshold of the object recognition system.

## Claims

### Claim 1

An object recognition method using queue-based model selection and optical flow in an autonomous driving environment, the method comprising:
preprocessing data through a dense flow in a form of a matrix by calculating an optical flow of images captured consecutively over time by a sensor for an autonomous vehicle,
wherein the preprocessing of the data comprises min-max normalizing the matrix for the dense flow representing a magnitude;
generating a confidence mask by generating a vectorized confidence threshold representing a probability that there is a moving object for each cell of the preprocessed matrix;
mapping the images captured consecutively over time to the confidence mask;
determining there is a moving object in the images when a confidence of each cell mapped to the confidence mask is lower than the corresponding confidence threshold;
determining there is no object in the images when the confidence of each cell mapped to the confidence mask is higher than the corresponding confidence threshold; and
selecting an object recognition model using a tradeoff constant between object recognition accuracy and queue stability in each time unit,
wherein the selecting of the object recognition model comprises selecting an optimal object recognition model using a size of a stored queue based on Lyapunov optimization, and
wherein the selecting of the object recognition model further comprises:
selecting a fastest object recognition model as the size of the queue is larger, and
selecting a highest accuracy object recognition model as the size of the queue is smaller.

### Claim 2 (depends on 1)

The object recognition method using queue-based model selection and optical flow in an autonomous driving environment according to claim 1, wherein preprocessing the data comprises:
removing static pixel information from the normalized matrix;
downsizing the matrix through interpolation;
vectorizing each element of the downsized matrix; and
replicating the vectorized elements at least one time.

### Claim 3 (depends on 2)

The object recognition method using queue-based model selection and optical flow in an autonomous driving environment according to claim 2, wherein the removing of the static pixel information from the normalized matrix comprises:
removing pixel information within a preset distance from a median value of the normalized matrix.

### Claim 4 (depends on 1)

A non-transitory computer-readable storage medium having recorded thereon a computer program for performing the object recognition method using queue-based model selection and optical flow in an autonomous driving environment according to claim 1.

### Claim 5

An object recognition device using queue-based model selection and optical flow in an autonomous driving environment, the object recognition device comprising at least one processor configured to run software to execute instructions, thereby configuring the processor to:
preprocess data through a dense flow in a form of a matrix by calculating an optical flow of images captured consecutively over time by a sensor for an autonomous vehicle,
wherein the at least one processor is further configured to preprocess the data by min-max normalizing the matrix for the dense flow representing a magnitude;
generate a confidence mask by generating a vectorized confidence threshold representing a probability that there is a moving object for each cell of the preprocessed matrix;
map the images captured consecutively over time to the confidence mask;
determine there is a moving object in the images when a confidence of each cell mapped to the confidence mask is lower than the corresponding confidence threshold;
determine there is no moving object in the images when the confidence of each cell mapped to the confidence mask is higher than the corresponding confidence threshold; and
select an object recognition model using a tradeoff constant between object recognition accuracy and queue stability in each time unit,
wherein the at least one processor is configured to select an optimal object recognition model using a size of a stored queue based on Lyapunov optimization, and
wherein the at least one processor is further configured to:
select a fastest object recognition model as the size of the queue is larger, and
select a highest accuracy object recognition model as the size of the queue is smaller.

### Claim 6 (depends on 5)

The object recognition device using queue-based model selection and optical flow in an autonomous driving environment according to claim 5,
wherein the at least one processor is configured to:
remove static pixel information from the normalized matrix,
downsize the matrix through interpolation,
vectorize each element of the downsized matrix, and
replicate the vectorized elements at least one time.