# US 11,113,563 B2 · Apparatus for detecting object and method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,563 B2
- Document ID: US11113563B2
- Status: Granted
- Kind: B2
- Application number: 16/296,017
- Filed: 2019-03-07
- Published: 2020-05-07
- Granted: 2021-09-07
- Earliest claimed: 2018-11-05
- Assignees: Hyundai Motor Company; Kia Motors Corporation
- Applicants: Hyundai Motor Company; Kia Motors Corporation
- Inventors: Young Hyun Kim
- Web page: https://patentgrep.com/patent/US11113563B2

## CPC

- G06N 3/08: Learning methods
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F 18/211: Selection of the most significant subset of features
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus for detecting an object includes a processor that extracts information for object detection from image data frames based on a hierarchical structure of a convolutional neural network (CNN) and transmits information for object detection extracted from an uppermost layer of the hierarchical structure to a lower layer to detect an object based on information received at each layer, and storage that stores the information for object detection and detected object information.

## Claims

### Claim 1

An apparatus for detecting an object, the apparatus comprising:
at least one processor configured to extract information for object detection from image data frames based on a hierarchical structure of a convolutional neural network (CNN) and transmit information for object detection extracted from an uppermost layer of the hierarchical structure to a lower layer to detect an object based on information received at each layer; and
storage configured to store the information for object detection and detected object information,
wherein the at least one processor further includes a hidden state top-down structure for receiving the feature information and the contextual information extracted for each layer of each image data frame; and
wherein the hidden state top-down structure includes a plurality of hidden state layers corresponding to a layer of the backbone network, and an uppermost hidden state layer sequentially transmits the contextual information received from the layer of the backbone network to a hidden state layer of the lower layer.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to detect the object in an one-stage scheme.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the at least one processor is configured to correct the detected object information by using the information for object detection.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the information for object detection includes feature information for large object detection and contextual information for small object detection.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the at least one processor includes a backbone network for extracting feature information for object detection and contextual information from the image data frames.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the backbone network includes a scale-based hierarchical feature structure for each of the image data frames, and is configured to substitute time-series data for data for each scale of the scale-based hierarchical feature structure and output the time-series data.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the hidden state top-down structure includes:
a first hidden state layer configured to perform the object detection by using first feature information and first contextual information which are received from a first layer which is an uppermost layer of the backbone network;
a second hidden state layer configured to perform the object detection by using the first contextual information received from the first hidden state layer and second feature information and second contextual information received from a second layer of the backbone network; and
a third hidden state layer configured to perform the object detection by using the first contextual information and the second contextual information received from the second hidden state layer, and third feature information and third contextual information received from a third layer of the backbone network.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the at least one processor further includes a recursive prediction module configured to receive the feature information and the contextual information from the hidden state top-down structure to correct a detection result received from the hidden state top-down structure.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the recursive prediction module is configured to repeatedly perform the correction of the detection result.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein the recursive prediction module is configured to remove a feature corresponding to a background of a detection result by the hidden state top-down structure by using an attention mask.

### Claim 11

A method of detecting an object, the method comprising:
extracting information for object detection from image data frames based on a hierarchical structure of a convolutional neural network (CNN); and
transmitting information for object detection extracted from an uppermost layer of the hierarchical structure to a lower layer to detect an object based on information received at each layer,
wherein the detecting of the objecting includes:
receiving feature information and contextual information extracted for each layer of the backbone network; and
detecting, by a plurality of hidden state layers corresponding to a layer of the backbone network, the object based on the feature information and the contextual information received fro the layer of the backbone network.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
correcting the detected object information by using the information for object detection.

### Claim 13 (depends on 12)

The method of claim 12, wherein the extracting of the information for the object detection includes:
in a backbone network having a scale-based hierarchical feature structure for each of the image data frames, substituting time-series data for data for each scale of the scale-based hierarchical feature structure.

### Claim 14 (depends on 11)

The method of claim 11, wherein the detecting of the object further includes:
sequentially transmitting, by an uppermost hidden state layer among the plurality of hidden state top-down layers, the contextual information received from the layer of the backbone network to a hidden state layer of the lower layer.

### Claim 15 (depends on 14)

The method of claim 14, wherein the sequential transmitting of the contextual information to the hidden state layer of the lower layer includes:
performing the object detection by using first feature information and first contextual information which are received from a first layer which is an uppermost layer of the backbone network in a first hidden state layer among the plurality of hidden state top-down layers;
performing the object detection in a second hidden state layer among the plurality of hidden state top-down layers by using the first contextual information received from the first hidden state layer and second feature information and second contextual information received from a second layer of the backbone network; and
performing the object detection in a third hidden state layer among the plurality of hidden state top-down layers by using the first contextual information and the second contextual information received from the second hidden state layer and third feature information and third contextual information received from received from a third layer of the backbone network.

### Claim 16 (depends on 12)

The method of claim 12, wherein the correcting of the detected object information includes:
receiving the feature information and the contextual information from the hidden state top-down layer to correct a detection result received from the hidden state top-down layer.

### Claim 17 (depends on 16)

The method of claim 16, wherein the correcting of the detected object information includes:
removing a feature corresponding to a background of a detection result by the hidden state top-down layer by using an attention mask.