# US 11,308,324 B2 · Object detecting system for detecting object by using hierarchical pyramid and object detecting method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,324 B2
- Document ID: US11308324B2
- Status: Granted
- Kind: B2
- Application number: 16/837,224
- Filed: 2020-04-01
- Published: 2021-03-04
- Granted: 2022-04-19
- Earliest claimed: 2019-08-26
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Byeoungsu Kim; Sangsoo Ko; Sanghyuck Ha
- Web page: https://patentgrep.com/patent/US11308324B2

## CPC

- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An object detecting system, which detects an object by using a hierarchical pyramid structure includes: a pyramid image generator configured to receive a plurality of input images respectively corresponding to a plurality of time points and generate a plurality of pyramid images corresponding to each of the plurality of input images; an object extractor configured to generate a plurality of pieces of object data by extracting at least one object from the plurality of pyramid images; and a buffer storing the plurality of pieces of object data on an object basis.

## Claims

### Claim 1

An object detecting system comprising:
a pyramid image generator configured to receive a first input image captured at a first time and a second input image captured at a second time, and generate a first pyramid image from the first input image and a second pyramid image from the second input image;
an object extractor configured to detect an object in the first pyramid image and the second pyramid image and generate a plurality of pieces of object data representing the object; and
a buffer storing the plurality of pieces of the object data representing the object detected in the first input image and the second input image,
wherein the object comprises a first object and a second object which is located relatively closer to the object detecting system than the first object,
wherein the pyramid image generator further configured to:
generate the first pyramid image having a first resolution by adding first time information corresponding to the first time to the first input image;
generate a third pyramid image having a second resolution which is lower than the first resolution, by down-sampling the first pyramid image; and
generate the second pyramid image having the first resolution by adding second time information corresponding to the second time to the second input image, and
wherein the object extractor further configured to:
generate first object data of the first object among the plurality of pieces of the object data by extracting the first object from the first pyramid image by using a deep learning model trained based on a neural network;
generate second object data of the first object among the plurality of pieces of the object data by extracting the first object from the second pyramid image by using the deep learning model; and
generate third object data of the second object among the plurality of pieces of the object data by extracting the second object from the third pyramid image by using the deep learning model.

### Claim 2 (depends on 1)

The object detecting system of claim 1, wherein the pyramid image generator is further configured to:
generate a fourth pyramid image having a third resolution which is lower than the second resolution, by down-sampling the third pyramid image; and
generate a fifth pyramid image having the second resolution by down-sampling the second pyramid image.

### Claim 3 (depends on 2)

The object detecting system of claim 2, wherein the object further comprises a third object which is located closer to the object detecting system than the second object, and
wherein the object extractor is further configured to:
generate forth object data of the third object among the plurality of pieces of the object data by extracting the third object from the fourth pyramid image by using a deep learning model trained based on a neural network; and
generate fifth object data of the second object among the plurality of pieces of the object data by extracting the second object from the fifth pyramid image by using the deep learning model.

### Claim 4 (depends on 3)

The object detecting system of claim 3, wherein the first object is located relatively far from an image-capturing device capturing the first input image and the second input image, and the second object is located relatively closer to the image-capturing device than the first object.

### Claim 5 (depends on 1)

The object detecting system of claim 1, wherein the object extractor is further configured to:
store the first object data of the first object and the second object data of the first object in a first area of the buffer; and
store the third object data of the second object in a second area of the buffer.

### Claim 6 (depends on 5)

The object detecting system of claim 5, further comprising:
an object tracker configured to track the first object by using at least one object data among the plurality of pieces of the object data selected from the first object data and the second object data stored in the first area and the first time information, and track the second object by using at least one object data among the plurality of pieces of the object data selected from the third object data stored in the second area and the second time information.

### Claim 7 (depends on 1)

The object detecting system of claim 1, wherein the pyramid image generator is further configured to:
generate a first pyramid image set including the first pyramid image and the second pyramid image; and
generate a second pyramid image set by down-sampling at least one of the first pyramid image and the second pyramid image in the first pyramid image set, and
wherein the object extractor is further configured to:
generate, from the first pyramid image set, N pieces of first object data respectively corresponding to N time points among the plurality of pieces of the object data, wherein N is a natural number; and
generate, from the second pyramid image set, M pieces of second object data respectively corresponding to M time points among the plurality of pieces of the object data, wherein M is a natural number.

### Claim 8 (depends on 7)

The object detecting system of claim 7, wherein N is greater than M.

### Claim 9 (depends on 7)

The object detecting system of claim 7, wherein the object comprises a first object and a second object, and
wherein the object detecting system further comprises an object tracker configured to:
track the first object by using P pieces of the first object data among the N pieces of first object data, wherein P is a natural number less than or equal to N; and
track the second object by using Q pieces of the second object data among the M pieces of second object data, wherein Q is a natural number that is less than or equal to M.

### Claim 10 (depends on 1)

The object detecting system of claim 1, further comprising:
a region-of-interest manager configured to set a first region of interest for the first input image and a second region of interest for the second input image,
wherein the object extractor is further configured to extract the object from a first region of the first input image and a second region of the second input image, the first region and the second region corresponding to the first region of interest and the second region of interest.

### Claim 11

An method of detecting an object using an object detecting system, the method comprising:
receiving a first input image captured at a first time and a second input image captured at a second time;
generating a first pyramid image associated with the first time from the first input image and a second pyramid image associated with the second time from the second input image;
generating a plurality of pieces of object data representing the object detected in the first input image and the second input image based on the first pyramid image and the second pyramid image; and
storing the plurality of pieces of the object data in a buffer,
wherein the object comprises a first object and a second object which is located relatively closer to the object detecting system than the first object,
wherein the generating the first pyramid image and the second pyramid image comprises:
generating the first pyramid image having a first resolution by adding first time information corresponding to the first time to the first input image;
generating a third pyramid image having a second resolution which is lower than the first resolution, by down-sampling the first pyramid image; and
generating the second pyramid image having the first resolution by adding second time information corresponding to the second time to the second input image, and
wherein the generating the plurality of pieces of the object data comprises:
generating first object data of the first object among the plurality of pieces of the object data by extracting the first object from the first pyramid image by using a deep learning model trained based on a neural network;
generating second object data of the first object among the plurality of pieces of the object data by extracting the first object from the second pyramid image by using the deep learning model; and
generating third object data of the second object among the plurality of pieces of the object data by extracting the second object from the third pyramid image by using the deep learning model.

### Claim 12 (depends on 11)

The method of claim 11, wherein the generating the first pyramid image and the second pyramid image further comprises:
generating a fourth pyramid image having a third resolution which is lower than the second resolution, by down-sampling the third pyramid image; and
generating a fifth pyramid image having the second resolution by down-sampling the second pyramid image.

### Claim 13 (depends on 12)

The method of claim 12, wherein the object further comprises a third object which is located closer to the object detecting system than the second object, and
wherein the generating of the plurality of pieces of the object data further comprises:
generating fourth object data of the third object among the plurality of pieces of the object data by extracting the third object from the fourth pyramid image by using a deep learning model trained based on a neural network; and
generating fifth object data of the second object among the plurality of pieces of the object data by extracting the second object from the fifth pyramid image by using the deep learning model.

### Claim 14 (depends on 13)

The method of claim 13, wherein the first object is located relatively far from an image-capturing device capturing the first input image and the second input image, and the second object is located relatively closer to the image-capturing device than the first object.

### Claim 15 (depends on 11)

The method of claim 11, wherein the storing comprises:
storing, in a first area of the buffer, the first object data of the first object and the second object data of the first object; and
storing, in a second area of the buffer, the third object data of the second object.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
tracking the first object by using at least one object data among the plurality of pieces of the object data selected from the first object data and the second object data stored in the first area and the first time information; and
tracking the second object by using at least one object data among the plurality of pieces of the object data selected from the third object data stored in the second area.

### Claim 17 (depends on 11)

The method of claim 11, wherein the generating first pyramid image and the second pyramid image comprises:
generating a first pyramid image set including the first pyramid image and the second pyramid image; and
generating a second pyramid image set by down-sampling at least one of the first pyramid image and the second pyramid image in the first pyramid image set, and
wherein the generating the plurality of pieces of the object data comprises:
generating, from the first pyramid image set, N pieces of first object data respectively corresponding to N time points, among the plurality of pieces of the object data, wherein N is a natural number; and
generating, from the second pyramid image set, M pieces of second object data respectively corresponding to M time points, among the plurality of pieces of the object data, wherein M is a natural number.

### Claim 18 (depends on 17)

The method of claim 17, wherein the object comprises a first object and a second object, and
wherein the method further comprises:
tracking the first object by using P pieces of the first object data among the N pieces of first object data, wherein P is a natural number less than or equal to N; and
tracking the second object by using Q pieces of the second object data among the M pieces of second object data, wherein Q is a natural number that is less than or equal to M.

### Claim 19

A driving assistance system for driving a vehicle by detecting an object, the driving assistance system comprising:
a pyramid image generator configured to receive a first input image captured at a first time and a second input image captured at a second time and generate a first pyramid image from the first input image and a second pyramid image from the second input image;
an object extractor configured to detect the object in the first pyramid image and the second pyramid image and generate a plurality of pieces of object data representing the object, by using deep learning that is based on a neural network;
a buffer storing the plurality of pieces of the object data representing the object detected in the first input image and the second input image; and
an object tracker configured to track the object based on the plurality of pieces of the object data stored in the buffer,
wherein the object comprises a first object and a second object which is located relatively closer to the driving assistance system than the first object,
wherein the pyramid image generator further configured to:
generate the first pyramid image having a first resolution by adding first time information corresponding to the first time to the first input image;
generate a third pyramid image having a second resolution which is lower than the first resolution, by down-sampling the first pyramid image; and
generate the second pyramid image having the first resolution by adding second time information corresponding to the second time to the second input image, and
wherein the object extractor further configured to:
generate first object data of the first object among the plurality of pieces of the object data by extracting the first object from the first pyramid image by using the deep learning model;
generate second object data of the first object among the plurality of pieces of the object data by extracting the first object from the second pyramid image by using the deep learning model; and
generate third object data of the second object among the plurality of pieces of the object data by extracting the second object from the third pyramid image by using the deep learning model.

### Claim 20 (depends on 19)

The driving assistance system of claim 19, wherein the pyramid image generator is further configured to:
generate a first pyramid image set including the first pyramid image and the second pyramid image; and
generate a second pyramid image set by down-sampling at least one of the first pyramid image and the second pyramid image, and
wherein the object extractor is further configured to:
generate, from the first pyramid image set, N pieces of first object data respectively corresponding to N time points among the plurality of pieces of the object data, wherein N is a natural number; and
generate, from the second pyramid image set, M pieces of second object data respectively corresponding to M time points among the plurality of pieces of the object data, wherein M is a natural number.