# US 2021/0174516 A1 · Method and apparatus with motion information estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174516 A1
- Document ID: US20210174516A1
- Status: Published application
- Kind: A1
- Application number: 16/916,669
- Filed: 2020-06-30
- Published: 2021-06-10
- Earliest claimed: 2019-12-10
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: CHUL WOO KANG; HYUN CHEOL JEON; KYUNGBOO JUNG
- Web page: https://patentgrep.com/patent/US20210174516A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G01C 21/165: {combined with non-inertial navigation instruments}
- G01C 21/28: with correlation of data from several navigational instruments
- G06F 18/253: {of extracted features}
- G06N 3/02: Neural networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06V 20/584: {of vehicle lights or traffic lights}
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A processor-implemented motion information estimating method includes: estimating motion information based on at least one set of initial motion information of a vehicle that is obtained from at least one sensor; predicting a plurality of sets of image feature information corresponding to a periphery of the vehicle based on the estimated motion information; obtaining a plurality of sets of detected image feature information detected from an input image obtained from an image sensor and an accuracy of each of the sets of the detected image feature information using a neural network; evaluating a reliability of each of the sets of the detected image feature information by comparing the sets of the predicted image feature information and the sets of the detected image feature information; and correcting the estimated motion information based on at least one set of the sets of the detected image feature information selected based on a result of the evaluating of the reliability and on the accuracy.

## Claims

### Claim 1

A processor-implemented motion information estimating method, comprising:
estimating motion information based on at least one set of initial motion information of a vehicle that is obtained from at least one sensor;
predicting a plurality of sets of image feature information corresponding to a periphery of the vehicle based on the estimated motion information;
obtaining a plurality of sets of detected image feature information detected from an input image obtained from an image sensor and an accuracy of each of the sets of the detected image feature information using a neural network;
evaluating a reliability of each of the sets of the detected image feature information by comparing the sets of the predicted image feature information and the sets of the detected image feature information; and
correcting the estimated motion information based on at least one set of the sets of the detected image feature information selected based on a result of the evaluating of the reliability and on the accuracy.

### Claim 2 (depends on 1)

The method of claim 1, wherein the correcting comprises:
selecting a fusion mode based on the result of the evaluating of the reliability and the accuracy; and
correcting the estimated motion information based on at least one set of the detected image feature information corresponding to the selected fusion mode.

### Claim 3 (depends on 2)

The method of claim 2, wherein the fusion mode comprises a predetermined number of axis position information and axis pose information to be derived from the at least one set of the detected feature information for the correcting of the estimated motion information.

### Claim 4 (depends on 2)

The method of claim 2, wherein the selecting of the fusion mode comprises:
selecting the fusion mode from a plurality of modes that uses the at least one set of the detected image feature information or the at least one set of the initial motion information for the correcting of the estimated motion information.

### Claim 5 (depends on 4)

The method of claim 4, wherein the selecting of the fusion mode comprises:
selecting the fusion mode from the modes based on a priority of each of the modes.

### Claim 6 (depends on 5)

The method of claim 5, wherein the selecting of the fusion mode further comprises:
in response to a priority of a fusion mode selected at a previous time being higher than a priority of a fusion mode to be selected at a current time, selecting the fusion mode for the current time after a time lag.

### Claim 7 (depends on 1)

The method of claim 1, wherein the comparing of the sets of the predicted image feature information and the sets of the detected image feature information comprises:
determining a difference between the sets of the predicted image feature information and the sets of the detected image feature information.

### Claim 8 (depends on 7)

The method of claim 7, wherein the difference between the sets of the predicted image feature information and the sets of the detected image feature information comprises a difference between predicted object positions of the sets of the predicted image feature information and detected object positions of the sets of the detected image feature information.

### Claim 9 (depends on 1)

The method of claim 1, wherein the predicting comprises:
predicting the sets of the image feature information by converting the estimated motion information to a two-dimensional (2D) coordinate on a 2D image based on a field of view (FOV) of the image sensor.

### Claim 10 (depends on 9)

The method of claim 9, wherein the 2D image includes a 2D bird's-eye-view map image.

### Claim 11 (depends on 1)

The method of claim 1, wherein the correcting comprises:
converting a 2D coordinate of the selected at least one set of the detected image feature information to a dimension of the estimated motion information; and
correcting the estimated motion information based on a result of the converting.

### Claim 12 (depends on 1)

The method of claim 1, wherein the estimating comprises:
determining a position, a speed, and a pose of the vehicle based on information received from an inertial measurement unit (IMU) sensor;
obtaining a position from a global positioning system (GPS) sensor;
obtaining a speed from a vehicle speedometer; and
estimating the motion information by fusing sets of information obtained from the IMU sensor, the GPS sensor, and the vehicle speedometer.

### Claim 13 (depends on 12)

The method of claim 12, wherein the at least one set of the initial motion information comprises any one of: the determined position, speed, and pose of the vehicle; the obtained position; and the obtained speed.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
determining whether to perform the motion information estimating method based on a speed obtained from a vehicle speedometer included in the initial motion information.

### Claim 15 (depends on 1)

A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1.

### Claim 16

A motion information estimating apparatus comprising:
an image sensor configured to obtain an input image of a periphery of a vehicle;
at least one non-image sensor configured to obtain at least one set of sensor information of a motion of the vehicle; and
one or more processors configured to:
determine at least one set of initial motion information of the vehicle based on the at least one set of the sensor information;
estimate motion information based on the determined at least one set of the initial motion information;
predict a plurality of sets of image feature information corresponding to the periphery of the vehicle based on the estimated motion information;
obtain a plurality of sets of detected image feature information detected from the input image and an accuracy of each of the sets of the detected image feature information, using a neural network;
evaluate a reliability of each of the sets of the detected image feature information by comparing the sets of the predicted image feature information and the sets of the detected image feature information; and
correct the estimated motion information based on at least one set of the sets of the detected image feature information selected based on a result of the evaluating of the reliability and on the accuracy.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein, for the correcting, the one or more processors are configured to:
select a fusion mode based on the result of the evaluating of the reliability and the accuracy; and
correct the estimated motion information based on at least one set of the detected image feature information corresponding to the selected fusion mode.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein, for the selecting of the fusion mode, the one or more processors are configured to:
select the fusion mode from a plurality of modes that uses the at least one set of the detected image feature information or the at least one set of the initial motion information for the correcting of the estimated motion information.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein, for the selecting of the fusion mode, the one or more processors are configured to:
select the fusion mode from the modes based on a priority of each of the modes.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein, for the selecting of the fusion mode, the one or more processors are configured to:
in response to a priority of a fusion mode selected at a previous time being higher than a priority of a fusion mode to be selected at a current time, select the fusion mode for the current time after a time lag.

### Claim 21 (depends on 16)

The apparatus of claim 16, wherein, for the comparing of the sets of the predicted image feature information and the sets of the detected image feature information, the one or more processors are configured to:
determine a difference between the sets of the predicted image feature information and the sets of the detected image feature information.

### Claim 22 (depends on 16)

The apparatus of claim 16, wherein, for the predicting, the one or more processors are configured to:
predict the sets of the image feature information by converting the estimated motion information to a two-dimensional (2D) coordinate on a 2D image based on a field of view (FOV) of the image sensor.

### Claim 23 (depends on 16)

The apparatus of claim 16, wherein, for the correcting, the one or more processors are configured to:
convert a 2D coordinate of the selected at least one set of the detected image feature information to a dimension of the estimated motion information; and
correct the estimated motion information based on a result of the converting.

### Claim 24 (depends on 16)

The apparatus of claim 16, wherein, for the estimating, the one or more processors are configured to:
determine a position, a speed, and a pose of the vehicle based on information received from an inertial measurement unit (IMU) sensor;
obtain a position from a global positioning system (GPS) sensor;
obtain a speed from a vehicle speedometer; and
estimate the motion information by fusing sets of information obtained from the IMU sensor, the GPS sensor, and the vehicle speedometer.

### Claim 25 (depends on 16)

The apparatus of claim 16, further comprising a memory storing instructions that, when executed by the one or more processors, configure the one or more processors to perform the estimating of the motion information, the predicting of the sets of the image feature information, the determining of the sets of detected image feature information, the evaluating of the reliability of each of the sets of the detected image feature information, and the correcting of the estimated motion information.

### Claim 26

A processor-implemented motion information estimating method, comprising:
estimating motion information of a vehicle based on information received from at least one sensor;
predicting image feature information corresponding to a periphery of the vehicle based on the estimated motion information;
detecting, using a neural network, image feature information based on an input image obtained from at least one other sensor;
determining a reliability of the detected image feature information by comparing the predicted image feature information with the detected image feature information; and
correcting the estimated motion information based on the detected image feature information, in response to the determined reliability being greater than or equal to a threshold.
