# US 2018/0322640 A1 · System and method for detecting moving obstacles based on sensory prediction from ego-motion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322640 A1
- Document ID: US20180322640A1
- Status: Published application
- Kind: A1
- Application number: 15/960,513
- Filed: 2018-04-23
- Published: 2018-11-08
- Earliest claimed: 2017-05-02
- Assignees: HRL Laboratories, LLC
- Applicants: HRL Laboratories, LLC
- Inventors: Kyungnam Kim; Hyukseong Kwon; Heiko Hoffmann
- Web page: https://patentgrep.com/patent/US20180322640A1

## CPC

- G06V 10/82: using neural networks
- B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering
- G01P 3/42: Devices characterised by the use of electric or magnetic means (G01P3/66 takes precedence; measuring electric or magnetic values in general G01R)
- G01S 17/50: Systems of measurement based on relative movement of target
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/93: for anti-collision purposes
- G01S 17/931: of land vehicles
- G01S 17/933: of aircraft or spacecraft
- G05D 1/024: {in combination with a laser (lasers per se H01S)}
- G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)}
- G05D 1/0253: {extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70)}
- G06F 18/24133: {Distances to prototypes}
- G06T 5/70: Denoising; Smoothing
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/254: involving subtraction of images
- G06T 7/292: Multi-camera tracking
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/74: {involving reference images or patches}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G08G 1/04: using optical or ultrasonic detectors
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- G08G 5/80: Anti-collision systems
- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G06T 2207/10012: Stereo images
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30261: Obstacle
- G08G 5/34: for flight plan modification
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01P: MEASURING LINEAR OR ANGULAR SPEED, ACCELERATION, DECELERATION, OR SHOCK; INDICATING PRESENCE, ABSENCE, OR DIRECTION, OF MOVEMENT (measuring or recording blood flow A61B5/02, A61B8/06; monitoring speed or deceleration of electrically-propelled vehicles B60L3/00; vehicle lighting systems adapted to indicate speed B60Q1/54; determining position or course in navigation, measuring ground distance in geodesy or surveying G01C; combined measuring devices for measuring two or more variables of movement G01C23/00; measuring velocity of sound G01H; measuring velocity of light G01J7/00; measuring direction or velocity of solid objects by reception or emission of radiowaves or other waves and based on propagation effects, e.g. Doppler effect, propagation time, direction of propagation, G01S; measuring speed of nuclear radiation G01T; measuring acceleration of gravity G01V; {measuring or recording the speed of trains B61L23/00; speed indicators incorporated in motor vehicles B60K35/00; measuring frequency or phase G01R; traffic control G08G})
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

Described is a system for detecting moving objects. During operation, the system obtains ego-motion velocity data of a moving platform and generates a predicted image of a scene proximate the moving platform by projecting three-dimensional (3D) data into an image plane based on pixel values of the scene. A contrast image is generated based on a difference between the predicted image and an actual image taken at a next step in time. Next, an actionable prediction map is then generated based on the contrast mage. Finally, one or more moving objects may be detected based on the actionable prediction map.

## Claims

### Claim 1

A system for detecting moving objects, the system comprising:
one or more processors and a memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of:
obtaining ego-motion velocity data of a moving platform;
generating a predicted image of a scene proximate the moving platform by projecting three-dimensional (3D) data into an image plane based on pixel values of the scene;
generating a contrast image based on a difference between the predicted image and an actual image taken at a next step in time;
generating an actionable prediction map based on the contrast mage; and
detecting one or more moving objects based on the actionable prediction map.

### Claim 2 (depends on 1)

The system as set forth in claim 1, wherein the predicted image is generated for multiple time steps, resulting in both forward and backward actionable prediction maps.

### Claim 3 (depends on 2)

The system as set forth in claim 2, further comprising an operation of generating a minimized actionable prediction map by removing ghost pixels from both the forward and backward actionable prediction maps.

### Claim 4 (depends on 3)

The system as set forth in claim 3, further comprising an operation of overlaying the minimized actionable prediction map on an actual image of the scene to generate a final overlay.

### Claim 5 (depends on 4)

The system as set forth in claim 4, wherein the 3D data is obtained from a stereo camera.

### Claim 6 (depends on 4)

The system as set forth in claim 4, wherein the 3D data is obtained from a LIDAR sensor mounted on the moving platform and the pixel values of the scene are obtained from an image taken from a camera mounted on the moving platform.

### Claim 7 (depends on 6)

The system as set forth in claim 6, wherein the LIDAR sensor and camera are mounted on the moving platform such that they are co-boresighted to look at a common field-of-view around the moving platform.

### Claim 8 (depends on 1)

The system as set forth in claim 1, further comprising an operation of controlling a device based on the detection of the one or more objects.

### Claim 9 (depends on 8)

The system as set forth in claim 8, wherein controlling the device includes causing a vehicle to maneuver to avoid collision with the object.

### Claim 10 (depends on 1)

The system as set forth in claim 1, wherein the 3D data is obtained from a stereo camera.

### Claim 11 (depends on 1)

The system as set forth in claim 1, wherein the 3D data is obtained from a LIDAR sensor mounted on the moving platform and the pixel values of the scene are obtained from an image taken from a camera mounted on the moving platform, and wherein the LIDAR sensor and camera are mounted on the moving platform such that they are co-boresighted to look at a common field-of-view around the moving platform.

### Claim 12

A computer program product for detecting moving objects, the computer program product comprising:
a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of:
obtaining ego-motion velocity data of a moving platform;
generating a predicted image of a scene proximate the moving platform by projecting three-dimensional (3D) data into an image plane based on pixel values of the scene;
generating a contrast image based on a difference between the predicted image and an actual image taken at a next step in time;
generating an actionable prediction map based on the contrast mage; and
detecting one or more moving objects based on the actionable prediction map.

### Claim 13 (depends on 12)

The computer program product as set forth in claim 12, wherein the predicted image is generated for multiple time steps, resulting in both forward and backward actionable prediction maps.

### Claim 14 (depends on 13)

The computer program product as set forth in claim 13, further comprising an operation of generating a minimized actionable prediction map by removing ghost pixels from both the forward and backward actionable prediction maps.

### Claim 15 (depends on 14)

The computer program product as set forth in claim 14, further comprising an operation of overlaying the minimized actionable prediction map on an actual image of the scene to generate a final overlay.

### Claim 16 (depends on 15)

The computer program product as set forth in claim 15, wherein the 3D data is obtained from a stereo camera.

### Claim 17 (depends on 15)

The computer program product as set forth in claim 15, wherein the 3D data is obtained from a LIDAR sensor mounted on the moving platform and the pixel values of the scene are obtained from an image taken from a camera mounted on the moving platform.

### Claim 18 (depends on 17)

The computer program product as set forth in claim 17, wherein the LIDAR sensor and camera are mounted on the moving platform such that they are co-boresighted to look at a common field-of-view around the moving platform.

### Claim 19 (depends on 12)

The computer program product as set forth in claim 12, further comprising an operation of controlling a device based on the detection of the one or more objects.

### Claim 20 (depends on 19)

The computer program product as set forth in claim 19, wherein controlling the device includes causing a vehicle to maneuver to avoid collision with the object.

### Claim 21 (depends on 12)

The computer program product as set forth in claim 12, wherein the 3D data is obtained from a stereo camera.

### Claim 22 (depends on 12)

The computer program product as set forth in claim 12, wherein the 3D data is obtained from a LIDAR sensor mounted on the moving platform and the pixel values of the scene are obtained from an image taken from a camera mounted on the moving platform, and wherein the LIDAR sensor and camera are mounted on the moving platform such that they are co-boresighted to look at a common field-of-view around the moving platform.

### Claim 23

A method for detecting moving objects, the method comprising an act of:
causing one or more processers to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of:
obtaining ego-motion velocity data of a moving platform;
generating a predicted image of a scene proximate the moving platform by projecting three-dimensional (3D) data into an image plane based on pixel values of the scene;
generating a contrast image based on a difference between the predicted image and an actual image taken at a next step in time;
generating an actionable prediction map based on the contrast mage; and
detecting one or more moving objects based on the actionable prediction map.

### Claim 24 (depends on 23)

The method as set forth in claim 23, wherein the predicted image is generated for multiple time steps, resulting in both forward and backward actionable prediction maps.

### Claim 25 (depends on 24)

The method as set forth in claim 24, further comprising an operation of generating a minimized actionable prediction map by removing ghost pixels from both the forward and backward actionable prediction maps.
