# US 2022/0044424 A1 · Machine learning and vision-based approach to zero velocity update object detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044424 A1
- Document ID: US20220044424A1
- Status: Published application
- Kind: A1
- Application number: 16/989,264
- Filed: 2020-08-10
- Published: 2022-02-10
- Earliest claimed: 2020-08-10
- Assignees: Honeywell International Inc.
- Applicants: Honeywell International Inc.
- Inventors: Vegnesh Jayaraman; Sai Krishnan Chandrasekar; Andrew Stewart; Vijay Venkataraman
- Web page: https://patentgrep.com/patent/US20220044424A1

## CPC

- G06T 7/215: Motion-based segmentation
- G06T 7/292: Multi-camera tracking
- G06F 18/24: Classification techniques
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06T 7/269: using gradient-based methods
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/579: from motion
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30241: Trajectory
- G06T 2207/30244: Camera pose
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques for detecting motion of a vehicle are disclosed. Optical flow techniques are applied to the entirety of the received images from an optical sensor mounted to a vehicle. Motion detection techniques are then imposed on the optical flow output to remove image portions that correspond to objects moving independent from the vehicle and determine the extent, if any, of movement by the vehicle from the remaining image portions. Motion detection can be performed via a machine learning classifier. In some aspects, motion can be detected by extracting the depth of received images in addition to optical flow. In additional or alternative aspects, the optical flow and/or motion detection techniques can be implemented by at least one artificial neural network.

## Claims

### Claim 1

A method of determining if a vehicle is moving, the method comprising:
acquiring a first image corresponding to a first time period;
acquiring a second image corresponding to a second time period, wherein the first image and the second image are taken from a camera from a frame of reference of the vehicle;
detecting, via an optical flow algorithm implemented over the entirety of the first and second images, a difference between one or more portions of the first image and a respective one or more portions of the second image, wherein the one or more image portions correspond to an object moving independent of the vehicle;
removing the one or more image portions of the first and second images based on the difference; and
determining movement of the vehicle based on the remaining image portions via a machine learning algorithm.

### Claim 2 (depends on 1)

The method of claim 1, wherein detecting a difference between the image portions between the first and second images further comprises detecting via a depth algorithm implemented over the entirety of the first and second images.

### Claim 3 (depends on 2)

The method of claim 2, wherein the depth algorithm and the optical flow algorithm is implemented by at least one artificial neural network.

### Claim 4 (depends on 2)

The method of claim 2, wherein the depth algorithm is performed by at least one artificial neural network or a light detecting and ranging (LiDAR) sensor.

### Claim 5 (depends on 3)

The method of claim 3, wherein the at least one artificial neural network comprises a deep neural network including at least one hidden layer.

### Claim 6 (depends on 1)

The method of claim 1, wherein the vehicle is one of a land traversing vehicle, an air traversing, water traversing vehicle and space traversing vehicle.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining movement of the vehicle further comprises comparing at least one mean value corresponding to optical flow of the first and second images to a threshold value, wherein the threshold value is indicative of movement of the vehicle.

### Claim 8

A system configured to detect motion of a vehicle, the system comprising:
one or more optical sensors, wherein the one or more optical sensors are configured to:
acquire a first image corresponding to a first time period, and
acquire a second image corresponding to a second time period, wherein the first image and the second image are taken from a frame of reference of the vehicle; and
one or more processors, wherein the one or more processors are configured to:
detect, via an optical flow algorithm implemented over the entirety of the first and second images, a difference between one or more portions of the first image and a respective one or more portions of the second image, wherein the one or more image portions correspond to an object moving independent of the vehicle;
remove the one or more image portions of the first and second images based on the difference; and
determine movement of the vehicle based on the remaining image portions via a machine learning algorithm.

### Claim 9 (depends on 8)

The system of claim 8, wherein to detect a difference between the image portions between the first and second images further comprises to detect via a depth algorithm implemented over the entirety of the first and second images.

### Claim 10 (depends on 9)

The system of claim 9, wherein the depth algorithm and the optical flow algorithm is implemented by at least one artificial neural network.

### Claim 11 (depends on 9)

The system of claim 9, wherein the one or more optical sensors comprises a LiDAR sensor, and wherein the depth algorithm is performed by the LiDAR sensor.

### Claim 12 (depends on 10)

The system of claim 10, wherein the at least one artificial neural network comprises a deep neural network including at least one hidden layer.

### Claim 13 (depends on 8)

The system of claim 8, wherein the vehicle includes at least one of: an automobile or an urban air mobility vehicle.

### Claim 14 (depends on 8)

The system of claim 8, wherein to determine movement of the vehicle further comprises to compare at least one mean value corresponding to optical flow of the first and second images to a threshold value, wherein the threshold value is indicative of movement of the vehicle.

### Claim 15

A non-transitory computer readable medium for implementing a method for determining movement of a vehicle, the non-transitory computer readable medium comprising a processor-readable medium on which program instructions are embodied, wherein the program instructions are operable, when executed by at least one processor in a system, to cause the system to:
receive a first image corresponding to a first time period;
receive a second image corresponding to a second time period, wherein the first image and the second image are taken from a camera from a frame of reference of the vehicle;
detect, via an optical flow algorithm implemented over the entirety of the first and second images, a difference between one or more portions of the first image and a respective one or more portions of the second image, wherein the one or more image portions correspond to an object moving independent of the vehicle;
remove the one or more image portions of the first and second images based on the difference; and
determine movement of the vehicle based on the remaining image portions via a machine learning algorithm.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein to detect a difference between the image portions between the first and second images further comprises to detect via a depth algorithm implemented over the entirety of the first and second images.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the at least one processor includes at least one artificial neural network, wherein the depth algorithm and the optical flow algorithm is implemented by at least one artificial neural network.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein the at least one artificial neural network comprises a deep neural network including at least one hidden layer.

### Claim 19 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein to determine movement of the vehicle further comprises to compare at least one mean value corresponding to optical flow of the first and second images to a threshold value, wherein the threshold value is indicative of movement of the vehicle.

### Claim 20 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the vehicle includes at least one of: an automobile and an urban air mobility vehicle.
