# US 2019/0244036 A1 · Quasi-Parametric Optical Flow Estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244036 A1
- Document ID: US20190244036A1
- Status: Published application
- Kind: A1
- Application number: 16/268,200
- Filed: 2019-02-05
- Published: 2019-08-08
- Earliest claimed: 2015-12-04
- Assignees: Texas Instruments Incorporated
- Applicants: Texas Instruments Incorporated
- Inventors: Hrushikesh Tukaram Garud; Manu Mathew; Soyeb Noormohammed Nagori
- Web page: https://patentgrep.com/patent/US20190244036A1

## CPC

- H04N 19/53: Multi-resolution motion estimation; Hierarchical motion estimation
- G06T 7/207: for motion estimation over a hierarchy of resolutions (multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53)
- G06T 7/223: using block-matching
- G06T 7/238: using non-full search, e.g. three-step search
- G06T 7/269: using gradient-based methods
- G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V 20/48: {Matching video sequences}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20032: Median filtering
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image processing system includes a processor and optical flow (OF) determination logic for quantifying relative motion of a feature present in a first frame of video and a second frame of video that provide at least one of temporally and spatially ordered images with respect to the two frames of video. The OF determination logic configures the processor to implement performing OF estimation between the first frame and second frame using a pyramidal block matching (PBM) method to generate an initial optical flow (OF) estimate at a base pyramid level having integer pixel resolution, and refining the initial OF estimate using at least one pass of a modified Lucas-Kanade (LK) method to provide a revised OF estimate having fractional pixel resolution.

## Claims

### Claim 1

An image processing system, comprising:
a processor enabled to implement:
performing optical flow (OF) estimation between a first frame of video and a second frame of video using a pyramidal block matching (PBM) method to generate an initial optical flow (OF) estimate at a base pyramid level having integer pixel resolution, and
refining the initial OF estimate using at least one pass of a modified Lucas-Kanade (LK) method to provide a revised OF estimate having fractional pixel resolution.

### Claim 2 (depends on 1)

The system of claim 1, wherein the PBM method utilizes a hierarchical coarse-to-fine search strategy using predictors and motion smoothness factors, a Binary Census Transform for pixel description, and a Hamming distance as a cost function.

### Claim 3 (depends on 1)

The system of claim 1, wherein the PBM method utilizes a paxel order of processing using a same set of predictors for each entire paxel and a pre-evaluation strategy for the set of predictors.

### Claim 4 (depends on 1)

The system of claim 1, wherein the PBM method uses pre-computation of a spatial gradient and temporal gradient during block matching steps of the PBM method.

### Claim 5 (depends on 1)

The system of claim 1, wherein the OF determination logic is implemented in part by a hardware accelerator (HWA), the HWA utilizing a combination of two levels of local memory with one memory for storing picture data from the first frame of video and from the second frame in a growing window fashion, and another memory to store a sliding window of said picture data from the second frame of video.

### Claim 6 (depends on 1)

The system of claim 1, wherein the modified LK method is exclusive of an image warping step and splits computing the revised OF estimate into sub-tasks of spatial and temporal gradient computation and then other operations, uses reduced precision gradient data for OF refinement, and uses only integer arithmetic to perform compute tasks including matrix inversion.

### Claim 7 (depends on 2)

The system of claim 2, wherein the OF determination logic configures the processor to include in said cost function for each sequential search of the search strategy a motion vector cost value that combines a motion smoothness factor and distance between a median predictor value and a candidate pixel.

### Claim 8 (depends on 1)

The system of claim 1, wherein the at least one pass consists of a single pass which applies the modified LK method at all pyramid levels of a search pyramid used by the PBM method including the base pyramid level after the PBM method concludes.

### Claim 9 (depends on 1)

The system of claim 1, further comprising a post process filter for post processing filtering using pseudo-1d separable median filters to remove impulsive noise from the revised OF estimate.

### Claim 10

A method of optical flow (OF) estimation, comprising:
using a processor implementing a stored quasi-parametric optical flow measurement (QP-OFM) algorithm which combines a pyramidal block matching (PBM) method with a modified Lucas-Kanade (LK) method, the QP-OFM algorithm performing:
optical flow (OF) estimating between a first frame of video and a second frame of video using the PBM method to generate an initial OF estimate at a base pyramid level having integer pixel resolution, and
refining the initial OF estimate using at least one pass of the modified LK method to provide a revised OF estimate having fractional pixel resolution.

### Claim 11 (depends on 10)

The method of claim 10, wherein the PBM method utilizes a hierarchical coarse-to-fine search strategy using predictors and motion smoothness factors, a Binary Census Transform for pixel description, and a Hamming distance as a cost function.

### Claim 12 (depends on 10)

The method of claim 10, wherein said PBM method utilizes a paxel order of processing using a same set of predictors for each entire paxel and a pre-evaluation strategy for the set of predictors.

### Claim 13 (depends on 10)

The a ethod of claim 10, wherein the PBM method uses pre-computation of a spatial gradient and temporal gradient during block matching steps of the PBM method.

### Claim 14 (depends on 10)

The method of claim 10, wherein the modified LK method is exclusive of an image warping step and splits computing the revised OF estimate into sub tasks of spatial gradient computation and then other operations, uses reduced precision gradient data for OF refinement, and uses only integer arithmetic to perform compute tasks including matrix inversion.

### Claim 15 (depends on 10)

The method of claim 10, wherein the QP-OFIVI algorithm is implemented at least in part as a System-on-a-chip (SOC) including a hardware accelerator (HWA), said HWA utilizing a combination of two levels of local memory with one memory for storing picture data from the first frame of video and from the second frame in a growing window fashion, and another memory to store a sliding window of said picture data from the second frame of video.

### Claim 16 (depends on 10)

The method of claim 10, wherein the method is implemented by an image processing system comprising a processor configured for generating a scene analysis.

### Claim 17 (depends on 16)

The method of claim 16, wherein the scene analysis is used by an Advanced Driver Assistance System (ADAS) for obstacle detection or collision avoidance auto-emergency braking.

### Claim 18 (depends on 11)

The method of claim 11, wherein the method includes in the cost function for each sequential search of the search strategy a motion vector cost value that combines a motion smoothness factor and distance between a median predictor value and a candidate pixel.

### Claim 19 (depends on 10)

The method of claim 10, further comprising post processing filtering using pseudo-1d separable median filters to remove impulsive noise from the revised OF estimate.

### Claim 20 (depends on 10)

The method of claim 10, wherein the at least one pass consists of a single pass which applies the modified LK method at all pyramid levels of a search pyramid used by the PBM method including the base pyramid level after the PBM method concludes.

### Claim 21 (depends on 1)

The system of claim 1, wherein the first frame of video is a reference image and the second frame of video is a query image.

### Claim 22 (depends on 10)

The method of claim 10, wherein the first frame of video is a reference image and the second frame of video is a query image.

### Claim 23

An image processing system, comprising:
circuitry for performing optical flow (OF) estimation between a first frame of video and a second frame of video using a pyramidal block matching (PBM) method to generate an initial optical flow (OF) estimate at a base pyramid level having integer pixel resolution, and
circuitry for refining the initial OF estimate using at least one pass of a modified Lucas-Kanade (LK) method to provide a revised OF estimate having fractional pixel resolution.

### Claim 24

An image processing system, comprising:
means for performing optical flow (OF) estimation between a first frame of video and a second frame of video using a pyramidal block matching (PBM) method to generate an initial optical flow (OF) estimate at a base pyramid level having integer pixel resolution, and
means for refining the initial OF estimate using at least one pass of a modified Lucas-Kanade (LK) method to provide a revised OF estimate having fractional pixel resolution.
