# US 2018/0324465 A1 · Edge-aware spatio-temporal filtering and optical flow estimation in real time

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0324465 A1
- Document ID: US20180324465A1
- Status: Published application
- Kind: A1
- Application number: 15/588,463
- Filed: 2017-05-05
- Published: 2018-11-08
- Earliest claimed: 2017-05-05
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: Tunc Ozan AYDIN; Florian Michael SCHEIDEGGER; Michael Stefano Fritz SCHAFFNER; Lukas CAVIGELLI; Luca BENINI; Aljosa Aleksej Andrej SMOLIC
- Web page: https://patentgrep.com/patent/US20180324465A1

## CPC

- G06T 5/70: Denoising; Smoothing
- G06T 5/20: using local operators
- H04N 19/51: Motion estimation or motion compensation
- H04N 19/615: using motion compensated temporal filtering [MCTF]
- H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence)
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T 2207/20208: High dynamic range [HDR] image processing
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The disclosure provides an approach for edge-aware spatio-temporal filtering. In one embodiment, a filtering application receives as input a guiding video sequence and video sequence(s) from additional channel(s). The filtering application estimates a sparse optical flow from the guiding video sequence using a novel binary feature descriptor integrated into the Coarse-to-fine PatchMatch method to compute a quasi-dense nearest neighbor field. The filtering application then performs spatial edge-aware filtering of the sparse optical flow (to obtain a dense flow) and the additional channel(s), using an efficient evaluation of the permeability filter with only two scan-line passes per iteration. Further, the filtering application performs temporal filtering of the optical flow using an infinite impulse response filter that only requires one filter state updated based on new guiding video sequence video frames. The resulting optical flow may then be used in temporal edge-aware filtering of the additional channel(s) using the nonlinear infinite impulse response filter.

## Claims

### Claim 1

A computer-implemented method for spatio-temporal video filtering, comprising:
receiving an optical flow for a video frame in a video sequence; and
temporally filtering the optical flow, generating a temporally filtered optical flow, using at least an edge-aware filter in time,
wherein the edge-aware filter in time uses at least an optical flow from a previous time step to align a filter state from the previous time step to the video frame.

### Claim 2 (depends on 1)

The method of claim 1, further comprising, spatially and temporally filtering video frames from one or more additional video channels using at least the temporally filtered optical flow.

### Claim 3 (depends on 2)

The method of claim 2, wherein the video frames from the one or more additional video channels are feature maps including one or more of disparity, saliency, or detail values.

### Claim 4 (depends on 1)

The method of claim 1, further comprising estimating the received optical flow by:
estimating a sparse optical flow for the video frame, wherein correspondences between the video frame and a previous frame are determined in the sparse optical flow estimation using at least a binary descriptor formulated as a collection of binarized octal orientation maps on spatial bins around a center of an associated region of the video frame; and
spatially filtering the sparse optical flow using an edge-aware spatial filter.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
the video frame is divided into a plurality of partially overlapping tiles with overlapping bordering regions; and
the spatial filtering is separately performed for each of the plurality of partially overlapping tiles and the results of the separate spatial filtering combined.

### Claim 6 (depends on 5)

The method of claim 5, wherein separately performing the spatial filtering for each of the plurality of partially overlapping tiles includes:
determining an approximation of a spatial edge-aware filtering output at a resolution lower than a full resolution;
computing border conditions for the plurality of partially overlapping tiles using at least the approximation of the spatial edge-aware filtering output; and
applying a permeability filter to the plurality of partially overlapping tiles.

### Claim 7 (depends on 4)

The method of claim 4, wherein the binary descriptor is determined with integer arithmetic.

### Claim 8 (depends on 4)

The method of claim 4, wherein the sparse optical flow is spatially filtered using at least one iteration of a permeability filter evaluated per iteration with two scan-line passes and a normalization map.

### Claim 9 (depends on 4)

The method of claim 4, wherein the sparse optical flow is estimated with a modified Coarse-to-fine PatchMatch (CPM) technique using at least the binary descriptor, forward-back checks on coarsest and finest levels, and a quadratic interpolation.

### Claim 10 (depends on 1)

The method of claim 1, wherein the temporal filtering is performed using at least a recursive permeability filter.

### Claim 11

A non-transitory computer-readable storage medium storing a program, which, when executed by a processor performs operations for spatio-temporal video filtering, the operations comprising:
receiving an optical flow for a video frame in a video sequence; and
temporally filtering the optical flow, generating a temporally filtered optical flow, using at least an edge-aware filter in time,
wherein the edge-aware filter in time uses at least an optical flow from a previous time step to align a filter state from the previous time step to the video frame.

### Claim 12 (depends on 11)

The computer-readable storage medium of claim 11, the operations further comprising, spatially and temporally filtering video frames from one or more additional video channels using at least the temporally filtered optical flow.

### Claim 13 (depends on 12)

The computer-readable storage medium of claim 12, wherein the video frames from the one or more additional video channels are feature maps including one or more of disparity, saliency, or detail values.

### Claim 14 (depends on 11)

The computer-readable storage medium of claim 11, the operations further comprising, estimating the received optical flow by:
estimating a sparse optical flow for the video frame, wherein correspondences between the video frame and a previous frame are determined in the sparse optical flow estimation using at least a binary descriptor formulated as a collection of binarized octal orientation maps on spatial bins around a center of an associated region of the video frame; and
spatially filtering the sparse optical flow using an edge-aware spatial filter.

### Claim 15 (depends on 14)

The computer-readable storage medium of claim 14, wherein:
the video frame is divided into a plurality of partially overlapping tiles with overlapping bordering regions; and
the spatial filtering is separately performed for each of the plurality of partially overlapping tiles and the results of the separate spatial filtering combined.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein separately performing the spatial filtering for each of the plurality of partially overlapping tiles includes:
determining an approximation of a spatial edge-aware filtering output at a resolution lower than a full resolution;
computing border conditions for the plurality of partially overlapping tiles using at least the approximation of the spatial edge-aware filtering output; and
applying a permeability filter to the plurality of partially overlapping tiles.

### Claim 17 (depends on 14)

The computer-readable storage medium of claim 14, wherein the binary descriptor is determined with integer arithmetic.

### Claim 18 (depends on 14)

The computer-readable storage medium of claim 14, wherein the sparse optical flow is spatially filtered using at least one iteration of a permeability filter evaluated per iteration with two scan-line passes and a normalization map.

### Claim 19 (depends on 14)

The computer-readable storage medium of claim 14, wherein the sparse optical flow is estimated with a modified Coarse-to-fine PatchMatch (CPM) technique using at least the binary descriptor, forward-back checks on coarsest and finest levels, and a quadratic interpolation.

### Claim 20

A computer-implemented method of determining correspondences between a first image and a second image, comprising:
determining, for each of a plurality of points in the first image and in the second image, values of a binary descriptor formulated as a collection of binarized octal orientation maps on spatial bins around a center of an associated region of the image; and
comparing the determined binary descriptor values of points in the first image with the determined binary descriptor values of points in the second image to determine correspondences.
