# US 2003/0213892 A1 · Method and apparatus for determining optical flow

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2003/0213892 A1
- Document ID: US20030213892A1
- Status: Published application
- Kind: A1
- Application number: 10/440,966
- Filed: 2003-05-19
- Published: 2003-11-20
- Earliest claimed: 2002-05-17
- Assignees: Sarnoff Corporation
- Inventors: Wenyi Zhao; Harpreet Singh Sawhney
- Web page: https://patentgrep.com/patent/US20030213892A1

## CPC

- H10F 39/12: Image sensors
- H10F: INORGANIC SEMICONDUCTOR DEVICES SENSITIVE TO INFRARED RADIATION, LIGHT, ELECTROMAGNETIC RADIATION OF SHORTER WAVELENGTH OR CORPUSCULAR RADIATION

## Abstract

A method and apparatus for determining the optical flow of a sequence of image frames. Optical flow fields are computed in a manner that enforces both brightness constancy and a consistency constraint.

## Claims

### Claim 1

A method for computing optical flow comprising the steps of:
a) obtaining a first image frame and a second image frame; and
b) computing an optical flow field using said first and second image frames, wherein said computed optical flow field is derived by enforcing an optical flow consistency constraint between said first and second image frames.

### Claim 2 (depends on 1)

The method of claim 1, wherein said computing step b) computes said optical flow field relative to a virtual reference frame.

### Claim 3 (depends on 1)

The method of claim 1, wherein said computed optical flow field is based on brightness constancy.

### Claim 4 (depends on 1)

The method of claim 1, wherein the computed optical flow field is determined according to a consistency constraint:
p2=p1+u1[p1] u2[p2]=−u1[p1]
where p1 are coordinates in said first image frame, where p2 are coordinates in said second image frame, where u1[p1] is a first flow field, and where field where u2[p2] is a second flow field.

### Claim 5 (depends on 1)

The method of claim 1, wherein said optical flow consistency constraint is provided by:
I(p)=I1(p−αu[p])=I2(p+(1−α)u[p]),
where α is a control parameter, where I(p) is a reference frame, where I1(p) is said first reference frame, where I2(p) is said second reference frame, and where u[p] is said optical flow field.

### Claim 6 (depends on 5)

The method of claim 5, wherein said optical flow consistency constraint is expressed in differential form:
8 I t ⁡ ( p ) ⁢ = def ⁢ ⁢ I 1 ⁡ ( p ) - I 2 ⁡ ( p ) ≈ ⁢ 1 2 ⁢ ( ∇ I 1 ⁡ ( p ) + ∇ I 2 ⁡ ( p ) ) T ⁢ u ⁡ [ p ] .
where said α is set to be 0.5.

### Claim 7 (depends on 5)

The method of claim 5, wherein said control parameter a is set in a range of [0,1].

### Claim 8 (depends on 1)

The method of claim 1, wherein the computed optical flow is determined to minimize the error: Errcons=[I1(p−αu[p]−I2(p+(1−α)u[p))]2,
where α is a control parameter, where I1(p) is said first reference frame, where I2(p) is said second reference frame, and where u[p] is said optical flow field.

### Claim 9 (depends on 1)

The method of claim 1, further comprising the step of:
c) obtaining flow fields in coordinates of said first image frame or said second image frame by warping said optical flow.

### Claim 10 (depends on 1)

The method of claim 1, where said optical flow field is used to detect salient motion.

### Claim 11 (depends on 1)

The method of claim 1, where said optical flow field is used to generate a reconstruction-based super-resolution image.

### Claim 12 (depends on 2)

The method of claim 2, wherein said virtual reference frame is used in a tweening method.

### Claim 13

A method of computing optical flow comprising the steps of:
a) obtaining a first image frame, a second image frame and a third frame; and
b) computing a plurality of optical flow fields using said first, second and third image frames, wherein said computed optical flow fields are derived by enforcing an optical flow consistency constraint between said first, second and third image frames.

### Claim 14 (depends on 13)

The method of claim 13, wherein said computing step b) computes said optical flow fields relative to said second frame, wherein said computed optical flow fields are such that an optical flow field computed from said first image frame to said second image frame is consistent with an optical flow field computed from said third image frame to said second image frame.

### Claim 15 (depends on 13)

The method of claim 13, wherein the computed optical flow field is based on brightness constancy.

### Claim 16 (depends on 13)

The method of claim 13, wherein said optical flow consistency constraint is provided by:
9 I t1 ′ = I 1 ′ - I ≈ 1 2 ⁢ ( ∇ I + ∇ I 1 ′ ) ) T ⁢ δ ⁢ ⁢ u 1 ⁢ ⁢ I t3 ′ = I 3 ′ - I ≈ 1 2 ⁢ ( ∇ I + ∇ I 3 ′ ) ) T ⁢ δ ⁢ ⁢ u 3 I t13 ′ = I 1 ′ - I 3 ′ ≈ 1 2 ⁡ [ ( ∇ I 1 ′ ) T ⁢ δ ⁢ ⁢ u 1 - ( ∇ I 3 ′ ) T ⁢ δ ⁢ ⁢ u 3 ]
where δu1 is an incremental optical flow field computed from said first image frame,
where δu3 is an incremental optical flow field computed from said third image frame,
and where I is a reference frame, and I′i are warped version of Ii.

### Claim 17 (depends on 16)

The method of claim 16, wherein said optical flow consistency constraint is expressed in a linear system of equations:
10 ⌊ 2 ⁢ ∑ ( ∇ I 1 ′ ) ⁢ ( ∇ I 1 ′ ) T - ∑ ( ∇ I 1 ′ ) ⁢ ( ∇ I 3 ′ ) T - ∑ ( ∇ I 3 ′ ) ⁢ ( ∇ I 1 ′ ) T ⁢ 2 ⁢ ∑ ( ∇ I 3 ′ ) ⁢ ( ∇ I 3 ′ ) T ⌋ ⁢ [ δ ⁢ ⁢ u 1 δ ⁢ ⁢ u 3 ] = [ I t1 ⁢ ∇ I 1 ′ + I t13 ⁢ ∇ I 1 ′ I t3 ⁢ ∇ I 3 ′ + I t31 ⁢ ∇ I 3 ′ ]
where It31=−It13.

### Claim 18 (depends on 13)

The method of claim 13, wherein the computed optical flow fields are computed so as to minimize an error between said first frame and said second frame and to minimize an error between said first frame and said third frame.

### Claim 19 (depends on 18)

The method of claim 18, wherein said errors are provided as:
11 Err cons = ∑ [ ( I 1 ⁡ ( p - u 1 ⁡ [ p ] ) - I ⁡ ( p ) ) 2 + ⁢ ⁢ ( I 3 ⁡ ( p - u 3 ⁡ [ p ] ) - I ⁡ ( p ) ) 2 + ⁢ ( I 1 ⁡ ( p - u 1 ⁡ [ p ] ) - I 3 ⁡ ( p - u 3 ⁡ [ p ] ) ) 2 ]
where I is said second image frame that is serving as a reference frame, where I1 is said first image frame, where I3 is said third image frame, where u1[p] is an optical flow field computed from said first image frame, and where u3[p] is an optical flow field computed from said third image frame.

### Claim 20 (depends on 13)

The method of claim 13, where said optical flow fields are used to detect salient motion.

### Claim 21 (depends on 13)

The method of claim 13, where said optical flow fields are used to generate a reconstruction-based super-resolution image.

### Claim 22

A method for computing optical flow comprising the steps of:
a) obtaining N number of image frames; and
b) computing N−1 optical flow fields using said N number of image frames, wherein said computed optical flow fields are derived by enforcing an optical flow consistency constraint between one of said N frames and a reference image frame r.

### Claim 23 (depends on 22)

The method of claim 22, wherein said computed optical flow fields are computed so as to minimize errors between one of said N frames and the reference frame r and errors between two of said N frames other than the reference frame r.

### Claim 24 (depends on 23)

The method of claim 23, wherein said computed optical flow fields are computed so as to minimize the following:
12 Err cons = ⁢ ∑ Err f ⁢ ⁢ 2 ⁢ r + Err f ⁢ ⁢ 2 ⁢ f = ⁢ ∑ i ≠ r ⁢ [ ( I i ⁡ ( p - u i ⁡ [ p ] ) - I r ⁡ ( p ) ) 2 + ⁢ ∑ i ≠ j ⁢ ( I i ⁡ ( p - u i ⁡ [ p ] ) - I j ⁡ ( p - u i ⁡ [ p ] ) ) 2 .
wherein Errf2r are said errors between one of said N frames and the reference frame r; wherein Errf2f are said errors between two of said N frames other than the reference frame r, where Ii is one of said N image frames, where Ij is one of said N image frames and where Ir is said reference image frame.

### Claim 25 (depends on 22)

The method of claim 22, wherein the computed optical flow fields are based on brightness constancy.

### Claim 26 (depends on 24)

The method of claim 24, wherein the computed optical flow fields are based the following linear system of equations:
13 [ ( n - 1 ) ⁢ ∑ ( ∇ I 1 ′ ) ⁢ ( ∇ I 1 ′ ) T - ∑ ( ∇ I 1 ′ ) ⁢ ( ∇ I 2 ′ ) T ⁢ … - ∑ ( ∇ I 1 ′ ) ⁢ ( ∇ I n ′ ) T ⋯ ⁢ ⋯ ⁢ ⁢ ⋯ ⁢ ⋯ - ∑ ( ∇ I n ′ ) ⁢ ( ∇ I 1 ′ ) T - ∑ ( ∇ I 1 ′ ) ⁢ ( ∇ I 2 ′ ) T ⁢ … ( n - 1 ) ⁢ ∑ ( ∇ I n ′ ) ⁢ ( ∇ I n ′ ) T ] ⁢ [AutoLeftMatch] [ δ ⁢ ⁢ u 1 ⋯ δ ⁢ ⁢ u n ] = [ ∑ I t1j ⁢ ∇ I 1 ′ ⋯ ∑ I tnj ⁢ ∇ n ′ ] .

### Claim 27

An apparatus for computing optical flow comprising:
means for obtaining a first image frame and a second image frame; and
means for computing an optical flow field using said first and second image frames, wherein said computed optical flow field is derived by enforcing an optical flow consistency constraint between said first and second image frames.

### Claim 28

An apparatus for computing optical flow comprising:
means for obtaining a first image frame, a second image frame and a third frame; and
means for computing a plurality of optical flow fields using said first, second and third image frames, wherein said computed optical flow fields are derived by enforcing an optical flow consistency constraint between said first, second and third image frames.
