# US 10,482,611 B2 · Determining optical flow > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,611 B2 - Document ID: US10482611B2 - Status: Granted - Kind: B2 - Application number: 16/266,844 - Filed: 2019-02-04 - Published: 2019-06-13 - Granted: 2019-11-19 - Earliest claimed: 2016-12-01 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Peter Z. Klier - Web page: https://patentgrep.com/patent/US10482611B2 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/269: using gradient-based methods - G06T 7/90: Determination of colour characteristics - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods of determining optical flow are provided. For instance, a plurality of input color vectors respectively associated with a plurality of input pixels can be obtained. Each input pixel can have a known position within an image. A color rate of change associated with each input pixel in the image can be obtained. An optical flow estimate associated with a subject pixel within the image can be determined based at least in part on the plurality of input color vectors and the color rates of change associated with the input pixels in the image. ## Claims ### Claim 1 A computer-implemented method of forming an optical flow vector describing estimated displacement of a subject pixel, the method comprising: obtaining, by one or more computing devices, a plurality of input color vectors respectively associated with a plurality of input pixels, each input pixel having a known position within an image; obtaining, by the one or more computing devices, a color rate of change associated with each input pixel in the image; and determining, by the one or more computing devices, an optical flow estimate associated with a subject pixel within the image based at least in part on the plurality of input color vectors and the color rates of change associated with the input pixels in the image. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein determining, by the one or more computing devices, the optical flow estimate comprises: accessing, by the one or more computing devices, an optical flow equation having a measured bilinear formula on a first side of the equation and a predicted bilinear formula on a second side of the equation; and evaluating, by the one or more computing devices, the optical flow equation to provide a plurality of scalar equations. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein evaluating, by the one or more computing devices, the optical flow equation to provide a plurality of scalar equations comprises determining a differentiated optical flow equation wherein each side of the differentiated optical flow equation is differentiated with respect to time. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the differentiated optical flow equation comprises the following equation: +x+y+xy=x′+y′+x′y+xy′ ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein: equals a first input color vector associated with a first input pixel; equals a second input color vector associated with a second input pixel minus the first input color vector; equals a third input color vector associated with a third input pixel minus the first input color vector; and equals the first input color vector minus the second input color vector minus the third input color vector plus a fourth input color vector associated with a fourth input pixel. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein evaluating, by the one or more computing devices, the optical flow equation to provide a plurality of scalar equations comprises expanding the differentiated optical flow equation into a plurality of scalar equations. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein determining, by the one or more computing devices, the optical flow estimate comprises performing, by the one or more computing devices, a least squares fit on a plurality of scalar equations associated with the optical flow equation. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the least squares fit is performed by solving the following equation: [ x ′ y ′ ] ⁢ ⁢ ⁢ ⁢ {  b ^ 24 ⁢ y  2 b ^ 24 ⁢ y · b ^ 34 ⁢ x b ^ 24 ⁢ y · b ^ 34 ⁢ x  b ^ 34 ⁢ x  2 } - 1 ⁡ [ ← b ^ 24 ⁢ y → ← b ^ 34 ⁢ x → ] ⁡ [ ↑ C ^ xy ′ ↓ ] . ### Claim 9 (depends on 5) The computer-implemented method of claim 5, further comprising determining a least squares error based at least in part on the optical flow estimate. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the input color vectors each comprise three components respectively associated with a red intensity component, a green intensity component, and a blue intensity component. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the input color vectors each comprises a plurality of components, and wherein at least one of the components is associated with an infrared intensity component, an ultraviolet intensity component, a multi-band intensity component, or a hyperspectral intensity component. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, further comprising determining, by the one or more computing devices, an interpolated color vector associated with the subject pixel. ### Claim 13 A computing system, comprising: one or more processors; and one or more memory devices, the one or more memory devices storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising: obtaining a plurality of input color vectors respectively associated with a plurality of input pixels, each input pixel having a known position within an image; obtaining a color rate of change associated with each input pixel in the image; and determining an optical flow estimate associated with a subject pixel within the image based at least in part on the plurality of input color vectors and the color rates of change associated with the input pixels in the image. ### Claim 14 (depends on 13) The computing system of claim 13, wherein determining an optical flow estimate comprises: accessing an optical flow equation having a measured bilinear formula on a first side of the equation and a predicted bilinear formula on a second side of the equation; and evaluating the optical flow equation to provide a plurality of scalar equations. ### Claim 15 (depends on 14) The computing system of claim 14, wherein evaluating the optical flow equation to provide a plurality of scalar equations comprises determining a differentiated optical flow equation wherein each side of the differentiated optical flow equation is differentiated with respect to time. ### Claim 16 (depends on 15) The computing system of claim 15, wherein the first side of the equation is differentiated such that the input color vectors vary with time and a position of the subject pixel is held constant, and wherein the second side of the equation is differentiated such that the position of the subject pixel varies and the input color vectors are held constant. ### Claim 17 (depends on 13) The computing system of claim 13, the operations further comprising further comprising determining an interpolated color vector associated with the subject pixel. ### Claim 18 One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising: obtaining a plurality of input color vectors respectively associated with a plurality of input pixels, each input pixel having a known position within an image; obtaining a color rate of change associated with each input pixel in the image; and determining an optical flow estimate associated with a subject pixel within the image based at least in part on the plurality of input color vectors and the color rates of change associated with the input pixels in the image. ### Claim 19 (depends on 18) The one or more tangible, non-transitory computer-readable media of claim 18, wherein determining an optical flow estimate comprises: accessing an optical flow equation having a measured bilinear formula on a first side of the equation and a predicted bilinear formula on a second side of the equation; and evaluating the optical flow equation to provide a plurality of scalar equations. ### Claim 20 (depends on 19) The one or more tangible, non-transitory computer-readable media of claim 19, wherein determining an optical flow estimate comprises solving the following equation: [ x ′ y ′ ] ⁢ ⁢ ⁢ ⁢ {  b ^ 24 ⁢ y  2 b ^ 24 ⁢ y · b ^ 34 ⁢ x b ^ 24 ⁢ y · b ^ 34 ⁢ x  b ^ 34 ⁢ x  2 } - 1 ⁡ [ ← b ^ 24 ⁢ y → ← b ^ 34 ⁢ x → ] ⁡ [ ↑ C ^ xy ′ ↓ ] .