# US 2019/0318486 A1 · Systems, Methods, and Media for Determining Object Motion in Three Dimensions from Light Field Image Data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318486 A1 - Document ID: US20190318486A1 - Status: Published application - Kind: A1 - Application number: 15/951,131 - Filed: 2018-04-11 - Published: 2019-10-17 - Earliest claimed: 2018-04-11 - Assignees: Wisconsin Alumni Research Foundation - Applicants: Wisconsin Alumni Research Foundation - Inventors: Mohit Gupta; Sizhuo Ma; Brandon Smith - Web page: https://patentgrep.com/patent/US20190318486A1 ## CPC - G06T 7/285: using a sequence of stereo image pairs - G06T 7/269: using gradient-based methods - H04N 13/232: using fly-eye lenses, e.g. arrangements of circular lenses - H04N 13/243: using three or more two-dimensional [2D] image sensors - G06T 2207/10016: Video; Image sequence - G06T 2207/10052: Images from lightfield camera - H04N 2013/0085: {Motion estimation from stereoscopic image signals} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract In accordance with some embodiments, systems, methods and media for determining object motion in three dimensions using light field image data are provided. In some embodiments, a system for three dimensional motion estimation is provided, comprising: an image sensor; optics that create many images of a scene; and a hardware processor configured to: cause the image sensor to capture a first plurality of images; generate a first light field; cause the image sensor to capture a second plurality of images at a second time; generate a second light field; calculate light field gradients using the first light field and second light field; and calculate, for each point in the scene, three dimensional motion using the light field gradients by applying a constraint to the motion in the scene. ## Claims ### Claim 1 A system for three dimensional motion estimation, the system comprising: an image sensor; optics configured to create a plurality of images of a scene; and one or more hardware processors that are configured to: cause the image sensor to capture at least a first image of the scene at a first time; generate a first light field using a first plurality of images of the scene including the first image; cause the image sensor to capture at least a second image of the scene at a second time, wherein the second time is subsequent to the first time; generate a second light field using the second plurality of images of the scene including the second image; calculate light field gradients using information from the first light field and information from the second light field; and calculate, for each point in the scene, three dimensional motion using the light field gradients by applying a constraint to the motion in the scene. ### Claim 2 (depends on 1) The system of claim 1, wherein the optics comprises an array of microlenses disposed between the image sensor and focusing optics, and the plurality of images of the scene are sub-aperture images projected by the microlenses onto the image sensor. ### Claim 3 (depends on 1) The system of claim 1, wherein the one or more hardware processors are further configured to: generate, for each of a plurality of rays in the first light field, a first matrix A of light field gradients that includes light field gradients corresponding to rays in a local neighborhood of the ray, wherein A is an n×3 matrix where n is the number of rays in the local neighborhood; generate, for each of the plurality of rays in the first light field, a second matrix b of temporal light field derivatives that includes the additive inverses of temporal light field derivatives corresponding to the rays in the local neighborhood of the ray, wherein b is a n×1 matrix; calculate, for each of the plurality of rays, a three element velocity vector V, such that V=(ATA)−1ATb, where the local neighborhood of rays are assumed to have the same velocity; and calculate motion in the scene based on the velocity vector V calculated for each of the plurality of rays. ### Claim 4 (depends on 1) The system of claim 1, wherein the one or more hardware processors are further configured to calculate, for each point in the scene, three dimensional motion by determining a k×3 velocity matrix V by finding a velocity vector V that minimizes the relationship: (L0(x)−L1(w(x V)))2, where L0 is the first light field, and L1 is the second light field, and w(x, V) is a warp function that is represented as w  ( x , V ) = ( x + V X - u Γ  V Z , y + V Y - v Γ  V Z , u , v ) , V X , V Y , and   V Z are the velocities in the X, Y, and Z directions respectively, x has coordinates (x, y, u, v), and Γ is the depth in the Z direction of the plane used to define the light field. ### Claim 5 (depends on 1) The system of claim 1, wherein the one or more hardware processors are further configured to find velocity vectors V for a plurality of light field coordinates that minimize a global functional E(V) that includes a smoothness term that penalizes departures from smoothness. ### Claim 6 (depends on 5) The system of claim 5, wherein the one or more hardware processors are further configured to find the velocity vectors V for each of the plurality of light field coordinates that minimize the global functional E(V) by solving a set of Euler-Lagrange equations using successive over-relaxation. ### Claim 7 (depends on 5) The system of claim 5, wherein the smoothness term includes a quadratic penalty functions λ and λZ, where λZ<λ. ### Claim 8 (depends on 5) The system of claim 5, wherein the smoothness term includes a generalized Charbonier function ρ(x)=(x2+ϵ2)a. ### Claim 9 (depends on 1) The system of claim 1, wherein the one or more hardware processors are further configured to find a velocity V that minimizes a functional that includes: a local term (ED(V)) that aggregates information from a plurality of rays emitted from the same scene point S that were detected in the first light field; and a smoothness term (ES(V)) that penalizes departures from smoothness. ### Claim 10 A method for three dimensional motion estimation, the method comprising: causing an image sensor to capture at least a first image of a scene at a first time, wherein the first image is formed on the image sensor via optics configured to create a plurality of images of the scene; generating a first light field using a first plurality of images of the scene including the first image; causing the image sensor to capture at least a second image of the scene at a second time, wherein the second time is subsequent to the first time and the second image is formed on the image sensor via the optics configured to create a plurality of images of the scene; generating a second light field using a second plurality of images of the scene including the second image; calculating light field gradients using information from the first light field and information from the second light field; and calculating, for each point in the scene, three dimensional motion using the light field gradients by applying a constraint to the motion in the scene. ### Claim 11 (depends on 10) The method of claim 10, wherein the optics comprises an array of microlenses disposed between the image sensor and focusing optics, and the plurality of images of the scene are sub-aperture images projected by the microlenses onto the image sensor. ### Claim 12 (depends on 10) The method of claim 10, further comprising: generating, for each of a plurality of rays in the first light field, a first matrix A of light field gradients that includes light field gradients corresponding to rays in a local neighborhood of the ray, wherein A is an n×3 matrix where n is the number of rays in the local neighborhood; generating, for each of the plurality of rays in the first light field, a second matrix b of temporal light field derivatives that includes the additive inverses of temporal light field derivatives corresponding to the rays in the local neighborhood of the ray, wherein b is a n×1 matrix; calculating, for each of the plurality of rays, a three element velocity vector V, such that V=(ATA)−1ATb, where the local neighborhood of rays are assumed to have the same velocity; and calculating motion in the scene based on the velocity vector V calculated for each of the plurality of rays. ### Claim 13 (depends on 10) The method of claim 10, further comprising calculating, for each point in the scene, three dimensional motion by determining a k×3 velocity matrix V by finding a velocity vector V that minimizes the relationship: (L0(x)−L1(w(x, V)))2, where L0 is the first light field, and L1 is the second light field, and w(x, V) is a warp function that is represented as w  ( x , V ) = ( x + V X - u Γ  V Z , y + V Y - v Γ  V Z , u , v ) , V X , V Y , and   V Z are the velocities in the X, Y, and Z directions respectively, x has coordinates (x, y, u, v), and r is the depth in the Z direction of the plane used to define the light field. ### Claim 14 (depends on 10) The method of claim 10, further comprising finding velocity vectors V for a plurality of light field coordinates that minimize a global functional E(V) that includes a smoothness term that penalizes departures from smoothness. ### Claim 15 (depends on 14) The method of claim 14, further comprising finding the velocity vectors V for the plurality of light field coordinates that minimize the global functional E(V) by solving a set of Euler-Lagrange equations using successive over-relaxation. ### Claim 16 (depends on 14) The method of claim 14, wherein the smoothness term includes a quadratic penalty functions λ and λZ, where λZ<λ. ### Claim 17 (depends on 14) The method of claim 14, wherein the smoothness term includes a generalized Charbonier function ρ(x)=(x2+ϵ2)a. ### Claim 18 (depends on 10) The method of claim 10, further comprising finding a velocity V that minimizes a functional that includes: a local term (ED(V)) that aggregates information from a plurality of rays emitted from the same scene point S that were detected in the first light field; and a smoothness term (ES(V)) that penalizes departures from smoothness. ### Claim 19 A non-transitory computer readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for three dimensional motion estimation, the method comprising: causing an image sensor to capture at least a first image of a scene at a first time, wherein the first image is formed on the image sensor via optics configured to create a plurality of images of the scene; generating a first light field using a first plurality of images of the scene including the first image; causing the image sensor to capture at least a second image of the scene at a second time, wherein the second time is subsequent to the first time and the first image is formed on the image sensor via optics configured to create a plurality of images of the scene; generating a second light field using a second plurality of images of the scene including the second image; calculating light field gradients using information from the first light field and information from the second light field; and calculating, for each point in the scene, three dimensional motion using the light field gradients by applying a constraint to the motion in the scene. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the optics comprises an array of microlenses disposed between the image sensor and focusing optics, and the plurality of images of the scene are sub-aperture images projected by the microlenses onto the image sensor.