Systems, methods, and media for determining object motion in three dimensions from light field image data

A method for three dimensional motion estimation, the method comprising:

• causing an image sensor to capture at least a first sub-aperture image of a scene and a second sub-aperture image of the scene at a first time, wherein the first sub-aperture image and the second sub-aperture image are formed on the image sensor via optics configured to create a plurality of images of the scene, wherein the optics comprises:
• focusing optics disposed between the scene and the image sensor;
• and
• an array comprising a plurality of microlenses disposed between the image sensor and the focusing optics, each of the plurality of microlenses projecting one of the plurality of images of the scene onto the image sensor as a sub-aperture image, wherein
• the first sub-aperture image is projected by a first microlens of the plurality of microlenses, and the second sub-aperture image is projected by a second microlens of the plurality of microlenses that is adjacent to the first microlens such that at least a portion of the first sub-aperture image and the second sub-aperture image represent the same portion of the scene;
• generating a first light field using a first plurality of sub-aperture images of the scene including the first sub-aperture image and the second sub-aperture image;
• causing the image sensor to capture at least a third sub-aperture image of the scene at a second time, wherein the second time is subsequent to the first time and the second image is formed on the image sensor via the optics configured to create a plurality of images of the scene;
• generating a second light field using a second plurality of images of the scene including the third sub-aperture image;
• calculating light field gradients using information from the first light field and information from the second light field;
• and
• calculating, for each point in the scene, three dimensional motion using the light field gradients by applying a constraint to the motion in the scene.

A non-transitory computer readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for three dimensional motion estimation, the method comprising:

• causing an image sensor to capture at least a first sub-aperture image of a scene and a second-sub-aperture image of the scene at a first time, wherein the first sub-aperture image and the second sub-aperture image are formed on the image sensor via optics configured to create a plurality of images of the scene, wherein the optics comprises:
• focusing optics disposed between the scene and the image sensor;
• and
• an array comprising a plurality of microlenses disposed between the image sensor and the focusing optics, each of the plurality of microlenses projecting one of the plurality of images of the scene onto the image sensor as a sub-aperture image, wherein
• the first sub-aperture image is projected by a first microlens of the plurality of microlenses, and the second sub-aperture image is projected by a second microlens of the plurality of microlenses that is adjacent to the first microlens such that at least a portion of the first sub-aperture image and the second sub-aperture image represent the same portion of the scene;
• generating a first light field using a first plurality of sub-aperture images of the scene including the first sub-aperture image and the second sub-aperture image;
• causing the image sensor to capture at least a third sub-aperture image of the scene at a second time, wherein the second time is subsequent to the first time and the first image is formed on the image sensor via optics configured to create a plurality of images of the scene;
• generating a second light field using a second plurality of images of the scene including the third sub-aperture image;
• calculating light field gradients using information from the first light field and information from the second light field;
• and
• calculating, for each point in the scene, three dimensional motion using the light field gradients by applying a constraint to the motion in the scene.