3D light field cameras, images and files, and methods of using, operating, processing and viewing same

A method of varying a three dimensional (3D) effect in a stereo image of a scene, the method comprising:

• a) at a light field data acquisition device, acquiring light field data representing light rays from the scene via one acquisition;
• b) in a processor, identifying a first set of at least two positions, wherein each position in the first set corresponds to a virtual aperture within a physical aperture of the light field data acquisition device and further corresponds to a subset of light rays of the acquired light field data, each subset of light rays coming from a direction, wherein a size of each virtual aperture is less than a size of the physical aperture, the virtual apertures comprising a first virtual aperture at a first position within the physical aperture and a second virtual aperture at a second position within the physical aperture, and wherein:
• the virtual aperture corresponding to each position in the first set is different from the virtual aperture corresponding to any other position in the first set;
• and
• the subset of light rays corresponding to each position in the first set comes from a direction that is different than the direction of the subset of light rays corresponding to any other position in the first set;
• c) in the processor, for each of at least two of the subsets of light rays corresponding to positions in the first set, generating image data of the scene using light field data for the subset of light rays associated with the corresponding virtual aperture, wherein the image data represents an image of the scene from a perspective corresponding to the position of the virtual aperture;
• d) in the processor, generating a first 3D stereo image of the scene using the image data generated from at least two of the subsets of light rays corresponding to positions in the first set, wherein the first 3D stereo image has a first 3D effect;
• e) in the processor, identifying a second set of at least two positions, wherein each position in the second set corresponds to a virtual aperture within the physical aperture of the light field data acquisition device and further corresponds to a subset of light rays of the acquired light field data, each subset of light rays coming from a direction, and wherein at least one of the positions within the second set is different from any of the positions identified in step b), and wherein:
• the virtual aperture corresponding to each position in the second set is different from the virtual aperture corresponding to any other position in the second set;
• and
• the subset of light rays corresponding to each position in the second set comes from a direction that is different than the direction of the subset of light rays corresponding to any other position in the second set;
• f) in the processor, for each subset of light rays associated with positions in the second set for which image data was not generated in step c), generating image data of the scene using light field data for the subset of light rays associated with the corresponding virtual aperture, wherein the image data represents an image of the scene from a perspective corresponding to the position of the virtual aperture;
• g) in the processor, generating a second 3D stereo image of the scene using the image data generated from at least two of the subsets of light rays corresponding to positions in the second set, wherein the second 3D stereo image has a second 3D effect that is different than the first 3D effect;
• and
• h) at an output device, outputting the first and second 3D stereo images.

A light field acquisition device comprising:

• a photosensor array, configured to acquire light field data representing light rays from a scene via one acquisition, wherein the photosensor array comprises a plurality of photosensors;
• processing circuitry, coupled to the photosensor array, configured to identify a first set of at least two positions associated with the light field data acquisition device, wherein each position in the first set corresponds to a virtual aperture and further corresponds to a subset of light rays of the acquired light field data, each subset of light rays coming from a direction, and wherein:
• the virtual aperture corresponding to each position in the first set is different from the virtual aperture corresponding to any other position in the first set;
• and
• the subset of light rays corresponding to each position in the first set comes from a direction that is different than the direction of the subset of light rays corresponding to any other position in the first set;
• and
• a microlens array, configured to direct light rays onto the photosensor array including a subset of light rays corresponding to each identified position, wherein the microlens array comprises a plurality of microlenses and wherein, for each identified position in the first set, each microlens of the microlens array is associated with a subset of photosensors associated with that identified position;
• wherein the processing circuitry is further configured to:
• for each of at least two of the subsets of light rays corresponding to positions in the first set, generate image data of the scene using light field data for the subset of light rays associated with the corresponding virtual aperture, the light field data being acquired by the corresponding subset of photosensors associated with each microlens of the microlens array, wherein the image data is representative of an image of the scene from a perspective corresponding to the position of the virtual aperture;
• generate a first 3D stereo image of the scene using the image data generated from the light field data from at least two of the subsets of photosensors corresponding to positions in the first set, wherein the 3D stereo image has a first 3D effect;
• identify a second set of at least two positions associated with the light field data acquisition device, wherein each position in the second set corresponds to a virtual aperture and further corresponds to a subset of light rays of the acquired light field data, each subset of light rays coming from a direction, and wherein at least one of the positions within the second set is different from any of the positions identified previously by the processing circuitry, and wherein:
• the virtual aperture corresponding to each position in the second set is different from the virtual aperture corresponding to any other position in the second set;
• the subset of light rays corresponding to each position in the second set comes from a direction that is different than the direction of the subset of light rays corresponding to any other position in the second set;
• and
• for each identified position in the second set, each microlens of the microlens array is associated with a subset of photosensors associated with that identified position;
• for each subset of light rays associated with positions in the second set for which image data was not previously generated, generate image data of the scene using light field data for the subset of light rays associated with the corresponding virtual aperture, the light field data being acquired by the corresponding subset of photosensors associated with each microlens of the microlens array, wherein the image data is representative of an image of the scene from a perspective corresponding to the position of the virtual aperture;
• generate a second 3D stereo image of the scene using the image data generated from the light field data from at least two of the subsets of photosensors corresponding to positions in the second set, wherein the second 3D stereo image has a second 3D effect that is different than the first 3D effect;
• the device further comprising an output device, coupled to the processing circuitry, configured to output the first and second 3D stereo images.

A light field acquisition device comprising:

• a photosensor array, configured to acquire light field data representing light rays from a scene via one acquisition, wherein the photosensor array comprises a plurality of photosensors;
• a physical aperture;
• a microlens array, configured to direct light onto the photosensor array, wherein the microlens array comprises a plurality of microlenses and each microlens of the microlens array directs light passing through the physical aperture to a plurality of associated photosensors;
• processing circuitry, coupled to the photosensor array, configured to:
• identify a first set of at least two positions associated with the light field data acquisition device, wherein each position in the first set corresponds to a virtual aperture within the physical aperture and further corresponds to a subset of light rays of the acquired light field data, each subset of light rays coming from a different direction, and wherein:
• the virtual aperture corresponding to each position in the first set is different from the virtual aperture corresponding to any other position in the first set;
• and
• the subset of light rays corresponding to each position in the first set comes from a direction that is different than the direction of the subset of light rays corresponding to any other position in the first set;
• and
• for each of the subsets of light rays corresponding to positions in the first set, generate image data of the scene using light field data which correlates to the corresponding virtual aperture, wherein the image data represents an image of the scene from a perspective corresponding to the position of the corresponding virtual aperture;
• generate a first 3D stereo image of the scene using the image data generated from the light field data correlating to at least two of the virtual apertures, wherein the 3D stereo image has a first 3D effect;
• identify a second set of at least two positions associated with the light field data acquisition device, wherein each position in the first set corresponds to a virtual aperture within the physical aperture and further corresponds to a subset of light rays of the acquired light field data, each subset of light rays coming from a direction, and wherein at least one of the positions within the second set is different from any of the positions identified previously by the processing circuitry, and wherein:
• the virtual aperture corresponding to each position in the second set is different from the virtual aperture corresponding to any other position in the second set;
• and
• the subset of light rays corresponding to each position in the second set comes from a direction that is different than the direction of the subset of light rays corresponding to any other position in the second set;
• for each of the subsets of light rays for which image data was not previously generated, generate image data of the scene using light field data which correlates to the corresponding virtual aperture, wherein the image data represents an image of the scene from a perspective corresponding to the position of the corresponding virtual aperture;
• and
• generate a second 3D stereo image of the scene using the image data generated from the light field data correlating to at least two of the virtual apertures, wherein the second 3D stereo image has a second 3D effect different than the first 3D effect;
• and
• an output device, coupled to the processing circuitry, configured to output the first and second 3D stereo images.