Light field capture and rendering for head-mounted displays

A system for imaging a physical scene, the system comprising:

• a camera assembly including a plurality of cameras configured to capture image data of the physical scene;
• a computing device communicatively coupled to the camera assembly, wherein the computing device has a memory containing computer-executable instructions and a processor for executing the computer-executable instructions contained in the memory, and wherein the computer-executable instructions, when executed by the processor, cause the processor to—
• receive the image data from the camera assembly;
• render a virtual camera;
• and
• generate an image from the virtual camera by determining a value for each of a plurality of virtual pixels in the image from the virtual camera, wherein determining the value for each of the plurality of virtual pixels comprises—
• determining a point in the physical scene corresponding to the virtual pixel along a virtual ray;
• determining a first pixel in the image data from a first one of the cameras corresponding to the point in the physical scene along a first ray;
• determining a second pixel in the image data from a second one of the cameras corresponding to the point in the physical scene along a second ray;
• and
• weighting a value of the first pixel and a value of the second pixel to determine a value of the virtual pixel based on (i) a first angular distance between the first ray and the virtual ray, (ii) a second angular distance between the second ray and the virtual ray, and (iii) a third angular distance between the virtual ray and a surface normal at the point in the physical scene;
• and
• a display communicatively coupled to the computing device, wherein the display is separate and spaced apart from the camera assembly, and wherein the display is configured to receive the image from the virtual camera from the computing device and display the image from the virtual camera.