Synthesizing an image from a virtual perspective using pixels from a physical imager array weighted based on depth error sensitivity

A non-transitory computer-readable storage medium storing instructions for

• synthesizing an image corresponding to a virtual perspective of a scene, the instructions
• when executed by one or more processors causing the one or more processors to perform
• steps including:
• obtaining a virtual pose of a virtual perspective, a plurality of poses of physical imagers of a camera array, and a geometry of the scene;
• identifying, for a target virtual pixel location of the image, a world point in the scene comprising an intersection point in the scene of a virtual ray extending to the target virtual pixel location;
• detecting, based on the world point, the plurality of poses of the physical imagers, and the geometry of the scene, candidate physical imager pixels corresponding to the world point;
• determining a plurality of error surfaces corresponding to different depth errors in the geometry of the scene;
• determining error surface points at intersections of the error surfaces by rays between the world point and the physical imager pixels;
• projecting rays from the error surface points to the virtual imager to determine virtual error pixel locations for each of the candidate physical imager pixels;
• determining virtual pixel error sensitivities representing a rate of change of errors between the virtual error pixel locations and the target pixel location in response to a change in depth error of the geometry of the scene;
• determining pixel weights associated with each of the candidate physical imager pixels as a function of the virtual pixel error sensitivities;
• and
• determining a value for the target virtual pixel based on a weighted combination of pixel values of the candidate physical imager pixels in accordance with the pixel weights.

An imaging system comprising:

• a camera array comprising a plurality of physical imagers;
• one or more processors;
• and
• a non-transitory computer-readable storage medium storing instructions for synthesizing an image corresponding to a virtual perspective of a scene, the instructions when executed by one or more processors causing the one or more processors to perform steps including:
• obtaining a virtual pose of a virtual perspective, a plurality of poses of the physical imagers of the camera array, and a geometry of the scene;
• identifying, for a target virtual pixel location of the image, a world point in the scene comprising an intersection point in the scene of a virtual ray extending to the target virtual pixel location;
• detecting, based on the world point, the plurality of poses of the physical imagers, and the geometry of the scene, candidate physical imager pixels corresponding to the world point;
• determining a plurality of error surfaces corresponding to different depth errors in the geometry of the scene;
• determining error surface points at intersections of the error surfaces by rays between the world point and the physical imager pixels;
• projecting rays from the error surface points to the virtual imager to determine virtual error pixel locations for each of the candidate physical imager pixels;
• determining virtual pixel error sensitivities representing a rate of change of errors between the virtual error pixel locations and the target pixel location in response to a change in depth error of the geometry of the scene;
• determining pixel weights associated with each of the candidate physical imager pixels as a function of the virtual pixel error sensitivities;
• and
• determining a value for the target virtual pixel based on a weighted combination of pixel values of the candidate physical imager pixels in accordance with the pixel weights.