Generating stereoscopic pairs of images from a single lens camera

A system for generating stereo pairs for virtual reality, the system comprising:

• a single lens camera,
• a processor, and
• memory storing one or more programs configured for execution by the processor, the one or more programs comprising instructions for:
• obtaining a monocular sequence of images using the single lens camera during a capture mode, the sequence of images being capture along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including an object;
• and
• generating stereo pairs for virtual reality using the sequence of images, the stereo pairs being generated for one or more points along the camera translation, wherein generating stereo pairs includes:
• selecting frames for each stereo pair based on a spatial baseline, each selected frame having a virtual or captured image;
• interpolating virtual images in between captured images in the sequence of images, wherein each virtual image is interpolated using infinite smoothing including:
• computing a plurality of transformations using keypoint correspondences between two captured images;
• determining an optimal subset of transformations from the plurality of transformations by selecting an optimal transformation candidate for each pixel of a first captured image, wherein selecting an optimal transformation candidate for each pixel is processed as a separate thread of a graphics processing unit (GPU) such that the processing time for each pixel is bounded by O(1) rather than O(n), wherein the plurality of transformations includes a subset of transformation candidates, wherein determining the optimal subset of transformations includes applying each transformation candidate in the subset of transformation candidates to each pixel in the first captured image, thereby generating a plurality of first interpolated pixel candidates, and to a corresponding pixel in the second captured image, thereby generating a plurality of second interpolated pixel candidates, and then comparing the plurality of first interpolated pixel candidates to the corresponding plurality of second interpolated pixel candidates to select an optimal transformation candidate having the smallest corresponding pixel candidate difference;
• calculating transformation parameters corresponding to the optimal subset of transformations;
• and
• interpolating the virtual image between the first and second captured images using the calculated transformation parameters;
• correcting selected frames by rotating the virtual or captured images in the selected frames, wherein rotating the virtual or captured images in the selected frames includes using Inertial Measurement Unit (IMU) and image data to identify regions that belong to the foreground and regions that belong to the background;
• and
• rendering the selected frames by assigning each virtual or captured image in the selected frames to left and right eyes.

A non-transitory computer readable medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

• obtaining a monocular sequence of images using the single lens camera during a capture mode, the sequence of images being capture along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including an object;
• and
• generating stereo pairs for virtual reality using the sequence of images, the stereo pairs being generated for one or more points along the camera translation, wherein generating stereo pairs includes:
• selecting frames for each stereo pair based on a spatial baseline, each selected frame having a virtual or captured image;
• interpolating virtual images in between captured images in the sequence of images, wherein each virtual image is interpolated using infinite smoothing including:
• computing a plurality of transformations using keypoint correspondences between two captured images;
• determining an optimal subset of transformations from the plurality of transformations by selecting an optimal transformation candidate for each pixel of a first captured image, wherein selecting an optimal transformation candidate for each pixel is processed as a separate thread of a graphics processing unit (GPU) such that the processing time for each pixel is bounded by O(1) rather than O(n), wherein the plurality of transformations includes a subset of transformation candidates, wherein determining the optimal subset of transformations includes applying each transformation candidate in the subset of transformation candidates to each pixel in the first captured image, thereby generating a plurality of first interpolated pixel candidates, and to a corresponding pixel in the second captured image, thereby generating a plurality of second interpolated pixel candidates, and then comparing the plurality of first interpolated pixel candidates to the corresponding plurality of second interpolated pixel candidates to select an optimal transformation candidate having the smallest corresponding pixel candidate difference;
• calculating transformation parameters corresponding to the optimal subset of transformations;
• and
• interpolating the virtual image between the first and second captured images using the calculated transformation parameters;
• correcting selected frames by rotating the virtual or captured images in the selected frames, wherein rotating the virtual or captured images in the selected frames includes using Inertial Measurement Unit (IMU) and image data to identify regions that belong to the foreground and regions that belong to the background;
• and
• rendering the selected frames by assigning each virtual or captured image in the selected frames to left and right eyes.