Generating virtual images based on captured image data

A system for generating a virtual view of a virtual camera based on an input image, the system comprising:

• a capturing device including a physical camera and a depth sensor, the capturing device configured for capturing the input image;
• a controller configured to:
• determine an actual pose of the capturing device;
• determine a desired pose of the virtual camera for showing the virtual view;
• define an epipolar geometry between the actual pose of the capturing device and the desired pose of the virtual camera;
• generate a virtual image depicting objects within the input image according to the desired pose of the virtual camera for the virtual camera based on an epipolar relation between the actual pose of the capturing device, the input image, and the desired pose of the virtual camera, wherein at least one pixel corresponding to the input image is selected based on a minimum matched angle index corresponding to the actual pose of the capturing device and the desired pose of the virtual camera;
• monotonize at least one minimum matched angle index based on a comparison of the at least one minimum matched angle index with an adjacent minimum matched angle index;
• compare the adjacent minimum matched angle index to the at least one minimum matched angle index;
• determine whether a difference between the adjacent minimum matched angle index and the at least one minimum matched angle index is greater than a difference threshold;
• and
• remove the at least one minimum matched angle index when the difference is greater than the difference threshold.

A vehicle, comprising a system for generating a virtual view of a virtual camera based on an input image, the system comprising:

• a capturing device including a physical camera and a depth sensor, the capturing device configured for capturing the input image;
• a controller, configured to:
• determine an actual pose of the capturing device;
• determine a desired pose of the virtual camera for showing the virtual view;
• define an epipolar geometry between the actual pose of the capturing device and the desired pose of the virtual camera;
• generate a virtual image depicting objects within the input image according to the desired pose of the virtual camera for the virtual camera based on an epipolar relation between the actual pose of the capturing device, the input image, and the desired pose of the virtual camera, wherein at least one pixel corresponding to the input image is selected based on a minimum matched angle index corresponding to the actual pose of the capturing device and the desired pose of the virtual camera;
• monotonize at least one minimum matched angle index based on a comparison of the at least one minimum matched angle index with an adjacent minimum matched angle index;
• compare the adjacent minimum matched angle index to the at least one minimum matched angle index;
• determine whether a difference between the adjacent minimum matched angle index and the at least one minimum matched angle index is greater than a difference threshold;
• and
• remove the at least one minimum matched angle index when the difference is greater than the difference threshold.