Back to results
US 10,706,582 B2Granted
Real-time monocular structure from motion
AssigneeNEC Corporation
InventorsManmohan Chandraker, Shiyu Song
Life of this patent
Estimating the term…
Abstract
Systems and methods are described for multithreaded navigation assistance by acquired with a single camera on-board a vehicle, using 2D-3D correspondences for continuous pose estimation, and combining the pose estimation with 2D-2D epipolar search to replenish 3D points.
Claims18
A system for autonomous vehicular navigation using a multithreaded monocular structure from motion (SFM) architecture, comprising:
- a single camera positioned on-board a vehicle;
- and
- a multithreaded processor coupled to the single camera and a non-transitory computer-readable storage medium, the multithreaded processor being included in the multithreaded SFM architecture, and being configured for:
- acquiring images using the single camera positioned on-board the vehicle;
- estimating camera motion using monocular SFM by performing continuous camera pose estimation of 2D-3D correspondences;
- detecting a local planarity of a road and correcting for scale drift using the monocular SFM based on the camera pose estimation;
- determining visual odometry correspondences and replenishing 3D points of the images by combining the pose estimation with a 2D-2D multi-threaded per-frame epipolar search, the per-frame epipolar search continuously generating, for each frame, redundantly validated 3D points persistent across comparatively long tracks, the determining visual odometry further comprising:
- parallelizing the epipolar search across a plurality of threads;
- validating the 3D points among intermediate points, performing a local bundle adjustment, and retaining the validated 3D points for insertion of new 3D points at a keyframe in a main thread to frontload the validated 3D points for the pose estimation;
- performing, in real-time, the pose estimation at each of a plurality of frames using only the validated 3D points to reduce processing requirements and increase processing speed of the pose estimation;
- executing a real-time global bundle adjustment in a thread-safe architecture in parallel with the real-time pose estimation;
- determining an optimized planar homography mapping between a road in two frames based on one or more determined ground estimation cues, the cues including triangulated sparse 3D points, inter-frame dense stereo matching, and a displacement computed based on a distance from a vanishing point;
- and
- estimating per-frame relative importances of the cues by computing observation covariances for each of the cues and performing ground plane estimation by combining the triangulated sparse 3D points and cues using a Kalman filter;
- and
- autonomously controlling driving functions of the vehicle based on the determined visual odometry and the local planarity of the road.
13 more elements
A method for vehicular navigation using a multithreaded monocular structure from motion (SFM) architecture, comprising:
- autonomously navigating a vehicle using a multithreaded processor coupled to a non-transitory computer-readable storage medium, the multithreaded processor being included in the multithreaded SFM architecture, and being configured for:
- acquiring images using a single camera positioned on-board a vehicle;
- estimating camera motion using monocular SFM by performing continuous camera pose estimation of 2D-3D correspondences;
- detecting a local planarity of a road and correcting for scale drift using the monocular SFM based on the camera pose estimation;
- determining visual odometry correspondences and replenishing 3D points of the images by combining the pose estimation with a 2D-2D multi-threaded per-frame epipolar search, the per-frame epipolar search continuously generating, for each frame, redundantly validated 3D points persistent across comparatively long tracks, the determining visual odometry further comprising:
- parallelizing the epipolar search across a plurality of threads;
- validating the 3D points among intermediate points, performing a local bundle adjustment, and retaining the validated 3D points for insertion of new 3D points at a keyframe in a main thread to frontload the validated 3D points for the pose estimation;
- performing, in real-time, the pose estimation at each of a plurality of frames using only the validated 3D points to reduce processing requirements and increase processing speed of the pose estimation;
- executing a real-time global bundle adjustment in a thread-safe architecture in parallel with the real-time pose estimation;
- determining an optimized planar homography mapping between a road in two frames based on one or more determined ground estimation cues, the cues including triangulated sparse 3D points, inter-frame dense stereo matching, and a displacement computed based on a distance from a vanishing point;
- and
- estimating per-frame relative importances of the cues by computing observation covariances for each of the cues and performing ground plane estimation by combining the triangulated sparse 3D points and cues using a Kalman filter;
- and
- autonomously controlling driving functions of the vehicle based on the determined visual odometry and the local planarity of the road.
11 more elements
Dependencies are inferred from the claim text.