Systems and methods for generating a 3D world model using velocity data of a vehicle

A system for generating a 3D world model, comprising:

• a hardware processor;
• a memory coupled to the at least one processor;
• an image receiver configured to receive, from an image capture device mounted on a movable entity, a sequence of images including a first image captured at a first time and a plurality of second images captured after the first time, the first image of the sequence of images including a first plurality of pixels having respective parameter values, the plurality of second images each including a respective second plurality of pixels having respective parameter values;
• a motion data receiver configured to receive motion data of the movable entity, the motion data including at least one of velocity data of the movable entity or acceleration data of the movable entity;
• a pixel movement tracker configured to track movement of at least a subset of the first plurality of pixels of the first image across the sequence of images, the subset corresponding to an object represented by the first plurality of pixels;
• a depth value estimator configured to estimate, responsive to the tracked subset of the first plurality of pixels and using the received at least one of the velocity data or the acceleration data, a depth value for each pixel of the subset of the first plurality of pixels;
• a keyframe generator configured to generate, using the estimated depth value for each pixel of the subset of the first plurality of pixels, a keyframe to populate a point cloud including a plurality of points, each point corresponding to a respective pixel of the subset of the first plurality of pixels;
• the keyframe generator configured to publish a depth map including the estimated depth values for each pixel of the subset of the first plurality of pixels of the first image being tracked across the sequence of images, the depth map including, for each tracked pixel, absolute distance information calculated based on a distance the tracked pixel moved across the sequence of images and the at least one of the velocity data or the acceleration data, the absolute distance information being between the movable entity and the object;
• and
• a 3D world model generator configured to generate a 3D world model using the generated keyframe.

A non-transitory computer readable storage medium including computer-executable instructions stored thereon, which when executed by a computer, causes the computer to:

• receive, from an image capture device mounted on a movable entity, a sequence of images including a first image captured at a first time and a plurality of second images captured after the first time, the first image of the sequence of images including a first plurality of pixels having respective parameter values, the plurality of second images each including a respective second plurality of pixels having respective parameter values;
• receive motion data of the movable entity, the motion data including at least one of velocity data of the movable entity or acceleration data of the movable entity;
• track movement of at least a subset of the first plurality of pixels of the first image across the sequence of images, the subset corresponding to an object represented by the first plurality of pixels;
• estimate, responsive to tracking the subset of the first plurality of pixels and using the received at least one of the velocity data or the acceleration data, a depth value for each pixel of the subset of the first plurality of pixels;
• publish a depth map including the estimated depth values for each pixel of the subset of the first plurality of pixels of the first image being tracked across the sequence of images, the depth map including, for each tracked pixel, absolute distance information calculated based on a distance the tracked pixel moved across the sequence of images and the at least one of the velocity data or the acceleration data;
• generate, using the estimated depth value for each pixel of the subset of the first plurality of pixels, a keyframe to populate a point cloud including a plurality of points, each point corresponding to a respective pixel of the subset of the first plurality of pixels;
• and
• generate a 3D world model using the generated keyframe.