Method and system for concurrent reconstruction of dynamic and static objects

An imaging system for localization and mapping of a scene including static and dynamic objects, wherein a sensor acquires sensor data while in motion, such that the data includes a sequence of frames and a memory stores a static map of one or multiple static objects in the scene and an object map of each dynamic object in the scene, wherein the static map includes a first set of stored landmarks, and the object map includes a set of stored segments, such that a descriptor of each of the stored set of segments is a function of multiple measurements forming the set of stored segments, the imaging system comprising:

• a processor in communication with the sensor and the memory, is configured to
• obtain a set of segments from each frame of the acquired sequence of frames, and register the set of segments from each frame which are observed from different angles by computing of a similarity between newly observed and the set of stored segments, wherein each frame of the acquired sequence of frames includes a set of keypoints representing intensity including color and depth measurements of the scene;
• employ a localizer to register at least some keypoints from the set of keypoints with corresponding landmarks in the stored static map using frame-based registration and to register at least some segments from the set of segments in each frame of the acquired sequence of frames with segments in the stored object map using a segment-based registration, wherein the landmarks correspond to discriminative points on an object in the scene, and wherein the segment-based registration performs appearance matching of segments in each frame of the acquired sequence of frames with the segments in the object map and geometric verification between matching segments, wherein the set of keypoints does not require hand-crafted point cloud filters because the frame-based registration rejects noise automatically;
• and
• a mapper to update each stored object map of each dynamic object in the scene with the at least some registered keypoints forming each segment and keypoints registered with the corresponding stored object map according to the segment-based registration, and to update the stored static map with the remaining keypoints in each frame of the acquired sequence of frames using the keypoints registered with the stored static map, wherein the reconstructed geometry is composed of unaltered intensity and depth measurements of the scene acquired from the sequence of frames.

An imaging system for localization and mapping of a scene including static and dynamic objects, comprising:

• a sensor to acquire sensor data while in motion or stationary, such that the sensor data includes a sequence of frames;
• an input interface receives data;
• a memory in communication with the input interface, to store the received data that includes a static map of one or multiple static objects in the scene and an object map of each dynamic object in the scene, wherein the static map includes a first set of landmarks, and the object map includes a second set of stored landmarks and a set of stored segments, such that a descriptor of each of the stored set of segments is a function of multiple measurements forming the set of stored segments, and wherein the measurements of the frame further include at least three-dimensional (3D) planes;
• a processor in communication with the sensor, the memory and the input interface, is configured to
• obtain a set of segments from each frame of the acquired sequence of frames, and register the set of segments from each frame which are observed from different angles by computing of a similarity between newly observed and the set of stored segments, wherein each frame of the acquired sequence of frames includes a set of keypoints representing intensity including color and depth measurements of the scene;
• employ a localizer to register at least some keypoints from the set of keypoints with corresponding landmarks in the stored static map using frame-based registration and to register at least some segments from the set of segments in each frame of the acquired sequence of frames with segments in the stored object map using a segment-based registration, such that the segment-based registration includes (a) matching the segment in each frame of the acquired sequence of frames with the segments in the stored object map using an appearance similarity search, (b) matching the measurements of the segment with landmarks of matching segments from the object map, and (c) identifying a transformation between matching segment pairs following random sample consensus (RANSAC) framework function, wherein the set of keypoints does not require hand-crafted point cloud filters because the frame-based registration rejects noise automatically;
• and
• a mapper to update each stored object map of each dynamic object in the scene with the at least some registered keypoints forming each segment and keypoints registered with the corresponding stored object map according to the segment-based registration, and to update the stored static map with the remaining keypoints in each frame of the acquired sequence of frames using the keypoints registered with the stored static map, wherein the reconstructed geometry is composed of unaltered intensity and depth measurements of the scene acquired from the sequence of frames.