Sparse simultaneous localization and matching with unified tracking

A computerized method of tracking a pose of one or more objects represented in a scene, the method comprising:

• a) capturing, by a sensor, a plurality of scans of one or more objects in a scene, each scan comprising a color and depth frame;
• b) receiving, by a computing device, a first one of the plurality of scans from the sensor;
• c) determining, by the computing device, two-dimensional (2D) feature points of the one or more objects using the color and depth frame of the received scan;
• d) retrieving, by the computing device, a key frame from a database that stores one or more key frames of the one or more objects in the scene, each key frame comprising a plurality of map points associated with the one or more objects;
• e) matching, by the computing device, one or more of the 2D feature points with one or more of the map points in the key frame;
• f) generating, by the computing device, a current pose of the one or more objects in the color and depth frame using the matched 2D feature points;
• g) inserting, by the computing device, the color and depth frame into the database as a new key frame, including the matched 2D feature points as map points for the new key frame;
• and
• h) repeating, by the computing device, steps b)-g) on each of the remaining scans, using the inserted new key frame for matching in step e);
• wherein the server computing device tracks the pose of the one or more objects in the scene across the plurality of scans.