Mobile camera localization using depth maps

A method of real time camera relocalization comprising:

• receiving a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a plurality of image elements each having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera;
• tracking the position and orientation of the mobile depth camera using the depth map frames and simultaneously forming a 3D model of the environment in which the mobile depth camera is moving using the depth map frames;
• detecting a loop closure error;
• modifying the 3D model in response to detecting the loop closure error;
• detecting a failure in the tracking of the position and orientation of the mobile depth camera by any of:
• comparing a change in a current and previous tracked position of the mobile depth camera with a threshold;
• detecting a failure in convergence of an iterative process used to track the position and orientation of the mobile depth camera;
• and comparing a current tracked position and orientation of the mobile depth camera with a position and orientation predicted using a model of motion of the mobile depth camera;
• relocalizing the mobile depth camera by recomputing its position and orientation using a current depth map captured by the mobile depth camera;
• and
• at least one of
• computing a track distribution of poses of the mobile depth camera being a predicted distribution of poses given known previous motion of the mobile depth camera;
• sampling a pose from the track distribution and using that sampled pose to align the current depth map and a previous depth map, or
• applying a random decision forest to patches from the current depth map and from a plurality of previous depth maps obtained from the 3D model of the environment to obtain histograms of textons being texture features of a depth map and selecting a previous depth map which is similar to the current depth map in terms of the histograms.

A real time camera relocalization system comprising:

• an input arranged to receive a sequence of depth map frames from a moving mobile depth camera each depth map frame comprising a plurality of image elements each having a depth value being related to a distance from the mobile depth camera to a surface in the scene captured by the mobile depth camera;
• a frame alignment engine arranged to track the position and orientation of the mobile depth camera using the depth map frames;
• a 3D model formation system arranged to form a 3D model of the environment in which the mobile depth camera is moving using the depth map frames, the 3D model formation system being further arranged to detect a loop closure error and to modify the 3D model in response to the loop closure error;
• a relocalization engine arranged to detect a failure in the tracking of the position and orientation of the mobile depth camera;
• and to relocalize the mobile depth camera by recomputing its position and orientation using a current depth map captured by the mobile depth camera and at least one previous depth map obtained from the 3D model, the relocalization engine being arranged to apply a random decision forest to patches from the current depth map and from a plurality of previous depth maps obtained from the 3D model of the environment to obtain histograms of textons being texture features of a depth map and to select a previous depth map which is similar to the current depth map in terms of the histograms.