Device for and method of determining a pose of a camera

A method of determining a pose of a camera, the method comprising:

• storing image data from the camera, depth data, and a motion prediction model output associated with frames captured by the camera;
• generating an edge map that is stored for a key frame associated with the image;
• generating a first pose of the camera using image data;
• generating a second pose of the camera using data related to rotation and translation motion of the camera;
• generating motion data using an inertial measurement unit;
• correcting the second pose using the motion data generated by the inertial measurement unit;
• determining whether one or more of the image data, the depth data, and the motion prediction model output indicate a pose failure;
• correcting the pose of the camera by re-localization using a matching of image data of a current frame to key frames and using the corrected second pose, wherein the re-localization of the camera comprises storing key frames until reliable features are obtained and a key frame to which an incoming frame is localized is determined;
• performing a fine adjustment for re-localization using a current frame and the depth data after performing re-localization using the corrected second pose;
• updating an edge map for a last tracked frame using foreground segmentation during re-localization to identify objects in the foreground;
• and
• storing the last tracked frame as a key frame and the edge map during re-localization.

A method of determining a pose of a camera, the method comprising:

• tracking rotation and translation motion of the camera;
• performing camera pose estimation using inputs from a plurality of sensors;
• detecting changes in an image based upon previous frames of the image using the camera, wherein the changes in the image indicate a pose failure;
• generating an edge map that is stored for a key frame associated with the image;
• identifying feature descriptors associated with a current frame;
• using image data from the camera to localize a pose of the camera to a nearest key frame based upon the feature descriptors of the current frame;
• performing re-localization using a matching of image data of a current frame to key frames and using the pose of the camera, wherein the re-localization of the camera comprises storing key frames until reliable features are obtained and a key frame to which an incoming frame is localized is determined;
• determining a type of problem associated with tracking the image;
• using depth data associated with a plurality of frames to perform finer adjustments of the pose using a current frame for re-localization after performing re-localization using the pose of the camera;
• updating an edge map for a last tracked frame using foreground segmentation during re-localization to identify objects in the foreground;
• and
• storing the last tracked frame as a key frame and the edge map during re-localization.

A device for determining a pose of a camera, the device comprising:

• a processor configured to:
• implement a plurality of sensors on the camera;
• analyze changes in a plurality of frames of an image detected by the camera to generate a first pose of the camera;
• generate an edge map that is stored for a key frame associated with the image;
• use data related to rotation and translation motion of the camera to generate a second pose of the camera;
• provide the first pose and the second pose to a filter to generate a filtered camera pose based upon the first pose of the camera associated with the image and the second pose of the camera associated with a motion of the camera;
• determine, using the filtered camera pose, when a pose of the camera is incorrect based upon the first pose and the filtered camera pose;
• determine which sensors of the plurality of sensors are providing reliable image data;
• perform re-localization using a matching of image data of a current frame to key frames and using the filtered camera pose, wherein the re-localization of the camera comprises storing key frames until reliable features are obtained and a key frame to which an incoming frame is localized is determined;
• perform a fine adjustment for re-localization using a current frame and depth data associated with the plurality of frames after performing re-localization using the filtered camera pose;
• update an edge map for a last tracked frame using foreground segmentation during re-localization to identify objects in the foreground;
• and
• store the last tracked frame as a key frame and the edge map during re-localization.

A non-transitory computer-readable storage medium having data stored therein representing software executable by a computer for determining a pose of a camera, the non-transitory computer-readable storage medium comprising:

• instructions for implementing a plurality of sensors on the camera;
• instructions for analyzing changes in a plurality of frames of an image detected by the camera to generate a first pose of the camera;
• instructions for generating an edge map that is stored for a key frame associated with the image;
• instructions for using data related to rotation and translation motion of the camera to generate a second pose of the camera;
• instructions for providing the first pose and the second pose to a filter to generate a filtered camera pose based upon the first pose of the camera associated with the image and the second pose of the camera associated with a motion of the camera;
• instructions for determining, using the filtered camera pose, when a pose of the camera is incorrect;
• instructions for determining which sensors of the plurality of sensors are providing reliable image data;
• instructions for performing re-localization using a matching of image data of a current frame to key frames and using the filtered camera pose, wherein the re-localization of the camera comprises storing key frames until reliable features are obtained and a key frame to which an incoming frame is localized is determined;
• instructions for performing a fine adjustment for re-localization using a current frame and depth data associated with the plurality of frames after performing re-localization using the filtered camera pose;
• instructions for updating an edge map for a last tracked frame using foreground segmentation during re-localization to identify objects in the foreground;
• and
• instructions for storing the last tracked frame as a key frame and the edge map during re-localization.