Matching between 2d and 3d for direct localization

An apparatus, comprising:

• a processor;
• a memory storing instructions that, when executed by the processor, perform a method of determining a pose of an entity, the pose comprising a 3D position and orientation of the entity, comprising:
• receiving a query comprising a three-dimensional (3D) point cloud depicting an environment of the entity, the 3D point cloud omitting visual imagery of the environment of the entity;
• searching for a match between the query and a 3D map of the environment,
• the 3D map comprising a two-dimensional (2D) image,
• the 2D image comprising a 3D position and a 3D orientation of a capture device used to capture the 2D image, and
• searching for the match comprising:
• extracting a point cloud descriptor from the 3D point cloud;
• extracting an image descriptor from the 2D image;
• and
• using the 2D image, directly localizing the entity with respect to the 3D map, directly localizing comprising:
• correlating the image descriptor with the point cloud descriptor, and
• based on the correlation, generating a numerical similarity value above a threshold representing a likelihood that the image descriptor and the point cloud descriptor depict a same element of the environment;
• and
• estimating, using the numerical similarity value, the pose of the entity with respect to the 3D map.

A computerized method of creating training data for training a localizer configured for direct localization, the method comprising:

• accessing a dataset, the dataset storing:
• a red-green-blue depth (RGB-D) image comprising a depiction of an environment in which an entity is to be localized and a depth map associated with the depiction, and
• a ground-truth pose associated with the RGB-D image;
• selecting the RGB-D image as a query image;
• finding, for the query image, a plurality of covisible images depicting at least in part the environment, the plurality of covisible images being a plurality of reference images;
• creating a point cloud, creating the point cloud comprising:
• projecting a pixel point of a reference image of the plurality of reference images having a valid depth to three dimensions (3D),
• transforming the pixel point to a world coordinate system,
• determining that the pixel point has the valid depth by comparing a depth value from the depth map of the RGB-D image with a threshold, and
• using a ground-truth transformation retrieved from the dataset, transforming the ground-truth pose of the query image to the world coordinate system;
• augmenting the point cloud;
• and
• storing the training data comprising the point cloud, the query image, and the ground-truth pose associated with the query image.