Method for semantic localization of an unmanned aerial vehicle

An unmanned aerial vehicle (UAV), comprising:

• a camera;
• and
• a control system configured to perform operations comprising:
• receiving an image captured by the camera, wherein the image depicts an environment below the UAV;
• generating a feature mask associated with the image via a machine learning model that is trained to identify and semantically label pixels representing the environment depicted in the image;
• retrieving one or more reference tiles associated with the environment, wherein the one or more reference tiles are associated with particular geographic locations and specify semantically labeled pixels representing the geographic locations;
• and
• correlating semantic labels of the semantically labeled pixels of the feature mask with semantic labels of the semantically labeled pixels of at least one of the one or more reference tiles to determine a geographic location of the UAV in the environment, wherein the correlating comprises:
• generating a likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles;
• generating, based on a plurality of likelihood grids associated with a corresponding plurality of images successively captured as the UAV moves through the environment, a cumulative likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles;
• and
• when the relative likelihood associated with a particular location of the one or more reference tiles exceeds a threshold likelihood in the cumulative likelihood grid, determining the geographic location of the UAV based at least in part on the particular location.

A non-transitory computer readable medium comprising program instructions executable by one or more processors to perform operations, the operations comprising:

• receiving an image captured by the camera, wherein the image depicts an environment below the UAV;
• generating a feature mask associated with the image via a machine learning model that is trained to identify and semantically label pixels representing the environment depicted in the image;
• retrieving one or more reference tiles associated with the environment, wherein the one or more reference tiles are associated with particular geographic locations and specify semantically labeled pixels representing the geographic locations;
• and
• correlating semantic labels of the semantically labeled pixels of the feature mask with semantic labels of the semantically labeled pixels of at least one of the one or more reference tiles to determine a geographic location of the UAV in the environment, wherein the correlating comprises:
• generating a likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles;
• generating, based on a plurality of likelihood grids associated with a corresponding plurality of images successively captured as the UAV moves through the environment, a cumulative likelihood grid that indicates relative likelihoods that the location of the UAV is at particular locations of the one or more reference tiles;
• and
• when the relative likelihood associated with a particular location of the one or more reference tiles exceeds a threshold likelihood in the cumulative likelihood grid, determining the geographic location of the UAV based at least in part on the particular location.