Long range localization with surfel maps

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• obtaining, for a particular location of a vehicle having one or more cameras and one or more detection sensors, surfel data comprising a plurality of surfels, wherein each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment;
• obtaining image data captured by a camera of the one or more cameras;
• determining that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for one of the one or more detection sensors;
• in response, determining that the image data for the region of interest matches surfel color data for one or more surfels corresponding to the region of interest;
• in response, designating the region of interest as having no unexpected objects;
• and
• performing the vehicle planning process with the region of interest designated as having no unexpected objects.

One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations comprising:

• obtaining, for a particular location of a vehicle having one or more cameras and one or more detection sensors, surfel data comprising a plurality of surfels, wherein each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment;
• obtaining image data captured by a camera of the one or more cameras;
• determining that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for one of the one or more detection sensors;
• in response, determining that the image data for the region of interest matches surfel color data for one or more surfels corresponding to the region of interest;
• in response, designating the region of interest as having no unexpected objects;
• and
• performing the vehicle planning process with the region of interest designated as having no unexpected objects.

A method performed by one or more computers, the method comprising:

• obtaining, for a particular location of a vehicle having one or more cameras and one or more detection sensors, surfel data comprising a plurality of surfels, wherein each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment;
• obtaining image data captured by a camera of the one or more cameras;
• determining that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for one of the one or more detection sensors;
• in response, determining that the image data for the region of interest does not match surfel color data for one or more surfels corresponding to the region of interest;
• in response, designating the region of interest as having an unexpected object;
• and
• performing the vehicle planning process with the region of interest designated as having the unexpected object.

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• obtaining, for a particular location of a vehicle having one or more cameras and one or more detection sensors, surfel data comprising a plurality of surfels, wherein each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment;
• obtaining image data captured by a camera of the one or more cameras;
• determining that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for one of the one or more detection sensors;
• in response, determining that the image data for the region of interest does not match surfel color data for one or more surfels corresponding to the region of interest;
• in response, designating the region of interest as having an unexpected object;
• and
• performing the vehicle planning process with the region of interest designated as having the unexpected object.

One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations comprising:

• obtaining, for a particular location of a vehicle having one or more cameras and one or more detection sensors, surfel data comprising a plurality of surfels, wherein each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment;
• obtaining image data captured by a camera of the one or more cameras;
• determining that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for one of the one or more detection sensors;
• in response, determining that the image data for the region of interest does not match surfel color data for one or more surfels corresponding to the region of interest;
• in response, designating the region of interest as having an unexpected object;
• and
• performing the vehicle planning process with the region of interest designated as having the unexpected object.