Autonomous driving with surfel maps

A system comprising:

• one or more computers;
• and
• one or more computer-readable media storing instructions that, when executed, cause the one or more computers to perform operations comprising:
• obtaining a three-dimensional representation of a real-world environment comprising a plurality of surfels, wherein each of the surfels corresponds to a respective point of plurality of points in a three-dimensional space of the real-world environment;
• receiving input sensor data from multiple sensors installed on the autonomous vehicle;
• detecting an animate object from the input sensor data;
• determining, from the input sensor data and the three-dimensional representation, that the animate object is located on an opposite side of a barrier relative to the autonomous vehicle;
• and
• updating a driving plan based on determining that the animate object is located on the opposite side of the barrier.

One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• obtaining a three-dimensional representation of a real-world environment comprising a plurality of surfels, wherein each of the surfels corresponds to a respective point of plurality of points in a three-dimensional space of the real-world environment;
• receiving input sensor data from multiple sensors installed on the autonomous vehicle;
• detecting an animate object from the input sensor data;
• determining, from the input sensor data and the three-dimensional representation, that the animate object is located on an opposite side of a barrier relative to the autonomous vehicle;
• and
• updating a driving plan based on determining that the animate object is located on the opposite side of the barrier.