Semantic and instance segmentation

The computer-implemented method of 2, wherein the plurality of portions of the sensor data includes a first portion and generating an instance label for the first portion comprises:

• determining, by the computing system, that the first portion is associated with a first class prediction;
• in response to determining that the first portion is associated with the first class prediction, determining, by the computing system, that an energy value of the first portion satisfies a threshold;
• and
• in response to determining that the energy value of the first portion satisfies the threshold, assigning, by the computing system, to the first portion a first instance label.

A computing system, comprising:

• one or more processors;
• a machine-learned model configured to receive sensor data representing an environment and in response to the sensor data to output an energy value in association with each of a plurality of portions of the sensor data, the machine-learned model trained to provide a common energy range to represent boundaries between objects in the environment;
• and
• one or more non-transitory computer-readable media that store instructions, that when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
• providing, as input to the machine-learned model, sensor data representing the environment;
• receiving, as output of the machine-learned model for each of a plurality of portions of the sensor data, a class prediction and an energy value, wherein the energy value is based on a distance of each portion relative to one or more object boundaries;
• and
• generating an instance label for each of the plurality of portions of sensor data based on a corresponding energy value and class prediction, wherein generating an instance label for each of the plurality of portions of the sensor data comprises:
• determining whether each of the plurality of portions of the sensor data is associated with a first class prediction;
• for portions of the sensor data that are associated with the first class prediction, identifying one or more object boundaries based on sensor data that corresponds to an energy value that does not satisfy a threshold;
• and
• for portions of the sensor data that are associated with the first class prediction and are within a common object boundary, generating a common instance label.

An autonomous vehicle, comprising:

• a sensor system configured to generate sensor data of an environment external to the autonomous vehicle;
• and
• a vehicle computing system, comprising:
• one or more processors;
• and
• one or more non-transitory computer-readable media that store instructions, that when executed by the one or more processors, cause the vehicle computing system to perform operations, the operations comprising:
• providing sensor data from the sensor system as input to a machine-learned neural network including one or more shared layers configured to jointly determine instance information and classification information for each of a plurality of portions of the sensor data;
• receiving, as a first output of one or more unshared first layers of the machine-learned neural network, an instance value for each of the plurality of portions of the sensor data based on the instance information from the one or more shared layers;
• receiving, as a second output of one or more unshared second layers of the machine-learned neural network, a classification for each of the plurality of portions of the sensor data based on the classification information from the one or more shared layers;
• and
• generating an instance label for each of the plurality of portions of the sensor data based on a corresponding classification and instance value from the machine-learned neural network, the instance value for each of the plurality of portions of the sensor data comprises an energy value based on a distance from each portion to a nearest object boundary, wherein the plurality of portions includes a first portion and generating an instance label for the first portion comprises:
• determining if the first portion is associated with a first classification;
• in response to determining that the first portion is associated with the first classification, determining if an energy value of the first portion satisfies a threshold;
• and
• in response to determining that the energy value of the first portion satisfies the threshold, assigning to the first portion a first instance label.

A computer-implemented method of detecting objects of interest comprising:

• receiving, by a computing system comprising one or more computing devices, sensor data from one or more sensors configured to generate sensor data associated with an environment;
• inputting, by the computing system, the sensor data to a machine-learned model configured to generate a class prediction and an instance prediction for each of a plurality of portions of the sensor data, the instance prediction for each of the plurality of portions of sensor data including an energy value based on a distance to at least one object boundary, the machine-learned model trained to generate a common energy value to represent the at least one object boundary, wherein the machine-learned model includes a neural network that uses feature sharing on the sensor data at a first portion of the neural network associated with both class prediction and instance prediction, the neural network includes a second portion that receives an output of the first portion of the neural network and determines the class prediction for each of the plurality of portions of the sensor data independently of the instance prediction, the neural network includes a third portion that receives the output of the first portion of the neural network and determines the instance prediction for each of the plurality of portions of the sensor data independently of the class prediction;
• generating, by the computing system as outputs of the machine-learned model, an instance prediction and a class prediction corresponding to each of the plurality of portions of the sensor data;
• generating, by the computing system, one or more object segments based at least in part on the instance predictions and the class predictions;
• and
• generating, by the computing system, an instance label for each of the plurality of portions of the sensor data based on a corresponding class prediction and instance prediction.