Streaming object detection and segmentation with polar pillars

A method, comprising:

• dividing, with at least one processor, streamed sectors of data points representing a three-dimensional (3D) space surrounding a vehicle into a plurality of polar pillars, wherein each polar pillar of the plurality of polar pillars comprises a slice of the 3D space that extends from a two-dimensional (2D) polar grid on a ground plane comprising wedged-shaped regions corresponding to the streamed sectors in the 3D space, wherein each data point of the sectors is assigned to a polar pillar in the plurality of polar pillars;
• encoding, with the at least one processor, the streamed sectors into a wedge-shaped region in a bird's eye view using polar pillars to obtain pillar-wise features on the polar grid, wherein 3D stacked polar pillar tensors are generated for non-empty polar pillars and convolution is iteratively applied to the 3D stacked polar pillar tensors to generate the pillar wise features;
• generating, with the at least one processor, a feature map based on the pillar-wise features, wherein each polar pillar of the plurality of polar pillars corresponds to a polar feature representation of data points assigned to the polar pillar;
• inputting, with the at least one processor, the feature map into a segmentation head, an object detection head, and a bounding box head of a network simultaneously;
• outputting, with the at least one processor, per-pixel segments, object classes, and bounding boxes from the segmentation head, object detection head, and bounding box head respectively, wherein the object detection head transforms the feature representation to a Cartesian representation for object detection and the bounding box head applies kernels to the data points of the feature representation based on a range for bounding box generation;
• and
• operating, with the at least one processor, the vehicle in the 3D space according to the per-pixel segments, object classes, and bounding boxes, wherein the streamed sectors are iteratively processed.

At least one non-transitory storage media storing instructions that, when executed by at least one processor, cause the at least one processor to:

• divide streamed sectors of data points representing a three-dimensional (3D) space surrounding a vehicle into a plurality of polar pillars, wherein each polar pillar of the plurality of polar pillars comprises a slice of the 3D space that extends from a two-dimensional (2D) polar grid on a ground plane comprising wedged-shaped regions corresponding to the streamed sectors in the 3D space, wherein each data point of the sectors is assigned to a polar pillar in the plurality of polar pillars;
• encode the streamed sectors into a wedge-shaped region in a bird's eye view using polar pillars to obtain pillar-wise features on the polar grid, wherein 3D stacked polar pillar tensors are generated for non-empty polar pillars and convolution is iteratively applied to the 3D stacked polar pillar tensors to generate the pillar wise features;
• generate a feature map based on the pillar-wise features, wherein each polar pillar of the plurality of polar pillars corresponds to a polar feature representation of data points assigned to the polar pillar;
• input the feature map into a segmentation head, an object detection head, and a bounding box head of a network simultaneously;
• output per-pixel segments, object classes, and bounding boxes from the segmentation head, object detection head, and bounding box head respectively, wherein the object detection head transforms the feature representation to a Cartesian representation for object detection and the bounding box head applies kernels to the data points of the feature representation based on a range for bounding box generation;
• and
• operate the vehicle in the 3D space according to the per-pixel segments, object classes, and bounding boxes, wherein the streamed sectors are iteratively processed.