Point cloud based 3D semantic segmentation

A device for three-dimension (3D) semantic segmentation, comprising:

• an interface, to obtain a point cloud data set for a time-ordered sequence of 3D frames, the 3D frames including a current 3D frame and one or more historical 3D frames previous to the current 3D frame;
• and
• processing circuitry, to:
• invoke a first artificial neural network (ANN) to estimate a 3D scene flow field for each of the one or more historical 3D frames by taking the current 3D frame as a reference frame;
• and
• invoke a second ANN to:
• produce an aggregated feature map, based on the reference frame and the estimated 3D scene flow field for each of the one or more historical 3D frames;
• and
• perform the 3D semantic segmentation based on the aggregated feature map,
• wherein the second ANN is configured to produce the aggregated feature map by:
• predicting a displacement of each point in point cloud data for the one or more historical 3D frames, based on the estimated 3D scene flow field for each of the one or more historical 3D frames;
• obtaining a warped 3D frame for each of the one or more historical 3D frames based on the predicted displacement of each point in the point cloud data for the one or more historical 3D frames and an initial position of the point in the corresponding historical 3D frame;
• obtaining a warped feature map for each of the one or more historical 3D frames from the warped 3D frame for the historical 3D frame;
• and
• aggregating the warped feature map for each of the one or more historical 3D frames to an original feature map of the current 3D frame.

An apparatus for three-dimension (3D) semantic segmentation, comprising:

• means for obtaining a point cloud data set for a time-ordered sequence of 3D frames, the 3D frames including a current 3D frame and one or more historical 3D frames previous to the current 3D frame;
• and
• means for invoking a first artificial neural network (ANN) to estimate a 3D scene flow field for each of the one or more historical 3D frames by taking the current 3D frame as a reference frame;
• and
• means for invoking a second ANN to:
• produce an aggregated feature map, based on the reference frame and the estimated 3D scene flow field for each of the one or more historical 3D frames, and perform the 3D semantic segmentation based on the aggregated feature map,
• wherein the second ANN includes a feature extract sub-network for each of the one or more historical 3D frames and the current 3D frame, to generate an origin feature map for each of the one or more historical 3D frames and the current 3D frame.

A non-transitory machine-readable storage medium having instructions stored thereon, which when executed by a processor, cause the processor to perform operations for three-dimension (3D) semantic segmentation, the operations comprises:

• obtaining a point cloud data set for a time-ordered sequence of 3D frames, the 3D frames including a current 3D frame and one or more historical 3D frames previous to the current 3D frame;
• and
• invoking a first artificial neural network (ANN) to estimate a 3D scene flow field for each of the one or more historical 3D frames by taking the current 3D frame as a reference frame;
• and
• invoking a second ANN to:
• produce an aggregated feature map, based on the reference frame and the estimated 3D scene flow field for each of the one or more historical 3D frames, and perform the 3D semantic segmentation based on the aggregated feature map,
• wherein the second ANN includes a feature extract sub-network for each of the one or more historical 3D frames and the current 3D frame, to generate an origin feature map for each of the one or more historical 3D frames and the current 3D frame.

A non-transitory machine-readable storage medium having instructions stored thereon, which when executed by a processor, cause the processor to perform operations for training a neural network for three-dimension (3D) semantic segmentation, the operations comprises:

• obtaining a point cloud data set for a time-ordered sequence of 3D frames, the 3D frames including a current 3D frame and one or more historical 3D frames previous to the current 3D frame;
• randomly selecting a historical 3D frame from the one or more historical 3D frames;
• producing a test result based on forward-propagating processing of the selected historical 3D frame through a training neural network;
• applying a loss function to evaluate the test result to produce a loss value;
• reducing the loss value by refining trainable parameters of the training neural network, based on backpropagation of the loss function through the training neural network;
• and
• supplying the refined trainable parameters to configure the neural network for 3D semantic segmentation.