Method and system of using a global transformer for efficient modeling of global context in point clouds

A system for object detection and part segmentation, the system comprising:

• a memory storing instructions;
• and
• a processor coupled to the memory, the processor is configured to execute instructions to:
• receive point cloud data;
• generate a three-dimensional (3D) point cloud including a plurality of points in a volume of a space of the point cloud based on the point cloud data;
• obtain a set of global feature tokens to model global context of the 3D point cloud and local feature points of the 3D point cloud;
• perform a global to local cross-attention (CA) to update the set of global feature tokens by aggregating local information from the local feature points to the set of global feature tokens via cross attention;
• and
• generate a set of new global feature tokens and new local feature points with modeled global context information for object detection and part segmentation, wherein the set of global feature tokens are initialized as at least two corners of a tightest bounding box of the 3D point cloud.

A non-transitory computer readable storage medium having embodied thereon a program, wherein the program is executable by a processor to perform a method of object detection and part segmentation, the method comprising:

• generating a three-dimensional (3D) point cloud including a plurality of points in a volume of a space of the point cloud based on receiving point cloud data;
• initially obtaining, with a global transformer, a set of global feature tokens to model global context of the 3D point cloud and local feature points of the 3D point cloud;
• enforcing with the global transformer an attention mechanism information flow across the set of global feature tokens and local feature points including a global to local cross-attention (CA) to update the set of global feature tokens by aggregating local information from the local feature points to the set of global feature tokens via cross attention;
• and
• generating, with the global transformer that is utilized for object detection and part segmentation, a set of new global feature tokens and new local feature points with modeled global context information based on the attention mechanism information flow.