# US 11,908,202 B2 · Method and system of using a global transformer for efficient modeling of global context in point clouds > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,202 B2 - Document ID: US11908202B2 - Status: Granted - Kind: B2 - Application number: 17/560,676 - Filed: 2021-12-23 - Published: 2023-06-29 - Granted: 2024-02-20 - Earliest claimed: 2021-12-23 - Assignees: GM Cruise Holdings LLC - Applicants: GM Cruise Holdings LLC - Inventors: Yong Jae Lee; Haotian Liu - Web page: https://patentgrep.com/patent/US11908202B2 ## CPC - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 20/64: Three-dimensional [3D] objects - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract For one embodiment of the present invention, a method of object detection and part segmentation is described. The method includes generating a three-dimensional (3D) point cloud including a plurality of points in a volume of a space of the point cloud based on receiving point cloud dat. The method further includes obtaining, with a global transformer, a set of global feature tokens and local feature points of the 3D point cloud, enforcing with an attention mechanism information flow across the set of global feature tokens and local feature point, and generating, with the global transformer, a set of new global feature tokens and new local feature points with modeled global context information. ## Claims ### Claim 1 A computer implemented method of object detection and part segmentation, the computer implemented method comprising: generating a three-dimensional (3D) point cloud including a plurality of points in a volume of a space of the point cloud in response to receiving point cloud data; initially obtaining, with a global transformer, a set of global feature tokens to model global context of the 3D point cloud and local feature points of the 3D point cloud; enforcing with the global transformer an attention mechanism information flow across the set of global feature tokens and local feature points including a global to local cross-attention (CA) to update the set of global feature tokens by aggregating local information from the local feature points to the set of global feature tokens via cross attention; and generating, with the global transformer that is utilized for object detection and part segmentation, a set of new global feature tokens and new local feature points with modeled global context information based on the attention mechanism information flow. ### Claim 2 (depends on 1) The computer implemented method of claim 1, wherein the set of global feature tokens are initialized as corners of a tightest bounding box of the 3D point cloud. ### Claim 3 (depends on 1) The computer implemented method of claim 1, wherein enforcing with an attention mechanism information flow across the set of global feature tokens and local feature points comprises: the global to local cross-attention (CA) to update the set of global feature tokens by aggregating local information from the local feature points to the set of global feature tokens via cross attention; a global self-attention (SA) to exchange information between the set of global feature tokens via self-attention; and a local to global cross-attention to update local features by propagating global information from the set of global feature tokens to the local feature points via cross attention. ### Claim 4 (depends on 1) The computer implemented method of claim 1, further comprising: applying global average pooling (GAP) separately to both of the new global feature tokens and new local feature points in a classification head to generate pooled local and global feature vectors; and concatenating pooled local and global feature vectors to generate a result. ### Claim 5 (depends on 4) The computer implemented method of claim 4, further comprising: sending the result into a multi-layer perceptron for classification of an object of the 3D point cloud. ### Claim 6 (depends on 1) The computer implemented method of claim 1, further comprising: utilizing a global transformer for each stage of a 3D point decoder for part segmentation. ### Claim 7 (depends on 1) The method of claim 1, wherein the set of global feature tokens attend to all local feature points. ### Claim 8 (depends on 7) The computer implemented method of claim 7, wherein the set of global feature tokens reduces a computational complexity for object detection from quadratic to linear with respect to input size of the point cloud data. ### Claim 9 (depends on 1) The computer implemented method of claim 1, wherein the point cloud data is received from one or more distance measurement sensors that are coupled to a vehicle. ### Claim 10 A system for object detection and part segmentation, the system comprising: a memory storing instructions; and a processor coupled to the memory, the processor is configured to execute instructions to: receive point cloud data; generate a three-dimensional (3D) point cloud including a plurality of points in a volume of a space of the point cloud based on the point cloud data; obtain a set of global feature tokens to model global context of the 3D point cloud and local feature points of the 3D point cloud; perform a global to local cross-attention (CA) to update the set of global feature tokens by aggregating local information from the local feature points to the set of global feature tokens via cross attention; and generate a set of new global feature tokens and new local feature points with modeled global context information for object detection and part segmentation, wherein the set of global feature tokens are initialized as at least two corners of a tightest bounding box of the 3D point cloud. ### Claim 11 (depends on 10) The system of claim 10, wherein the set of global feature tokens are initialized as at least four corners of a tightest bounding box of the 3D point cloud. ### Claim 12 (depends on 10) The system of claim 10, wherein the processor is configured to execute instructions to: perform a global self-attention (SA) to exchange information between the set of global feature tokens via self-attention; and perform a local to global cross-attention to update local features by propagating global information from the set of global feature tokens to the local feature points via cross attention. ### Claim 13 (depends on 10) The system of claim 10, wherein the processor is configured to execute instructions to: apply global average pooling (GAP) separately to both of the new global feature tokens and new local feature points to generate pooled local and global feature vectors; concatenate the pooled local and global feature vectors to generate a result; send the result into a multi-layer perceptron for classification of an object of the 3D point cloud. ### Claim 14 (depends on 13) The system of claim 13, wherein the processor is configured to execute instructions to: perform decoding prior to part segmentation. ### Claim 15 (depends on 10) The system of claim 10, wherein the set of global feature tokens reduces a computational complexity for object detection from quadratic to linear. ### Claim 16 (depends on 10) The system of claim 10, wherein the point cloud data is received from one or more distance measurement sensors. ### Claim 17 A non-transitory computer readable storage medium having embodied thereon a program, wherein the program is executable by a processor to perform a method of object detection and part segmentation, the method comprising: generating a three-dimensional (3D) point cloud including a plurality of points in a volume of a space of the point cloud based on receiving point cloud data; initially obtaining, with a global transformer, a set of global feature tokens to model global context of the 3D point cloud and local feature points of the 3D point cloud; enforcing with the global transformer an attention mechanism information flow across the set of global feature tokens and local feature points including a global to local cross-attention (CA) to update the set of global feature tokens by aggregating local information from the local feature points to the set of global feature tokens via cross attention; and generating, with the global transformer that is utilized for object detection and part segmentation, a set of new global feature tokens and new local feature points with modeled global context information based on the attention mechanism information flow. ### Claim 18 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein the set of global feature tokens are initialized as corners of a tightest bounding box of the 3D point cloud. ### Claim 19 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein enforcing with an attention mechanism information flow across the set of global feature tokens and local feature points comprises: the global to local cross-attention (CA) operation to update the set of global feature tokens by aggregating local information from the local feature points to the set of global feature tokens via cross attention; a global self-attention (SA) operation to exchange information between the set of global feature tokens via self-attention; and a local to global cross-attention operation to update local features by propagating global information from the set of global feature tokens to the local feature points via cross attention. ### Claim 20 (depends on 17) The non-transitory computer readable storage medium of claim 17, the method further comprising: applying global average pooling (GAP) separately to both of the new global feature tokens and new local feature points to generate pooled local and global feature vectors; and concatenating the pooled local and global feature vectors to generate a result.