# US 12,762,827 B2 · Semantic guided scene flow estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,827 B2 - Document ID: US12762827B2 - Status: Granted - Kind: B2 - Application number: 18/609,469 - Filed: 2024-03-19 - Published: 2025-09-25 - Granted: 2026-10-13 - Earliest claimed: 2024-03-19 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Varun Ravi Kumar; Jun Wang; Senthil Kumar Yogamani - Web page: https://patentgrep.com/patent/US12762827B2 ## CPC - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G01S 17/89: for mapping or imaging - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/60: Analysis of geometric attributes - G06V 10/806: {of extracted features} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method for scene flow estimation includes receiving multimodal data having at least a first modality and a second modality, wherein the multimodal data represents a plurality of points in a scene. The method also includes extracting a first set of features from the first modality and extracting a second set of features from the second modality; and projecting the first set of features and the second set of features into a shared latent space to generate a first latent representation. Additionally, the method includes estimating a flow of the plurality of points of the scene based on one or more relationships between the first set of features and the second set of features by using a model trained to learn the one or more relationships between the first set of features and the second set of features based on the first latent representation and the second latent representation. ## Claims ### Claim 1 A method for scene flow estimation comprising: receiving multimodal data having at least a first modality and a second modality, wherein the multimodal data represents a plurality of points in a scene; extracting a first set of features from the first modality and extracting a second set of features from the second modality; projecting the first set of features and the second set of features into a shared latent space to generate a first latent representation of the first set of features and a second latent representation of the second set of features; and estimating a flow of the plurality of points of the scene based on one or more relationships between the first set of features and the second set of features comprising using a model configured to represent the one or more relationships between the first set of features and the second set of features based on the first latent representation and the second latent representation, wherein the model is configured to integrate the first set of features with the second set of features to generate an integrated representation by concatenating the first set of features with the second set of features using cross-modal attention and using one or more attention weights comprising scalar values indicating a degree of influence the first set of features and the second set of features have on the integrated representation. ### Claim 2 (depends on 1) The method of claim 1, further comprising: defining a surface of each object depicted in the scene using a local coordinate system corresponding to each object. ### Claim 3 (depends on 1) The method of claim 1, wherein the first modality comprises image data and the second modality comprises LiDAR point cloud data, wherein extracting the first set of features from the first modality comprises extracting one or more semantic priors providing semantic information about the scene and wherein extracting the second set of features from the second modality comprises extracting geometric features capturing a 3D structure and layout of the scene. ### Claim 4 (depends on 3) The method of claim 3, further comprising: training the model to learn the one or more relationships between the first set of features and the second set of features based on the first latent representation and the second latent representation, wherein learning the one or more relationships between the first set of features and the second set of features comprises integrating the extracted semantic information with the extracted geometric features to generate the integrated representation. ### Claim 5 (depends on 4) The method of claim 4, wherein estimating the flow of the plurality of points of the scene comprises using the extracted semantic information to guide geometric reasoning and point flow predictions. ### Claim 6 (depends on 4) The method of claim 4, wherein integrating the extracted semantic information with the extracted geometric features further comprises generating a plurality of fused multi-modal features that combine the semantic information and the geometric features. ### Claim 7 (depends on 1) The method of claim 1, wherein estimating the flow of the plurality of points of the scene comprises generating a Graph Neural Network (GNN) having a plurality of nodes and a plurality of edges interconnecting the plurality of nodes, wherein each of the plurality of nodes represents the semantic information and the geometric features associated with each of the plurality of points, wherein neighboring nodes of the GNN exchange the semantic information and the geometric features and wherein each of the plurality of nodes aggregates, using an update function, the semantic information and the geometric features associated with a corresponding node with the semantic information and the geometric features received from the neighboring nodes. ### Claim 8 (depends on 7) The method of claim 7, wherein a first node of the plurality of nodes representing a first point of the plurality of points is connected by one of the plurality of edges to a second node of the plurality of nodes representing a second point of the plurality of points if a distance between the first point and the second point is less than a predefined threshold. ### Claim 9 (depends on 7) The method of claim 7, wherein the scene flow estimation is used to determine velocity of an object associated with a 3D scene. ### Claim 10 (depends on 1) The method of claim 1, further comprising operating an Advanced Driver Assistance Systems (ADAS) system based on an estimation of the flow of the plurality of points of the scene. ### Claim 11 An apparatus for scene flow estimation, the apparatus comprising: a memory for storing multimodal data; and processing circuitry in communication with the memory, wherein the processing circuitry is configured to: receive the multimodal data having at least a first modality and a second modality, wherein the multimodal data represents a plurality of points in a scene; extract a first set of features from the first modality and extract a second set of features from the second modality; project the first set of features and the second set of features into a shared latent space to generate a first latent representation of the first set of features and a second latent representation of the second set of features; and estimate a flow of the plurality of points of the scene based on one or more relationships between the first set of features and the second set of features by using a model configured to represent the one or more relationships between the first set of features and the second set of features based on the first latent representation and the second latent representation, wherein the model is configured to integrate the first set of features with the second set of features to generate an integrated representation by concatenating the first set of features with the second set of features using cross-modal attention and using one or more attention weights comprising scalar values indicating a degree of influence the first set of features and the second set of features have on the integrated representation. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein the processing circuitry is further configured to: define a surface of each object depicted in the scene using a local coordinate system corresponding to each object. ### Claim 13 (depends on 11) The apparatus of claim 11, wherein the first modality comprises image data and the second modality comprises LiDAR point cloud data, wherein the processing circuitry configured to extract the first set of features from the first modality is further configured to extract one or more semantic priors providing semantic information about the scene and wherein the processing circuitry configured to extract the second set of features from the second modality is further configured to extract geometric features capturing a 3D structure and layout of the scene. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the processing circuitry is further configured to: train the model to learn the one or more relationships between the first set of features and the second set of features based on the first latent representation and the second latent representation, and wherein the model configured to learn the one or more relationships between the first set of features and the second set of features is further configured to integrate the extracted semantic information with the extracted geometric features to generate the integrated representation. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the processing circuitry configured to estimate the flow of the plurality of points of the scene is further configured to use the extracted semantic information to guide geometric reasoning and point flow predictions. ### Claim 16 (depends on 14) The apparatus of claim 14, wherein the processing circuitry configured to integrate the extracted semantic information with the extracted geometric features is further configured to generate a plurality of fused multi-modal features that combine the semantic information and the geometric features. ### Claim 17 (depends on 11) The apparatus of claim 11, wherein the processing circuitry configured to estimate the flow of the plurality of points of the scene is further configured to generate a Graph Neural Network (GNN) having a plurality of nodes and a plurality of edges interconnecting the plurality of nodes, wherein each of the plurality of nodes represents the semantic information and the geometric features associated with each of the plurality of points, wherein neighboring nodes of the GNN exchange the semantic information and the geometric features and wherein each of the plurality of nodes aggregates, using an update function, the semantic information and the geometric features associated with a corresponding node with the semantic information and the geometric features received from the neighboring nodes. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein a first node of the plurality of nodes representing a first point of the plurality of points is connected by one of the plurality of edges to a second node of the plurality of nodes representing a second point of the plurality of points if a distance between the first point and the second point is less than a predefined threshold. ### Claim 19 (depends on 17) The apparatus of claim 17, wherein the scene flow estimation is used to determine velocity of an object associated with a 3D scene. ### Claim 20 (depends on 11) The apparatus of claim 11, wherein the processing circuitry is further configured to operate an Advanced Driver Assistance Systems (ADAS) system based on an estimation of the flow of the plurality of points of the scene. ### Claim 21 A non-transitory computer-readable medium storing instructions that, when applied by processing circuitry, causes the processing circuitry to: receive the multimodal data having at least a first modality and a second modality, wherein the multimodal data represents a plurality of points in a scene; extract a first set of features from the first modality and extract a second set of features from the second modality; project the first set of features and the second set of features into a shared latent space to generate a first latent representation of the first set of features and a second latent representation of the second set of features; and estimate a flow of the plurality of points of the scene based on the one or more relationships between the first set of features and the second set of features by using a model configured to represent the one or more relationships between the first set of features and the second set of features based on the first latent representation and the second latent representation, wherein the model is configured to integrate the first set of features with the second set of features to generate an integrated representation by concatenating the first set of features with the second set of features using cross-modal attention and using one or more attention weights comprising scalar values indicating a degree of influence the first set of features and the second set of features have on the integrated representation.