# US 2025/0095345 A1 · Systems and methods for two-stage 3d object detection network for sensor fusion > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0095345 A1 - Document ID: US20250095345A1 - Status: Published application - Kind: A1 - Application number: 18/471,221 - Filed: 2023-09-20 - Published: 2025-03-20 - Earliest claimed: 2023-09-20 - Assignees: Toyota Motor Engineering & Manufacturing North America, Inc.; Toyota Jidosha Kabushiki Kaisha - Applicants: Toyota Motor Engineering & Manufacturing North America, Inc. - Inventors: ROHIT GUPTA; Qi Chen - Web page: https://patentgrep.com/patent/US20250095345A1 ## CPC - G06V 10/803: {of input or preprocessed data} - B60W 50/0097: {Predicting future conditions} - B60W 60/0015: {specially adapted for safety} - G01S 7/415: {Identification of targets based on measurements of movement associated with the target} - G01S 13/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S13/46) - G01S 13/58: Velocity or trajectory determination systems; Sense-of-movement determination systems - G01S 13/867: {Combination of radar systems with cameras} - G01S 13/89: for mapping or imaging - G01S 13/931: of land vehicles - G06T 7/11: Region-based segmentation - G06T 7/194: involving foreground-background segmentation - G06T 7/73: using feature-based methods - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/82: using neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2554/4029: Pedestrians - B60W 2554/4041: Position - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30261: Obstacle - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods are provided that implement two-stage three-dimensional (3D) object detection, which includes a distinct fusion of radar data and camera data. The radar data includes a four-dimensional (4D) millimeter-wave (MMW) radar point cloud, and the camera data includes a high-resolution image in the two-dimensional space (2D). Thereafter, a 3D ROI proposal is fused with 2D image data generating a 2D proposal projection. The 2D proposal projection comprises proposals that predict the position of objects in the high-resolution image. In some cases, proposals in the 2D proposal projection correspond to anchors in the 3D ROI proposal. Then, 2D object detection results are fused with the 3D ROI proposal to generate 3D object detection results. Two-stage 3D object detection allows vehicles to detect objects in a 3D space of the driving environment to navigate safely, avoid obstacles, and respond to dynamic changes in the surrounding environment while operating autonomously. ## Claims ### Claim 1 A vehicle comprising: sensors capturing camera data and radar data, wherein the camera data and the radar data is associated with a driving environment of the vehicle including one or more objects; and a controller performing fusion of the camera data and the radar data to generate a three-dimensional (3D) object detection result including a position of the one or more objects in a 3D space associated with the driving environment of the vehicle, and executing autonomous control of the vehicle based on the 3D object detection result, wherein the autonomous control is associated with the driving environment. ### Claim 2 (depends on 1) The vehicle of claim 1, wherein the radar data comprises a four-dimensional (4D) millimeter-wave (MMW) radar point cloud. ### Claim 3 (depends on 1) The vehicle of claim 1, wherein the controller performs analysis of the 4D MMW radar point cloud to detect key points associated with the one or more objects. ### Claim 4 (depends on 3) The vehicle of claim 3, wherein the analysis of the 4D MMW radar point cloud comprises foreground point cloud segmentation. ### Claim 5 (depends on 3) The vehicle of claim 3, wherein the controller generates a 3D region of interest (ROI) proposal including anchors correspond to the detected key points in the 4D MMW radar point cloud. ### Claim 6 The vehicle of 5, wherein the camera data comprises a high-resolution image in the two-dimensional space (2D). ### Claim 7 (depends on 6) The vehicle of claim 6, wherein the controller fuses the 3D ROI proposal and the high-resolution image to generates a 2D proposal projection. ### Claim 8 (depends on 7) The vehicle of claim 7, wherein the 2D proposal projection comprises proposals predicting the position of the one or more objects in the high-resolution image. ### Claim 9 (depends on 8) The vehicle of claim 8, wherein the proposals in the 2D proposal projection correspond to the anchors in the 3D ROI proposal. ### Claim 10 (depends on 9) The vehicle of claim 9, wherein the controller applies anchor-based object detection networks to the 2D proposal projection and generates 2D object detection results. ### Claim 11 (depends on 10) The vehicle of claim 10, wherein the 2D object detection result comprises 2D bounding boxes corresponding to the positions of the one or more objects in the high-resolution image detected using the anchor-based object detection networks. ### Claim 12 (depends on 11) The vehicle of claim 11, wherein the controller fuses the 2D object detection results with the 3D ROI proposal to generate 3D object detection results. ### Claim 13 (depends on 11) The vehicle of claim 11, wherein the 3D object detection result comprises 3D bounding boxes in the 3D space corresponding to the 2D bounding boxes of the 2D object detection result in the 2D space. ### Claim 14 (depends on 13) The vehicle of claim 13, wherein the 3D bounding boxes indicate the positions of the one or more objects in a 3D space associated with the driving environment of the vehicle. ### Claim 15 (depends on 1) The vehicle of claim 1, wherein the vehicle comprises an autonomous vehicle. ### Claim 16 A method comprising: receiving camera data and radar data, wherein the camera data and the radar data is associated with a driving environment of the vehicle including one or more objects; fusing of the camera data and the radar data to generate a three-dimensional (3D) object detection result including a position of the one or more objects in a 3D space associated with the driving environment of the vehicle, and executing autonomous control of the vehicle based on the 3D object detection result, wherein the autonomous control is associated with the driving environment. ### Claim 17 (depends on 16) The method of claim 16, wherein the radar data comprises a four-dimensional (4D) millimeter-wave (MMW) radar point cloud. ### Claim 18 (depends on 17) The method of claim 17, wherein the camera data comprises a high-resolution image in the two-dimensional space (2D). ### Claim 19 (depends on 18) The method of claim 18, wherein the 3D ROI proposal generated from the 4D MMW radar point cloud is fused with the high-resolution image to generate a 2D proposal projection. ### Claim 20 (depends on 19) The method of claim 19, wherein the 2D proposal projection comprises proposals predicting the position of the one or more objects in the high-resolution image.