# US 2025/0148801 A1 · Apparatus and method for controlling a vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0148801 A1 - Document ID: US20250148801A1 - Status: Published application - Kind: A1 - Application number: 18/640,426 - Filed: 2024-04-19 - Published: 2025-05-08 - Earliest claimed: 2023-11-06 - Assignees: Hyundai Motor Company; Kia Corporation - Applicants: Hyundai Motor Company; Kia Corporation - Inventors: Jong Hyun Choi - Web page: https://patentgrep.com/patent/US20250148801A1 ## CPC - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60W 40/02: related to ambient conditions - B60W 50/00: Details of control systems for road vehicle drive control not related to the control of a particular sub-unit {, e.g. process diagnostic or vehicle driver interfaces} - G01S 7/4865: Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak (peak detection in noise, signal conditioning G01S7/487) - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G06T 7/55: from multiple images - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 10/806: {of extracted features} - B60W 60/00: Drive control systems specially adapted for autonomous road vehicles - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2420/408: Radar; Laser, e.g. lidar - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An apparatus for controlling a vehicle includes a camera configured to obtain a surrounding image of the vehicle and a Lidar configured to obtain point cloud data detected from one or more object positioned around the vehicle. The apparatus also includes a controller configured to generate information about a three dimensional (3D) bounding box and information about a two dimensional (2D) keypoint, corresponding to each object among the one or more objects, based on sensor data obtained from the camera and the Lidar. The controller is further configured to estimate a depth of each keypoint, based on the information about the 3D bounding box and the information about the information about the 2D keypoint, to generate information about a 3D keypoint for each object among the one or more objects. ## Claims ### Claim 1 An apparatus for controlling a vehicle, the apparatus comprising: a camera configured to obtain a surrounding image of the vehicle; a Lidar configured to obtain point cloud data detected from one or more objects positioned around the vehicle; and a controller configured to generate information about a three dimensional (3D) bounding box and information about a two dimensional (2D) keypoint, corresponding to each object among the one or more objects, based on sensor data obtained from the camera and the Lidar, estimate a depth of each keypoint based on the information about the 3D bounding box and the information about the 2D keypoint to generate information about a 3D keypoint for each object among the one or more objects, and recognize each object among the one or more objects based on the information about the 3D bounding box and the information about the 3D keypoint. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the controller is further configured to: when generating the information about the 3D keypoint for each object among the one or more objects, match a bounding box for the object with a keypoint cluster, select Lidar points, among Lidar points obtained from the Lidar, that are positioned inside the bounding box, and project the selected Lidar points onto an image obtained from the camera. ### Claim 3 (depends on 2) The apparatus of claim 2, wherein the controller is further configured to: when generating the information about the 3D keypoint for each object among the one or more objects, calculate a keypoint weight using the selected Lidar points and information about keypoints matched to the bounding box; and calculate depths of the keypoints by applying the keypoint weight to depths of the Lidar points. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the keypoint weight is calculated by using an exponential function having distances between the Lidar points and the keypoints matched with the bounding box, as exponents. ### Claim 5 (depends on 3) The apparatus of claim 3, wherein the controller is configured to: when generating the information about the 3D keypoint for each object among the one or more objects, calculate reliability for the depths of a keypoint based on a distribution state of the Lidar points for a surrounding region of the keypoint. ### Claim 6 (depends on 5) The apparatus of claim 5, wherein the controller is configured to: when generating the information about the 3D keypoint for each object among the one or more objects, calculate the reliability for the depths of the keypoints by using distances between the Lidar points and the keypoints matched with the bounding box and a mean value and a standard deviation of the distances. ### Claim 7 (depends on 5) The apparatus of claim 5, wherein a reliability for the depths of the keypoints has a greater value as compared to the Lidar points. ### Claim 8 (depends on 5) The apparatus of claim 5, wherein the controller is further configured to: when generating the information about the 3D keypoint for each object among the one or more objects, generate pseudo keypoint coordinates based on coordinates and depth values of the keypoints, and project the generated pseudo keypoint coordinates onto a Lidar space. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the controller is further configured to: allow the pseudo keypoint coordinates, that are projected onto the Lidar space, to be bilaterally symmetrical to each other about a center of the bounding box of each object among the one or more objects, in a cross-sectional view; and prevent keypoint coordinates, among the pseudo keypoint coordinates that are projected onto the Lidar space, that are equal to or less than a reference value in the reliability for the depths of the keypoints, from being bilaterally symmetrical to each other. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the controller is configured to: when i) the pseudo keypoint coordinates are projected onto the Lidar space and ii) keypoint coordinates are present at a relevant position in the Lidar space, compare reliability between the depths of the keypoint and the pseudo keypoint, and select a keypoint having higher reliability. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein the controller is further configured to: remove, from the Lidar space, keypoint coordinates, among the keypoint coordinates projected onto the Lidar space, that are outside of the bounding box for each object among the one or more objects, and correct reliability of the keypoint coordinates to zero if the pseudo keypoint coordinates are projected onto the Lidar space. ### Claim 12 (depends on 1) The apparatus of claim 1, wherein the controller is further configured to: learn an operation for generating the information about the 3D bounding box corresponding to the one or more objects and the information about the 3D keypoint corresponding to the one or more objects, based on the sensor data obtained from the camera and the Lidar; and output learning data generated as a learning result. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the controller is configured to: learn information about the 3D keypoint if reliability for a depth of the 3D keypoint exceeds a reference value. ### Claim 14 (depends on 12) The apparatus of claim 12, wherein the controller is configured to: when learning the information about the 3D keypoint, calculate loss of the depth of the 3D keypoint by employing a reliability for the depth of the 3D keypoint as a weight, and when estimating the 3D keypoint based on learning data, reflect the loss of the depth of the 3D keypoint. ### Claim 15 A method for controlling vehicle, the method comprising: generating information about a 3D bounding box and information about a 2D keypoint, corresponding to each object among one or more objects positioned around the vehicle, based on sensor data obtained from a camera and a Lidar; estimating a depth of keypoints of each object among the one or more objects, based on the information about the 3D bounding box and the information about the information about the 2D keypoint to generate information about a 3D keypoint for each object among the one or more objects; and recognizing each object, among the one or more objects, based on the information about the 3D bounding box and the information about the 3D keypoint. ### Claim 16 (depends on 15) The method of claim 15, wherein generating the information about the 3D keypoint for each object among the one or more objects includes: matching a bounding box for the object with a keypoint cluster; selecting Lidar points, among Lidar points obtained from the Lidar, that are positioned inside each bounding box; projecting the selected Lidar points onto an image obtained from the camera; calculating a keypoint weight using the selected Lidar points and information about keypoints matched to the bounding box; calculating depths of the keypoints by applying the keypoint weight to depths of the Lidar points; calculating reliability for the depths of the keypoints based on a distribution state of the Lidar points for a surrounding region of the keypoint; generating pseudo keypoint coordinates based on coordinates and depth values of the keypoints; and projecting the generated pseudo keypoint coordinates onto a Lidar space. ### Claim 17 (depends on 16) The method of claim 16, wherein generating the information about the 3D keypoint for each object among the one or more objects includes: allowing pseudo keypoint coordinates, that are projected onto the Lidar space, to be bilaterally symmetrical to each other about a center of the bounding box of each object among the one or more objects, in a cross-sectional view; and preventing keypoint coordinates, among the pseudo keypoint coordinates that are projected onto the Lidar space, that are equal to or less than a reference value in the reliability for the depths of the keypoints, from being bilaterally symmetrical to each other. ### Claim 18 (depends on 16) The method of claim 16, wherein generating the information about the 3D keypoint for each object among the one or more objects includes: when i) the pseudo keypoint coordinates are projected onto the Lidar space and ii) keypoint coordinates are present at a relevant position in the Lidar space comparing reliability between the depths of the keypoint and the pseudo keypoint, and selecting a keypoint having higher reliability. ### Claim 19 (depends on 16) The method of claim 16, wherein generating the information about the 3D keypoint for each object among the one or more objects includes: removing, from the Lidar space, keypoint coordinates, among keypoint coordinates projected onto the Lidar space, that are out of the bounding box of the object; and correcting reliability for relevant keypoint coordinates to zero when the pseudo keypoint coordinates are projected onto the Lidar space. ### Claim 20 (depends on 15) The method of claim 15, further comprising: learning an operation for generating the information about the 3D bounding box corresponding to the one or more objects and the information about the 3D keypoint corresponding to the one or more objects, based on the sensor data obtained from the camera and the Lidar; and outputting learning data generated as a learning result.