# US 2024/0203129 A1 · Ground plane fitting method, vehicle-mounted device and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0203129 A1 - Document ID: US20240203129A1 - Status: Published application - Kind: A1 - Application number: 18/135,037 - Filed: 2023-04-14 - Published: 2024-06-20 - Earliest claimed: 2022-12-19 - Assignees: Hon Hai Precision Industry Co., Ltd. - Applicants: Hon Hai Precision Industry Co., Ltd. - Inventors: JUNG-HAO YANG; CHIN-PIN KUO; CHIH-TE LU - Web page: https://patentgrep.com/patent/US20240203129A1 ## CPC - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - B60W 40/10: related to vehicle motion - B60W 2420/403: Image sensing, e.g. optical camera - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract A ground plane fitting method applied to a vehicle-mounted device is provided. In the method, the vehicle-mounted device acquires a plurality of point clouds of a scene front of a vehicle along a traveling direction and a target image and determines a set of ground point clouds corresponding to the target image according to the plurality of point clouds and the target image. The vehicle-mounted device further obtains multiple ground normal vectors by correcting multiple normal vectors of multiple cameras using to acquire the target images; and fits the ground plane in the traveling direction of the vehicle according to the set of ground point clouds and the obtained ground normal vectors to obtain a fitted ground plane. The method can improve an accuracy of the obtained ground normal vector, thereby effectively improving the accuracy of fitting the ground plane and assisting the safe of the self-driving vehicle. ## Claims ### Claim 1 A ground plane fitting method using a vehicle-mounted device, the method comprising: acquiring a plurality of point clouds of a scene front of a vehicle along a traveling direction and a target image; determining a set of ground point clouds corresponding to the target image according to the plurality of point clouds and the target image; obtaining a plurality of ground normal vectors by correcting a plurality of camera normal vectors of a camera that acquires the target image; and obtaining a fitted ground plane of a ground of the scene along the traveling direction of the vehicle by fitting the set of ground point clouds and the plurality of ground normal vectors. ### Claim 2 (depends on 1) The ground plane fitting method according to claim 1, wherein determining the set of ground point clouds corresponding to the target image according to the plurality of point clouds and the target image comprises: determining an area of the ground of the target image; and setting a set of point clouds corresponding to the area of the ground as a set of ground point clouds by projecting the point clouds onto the target image. ### Claim 3 (depends on 2) The ground plane fitting method according to claim 2, wherein the area of the ground is determined by using a semantic segmentation algorithm to the target image. ### Claim 4 (depends on 1) The ground plane fitting method according to claim 1, wherein obtaining the plurality of ground normal vectors by correcting the plurality of camera normal vectors of the camera that acquires the target image comprises: obtaining the plurality of camera normal vectors, which comprise timestamps of the camera; and acquiring movement data of the camera and obtaining the plurality of ground normal vectors by correcting the plurality of camera normal vectors according to the movement data. ### Claim 5 (depends on 4) The ground plane fitting method according to claim 4, further comprising: setting a coordinate axis pointing to a sky of a camera coordinate system of the camera as the camera normal vector. ### Claim 6 (depends on 5) The ground plane fitting method according to claim 5, further comprising: acquiring a plurality of images comprising the target image that are taken by the camera, each image of the plurality of images corresponding to a timestamp; and determining a camera coordinate system corresponding to each image. ### Claim 7 (depends on 4) The ground plane fitting method according to claim 4, wherein obtaining the plurality of ground normal vectors by correcting the plurality of camera normal vectors according to the movement data comprises: obtaining a gravitational acceleration from the movement data; and correcting the camera normal vector to the ground normal vector by correcting a rotation matrix of a camera pose of the camera according to a direction of the gravitational acceleration. ### Claim 8 A vehicle-mounted device comprising: a storage device; at least one processor; and the storage device storing one or more programs, which when executed by the at least one processor, cause the at least one processor to: acquire a plurality of point clouds of a scene front of a vehicle along a traveling direction and a target image; determine a set of ground point clouds corresponding to the target image according to the plurality of point clouds and the target image; obtain a plurality of ground normal vectors by correcting a plurality of camera normal vectors of a camera that acquires the target image; and obtain a fitted ground plane of a ground of the scene along the traveling direction of the vehicle by fitting the set of ground point clouds and the plurality of ground normal vectors. ### Claim 9 (depends on 8) The vehicle-mounted device according to claim 8, wherein the at least one processor determines the set of ground point clouds corresponding to the target image according to the plurality of point clouds and the target image by: determining an area of the ground of the target image; and setting a set of point clouds corresponding to the area of the ground as a set of ground point clouds by projecting the point clouds onto the target image. ### Claim 10 (depends on 9) The vehicle-mounted device according to claim 9, wherein the area of the ground is determined by using a semantic segmentation algorithm to the target image. ### Claim 11 (depends on 8) The vehicle-mounted device according to claim 8, wherein the at least one processor obtains the plurality of ground normal vectors by correcting the plurality of camera normal vectors of the camera that acquires the target image by: obtaining the plurality of camera normal vectors, which comprise timestamps of the camera; and acquiring movement data of the camera and obtaining the plurality of ground normal vectors by correcting the plurality of camera normal vectors according to the movement data. ### Claim 12 (depends on 11) The vehicle-mounted device according to claim 11, wherein the at least one processor is further caused to: set a coordinate axis pointing to a sky of a camera coordinate system of the camera as the camera normal vector. ### Claim 13 (depends on 12) The vehicle-mounted device according to claim 12, wherein the at least one processor is further caused to: acquire a plurality of images comprising the target image that are taken by the camera, each image of the plurality of images corresponding to a timestamp; and determine a camera coordinate system corresponding to each image. ### Claim 14 (depends on 11) The vehicle-mounted device according to claim 11, wherein the at least one processor obtains the plurality of ground normal vectors by correcting the plurality of camera normal vectors according to the movement data by: obtaining a gravitational acceleration from the movement data; and correcting the camera normal vector to the ground normal vector by correcting a rotation matrix of a camera pose of the camera according to a direction of the gravitational acceleration. ### Claim 15 A non-transitory storage medium having instructions stored thereon, when the instructions are executed by a processor of a vehicle-mounted device, the processor is caused to perform a ground plane fitting method, wherein the method comprises: acquiring a plurality of point clouds of a scene front of a vehicle along a traveling direction and a target image; determining a set of ground point clouds corresponding to the target image according to the plurality of point clouds and the target image; obtaining a plurality of ground normal vectors by correcting a plurality of camera normal vectors of a camera that acquires the target image; and obtaining a fitted ground plane of a ground of the scene along the traveling direction of the vehicle by fitting the set of ground point clouds and the plurality of ground normal vectors. ### Claim 16 (depends on 15) The non-transitory storage medium according to claim 15, wherein determining the set of ground point clouds corresponding to the target image according to the plurality of point clouds and the target image comprises: determining an area of the ground of the target image; and setting a set of point clouds corresponding to the area of the ground as a set of ground point clouds by projecting the point clouds onto the target image. ### Claim 17 (depends on 16) The non-transitory storage medium according to claim 16, wherein the area of the ground is determined by using a semantic segmentation algorithm to the target image. ### Claim 18 (depends on 15) The non-transitory storage medium according to claim 15, wherein obtaining the plurality of ground normal vectors by correcting the plurality of camera normal vectors of the camera that acquires the target image comprises: obtaining the plurality of camera normal vectors, which comprise timestamps of the camera; and acquiring movement data of the camera and obtaining the plurality of ground normal vectors by correcting the plurality of camera normal vectors according to the movement data. ### Claim 19 (depends on 18) The non-transitory storage medium according to claim 18, wherein the method further comprises: setting a coordinate axis pointing to a sky of a camera coordinate system of the camera as the camera normal vector. ### Claim 20 (depends on 19) The non-transitory storage medium according to claim 19, wherein the method further comprises: acquiring a plurality of images comprising the target image that are taken by the camera, each image of the plurality of images corresponding to a timestamp; and determining a camera coordinate system corresponding to each image.