# US 11,908,198 B2 · Contextualization and refinement of simultaneous localization and mapping > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,198 B2 - Document ID: US11908198B2 - Status: Granted - Kind: B2 - Application number: 17/205,314 - Filed: 2021-03-18 - Published: 2022-09-22 - Granted: 2024-02-20 - Earliest claimed: 2021-03-18 - Assignees: Pony AI Inc. - Applicants: Pony AI Inc. - Inventors: Mengda Yang; Weixin Jiang; Zheng Yan - Web page: https://patentgrep.com/patent/US11908198B2 ## CPC - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60W 60/001: {Planning or execution of driving tasks} - G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) - B60W 2420/408: Radar; Laser, e.g. lidar - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract A computing system includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations. The operations include obtaining sensor data from a sensor of a vehicle, the sensor data including point cloud frames at different positions, orientations, and times, the sensor data used to generate a map, determining a position and an orientation of the sensor corresponding to a capture of each of the point cloud frames according to a simultaneous localization and mapping (SLAM) algorithm, and depicting, on an interface, a graphical illustration of the determined positions at which the point cloud frames were captured. ## Claims ### Claim 1 A computing system comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the computing system to perform: obtaining sensor data from a sensor of a vehicle, the sensor data including point cloud frames at different positions, orientations, and times, the sensor data used to generate a map; determining a position and an orientation of the sensor corresponding to a capture of each of the point cloud frames according to a simultaneous localization and mapping (SLAM) algorithm; and depicting, on an interface, a graphical illustration of the determined positions at which the point cloud frames were captured, wherein the graphical illustration includes trajectory points indicating the determined positions and connectors connecting the trajectory points to indicate loop closure constraints among the trajectory points; receiving a selection of a loop closure constraint; and in response to receiving the selection of the loop closure constraint, displaying data of the loop closure constraint, the data indicating second trajectory points subject to the loop closure constraint. ### Claim 2 (depends on 1) The computing system of claim 1, wherein the graphical illustration includes trajectory points indicating the determined positions and lines connecting the trajectory points to indicate time relationships or constraints between the trajectory points. ### Claim 3 (depends on 2) The computing system of claim 2, wherein the trajectory points are depicted at different stages of the SLAM algorithm, the stages including after capture of the sensor data, after GNSS-IMU post processing, and after point cloud registration. ### Claim 4 (depends on 2) The computing system of claim 2, wherein the instructions further cause the computing system to perform: receiving a selection of a trajectory point of the trajectory points; and displaying a point cloud frame corresponding to the trajectory point. ### Claim 5 (depends on 4) The computing system of claim 4, wherein the instructions further cause the computing system to perform: displaying a second point cloud frame resulting from a registration of the point cloud frame according to a source point cloud. ### Claim 6 (depends on 2) The computing system of claim 2, wherein the instructions further cause the computing system to perform: receiving a selection of a constraint; and displaying data of the constraint, the data indicating second trajectory points subject to the constraint and whether the constraint is a soft constraint or a hard constraint. ### Claim 7 (depends on 6) The computing system of claim 6, wherein the instructions further cause the computing system to perform: displaying information of the trajectory points connected by the constraint and associated with the constraint; displaying point cloud frames corresponding to the trajectory points; and displaying a resulting point cloud frame generated from a merging or an aligning operation of the point cloud frames corresponding to the trajectory points. ### Claim 8 (depends on 7) The computing system of claim 7, wherein the resulting point cloud frame is generated under a condition that the constraint is only partially satisfied; and the instructions further cause the one or more processors to perform: displaying a second resulting point cloud frame generated from a merging or an aligning operation of the point cloud frames under a second condition that the constraint is fully satisfied. ### Claim 9 (depends on 2) The computing system of claim 2, wherein the instructions further cause computing system to perform: receiving an indication to delete a constraint of the constraints; redetermining the position and the orientation of the sensor corresponding to the capture of each of the point cloud frames according to the deleted constraint; and depicting, on an interface, a graphical illustration of the redetermined positions at which the point cloud frames were captured. ### Claim 10 (depends on 9) The computing system of claim 9, wherein the instructions further cause the computing system to perform: generating the map according to the sensor data and the determined position and orientation of the sensor; and transforming the map according to the redetermined position and the orientation of the sensor and the removal of the constraint. ### Claim 11 (depends on 2) The computing system of claim 2, wherein at least a portion of the constraints correspond to a fusion of GNSS data and IMU data. ### Claim 12 A computer-implemented method, comprising: obtaining sensor data from a sensor of a vehicle, the sensor data including point cloud frames at different positions, orientations, and times, the sensor data used to generate a map; determining a position and an orientation of the sensor corresponding to a capture of each of the point cloud frames according to a simultaneous localization and mapping (SLAM) algorithm; and depicting, on an interface, a graphical illustration of the determined positions at which the point cloud frames were captured, wherein the graphical illustration includes trajectory points indicating the determined positions and connectors connecting the trajectory points to indicate loop closure constraints among the trajectory points; receiving a selection of a loop closure constraint; and in response to receiving the selection of the loop closure constraint, displaying data of the loop closure constraint, the data indicating second trajectory points subject to the loop closure constraint. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein the graphical illustration includes trajectory points indicating the determined positions and lines connecting the trajectory points to indicate time relationships or constraints between the trajectory points. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein the trajectory points are depicted at different stages of the SLAM algorithm, the stages including after capture of the sensor data, after GNSS-IMU post processing, and after point cloud registration. ### Claim 15 (depends on 13) The computer-implemented method of claim 13, further comprising: receiving a selection of a trajectory point of the trajectory points; and displaying a point cloud frame corresponding to the trajectory point. ### Claim 16 (depends on 15) The computer-implemented method of claim 15, further comprising: displaying a second point cloud frame resulting from a registration of the point cloud frame according to a source point cloud. ### Claim 17 (depends on 13) The computer-implemented method of claim 13, further comprising: receiving a selection of a constraint; and displaying data of the constraint, the data indicating second trajectory points subject to the constraint and whether the constraint is a soft constraint or a hard constraint. ### Claim 18 (depends on 17) The computer-implemented method of claim 17, further comprising: displaying information of the trajectory points connected by the constraint and associated with the constraint; displaying point cloud frames corresponding to the trajectory points; and displaying a resulting point cloud frame generated from a merging or an aligning operation of the point cloud frames corresponding to the trajectory points. ### Claim 19 (depends on 18) The computer-implemented method of claim 18, wherein the resulting point cloud frame is generated under a condition that the constraint is only partially satisfied, and the computer-implemented method further comprises: displaying a second resulting point cloud frame generated from a merging or an aligning operation of the point cloud frames under a second condition that the constraint is fully satisfied. ### Claim 20 (depends on 13) The computer-implemented method of claim 13, further comprising: receiving an indication to delete a constraint of the constraints; redetermining the position and the orientation of the sensor corresponding to the capture of each of the point cloud frames according to the deleted constraint; and depicting, on an interface, a graphical illustration of the redetermined positions at which the point cloud frames were captured.