# US 12,001,175 B2 · Long tail lidar 3-D object detection improvement with targeted simulation data injection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,175 B2 - Document ID: US12001175B2 - Status: Granted - Kind: B2 - Application number: 17/556,561 - Filed: 2021-12-20 - Published: 2023-06-22 - Granted: 2024-06-04 - Earliest claimed: 2021-12-20 - Assignees: GM Cruise Holdings LLC - Applicants: GM Cruise Holdings LLC - Inventors: Abdelrahman Elogeel; Andres Hasfura; Alexander Pon; Debanjan Nandi; Carden Bagwell; Marzieh Parandehgheibi; Teng Liu - Web page: https://patentgrep.com/patent/US12001175B2 ## CPC - G05B 13/0265: {the criterion being a learning criterion} - G06N 20/00: Machine learning - B60W 50/0098: {Details of control systems ensuring comfort, safety or stability not otherwise provided for} - B60W 60/001: {Planning or execution of driving tasks} - B60W 2050/0088: {Adaptive recalibration} - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2554/402: Type - B60W 2554/801: Lateral distance - B60W 2554/802: Longitudinal distance - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract The subject disclosure relates to techniques for improving performance of a machine learning algorithm that at least receives data descriptive of a 3-D object and provides an output, where the 3-D object occurs infrequently in a training dataset. A process of the disclosed technology can include determining that the machine learning algorithm performed below a threshold performance score when receiving data descriptive of the 3-D object in a real-world scene, wherein the 3-D object is classified as a first type of object, creating at least one 3-D representation of the first type of object for use in a simulation, modifying a plurality of simulated scenes to include the at least one 3-D representation of the first type of object, and training the machine learning algorithm with the modified simulated scenes, whereby the machine learning algorithm has greater exposure to the first type of object. ## Claims ### Claim 1 A computer-implemented method comprising: determining that a machine learning algorithm performed below a threshold performance score when receiving data descriptive of a 3-D object in a real-world scene, wherein the 3-D object is classified as a first type of object; determining that the first type of object corresponds to a long-tail object, wherein the long-tail object has a low frequency of occurrence relative to other objects in a training dataset used to train the machine learning algorithm; searching a database of saved real-world scenarios to identify objects classified as the first type of object in the saved real-world scenarios, wherein the saved real-world scenarios include sensor data associated with the first type of object; generating at least one 3-D representation of the first type of object from the sensor data associated with the first type of object; modifying a plurality of simulated scenes to include the at least one 3-D representation of the first type of object to yield a plurality of modified simulated scenes; and training the machine learning algorithm with the plurality of modified simulated scenes, whereby the machine learning algorithm has greater exposure to the first type of object. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein modifying the plurality of simulated scenes to include the at least one 3-D representation of the first type of object further comprises: varying a pose of the at least one 3-D representation of the first type of object across the plurality of simulated scenes. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein modifying the plurality of simulated scenes to include the at least one 3-D representation of the first type of object further comprises: varying a distance of the at least one 3-D representation of the first type of object relative to a simulated sensor capturing the data descriptive of the at least one 3-D representation of the first type of object. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the data descriptive of the 3-D object in the real-world scene is Light Detection and Ranging (LiDAR) point cloud data. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the machine learning algorithm is part of an autonomous vehicle (AV) computing stack of an AV responsible for receiving data from at least one sensor and providing outputs effective to control actuators to autonomously pilot the AV. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: storing the at least one 3 D representation of the first type of object for use in a database of the first type of object, wherein the at least one 3-D representation of the first type of object is one of a plurality of the first type of object in the database. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, further comprising: randomly selecting one of the first type of object from the database each time a simulated scene is to be modified to include at least one 3-D representation of the plurality of the first type of object. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein modifying the plurality of simulated scenes to include the at least one 3-D representation of the first type of object further comprises: modifying the plurality of simulated scenes based on one or more criteria. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the one or more criteria specifies where objects of the first type of object should be located within the plurality of simulated scenes. ### Claim 10 (depends on 8) The computer-implemented method of claim 8, wherein the plurality of simulated scenes is a subset of a total number of simulated scenes, wherein the one or more criteria specifies a ratio of the plurality of simulated scenes that are modified to include the first type of object to the total number of simulated scenes. ### Claim 11 A system comprising: a memory configured to store instructions; and a processor configured to execute the instructions and cause the processor to: determine that a machine learning algorithm performed below a threshold performance score when receiving data descriptive of a 3-D object in a real-world scene, wherein the 3-D object is classified as a first type of object; determine that the first type of object corresponds to a long-tail object, wherein the long-tail object has a low frequency of occurrence relative to other objects in a training dataset used to train the machine learning algorithm; search a database of saved real-world scenarios to identify objects classified as the first type of object in the saved real-world scenarios, wherein the saved real-world scenarios include sensor data associated with the first type of object; generate at least one 3-D representation of the first type of object from the sensor data associated with the first type of object; modify a plurality of simulated scenes to include the at least one 3-D representation of the first type of object to yield a plurality of modified simulated scenes; and train the machine learning algorithm with the plurality of modified simulated scenes, whereby the machine learning algorithm has greater exposure to the first type of object. ### Claim 12 (depends on 11) The system of claim 11, wherein the processor is configured to execute the instructions and cause the processor to: vary a pose of the at least one 3-D representation of the first type of object across the plurality of simulated scenes. ### Claim 13 (depends on 11) The system of claim 11, wherein the processor is configured to execute the instructions and cause the processor to: vary a distance of the at least one 3-D representation of the first type of object relative to a simulated sensor capturing the data descriptive of the at least one 3-D representation of the first type of object. ### Claim 14 (depends on 11) The system of claim 11, wherein the data descriptive of the 3-D object in the real-world scene is Light Detection and Ranging (LiDAR) point cloud data. ### Claim 15 (depends on 11) The system of claim 11, wherein the machine learning algorithm is part of an autonomous vehicle (AV) computing stack of an AV responsible for receiving data from at least one sensor and providing outputs effective to control actuators to autonomously pilot the AV. ### Claim 16 A non-transitory computer readable medium comprising instructions, the instructions, when executed by a computing system, cause the computing system to: determine that a machine learning algorithm performed below a threshold performance score when receiving data descriptive of a 3-D object in a real-world scene, wherein the 3-D object is classified as a first type of object; determine that the first type of object corresponds to a long-tail object, wherein the long-tail object has a low frequency of occurrence relative to other objects in a training dataset used to train the machine learning algorithm; search a database of saved real-world scenarios to identify objects classified as the first type of object in the saved real-world scenarios, wherein the saved real-world scenarios include sensor data associated with the first type of object; generate at least one 3-D representation of the first type of object from the sensor data associated with the first type of object; modify a plurality of simulated scenes to include the at least one 3-D representation of the first type of object to yield a plurality of modified simulated scenes; and train the machine learning algorithm with the plurality of modified simulated scenes, whereby the machine learning algorithm has greater exposure to the first type of object. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, further comprising instructions that, when executed by the computing system, cause the computing system to: vary a pose of the at least one 3-D representation of the first type of object across the plurality of simulated scenes. ### Claim 18 (depends on 16) The non-transitory computer readable medium of claim 16 further comprising instructions that, when executed by the computing system, cause the computing system to: vary a distance of the at least one 3-D representation of the first type of object relative to a simulated sensor capturing the data descriptive of the at least one 3-D representation of the first type of object. ### Claim 19 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the data descriptive of the 3-D object in the real-world scene is Light Detection and Ranging (LiDAR) point cloud data.