# US 11,443,531 B1 · Methods and apparatus for automatic collection of under-represented data for improving a training of a machine learning model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,531 B1 - Document ID: US11443531B1 - Status: Granted - Kind: B1 - Application number: 17/553,276 - Filed: 2021-12-16 - Granted: 2022-09-13 - Earliest claimed: 2021-12-16 - Assignees: PlusAI, Inc. - Applicants: PlusAI, Inc. - Inventors: Inderjot Saggu; Mianwei Zhou; Ankur Agarwal; Anurag Ganguli - Web page: https://patentgrep.com/patent/US11443531B1 ## CPC - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - G06N 20/00: Machine learning - G06T 7/74: {involving reference images or patches} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06T 2207/20081: Training; Learning - G06T 2207/30256: Lane; Road marking - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract In some embodiments, a method can include executing a first machine learning model to detect at least one lane in each image from a first set of images. The method can further include determining an estimate location of a vehicle for each image, based on localization data captured using at least one localization sensor disposed at the vehicle. The method can further include selecting lane geometry data for each image, from a map and based on the estimate location of the vehicle. The method can further include executing a localization model to generate a set of offset values for the first set of images based on the lane geometry data and the at least one lane in each image. The method can further include selecting a second set of images from the first set of images based on the set of offset values and a previously-determined offset threshold. ## Claims ### Claim 1 A method, comprising: receiving a first plurality of images; executing a first machine learning model to detect at least one lane in each image from the first plurality of images; determining an estimate location of a vehicle for each image from the first plurality of images, based on localization data captured using at least one localization sensor disposed at the vehicle; selecting lane geometry data for each image from the first plurality of images, from a map and based on the estimate location of the vehicle; executing a localization model to generate a plurality of offset values for the first plurality of images based on the lane geometry data and the at least one lane in each image from the first plurality of images; selecting a second plurality of images from the first plurality of images based on the plurality of offset values and a previously-determined offset threshold; and collecting training data based on the second plurality of images. ### Claim 2 (depends on 1) The method of claim 1, further comprising: training the first machine learning model based on the training data. ### Claim 3 (depends on 1) The method of claim 1, further comprising: executing a second machine learning model to classify each image from the second plurality of images to a scenario from a plurality of scenarios, to produce a plurality of classified images; and selecting a third plurality images from the plurality of classified images based on at least one previously-determined scenario of interest from the plurality of scenarios. ### Claim 4 (depends on 1) The method of claim 1, wherein the first plurality of images is ordered in a time sequence, the method further comprising: selecting a first image from the first plurality of images, the first image being associated with a first time in the time sequence and based on a previously-determined offset threshold; selecting a second image from the first plurality of images, the second image being associated with a second time being after the first time in the time sequence and based on a previously-determined offset threshold; and selecting the second plurality of images such that every image in the time sequence from the first image to the second image being included in the second plurality of images. ### Claim 5 (depends on 1) The method of claim 1, wherein the first plurality of images are captured by at least one forward-facing camera disposed at a vehicle. ### Claim 6 (depends on 1) The method of claim 1, wherein each offset value from the plurality of offset values is a physical measure of distance. ### Claim 7 (depends on 1) The method of claim 1, wherein the at least one localization sensor includes at least one of a Global Position System (GPS) sensor, an odometry sensor, a Light Detection and Ranging (LIDAR) sensor, a Real-Time Kinematic (RTK) sensor, or an Inertial Measurement Unit (IMU) sensor. ### Claim 8 (depends on 1) The method of claim 1, further comprising: capturing a sequence of inertial measurement unit (IMU) data using at least one IMU sensor; associating the sequence of IMU data to the first plurality of images; and projecting the at least one lane detected from each image from the first plurality of images to a subset of IMU data from the sequence of IMU data, associated with the image. ### Claim 9 (depends on 1) The method of claim 1, further comprising: executing a third machine learning model to generate an indication of a location of the vehicle with respect to the at least one lane based on the at least one lane, each image from the first plurality of images, and the offset value. ### Claim 10 (depends on 1) The method of claim 1, wherein the plurality of offset values includes at least one of a pitch offset value, a yaw offset value, a lateral offset value, or a longitudinal offset value. ### Claim 11 (depends on 1) The method of claim 1, wherein each offset value from the plurality of offset values is a quantification of a difference in a shape of at least one lane and a shape of the lane geometry data. ### Claim 12 (depends on 1) The method of claim 1, wherein the first plurality of images are generated using point intensities acquired using a LIDAR sensor. ### Claim 13 A system, comprising: at least one camera disposed at a vehicle and configured to capture a first plurality of images; at least one localization sensor disposed at the vehicle and configured to capture localization data; a memory operatively coupled to the at least one camera and the at least one localization sensor, the memory configured to store the first plurality of images and the localization data; and a processor operatively coupled to the memory, the processor configured to: execute a machine learning model to detect at least one lane in each image from the first plurality of images; determine an estimate location of the vehicle for each image from the first plurality of images, based on the localization data; select lane geometry data for each image from the first plurality of images, from a map and based on the estimate location of the vehicle; execute a localization model to generate a plurality of offset values for the first plurality of images based on the lane geometry data and the at least one lane in each image from the first plurality of images; selecting a second plurality of images from the first plurality of images based on the plurality of offset values and a previously-determined offset threshold; and training the machine learning model based on the second plurality of images. ### Claim 14 (depends on 13) The system of claim 13, wherein the first plurality of images is ordered in a time sequence, the method further comprising: selecting a first image from the first plurality of images, associated with a first time in the time sequence and based on a previously-determined offset threshold; selecting a second image from the first plurality of images, associated with a second time being after the first time in the time sequence and based on a previously-determined offset threshold; and selecting the second plurality of images such that every image in the time sequence from the first image associated with the first time to the second image associated with the second time being included in the second plurality of images. ### Claim 15 (depends on 13) The system of claim 13, wherein each offset value from the plurality of offset values is a physical measure of distance. ### Claim 16 (depends on 13) The system of claim 13, wherein the at least one localization sensor includes at least one of a Global Position System (GPS) sensor, an odometry sensor, a Light Detection and Ranging (LIDAR) sensor, a Real-Time Kinematic (RTK) sensor, or an Inertial Measurement Unit (IMU) sensor. ### Claim 17 (depends on 13) The system of claim 13, wherein the plurality of offset values includes at least one of a pitch offset value, a yaw offset value, a lateral offset value, or a longitudinal offset value. ### Claim 18 A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the code comprising code to cause the processor to: receive a first plurality of images; execute a first machine learning model to detect at least one lane in each image from the first plurality of images; determine an estimate location of a vehicle for each image from the first plurality of images, based on localization data captured using at least one localization sensor disposed at the vehicle; select lane geometry data for each image from the first plurality of images, from a map and based on the estimate location of the vehicle; execute a localization model to generate a plurality of offset values for the first plurality of images based on the lane geometry data and the at least one lane in each image from the first plurality of images; select a second plurality of images from the first plurality of images based on the plurality of offset values and a previously-determined offset threshold; and collect training data based on the second plurality of images. ### Claim 19 (depends on 18) The non-transitory processor-readable medium of claim 18, wherein the code further comprise code to cause the processor to: train the first machine learning model based on the training data. ### Claim 20 (depends on 18) The non-transitory processor-readable medium of claim 18, wherein the code further comprise code to cause the processor to: execute a second machine learning model to classify each image from the second plurality of images to a scenario from a plurality of scenarios, to produce a plurality of classified images; and select a third plurality images from the plurality of classified images based on at least one previously-determined scenario of interest from the plurality of scenarios. ### Claim 21 (depends on 18) The non-transitory processor-readable medium of claim 18, wherein: the first plurality of images is ordered in a time sequence; and the code further comprise code to cause the processor to: select a first image from the first plurality of images, the first image being associated with a first time in the time sequence and based on a previously-determined offset threshold; select a second image from the first plurality of images, the second image being associated with a second time being after the first time in the time sequence and based on a previously-determined offset threshold; and select the second plurality of images such that every image in the time sequence from the first image to the second image being included in the second plurality of images. ### Claim 22 (depends on 18) The non-transitory processor-readable medium of claim 18, wherein the first plurality of images are captured by at least one forward-facing camera disposed at the vehicle.