# US 12,759,583 B2 · System and method of gamified active learning for vehicle perception systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,759,583 B2 - Document ID: US12759583B2 - Status: Granted - Kind: B2 - Application number: 18/469,597 - Filed: 2023-09-19 - Published: 2025-03-20 - Granted: 2026-10-13 - Earliest claimed: 2023-09-19 - Assignees: GM Global Technology Operations LLC - Applicants: GM Global Technology Operations LLC - Inventors: Alireza Esna Ashari Esfahani; Brent Navin Roger Bacchus; Upali P. Mudalige - Web page: https://patentgrep.com/patent/US12759583B2 ## CPC - B60W 60/0015: {specially adapted for safety} - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2554/4049: Relationship among other objects, e.g. converging dynamic objects - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system and method of gamified learning for a vehicle is provided. The system includes a perception model configured to receive perception data from a camera, execute a perception task to obtain a task result for the object, determine the perception confidence level of the task result is below a predetermined perception confidence threshold, show an image of the object to an on-scene user, request the on-scene user to annotate the object, and reward the on-scene user in response to receiving the annotation from the on-scene user. The system further includes a server configured to aggregate a plurality of annotations from a plurality of users, apply a probabilistic model to determine a ground truth annotation, and input the ground truth annotation and an image to a machine learning model to update the perception model. ## Claims ### Claim 1 A method of gamified active learning for a perception system of a vehicle, comprising: applying a perception model to execute a perception task on an object to obtain a task result, wherein the perception model is based on a machine learning model; determining a perception confidence level for the task result; determining the perception confidence level is below a predetermined perception confidence level; showing an image of the object to an on-scene user; requesting the on-scene user to annotate the object; providing an annotation of the object by the on-scene user, wherein the annotation is based on a visual observation of the object by the on-scene user; rewarding the on-scene user in response to the on-scene user providing the annotation of the object; and training the machine learning model with the image of the object and the annotation of the object by the on-scene user to update the perception model; wherein requesting the on-scene user to annotate the object includes: requesting the on-scene user to answer a predetermined question directed to the object, wherein the predetermined question is selected from a group consisting of a freeform answer question, a multiple choice question, and a binary choice answer question. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining a choice of annotation for the object does not exist on a server before requesting the on-scene user to annotate the object; and selecting the freeform answer question in response to the choice of annotation does not exist on the server. ### Claim 3 (depends on 1) The method of claim 1, further comprising: determining a choice of annotation for the object exists on a server before requesting the on-scene user to annotate the object; determining an annotation confidence level of the choice of annotation for the object is less than a predetermined annotation confidence threshold; and selecting the multiple choice question in response to the choice of annotation for the object is less than the predetermined annotation confidence threshold. ### Claim 4 (depends on 1) The method of claim 1, further comprising: determining a choice of annotation for the object exists on a server before requesting the on-scene user to annotate the object; determining an annotation confidence level of the choice of annotation for the object is equal to or greater a predetermined annotation confidence threshold; and selecting the binary choice question in response to the annotation confidence level of the choice of annotation for the object is equal to or greater than the predetermined annotation confidence threshold and the choice of annotation for the object exists on the server before requesting the on-scene user to annotate the object. ### Claim 5 (depends on 4) The method of claim 4, further comprises: providing annotations of a same object by a plurality of on-scene users; aggregating the annotations of the same object; applying a probabilistic model on the aggregated annotations to determine a ground truth annotation of the same object; and training the machine learning model with an image of the same object and the ground truth annotation of the same object. ### Claim 6 (depends on 5) The method of claim 5, wherein the annotations provided by the plurality of on-scene users are based on the binary choice question. ### Claim 7 (depends on 5) The method of claim 5, wherein the probabilistic model is a Dirichlet distribution. ### Claim 8 (depends on 5) The method of claim 5, wherein the same object is observed at different time periods and at different viewing perspectives. ### Claim 9 (depends on 1) The method of claim 1, further comprising determining the on-scene user is credible before requesting the on-scene user to answer the predetermined question. ### Claim 10 A gamified active learning system for a vehicle, comprising: a human-machine-interface (HMI) configured to display an image to an occupant of the vehicle; at least one external viewing camera; and a perception module having a perception model configured to: receive perception data from the at least one external viewing camera, analyze the received perception data to detect an object, execute a perception task to obtain a task result for the object, determine a perception confidence level of the task result, determine the perception confidence level of the task result is below a predetermined perception confidence threshold, send a signal to the HMI to show an image of the object to an on-scene user, request the on-scene user to annotate the object based on a visual inspection of the object by the on-scene user, wherein the request to the on-scene user to annotate includes answering a predetermined question selected from a group consisting of a freeform answer question directed to the object, a multiple choice question directed to the object, and a binary choice answer question directed to the object; receive an annotation from the on-scene user, and reward the on-scene user in response to receiving the annotation from the on-scene user; wherein the perception module having the perception model is further configured to determine one of: (i) a choice of annotation for the object exists on a server, and (ii) the choice of annotation for the object does not exist on a server; determine one of: (i) an annotation confidence level of the choice of annotation for the object is less than a predetermined annotation confidence threshold, and (ii) the annotation confidence level of the choice of annotation for the object is equal to or greater than the predetermined annotation confidence threshold; and selecting one of: (i) the freeform answer question in response to the choice of annotation for the object does not exist on the server; (ii) the multiple choice question in response to the choice of annotation for the object exists on the server and the annotation confidence level of the choice of annotation is less than the predetermined annotation confidence threshold; and (iii) the binary question in response to the choice of annotation for the object exists on the server and the annotation confidence level of the choice of annotation is equal to or greater than the predetermined annotation confidence threshold. ### Claim 11 (depends on 10) The gamified active learning system for a vehicle of claim 10, further comprising: a remote server configured to: aggregate a plurality of annotations from a plurality of users; apply a probabilistic model on the aggregated annotations to determine a ground truth annotation of the same object; and input the ground truth annotation and an image of the object to a machine learning model to update the perception model. ### Claim 12 (depends on 11) The gamified active learning system for a vehicle of claim 11, wherein the perception task is to determine a classification of the object and the task result is a predicted classification of the object. ### Claim 13 (depends on 11) The gamified active learning system for a vehicle of claim 11, wherein the object is a street sign, the perception task is an extraction of text from the street sign, and the task result is a prediction of a name on the street sign based on the extracted text. ### Claim 14 (depends on 10) The gamified active learning system for a vehicle of claim 10, wherein the HMI is a personal electronic device off-board the vehicle. ### Claim 15 A perception module of a vehicle comprising: a processor; and a non-transitory computer readable storage device comprising a perception model stored thereon for gamified active learning, that upon execution of the perception model by the processor, cause the processor to: execute a perception task to obtain a task result for a detected object; determine a perception confidence level of the task result; determine the perception confidence level of the task result is below a predetermined perception confidence threshold; send a signal to a visual display to show an image of the object to an on-scene user in response to the perception confidence level of the task result is below the predetermined perception confidence threshold; request the on-scene user to answer a predetermined question regarding the object; receive an answer from the on-scene user based on a visual observation of the object by the on-scene user; determine one of: (i) a choice of annotation for the object exists on a server, and (ii) the choice of annotation for the object does not exist on a server; and determine one of: (i) an annotation confidence level of the choice of annotation for the object is less than a predetermined annotation confidence threshold, and (ii) the annotation confidence level of the choice of annotation for the object is equal to or greater than the predetermined annotation confidence threshold; and wherein the predetermined question regarding the object is selected from a group consisting of: (i) a freeform answer question in response to the choice of annotation for the object does not exist on the server; (ii) a multiple choice question in response to the choice of annotation for the object exists on the server and the annotation confidence level of the choice of annotation is less than the predetermined annotation confidence threshold; and (iii) a binary question in response to the choice of annotation for the object exists on the server and the annotation confidence level of the choice of annotation is equal to or greater than the predetermined annotation confidence threshold. ### Claim 16 (depends on 15) The perception module of claim 15, wherein the upon execution of the perception model by the processor, further cause the processor to: annotate the image based on the answer from the on-scene user; upload the annotated image to a remote server to train a machine learning model to update the perception model. ### Claim 17 (depends on 15) The perception module of claim 15, wherein the upon execution of the perception model by the processor, further cause the processor to: determine that the on-scene user is credible before requesting the on-scene user to answer a predetermined question regarding the object. ### Claim 18 (depends on 17) The perception module of claim 17, wherein the perception module is part of a perception system on a vehicle.