Automated Training Data Labeling for Fine-Tuning a Vision Language Model

A non-transitory computer-readable medium storing instructions that, when executed by a computing system, cause the computing system to perform operations comprising:

• accessing sensor data captured by a sensor coupled to a smart cart system;
• detecting, based on the sensor data, that an action has occurred with respect to the smart cart system, wherein the action comprises an item being added to a storage area of the smart cart system by a user;
• identifying a timestamp at which the action occurred based on the sensor data;
• accessing video data captured by a camera coupled to the smart cart system;
• identifying a portion of the video data that corresponds to the timestamp at which the action occurred;
• accessing label scanner data captured by a label scanner coupled to the smart cart system, wherein the label scanner data describes machine-readable labels that were scanned by the label scanner;
• identifying a machine-readable label corresponding to the item added to the storage area of the smart cart system based on the identified timestamp and the label scanner data;
• generating a training example for a vision language model based on the portion of the video data and the machine-readable label, wherein the training example comprises input data to the vision language model and a label for an intended output of the vision language model, wherein the input data comprises the identified portion of the video data and the label comprises an item identifier corresponding to the identified machine-readable label;
• updating a plurality of parameters of the vision language model based on the training example by:
• applying the vision language model to the input data of the training example to generate an output;
• and
• comparing the output of the vision language model with the label of the training example;
• and
• storing the updated parameters for the vision language model to a computer-readable medium.

A computing system comprising:

• a processor;
• and
• a non-transitory computer-readable medium storing instructions that, when executed by a computing system, cause the computing system to perform operations comprising:
• accessing sensor data captured by a sensor coupled to a smart cart system;
• detecting, based on the sensor data, that an action has occurred with respect to the smart cart system, wherein the action comprises an item being added to a storage area of the smart cart system by a user;
• identifying a timestamp at which the action occurred based on the sensor data;
• accessing video data captured by a camera coupled to the smart cart system;
• identifying a portion of the video data that corresponds to the timestamp at which the action occurred;
• accessing label scanner data captured by a label scanner coupled to the smart cart system, wherein the label scanner data describes machine-readable labels that were scanned by the label scanner;
• identifying a machine-readable label corresponding to the item added to the storage area of the smart cart system based on the identified timestamp and the label scanner data;
• generating a training example for a vision language model based on the portion of the video data and the machine-readable label, wherein the training example comprises input data to the vision language model and a label for an intended output of the vision language model, wherein the input data comprises the identified portion of the video data and the label comprises an item identifier corresponding to the identified machine-readable label;
• updating a plurality of parameters of the vision language model based on the training example by:
• applying the vision language model to the input data of the training example to generate an output;
• and
• comparing the output of the vision language model with the label of the training example;
• and
• storing the updated parameters for the vision language model to a computer-readable medium.