Augmented content generation with language model for assisting operation of smart cart

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

• receiving sensor data from one or more sensors of a smart cart in operation by a picker at a source location;
• identifying a triggering event from a plurality of types of triggering events based on the sensor data;
• retrieving a template associated with the identified triggering event, wherein the template comprises instructions for generating one or more suggestions for the picker to augment operation of the smart cart;
• obtaining contextual information associated with operation of the smart cart by the picker;
• generating a prompt by modifying the template comprising the instructions to include the sensor data and the contextual information;
• transmitting, to a model serving system, the prompt for execution by a language model;
• receiving, from the model serving system, a response output by the language model;
• generating augmented content including visual content including text describing one or more suggestions for the picker by parsing the response output by the language model;
• and
• causing display of the augmented content to on an electronic display of the smart cart during operation of the smart cart.

A system comprising:

• a processor;
• and
• a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform operations comprising:
• receiving sensor data from one or more sensors of a smart cart in operation by a picker at a source location;
• identifying a triggering event from a plurality of types of triggering events based on the sensor data;
• retrieving a template associated with the identified triggering event, wherein the template comprises instructions for generating one or more suggestions for the picker to augment operation of the smart cart;
• obtaining contextual information associated with operation of the smart cart by the picker;
• generating a prompt by modifying the template comprising the instructions to include the sensor data and the contextual information;
• transmitting, to a model serving system, the prompt for execution by a language model;
• receiving, from the model serving system, a response output by the language model;
• generating augmented content including visual content including text describing one or more suggestions for the picker by parsing the response output by the language model;
• and
• causing display of the augmented content on an electronic display of the smart cart during operation of the smart cart.