Visual event processing using language models

A method, comprising:

• receiving first user input data representing a first user command to store an activity indictor representing an occurrence of a visual event, the visual event associated with an activity in an environment;
• storing an activity indicator in a device;
• generating, utilizing the activity indicator stored in the device, a first query requesting that a language model (LM) identify the activity from image data, wherein the LM is disposed on the device and is configured to determine results utilizing the image data as the input;
• determining, at the device utilizing the LM and the first query, that the image data depicts the activity;
• storing first data indicating occurrence of the visual event based at least in part on the LM determining that the image data depicts the activity;
• receiving second user input data representing a natural language request associated with stored visual events;
• determining, utilizing the LM and based at least in part on the second user input data, that the natural language request is associated with the visual event;
• and
• generating a first natural language response to the second user input data indicating that the visual event has been detected, wherein generating the first natural language response is based at least in part on the first data indicating occurrence of the visual event.

A system, comprising:

• one or more processors;
• and
• non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
• receiving first user input data representing a first user command to store an activity indictor representing an occurrence of a visual event, the visual event associated with an activity in an environment;
• storing an activity indicator in a device;
• generating, utilizing the activity indicator stored in the device, a first query requesting that a language model (LM) identify the activity from image data, wherein the LM is disposed on the device and is configured to determine results utilizing the image data as the input;
• determining, at the device utilizing the LM and the first query, that the image data depicts the activity;
• storing first data indicating occurrence of the visual event based at least in part on the LM determining that the image data depicts the activity;
• receiving second user input data representing a natural language request associated with stored visual events;
• determining, utilizing the LM and based at least in part on the second user input data, that the natural language request is associated with the visual event;
• and
• generating a first natural language response to the second user input data indicating that the visual event has been detected, wherein generating the first natural language response is based at least in part on the first data indicating occurrence of the visual event.