Scalable and optimized edge ai architecture for interactive streaming applications

One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform artificial intelligence (AI) processing in a streaming environment on an endpoint device, by performing the operations of:

• obtaining media data associated with playback of content by a streaming application executing on the endpoint device;
• identifying, via at least one AI inference operation executed on the media data, an object represented within the media data;
• generating refined object data corresponding to the object;
• generating feature data associated with the object based on the refined object data;
• obtaining item information associated with the object based on a comparison between the feature data and reference data stored on the endpoint device;
• and
• outputting presentation data associated with the item information for display in association with the playback of the content.

A computer system, comprising:

• one or more memories that include instructions;
• and
• one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform artificial intelligence (AI) processing in a streaming environment on the computer system, by performing the operations of:
• obtaining media data associated with playback of content by a streaming application executing on the computer system;
• identifying, via at least one AI inference operation executed on the media data, an object represented within the media data;
• generating refined object data corresponding to the object;
• generating feature data associated with the object based on the refined object data;
• obtaining item information associated with the object based on a comparison between the feature data and reference data stored on the computer system;
• and
• outputting presentation data associated with the item information for display in association with the playback of the content.