Analyzing semantic user sessions using screen understanding

A method for analyzing semantic sessions, the method comprising:

• receiving a plurality of screenshot images;
• for each of the plurality of screenshot images:
• identifying one or more image regions and one or more text regions on the screenshot image, and
• generating features for the screenshot image based on the one or more image regions and the one or more text regions using an artificial intelligence (AI) module, wherein the features comprise textual embeddings and visual embeddings of the screenshot image;
• generating a user activity graph comprising nodes and edges, wherein the nodes correspond to the plurality of screenshot images, and wherein the edges correspond to commonalities between node pairs based on the generated features of the plurality of screenshot images;
• and
• identifying one or more node clusters from the user activity graph based on semantic relationships.

A computing system for analyzing semantic sessions, the computing system comprising:

• processing circuitry and memory storing instructions that, during execution, cause the processing circuitry to perform the steps of:
• generating a user activity graph comprising nodes and edges, wherein the nodes correspond to a plurality of screenshot images, and wherein the edges correspond to commonalities between node pairs based on features of the plurality of screenshot images;
• receiving a search query;
• identifying a node cluster from the user activity graph based on semantic relationships relevant to the search query;
• and
• presenting search results comprising one or more relevant screenshot images selected from the plurality of screenshot images corresponding to the node cluster.