Obtaining Actionable Application Feedback via Generation of LLM-Based Survey Prompts and Application Insights

An application service provider server, comprising:

• a processor;
• an application;
• a communication connection for connecting remote computing systems to the application service provider server via a network;
• and
• a computer-readable storage medium operatively coupled to the processor, the computer-readable storage medium comprising computer-executable instructions that, when executed by the processor, cause the processor to:
• cause execution of the application on the remote computing systems;
• during the execution of the application on each of the remote computing systems:
• cause surfacing of a user interface on a display of the remote computing system, wherein the user interface comprises a survey user interface element;
• cause surfacing of a first survey prompt via the survey user interface element of the user interface;
• receive, via the survey user interface element of the user interface, a first user input comprising a response to the first survey prompt;
• input the first user input to a large language model (LLM);
• generate, via the LLM, a second survey prompt based on the provided first user input;
• cause surfacing of the second survey prompt via the survey user interface element of the user interface;
• receive, via the survey user interface element of the user interface, a second user input comprising a verbatim response to the second survey prompt;
• and
• generate, via the LLM, at least one topic tag corresponding to the verbatim response;
• aggregate the verbatim responses received during the execution of the application on all the remote computing systems, wherein the verbatim responses are aggregated according to the at least one topic tag corresponding to each verbatim response;
• and
• generate, via the LLM, application insights corresponding to the application based on the aggregated verbatim responses, wherein the application insights comprise at least one metric relating to at least one of application performance, application usage, subscription value, or user satisfaction with respect to the application.

A computer-readable storage medium comprising computer-executable instructions that, when executed by a processor, cause the processor to:

• cause execution of an application on multiple remote computing systems;
• during the execution of the application on each of the remote computing systems:
• cause surfacing of a user interface on a display of the remote computing system, wherein the user interface comprises a survey user interface element;
• cause surfacing of a first survey prompt via the survey user interface element of the user interface;
• receive, via the survey user interface element of the user interface, a first user input comprising a response to the first survey prompt;
• input the first user input and user-specific data to a large language model (LLM);
• generate, via the LLM, a second survey prompt based on the provided first user input and the user-specific data;
• cause surfacing of the second survey prompt via the survey user interface element of the user interface;
• receive, via the survey user interface element of the user interface, a second user input comprising a verbatim response to the second survey prompt;
• and
• generate, via the LLM, at least one topic tag corresponding to the verbatim response;
• aggregate the verbatim responses received during the execution of the application on all the remote computing systems, wherein the verbatim responses are aggregated according to the at least one topic tag corresponding to each verbatim response;
• and
• generate, via the LLM, application insights corresponding to the application based on the aggregated verbatim responses, wherein the application insights comprise at least one metric relating to at least one of application performance, application usage, subscription value, or user satisfaction with respect to the application.