Query recognition resiliency determination in virtual agent systems

A computer program product for updating a natural language process associated with a virtual agent system based on the resiliency of user queries, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:

• generate a plurality of query samples based on a query that maps to a predetermined categorization;
• process each of the plurality of query samples with a first natural language process, wherein the processing produces respective outputs comprising respective categorizations for each of the plurality of query samples;
• store data regarding whether the respective categorizations for each of the plurality of query samples matches the predetermined categorization;
• generate a report based on the stored data indicating a percentage of the plurality of query samples that have categorizations that match the predetermined characterization;
• output the report;
• and
• receive a second natural language process comprising updates to the first natural language process based on the report, wherein the updates increase the percentage of the plurality of query samples that match the predetermined categorization when processed with the second natural language process relative to the first natural language process.

A system comprising:

• a CPU, a computer readable memory and a computer readable storage medium associated with a computing device;
• program instructions to select a predetermined categorization to analyze for resiliency;
• program instructions to select a query associated with the predetermined categorization;
• program instructions to alter the query to produce a query sample;
• program instructions to apply a first natural language process to process the query sample;
• program instructions to store an output of the processed query sample, wherein the output indicates whether a categorization of the processed query sample matches the predetermined categorization;
• program instructions to generate a report based on the stored data indicating a variance for the predetermined categorization based on the output of the processed query sample and respective outputs of a plurality of other query samples associated with the predetermined categorization;
• program instructions to output the report;
• and
• program instructions to receive a second natural language process comprising updates to the first natural language process based on the report, wherein the updates increase the percentage of the plurality of query samples that match the predetermined categorization when processed with the second natural language process relative to the first natural language process,
• wherein the program instructions are stored on the computer readable storage medium for execution by the CPU via the computer readable memory.

A computer implemented method comprising:

• altering, by a computing device, a query associated with a predetermined categorization to produce a query sample;
• applying, by the computing device, first a natural language process to process the query sample;
• storing, by the computing device, an output of the processed query sample, wherein the output indicates whether a categorization of the processed query sample matches the predetermined categorization;
• generating, by the computing device, a report based on the stored data indicating a variance for the predetermined categorization based on the output of the processed query sample and respective outputs of a plurality of other query samples associated with the predetermined categorization;
• outputting, by the computing device, the report;
• and
• receiving, by the computing device, a second natural language process comprising updates to the first natural language process based on the report, wherein the updates increase the percentage of the plurality of samples that match the predetermined categorization when processed with the second natural language process relative to when the plurality of samples was processed with the first natural language process.