System, method and computer program product for assessing the capabilities of a conversation agent via black box testing
AssigneeInternational Business Machines Corporation
InventorsAlan BRAZ, Heloisa Caroline De Souza Pereira Candello, Claudio Santos Pinhanez, Marisa Affonso Vasconcelos
Life of this patent
Estimating the term…
Abstract
A conversational agent capability assessment method, system, and computer program product, includes obtaining data to create at least one scenario for testing a conversational agent, performing a set of tests using a scenario of the at least one scenario created to assess a capability of the conversational agent, and comparing a result of the capability from the set of tests with an expected result of the scenario.
Claims20
A computer-implemented conversational agent capability assessment method, the method comprising:
- obtaining data to create at least one scenario for testing a conversational agent having a non-deterministic output;
- performing a set of tests using a scenario of the at least one scenario created to assess a capability of the conversational agent based on evaluation metrics generated from the set of tests while interacting with the conversational agent;
- and
- comparing a result of the capability from the set of tests with an expected result of the scenario based on results from a different conversational agent.
A computer program product for conversational agent capability assessment, the computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform:
- obtaining data to create at least one scenario for testing a conversational agent having a non-deterministic output;
- performing a set of tests using a scenario of the at least one scenario created to assess a capability of the conversational agent based on evaluation metrics generated from the set of tests while interacting with the conversational agent;
- and
- comparing a result of the capability from the set of tests with an expected result of the scenario based on results from a different conversational agent.
A conversational agent capability assessment system, said system comprising:
- a processor;
- and
- a memory, the memory storing instructions to cause the processor to perform:
- obtaining data to create at least one scenario for testing a conversational agent having a non-deterministic output;
- performing a set of tests using a scenario of the at least one scenario created to assess a capability of the conversational agent based on evaluation metrics generated from the set of tests while interacting with the conversational agent;
- and
- comparing a result of the capability from the set of tests with an expected result of the scenario based on results from a different conversational agent.
Dependencies are inferred from the claim text.