US 11,238,241 B1Granted

Natural language dialog scoring

AssigneeAmazon Technologies, Inc.

InventorsRavi Chikkanayakanahalli Mallikarjuniah, Priya Rao Chagaleti, Shiladitya Roy, Christopher Forbes Will, Cole Ira Brendel, Wei Huang, Sarthak Anand

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

The present disclosure provides various metrics that may be used to generate a score representing a conversational quality of a user/computing system dialog. User input data, corresponding to a dialog with a natural language processing system, may be determined and associated with a dialog identifier. System output data, corresponding to the dialog, may also be determined and associated with the dialog identifier. After the system output data is output to a user, the user input data and system output data may be processed to determine a score corresponding to the dialog exchange.

Claims20

Dependent claims

A method comprising:

receiving first user input data corresponding to a request to be performed by a skill component, the first user input data representing a first user input of a dialog exchange;
associating the first user input data with a dialog identifier;
receiving, from the skill component, first system output data responsive to the first user input data;
associating the first system output data with the dialog identifier;
causing the first system output data to be presented to a user;
receiving, after causing the first system output data to be presented, second user input data representing a second user input of the dialog exchange;
associating the second user input data with the dialog identifier;
receiving, from the skill component, second system output data responsive to the second user input data;
associating the second system output data with the dialog identifier;
causing the second system output data to be presented to the user;
determining that the dialog exchange has ended;
determining the first system output data comprises a prompt;
determining a first number of variants of the prompt;
determining a second number corresponding to variants of the prompt output during the dialog exchange;
processing, using at least a first machine learning model and after determining the dialog exchange has ended, the first user input data, the first system output data, the second user input data, and the second system output data to determine:
first data, representing a naturalness of the dialog exchange, based at least in part on the first number and the second number,
second data representing a relevance of the dialog exchange, and
third data representing an accuracy of the first system output data and the second system output data;
and
sending, to the skill component, the first data, the second data, and the third data.

17 more elements

A system comprising:

at least one processor;
and
at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
determine first user input data corresponding to a dialog exchange with a natural language processing system;
associate the first user input data with a dialog identifier;
determine first system output data responsive to the first user input data;
associate the first system output data with the dialog identifier;
cause the first system output data to be output;
determine the first system output data comprises a prompt;
and
determine first data corresponding to the dialog exchange based at least in part on a first number of variants of the prompt and a second number corresponding to variants of the prompt output during the dialog exchange.

8 more elements

Dependencies are inferred from the claim text.