US 2019/0103095 A1Published application

Utterance quality estimation

AssigneeOracle International Corporation

InventorsGautam Singaraju, Jiarui Ding, Vishal Vishnoi, Mark Joseph Sugg, Edward E. Wong

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques disclosed herein relate to improving quality of classification models for differentiating different user intents by improving the quality of training samples used to train the classification models. Pairs of user intents that are difficult to differentiate by classification models trained using the given training samples are identified based upon distinguishability scores (e.g., F-scores). For each of the identified pairs of intents, pairs of training samples each including a training sample associated with a first intent and a training sample associated with a second intent in the pair of intents are ranked based upon a similarity score between the two training samples in each pair of training samples. The identified pairs of intents and the pairs of training samples having the highest similarity scores may be presented to users through a user interface, along with user-selectable options or suggestions for improving the training samples.

Claims20

Dependent claims

A method of improving training samples for training an intent classification model to classify end user utterances into a plurality of intents, the method comprising, by a computing system:

selecting, from the plurality of intents, a first pair of intents including a first intent and a second intent, wherein a first subset of the training samples is associated with the first intent and a second subset of the training samples is associated with the second intent;
training, based upon the first subset of the training samples and the second subset of the training samples, a classification model for classifying end user utterances associated with the first pair of intents;
determining a distinguishability score for the first pair of intents, the distinguishability score indicating classification quality of the trained classification model;
determining that the distinguishability score for the first pair of intents is below a threshold value;
determining a similarity score for each respective pair of training samples in pairs of training samples that each include one training sample from the first subset of the training samples and one training sample from the second subset of the training samples;
selecting a pair of training samples that has a highest similarity score among the pairs of training samples;
and
providing, through a user interface:
information regarding the first pair of intents;
the selected pair of training samples;
and
a user-selectable option for improving the training samples.

9 more elements

A non-transitory computer readable medium storing a plurality of instructions executable by one or more processors, wherein the plurality of instructions, when executed by the one or more processors, causes the one or more processors to perform processing comprising:

selecting, from a plurality of intents, a first pair of intents including a first intent and a second intent, wherein a first subset of training samples is associated with the first intent and a second subset of training samples is associated with the second intent;
training, based upon the first subset of training samples and the second subset of training samples, a classification model for classifying end user utterances associated with the first pair of intents;
determining a distinguishability score for the first pair of intents, the distinguishability score indicating classification quality of the trained classification model;
determining that the distinguishability score for the first pair of intents is below a threshold value;
determining a similarity score for each respective pair of training samples in pairs of training samples that each include one training sample from the first subset of training samples and one training sample from the second subset of training samples;
selecting a pair of training samples that has a highest similarity score among the pairs of training samples;
and
providing, through a user interface:
information regarding the first pair of intents;
the selected pair of training samples;
and
a user-selectable option for improving the first subset of training samples or the second subset of training samples.

9 more elements

Dependencies are inferred from the claim text.