Training a dialog system using user feedback for answers to questions

A system comprising:

• one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to execute a dialog engine on a mobile device and perform operations comprising:
• receiving, at the dialog engine, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user;
• in response to receiving the voice query:
• identifying, a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query;
• providing a first answer to the voice query for output using a speaker of the mobile device, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query;
• receiving feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query
• determining whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device;
• and
• when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query.

A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to execute a dialog engine on a mobile device and perform operations comprising:

• receiving, at the dialog engine, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user;
• in response to receiving the voice query:
• identifying a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query;
• providing a first answer to the voice query for output using a speaker of the system, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query;
• receiving feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query
• determining whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device;
• and
• when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query.