# US 10,872,604 B2 · User experience evaluation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,604 B2
- Document ID: US10872604B2
- Status: Granted
- Kind: B2
- Application number: 15/982,851
- Filed: 2018-05-17
- Published: 2019-11-21
- Granted: 2020-12-22
- Earliest claimed: 2018-05-17
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Lae-Hoon Kim; Yinyi Guo; Ravi Choudhary; Sunkuk Moon; Erik Visser; Fatemeh Saki
- Web page: https://patentgrep.com/patent/US10872604B2

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04N 21/42203: {sound input device, e.g. microphone}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/18: for vehicle drivers {or machine operators}
- B60W 50/087: {where the control system corrects or modifies a request from the driver}
- B60W 50/10: Interpretation of driver requests or demands
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/1807: {using prosody or stress}
- G10L 25/63: for estimating an emotional state
- G16H 20/70: relating to mental therapies, e.g. psychological therapy or autogenous training
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A device includes a memory configured to store a user experience evaluation unit. A processor is configured to receive a first user input corresponding to a user command to initiate a particular task, the first user input received via a first sensor. The processor is configured to, after receiving the first user input, receive one or more subsequent user inputs, the one or subsequent user inputs including a second user input received via a second sensor. The processor is configured to initiate a remedial action in response to determining, based on the user experience evaluation unit, that the one or more subsequent user inputs correspond to a negative user experience.

## Claims

### Claim 1

A device comprising:
a memory configured to store a user experience evaluation unit; and
a processor coupled to the memory, the processor configured to:
receive a first user input corresponding to a user command to initiate a particular task, the first user input received via a first sensor;
initiate performance of the particular task in response to the first user input;
after initiation of performance of the particular task, receive one or more subsequent user inputs, the one or more subsequent user inputs including a second user input received via a second sensor and the one or more subsequent user inputs associated with user experience associated with the performance of the particular task; and
initiate a remedial action associated with the performance of the particular task in response to determining, based on evaluation of the one or more subsequent user inputs by the user experience evaluation unit, that the user experience associated with the performance of the particular task corresponds to a negative user experience.

### Claim 2 (depends on 1)

The device of claim 1, further comprising:
a microphone; and
a camera, wherein the one or more subsequent user inputs include an audio input captured by the microphone, a video input captured by the camera, or a combination thereof.

### Claim 3 (depends on 2)

The device of claim 2, wherein the first sensor includes the microphone and the second sensor includes the camera.

### Claim 4 (depends on 1)

The device of claim 1, wherein the user experience evaluation unit is executable to perform at least one of: speech keyword detection, audio emotion analytics, video emotion analytics, prosody analytics, or audio event detection.

### Claim 5 (depends on 1)

The device of claim 1, wherein the first user input includes an audio input of a user, wherein the memory is further configured to store a mapping unit that maps user commands to associated tasks, and wherein the processor is further configured to:
determine the user command based on the audio input; and
access the mapping unit using the user command to select the particular task.

### Claim 6 (depends on 5)

The device of claim 5, wherein the remedial action includes prompting the user for a non-audio input indicating a user-selected task to be associated with the user command.

### Claim 7 (depends on 5)

The device of claim 5, wherein the remedial action includes suggesting that the user perform one or more actions to enhance speech recognition of audio captured by a microphone.

### Claim 8 (depends on 1)

The device of claim 1, wherein the remedial action includes at least one of: playing soothing music, adjusting a voice interface to speak in a calming manner, or recommending a relaxing activity for a user.

### Claim 9 (depends on 1)

The device of claim 1, wherein the processor is implemented in a vehicle, wherein the remedial action includes prompting a user of the vehicle for information regarding the user experience, and wherein the processor is configured to delay performing the remedial action until the user is detected to have a non-negative user experience, the user has shut off the vehicle, or both.

### Claim 10

A method comprising:
receiving, at a processor, a first user input corresponding to a user command to initiate a particular task, the first user input received via a first sensor;
initiating, at the processor, performance of the particular task in response to the first user input;
after initiation of performance of the particular task, receiving one or more subsequent user inputs, the one or more subsequent user inputs including a second user input received via a second sensor and the one or more subsequent user inputs associated with user experience associated with the performance of the particular task; and
initiating a remedial action associated with the performance of the particular task in response to determining, based on evaluation of the one or more subsequent user inputs by a user experience evaluation unit, that the user experience associated with the performance of the particular task corresponds to a negative user experience.

### Claim 11 (depends on 10)

The method of claim 10, wherein the first sensor includes a microphone and the second sensor includes a camera.

### Claim 12 (depends on 10)

The method of claim 10, wherein determining that the one or more subsequent user inputs correspond to a negative user experience includes performing at least one of speech keyword detection, audio emotion analytics, video emotion analytics, prosody analytics, or audio event detection.

### Claim 13 (depends on 10)

The method of claim 10, wherein the first user input includes an audio input of a user, and further comprising:
determining the user command based on the audio input; and
selecting the particular task based on the user command.

### Claim 14 (depends on 13)

The method of claim 13, wherein the remedial action includes prompting the user for a non-audio input indicating a user-selected task to be associated with the user command.

### Claim 15 (depends on 13)

The method of claim 13, wherein the remedial action includes suggesting that the user close a window, direct speech toward a microphone, speak more loudly or distinctly, or a combination thereof.

### Claim 16 (depends on 10)

The method of claim 10, wherein the remedial action includes at least one of: playing soothing music, adjusting a voice interface to generate speech to have a calming effect, or recommending a relaxing activity for a user.

### Claim 17 (depends on 10)

The method of claim 10, wherein the remedial action includes prompting a user of a vehicle for information regarding the user experience, and wherein performance of the remedial action is delayed until the user is detected to have a non-negative user experience, the user has shut off the vehicle, or both.

### Claim 18

A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to initiate, perform, or control operations comprising:
receiving a first user input corresponding to a user command to initiate a particular task, the first user input received via a first sensor;
initiating performance of the particular task in response to the first user input;
after initiation of performance of the particular task, receiving one or more subsequent user inputs, the one or more subsequent user inputs including a second user input received via a second sensor and the one or more subsequent user inputs associated with user experience associated with the performance of the particular task; and
initiating a remedial action associated with the performance of the particular task in response to determining, based on evaluation of the one or more subsequent user inputs by a user experience evaluation unit, that the user experience associated with the performance of the particular task corresponds to a negative user experience.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the first sensor includes a microphone and the second sensor includes a camera.

### Claim 20 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein determining that the one or more subsequent user inputs correspond to a negative user experience includes performing at least one of audio emotion analytics or video emotion analytics.

### Claim 21 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the first user input includes an audio input of a user, and wherein the operations further comprise:
determining the user command based on speech recognition of the audio input; and
selecting the particular task based on the user command.

### Claim 22 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein the remedial action includes prompting the user to perform one or more actions to enhance speech recognition of subsequent audio input.

### Claim 23 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the remedial action includes at least one of: playing soothing music, adjusting a voice interface to generate speech to have a calming effect, or recommending a relaxing activity for a user.

### Claim 24 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein performance of the remedial action is delayed until the negative user experience is detected to have ended.

### Claim 25

An apparatus comprising:
means for receiving a first user input corresponding to a user command to initiate a particular task, the first user input received via a first sensor;
means for initiating performance of the particular task in response to the first user input;
means for receiving one or more subsequent user inputs, the one or more subsequent user inputs including a second user input received via a second sensor after the first user input is received, the one or more subsequent user inputs received after initiation of performance of the particular task, and the one or more subsequent user inputs associated with user experience associated with the performance of the particular task; and
means for initiating a remedial action associated with the performance of the particular task in response to determining, based on evaluation of the one or more subsequent user inputs by a user experience evaluation unit, that the user experience associated with the performance of the particular task corresponds to a negative user experience.

### Claim 26 (depends on 25)

The apparatus of claim 25, wherein the first sensor includes a microphone and the second sensor includes a camera.

### Claim 27 (depends on 25)

The apparatus of claim 25, further comprising means for determining that the one or more subsequent user inputs correspond to a negative user experience.

### Claim 28 (depends on 25)

The apparatus of claim 25, wherein the first user input includes an audio input of a user.

### Claim 29 (depends on 28)

The apparatus of claim 28, further comprising means for prompting the user for a non-audio input indicating a user-selected task to be associated with the user command.

### Claim 30 (depends on 25)

The apparatus of claim 25, wherein the remedial action includes at least one of: playing soothing music, adjusting a voice interface to generate speech to have a calming effect, or recommending a relaxing activity for a user.