# US 10,504,514 B2 · Human machine interface system and method for improving user experience based on history of voice activity

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,514 B2
- Document ID: US10504514B2
- Status: Granted
- Kind: B2
- Application number: 15/720,458
- Filed: 2017-09-29
- Published: 2019-04-04
- Granted: 2019-12-10
- Earliest claimed: 2017-09-29
- Assignees: Visteon Global Technologies, Inc.
- Applicants: Sorin Marius Panainte
- Inventors: Sorin Marius Panainte
- Web page: https://patentgrep.com/patent/US10504514B2

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G10L 15/063: {Training}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G10L 2015/223: {Execution procedure of a spoken command}
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04L 67/306: User profiles
- H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A human machine interface system and method of operating the human machine interface system are provided. The system includes a voice user interface module with an audio input unit to receive a voice signal from a microphone and digitize the voice signal as an audio file. A recognition engine unit is coupled to the audio input unit and is configured to analyze the audio file and recognize a plurality of voice commands and output a plurality of recognized commands. A logger unit is coupled to the recognition engine unit to receive the plurality of recognized commands and output a user intent output and a normalized vector representation based on the plurality of recognized commands. A classification component receives the normalized vector representation from the logger unit and outputs a classified user profile to a plurality of human machine interface applications based on the normalized vector representation.

## Claims

### Claim 1

A human machine interface system, comprising:
a processor; and
a memory that includes instructions that, when executed by the processor, cause the processor to:
generate an audio file corresponding to a voice input of a user;
identify, using a predetermined set of recognized commands, a plurality of recognized commands corresponding to the audio file;
receive, from a cloud computing system remotely located from the processor, a user sentiment corresponding to the audio file;
generate a normalized vector representation based on the plurality of recognized commands and the user sentiment;
determine, based on at least the plurality of recognized commands, a user intent output corresponding to an experience of the user;
control a plurality of human machine interface applications using the user intent output;
receive, from the cloud computing system, a classified user profile based on the normalized vector representation; and
output the classified user profile to the plurality of human machine interface applications.

### Claim 2 (depends on 1)

The human machine interface system as set forth in claim 1, wherein cloud computing is configured to:
cluster the normalized vector representation to a clustered vector representation using a clustering algorithm; and
analyze the clustered vector representation using a machine learning algorithm to output the classified user profile to the plurality of human machine interface applications.

### Claim 3 (depends on 2)

The human machine interface system as set forth in claim 2, further including a short term memory and a long term memory and wherein the cloud computing system is further configured to analyze the clustered vector representation in view of recorded short term functions of the clustered vector representation and recorded short term neuron parameters stored in the short term memory and recorded long term functions of the clustered vector representation and recorded long term neuron parameters stored in the long term memory using the machine learning algorithm.

### Claim 4 (depends on 2)

The human machine interface system as set forth in claim 2, wherein the machine learning algorithm includes a radial basis function.

### Claim 5 (depends on 2)

The human machine interface system as set forth in claim 2, wherein the clustering algorithm is a k-means algorithm.

### Claim 6 (depends on 1)

The human machine interface system as set forth in claim 1, wherein the instructions further cause the processor to:
identify a failure rate based on at least one of a number of times that the user retries a particular command and a number of times in a session that the user is prompted by the system to repeat one of the plurality of voice commands and the user switching between the plurality of voice commands in a fixed amount of time and there being unexpected pauses initiating one of the plurality of voice commands;
modify the user intent output based on the failure rate; and
provide additional help and guidance to the user in response to receiving the modified user intent output.

### Claim 7 (depends on 1)

The human machine interface system as set forth in claim 1, wherein the plurality of human machine interface applications include a phone application and a media application and a navigation application.

### Claim 8 (depends on 1)

The human machine interface system as set forth in claim 1, wherein the cloud computing system includes a neural network used to generate the classified user profile.

### Claim 9 (depends on 1)

The human machine interface system as set forth in claim 1, wherein the cloud computing system includes a support vector machine used to generate the classified user profile.

### Claim 10 (depends on 1)

The human machine interface system as set forth in claim 1, wherein the plurality of human machine interface applications are configured to receive and utilize the user sentiment.

### Claim 11

A system for a vehicle, the system comprising:
a processor; and
a memory that includes instructions that, when executed by the processor, cause the processor to:
identify a plurality of recognized commands based on an audio file corresponding to a voice input;
generate, using a deep neural algorithm, a user sentiment corresponding to the audio file;
generate a normalized vector representation based on the plurality of recognized commands and the user sentiment;
generate, using a neural network, a classified user profile based on the normalized vector representation; and
communicate the classified user profile to at least one human machine interface application.

### Claim 12 (depends on 11)

The system as set forth in claim 11, wherein the at least one human machine interface application is configured to receive and utilize the user sentiment.

### Claim 13 (depends on 11)

The system as set forth in claim 11, wherein the at least one human machine interface application includes one of a phone application, a media application, and a navigation application.

### Claim 14 (depends on 11)

The system as set forth in claim 11, wherein the instructions further cause the processor to:
cluster the normalized vector representation to a clustered vector representation using a clustering algorithm; and
analyze the clustered vector representation using a machine learning algorithm to output the classified user profile to the at least one human machine interface application.

### Claim 15 (depends on 14)

The system as set forth in claim 14, further including a short term memory and a long term memory and wherein the instructions further cause the processor to analyze the clustered vector representation in view of recorded short term functions of the clustered vector representation and recorded short term neuron parameters stored in the short term memory and recorded long term functions of the clustered vector representation and recorded long term neuron parameters stored in the long term memory using the machine learning algorithm.

### Claim 16 (depends on 14)

The system as set forth in claim 14, wherein the machine learning algorithm includes a radial basis function.

### Claim 17 (depends on 14)

The system as set forth in claim 14, wherein the clustering algorithm is a k-means algorithm.

### Claim 18

A voice recognition system, comprising:
a processor configured to:
receive a voice input from an input device;
generate an audio file based on the voice input; and
identify a plurality of recognized commands corresponding to the audio file;
at least one cloud computing device remotely located from the processor and configured to generate, using a deep neural algorithm, a user sentiment corresponding to the audio file; and
a neural network in communication with the processor and configured to:
receive, from the processor, a normalized vector representation corresponding to the plurality of recognized commands and the user sentiment; and
generate, based on the normalized vector representation, a classified user profile.

### Claim 19 (depends on 18)

The system of claim 18, wherein the processor is further configured to communicate the classified user profile to at least one human machine interface application.

### Claim 20 (depends on 19)

The system as set forth in claim 19, wherein the at least one human machine interface application is configured to receive and utilize the user sentiment.