# US 12,223,946 B2 · Artificial intelligence voice response system for speech impaired users > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,946 B2 - Document ID: US12223946B2 - Status: Granted - Kind: B2 - Application number: 16/948,310 - Filed: 2020-09-11 - Published: 2022-03-17 - Granted: 2025-02-11 - Earliest claimed: 2020-09-11 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Shikhar Kwatra; Laura Grace Ellis; Kaitlin McGoldrick; Sarbajit K. Rakshit - Web page: https://patentgrep.com/patent/US12223946B2 ## CPC - G10L 15/08: Speech classification or search - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G10L 15/063: {Training} - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 15/06: Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice (G10L15/14 takes precedence) - G10L 15/1822: {Parsing for meaning understanding} - G10L 2015/088: {Word spotting} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method, computer system, and a computer program product for voice responses is provided. The present invention may include gathering user data from at least one connected device. The present invention may include training a voice response system based on the gathered user data. The present invention may include identifying a wakeup signal based on the trained voice response system. The present invention may include determining that user engagement is intended based on identifying the wakeup signal. The present invention may include engaging with the user through the at least one connected device. ## Claims ### Claim 1 A method for voice responses, the method comprising: gathering user data from at least one connected device, wherein the connected device comprises an artificial intelligence (AI) system that can observe human conversations and detect behavioral signals or biometric patterns of a user within a range of sensors used to detect the user's behavioral signals or biometric patterns; predicting when a user desires to submit a voice command and whether the user has the ability to submit the voice command, wherein heuristics and health conditions of the user are used to predict a topic of a voice command and voice request and to provide a spoken menu from which the user may choose at least one voice command; analyzing user speech stream data via natural language processing (NLP) algorithms to dynamically determine a satisfaction or frustration level of a user; passing the gathered user data to a random forest algorithm to perform a binary classification; training a voice response system based on the gathered user data, wherein the trained voice response system includes a plurality of voice menus, one of the plurality of voice menus being dynamically modified to include a set of hierarchical questions relating to an observed interaction or set of interactions with a user, until a customized menu is determined; using a deep reinforcement learning model, such as a long short term memory recurrent neural network model (LSTM-RNN), to improve a knowledge corpus by correlating gathered behavioral input, body language, and biometric signals with an intended topic or a hierarchical voice menu related to the intended topic; identifying a wakeup signal for the connected device based on observed behavioral signals or biometric patterns of the user within the range of the sensors used to detect the user's behavioral signals or biometric patterns, wherein the identified wakeup signal comprises a change in a behavior or a biometric parameter of the user; determining that user engagement is intended based on identifying the wakeup signal; handling a non-routine event of the user by initiating a phone call to a live person who may assist in understanding the non-routine event; determining the customized menu based on the intended user engagement; and engaging with the user through the at least one connected device. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one connected device is an augmentative and alternative communication device. ### Claim 3 (depends on 1) The method of claim 1, wherein training the voice response system based on the gathered user data further comprises: predicting a topic of a voice request using a long short term memory recurrent neural network. ### Claim 4 (depends on 1) The method of claim 1, wherein the wakeup signal is a change in a biometric parameter recorded on a connected Internet of Things (IoT) device. ### Claim 5 (depends on 1) The method of claim 1, wherein determining that user engagement is intended further comprises: using a random forest algorithm to perform a binary classification of the gathered user data; and using a long short-term memory (LSTM-RNN) model to determine how to proceed on engaging with the user. ### Claim 6 (depends on 1) The method of claim 1, wherein engaging with the user through the at least one connected device further comprises: providing the user with a customized menu based on the user data; analyzing user feedback; and predicting a user topic. ### Claim 7 (depends on 1) The method of claim 1, wherein the user data is stored in a database, and wherein the database is updated, based on engagement with the user, to correlate the user data with a user topic predicted by a long short term memory recurrent neural network. ### Claim 8 (depends on 7) The method of claim 7, wherein the plurality of voice menus are stored in the database, and wherein each of the plurality of voice menus is dynamically amended by the voice response system based on the gathered data. ### Claim 9 A computer system for voice responses, comprising: one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising: gathering user data from at least one connected device, wherein the connected device comprises an artificial intelligence (AI) system that can observe human conversations and detect behavioral signals or biometric patterns of a user within a range of sensors used to detect the user's behavioral signals or biometric patterns; predicting when a user desires to submit a voice command and whether the user has the ability to submit the voice command, wherein heuristics and health conditions of the user are used to predict a topic of a voice command and voice request and to provide a spoken menu from which the user may choose at least one voice command; analyzing user speech stream data via natural language processing (NLP) algorithms to dynamically determine a satisfaction or frustration level of a user; passing the gathered user data to a random forest algorithm to perform a binary classification; training a voice response system based on the gathered user data, wherein the trained voice response system includes a plurality of voice menus, one of the plurality of voice menus being dynamically modified to include a set of hierarchical questions relating to an observed interaction or set of interactions with a user, until a customized menu is determined; using a deep reinforcement learning model, such as a long short term memory recurrent neural network model (LSTM-RNN), to improve a knowledge corpus by correlating gathered behavioral input, body language, and biometric signals with an intended topic or a hierarchical voice menu related to the intended topic; identifying a wakeup signal for the connected device based on observed behavioral signals or biometric patterns of the user within the range of the sensors used to detect the user's behavioral signals or biometric patterns, wherein the identified wakeup signal comprises a change in a behavior or a biometric parameter of the user; determining that user engagement is intended based on identifying the wakeup signal; handling a non-routine event of the user by initiating a phone call to a live person who may assist in understanding the non-routine event; determining the customized menu based on the intended user engagement; and engaging with the user through the at least one connected device. ### Claim 10 (depends on 9) The computer system of claim 9, wherein the at least one connected device is an augmentative and alternative communication device. ### Claim 11 (depends on 9) The computer system of claim 9, wherein training the voice response system based on the gathered user data further comprises: predicting a topic of a voice request using a long short term memory recurrent neural network. ### Claim 12 (depends on 9) The computer system of claim 9, wherein the wakeup signal is a change in a biometric parameter recorded on a connected Internet of Things (IoT) device. ### Claim 13 (depends on 9) The computer system of claim 9, wherein determining that user engagement is intended further comprises: using a random forest algorithm to perform a binary classification of the gathered user data; and using a long short-term memory (LSTM-RNN) model to determine how to proceed on engaging with the user. ### Claim 14 (depends on 9) The computer system of claim 9, wherein engaging with the user through the at least one connected device further comprises: providing the user with a customized menu based on the user data; analyzing user feedback; and predicting a user topic. ### Claim 15 (depends on 9) The computer system of claim 9, wherein the user data is stored in a database, and wherein the database is updated, based on engagement with the user, to correlate the user data with a user topic predicted by a long short term memory recurrent neural network. ### Claim 16 (depends on 15) The computer system of claim 15, wherein the plurality of voice menus are stored in the database, and wherein each of the plurality of voice menus is dynamically amended by the voice response system based on the gathered data. ### Claim 17 A computer program product for voice responses, comprising: one or more non-transitory computer-readable storage media and program instructions stored on at least one of the one or more tangible storage media, the program instructions executable by a processor to cause the processor to perform a method comprising: gathering user data from at least one connected device, wherein the connected device comprises an artificial intelligence (AI) system that can observe human conversations and detect behavioral signals or biometric patterns of a user within a range of sensors used to detect the user's behavioral signals or biometric patterns; predicting when a user desires to submit a voice command and whether the user has the ability to submit the voice command, wherein heuristics and health conditions of the user are used to predict a topic of a voice command and voice request and to provide a spoken menu from which the user may choose at least one voice command; analyzing user speech stream data via natural language processing (NLP) algorithms to dynamically determine a satisfaction or frustration level of a user; passing the gathered user data to a random forest algorithm to perform a binary classification; training a voice response system based on the gathered user data, wherein the trained voice response system includes a plurality of voice menus, one of the plurality of voice menus being dynamically modified to include a set of hierarchical questions relating to an observed interaction or set of interactions with a user, until a customized menu is determined; using a deep reinforcement learning model, such as a long short term memory recurrent neural network model (LSTM-RNN), to improve a knowledge corpus by correlating gathered behavioral input, body language, and biometric signals with an intended topic or a hierarchical voice menu related to the intended topic; identifying a wakeup signal for the connected device based on observed behavioral signals or biometric patterns of the user within the range of the sensors used to detect the user's behavioral signals or biometric patterns, wherein the identified wakeup signal comprises a change in a behavior or a biometric parameter of the user; determining that user engagement is intended based on identifying the wakeup signal; handling a non-routine event of the user by initiating a phone call to a live person who may assist in understanding the non-routine event; determining the customized menu based on the intended user engagement; and engaging with the user through the at least one connected device. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein the at least one connected device is an augmentative and alternative communication device. ### Claim 19 (depends on 17) The computer program product of claim 17, wherein training the voice response system based on the gathered user data further comprises: predicting a topic of a voice request using a long short term memory recurrent neural network. ### Claim 20 (depends on 17) The computer program product of claim 17, wherein the wakeup signal is a change in a biometric parameter recorded on a connected Internet of Things (IoT) device. ### Claim 21 (depends on 17) The computer program product of claim 17, wherein determining that user engagement is intended further comprises: using a random forest algorithm to perform a binary classification of the gathered user data; and using a long short-term memory (LSTM-RNN) model to determine how to proceed on engaging with the user. ### Claim 22 (depends on 17) The computer program product of claim 17, wherein engaging with the user through the at least one connected device further comprises: providing the user with a customized menu based on the user data; analyzing user feedback; and predicting a user topic. ### Claim 23 (depends on 17) The computer program product of claim 17, wherein the user data is stored in a database, and wherein the database is updated, based on engagement with the user, to correlate the user data with a user topic predicted by a long short term memory recurrent neural network.