# US 2020/0286480 A1 · Brain-inspired spoken language understanding system, a device for implementing the system, and method of operation thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286480 A1
- Document ID: US20200286480A1
- Status: Published application
- Kind: A1
- Application number: 16/810,358
- Filed: 2020-03-05
- Published: 2020-09-10
- Earliest claimed: 2019-03-05
- Assignees: Medyug Technology Private Limited
- Applicants: Medyug Technology Private Limited
- Inventors: Baljit Singh; Praveen Prakash
- Web page: https://patentgrep.com/patent/US20200286480A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G16H 10/20: for electronic clinical trials or questionnaires
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/08: Speech classification or search
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G10L 15/25: using position of the lips, movement of the lips or face analysis
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G16H 40/63: for local operation
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/228: {of application context}
- Y02A 90/10: Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- Y02A: TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE

## Abstract

A brain-inspired spoken language understanding system, comprises: a first module that facilitates the conversion of a voice input into phoneme sequences; a buffer module that facilitates the storing of the phoneme sequences until they are clustered into a first storage as one or more meaningful thought representations; a second module that facilitates the monitoring of the reasoning, disambiguation, and prioritization of the system, in addition to controlling the system; a seventh module and an eighth module that facilitate the capturing of at least one non-verbal signal from a conversation, and create at least one time-synchronized non-verbal object, with the help of a third knowledge base; a third module that facilitates the conversion of the phoneme sequences and at least one time-synchronized non-verbal object into sequences of phonetics-based words; a fourth module that facilitates the conversion of the phonetics-based words into sequences of thought representations, with the help of a first knowledge base; a fifth module that facilitates the fine-tuning of the thought representations, with the help of the first knowledge base and a second knowledge base; and a sixth module that maintains the instance of conversation as thought representations in the form of one or more connected stories. A device for implementing the brain-inspired spoken language understanding system and its method of functioning are also disclosed.

## Claims

### Claim 1

A brain-inspired spoken language understanding system, comprising:
a first module that facilitates the conversion of a voice input received through one or more microphones into phoneme sequences, said first module comprising: a first unit that converts the voice input received into spectrograms, and a second unit that converts the spectrograms into phonemes, with pause time between them, if applicable;
a buffer module that facilitates the storing of the phoneme sequences until they are clustered into a first storage as one or more meaningful thought representations;
a second module that facilitates the monitoring of the reasoning, disambiguation, and prioritization of the system, in addition to controlling the system;
a seventh module that comprises: at least one sensor that captures at least one non-verbal signal from a conversation, and a pre-processing unit that facilitates the pre-processing of the at least one non-verbal signal received from the at least one sensor;
an eighth module that is configured to classify and transform the at least one non-verbal signal received from the seventh module into at least one symbol, and creates at least one time-synchronized non-verbal object by permuting and matching the at least one symbol against a third knowledge base, said third knowledge base providing a look-up for non-verbal symbols into known complex objects;
a ninth module that facilitates the transmission of the at least one object received from the eighth module, along with time-stamp, to a third module;
the third module that facilitates the conversion of the phoneme sequences and at least one time-synchronized non-verbal object into sequences of phonetics-based words;
the fourth module that facilitates the conversion of the phonetics-based words into sequences of thought representations, with the help of a first knowledge base;
a fifth module that facilitates the fine-tuning of the thought representations received from the fourth module by: merging two or more thought representations, inferring new thought representations, correcting the thought representations, and/or filling one or more missing information in the thought representations, with the help of the first knowledge base and a second knowledge base; and
a sixth module that maintains the instance of conversation as thought representations in the form of one or more connected stories as thought cloud, with said third module, said fourth module, and said fifth module interacting with each other, with the help of the second module, and with the system running the sequence possibilities throughout said third module, said fourth module, and said fifth module like a pipeline with feedback coming from forward modules to prior modules.

### Claim 2 (depends on 1)

The brain-inspired spoken language understanding system as claimed in claim 1, wherein the third module comprises: a third unit that collects phonemes from the buffer module and uses sliding dynamic width window technique over these phonemes to create a sequence of best possible phonetics-based words, said third unit receiving one or more feedback from the other modules through the second module to decide whether the phonetics-based words are meaningful or not, and sending status signals to the second module after the completion of a task; a fourth unit that receives the phoneme sequences from the third unit and converts them into phonetics-based words; a fifth unit that receives the sequence of phonetics-based words identified by the third unit and refines the phonetics-based words; and a sixth unit, which is a phoneme similarity matrix that resolves conflicting words by using similar sounding phonemes.

### Claim 3 (depends on 1)

The brain-inspired spoken language understanding system as claimed in claim 1, wherein the fourth module comprises: a seventh unit that takes sequences of phonetics-based words as input, and, while looking at expected output from an eighth unit, fetches possible concepts/entities from a single phonetics-based word or a group of phonetics-based words, with the help of the first knowledge base, with the output concepts being sent to an entity possibility buffer unit; an eighth unit that holds sequences of phonetics-based words since the last completion of a thought representation and outputs: expected phonetics-based words or concept type for the given sequence of previous phonetics-based words and thought representation for sequence of phonetics-based words; and a ninth unit that facilitates the identification of the best possible and meaningful thought representations by reasoning the phonetics-based words in the entity possibility buffer unit with all the thought representations obtained since the last completed thought representation process, said ninth unit receiving one or more feedback from the other modules through the second module to decide whether the thought representations are meaningful or not, and sending status signal to the second module during the current task or after completion of the task.

### Claim 4 (depends on 1)

The brain-inspired spoken language understanding system as claimed in claim 1, wherein the fifth module comprises: an entity de-referencing (EDR) unit that receives the incoming thought representation sequences from the fourth module, keeps track of all the entities mentioned throughout the conversation, and updates the incoming thought representation sequences with well-specified entities; a tenth unit that checks whether the thought representations are well-connected and switching between their contexts is meaningful, with the help of the first knowledge base; an eleventh unit that explores all possible thought representation context connectivity; and a twelfth unit that facilitates the fine-tuning of the thought representations received from the fourth module by: merging two or more thought representations, inferring new thought representations, correcting the thought representations, and/or filling one or more missing information in previous thought representations, with the help of the second knowledge base, and creates a story around the thought representation sequence, said twelfth unit receiving one or more feedback from the second module to decide whether to process further or stop processing, and sending status signal to the second module during the current task or after the completion of the task.

### Claim 5 (depends on 1)

The brain-inspired spoken language understanding system as claimed in claim 1, wherein the first knowledge base is based on Unified Medical Language System and comprises all the words or vocabularies or lingo related to clinical concepts and limited worldly knowledge concepts.

### Claim 6 (depends on 1)

The brain-inspired spoken language understanding system as claimed in claim 1, wherein the first knowledge base and the second knowledge base are configured to have prioritized access based on frequency and recentness of access.

### Claim 7 (depends on 2)

The brain-inspired spoken language understanding system as claimed in claim 2, wherein the fourth unit is configured as a predictive memory.

### Claim 8 (depends on 3)

The brain-inspired spoken language understanding system as claimed in claim 3, wherein the eighth unit is configured as a predictive memory.

### Claim 9 (depends on 4)

The brain-inspired spoken language understanding system as claimed in claim 4, wherein the eleventh unit is configured as a predictive memory.

### Claim 10 (depends on 4)

The brain-inspired spoken language understanding system as claimed in claim 4, wherein the tenth unit detects context-switching anomalies and thought-specific anomalies.

### Claim 11

A device for implementing the brain-inspired spoken language understanding system comprising: at least one central processing unit, said central processing unit being in communication with at least one memory where at least one module of the brain-inspired spoken language understanding system resides; one or more communication units which facilitate: the connecting of the device with one or more health care devices to collect the relevant data from a user, and the connecting of the device with a network to transmit or store the output; at least one input unit; a display unit that presents the output to the user; at least one microphone that captures a voice of the user; at least one speaker that delivers an audio output to the user; at least one camera that captures images/videos of the user for lip-reading and gesture identification; and one or more supplementary processing units with at least one memory and one or more communication units to support the at least one central processing unit in handling a plurality of tasks.

### Claim 12 (depends on 11)

The device for implementing the brain-inspired spoken language understanding system as claimed in claim 11, wherein the one or more health care devices comprise one or more of: digital BP, digital Pulse Oximetry, digital Stethoscope, digital Weighing device, digital Height measuring device, digital Thermometer, ECG, and digital Glucometer.

### Claim 13 (depends on 11)

The device for implementing the brain-inspired spoken language understanding system as claimed in claim 11, wherein the one or more communication units in the central processing units support any wired or wireless communication technology.

### Claim 14 (depends on 11)

The device for implementing the brain-inspired spoken language understanding system as claimed in claim 11, wherein the at least one input unit is a keyboard or a mouse.

### Claim 15 (depends on 11)

The device for implementing the brain-inspired spoken language understanding system as claimed in claim 11, wherein the device communicates with a hospital management system or an electronic medical record system for sharing its results.

### Claim 16

A method of working of the device in which the brain-inspired spoken language understanding system is implemented, comprising:
(i) Listening to the complaints explained by the patient or his/her caretaker;
(ii) Understanding the complaints by the device, with the help of the brain-inspired spoken language understanding system;
(iii) Using the at least one camera to identify certain phonemes from the patient's or caretaker's' gestures and lip movements;
(iv) Clarifying certain sentences uttered by the patient or his/her caretaker to disambiguate;
(v) Asking questions for further understanding of the patient's medical condition/problem;
(vi) Collecting one or more clinical vital data of the patient, with the help of the relevant health care devices connected with the device, depending on the patient's medical problem, or specific to the preferences of the doctor;
(vii) Preparing a summary of the discussion, along with patient's medical/family history;
(viii) Presenting the summary to the patient for his/her verification, and making corrections as requested by the patient; and
(ix) Preparing a doctor-specific summary and forwarding the summary to the doctor for his/her review.

### Claim 17 (depends on 16)

The method of working of the device in which the brain-inspired spoken language understanding system is implemented as claimed in claim 16, wherein the one or more clinical vital data comprises one or more of: temperature, BP, weight, height, sugar level, and ECG.

### Claim 18

A method of working of the device in which the brain-inspired spoken language understanding system is implemented, comprising:
(i) Listening to the conversation between the doctor and the patient or his/her caretaker, and interacting in a limited manner with the patient, if allowed by the doctor;
(ii) Understanding the conversation, with the help of the brain-inspired spoken language understanding system by continuously listening to the doctor and the patient;
(iii) Modifying the patient summary prepared by the device during pre-consultation, if the patient makes some new revelations or corrections, or as instructed by the doctor;
(iv) Taking notes of the instructions as dictated by the doctor;
(v) Preparing a final consultation summary, along with a patient treatment advice, as instructed by the doctor;
(vi) Presenting the final summary to the doctor for his/her review, and making changes as instructed by the doctor;
(vii) Printing the patient treatment advice to be given to the patient;
(viii) Sharing (the final medical record with an external Electronic Medical Records System/Hospital Management System connected with the device; and
(x) Archiving the information and deleting it from its memory.
