# US 2019/0171348 A1 · System and method for aiding communication > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0171348 A1 - Document ID: US20190171348A1 - Status: Published application - Kind: A1 - Application number: 16/321,286 - Filed: 2017-05-23 - Published: 2019-06-06 - Earliest claimed: 2016-07-28 - Assignees: Tata Consultancy Services Limited - Applicants: Tata Consultancy Services Limited - Inventors: Arpit VISHWAKARMA; Aniruddha SINHA; Ratnamala MANNA; Debatri CHATTERJEE; Vedraj VEDRAJ; Kingshuk CHAKRAVARTY; Rajat Kumar DAS; Anagha Nikhil MEHROTRA; Arpan PAL; Rahul Dasharath GAVAS; Anwesha KHASNOBISH; Abhijit DAS - Web page: https://patentgrep.com/patent/US20190171348A1 ## CPC - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - A61B 5/1114: {Tracking parts of the body} - A61F 4/00: Methods or devices enabling patients or disabled persons to operate an apparatus or a device not forming part of the body - G06F 3/00: Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06N 20/00: Machine learning - G09B 5/06: with both visual and audible presentation of the material to be studied - H04N 7/185: {from a mobile camera, e.g. for remote control} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract System and method for aiding communication for subjects suffering from paralysis of muscles controlled by peripheral nervous system are disclosed. A method for aiding communication for said subjects includes capturing, from a plurality of sensors, sensor data generated based on an interaction of a subject with an interactive UI. The plurality of sensors includes one or more body-parts movement tracking sensors for tracking motion of body-parts of the subject and one or more physiological signal sensors for monitoring physiological signals generated from the subject during the interaction. A plurality of model parameters indicative of characteristics of the subject related to the interaction are determined based on the sensor data. The navigation at the interactive UI is controlled on the plurality of model parameters. ## Claims ### Claim 1 A processor-implemented method for aiding communication, the method comprising: capturing, from a plurality of sensors, sensor data generated based on an interaction of a subject with an interactive UI, via one or more hardware processors, the plurality of sensors comprising one or more body-parts movement tracking sensors for tracking motion of body-parts of the subject and one or more physiological signal sensors for monitoring physiological signals generated from the subject during the interaction; determining, based on the sensor data, a plurality of model parameters indicative of characteristics of the subject related to the interaction, via the one or more hardware processors; and controlling navigation at the interactive UI based on the plurality of model parameters, via the one or more hardware processors. ### Claim 2 (depends on 1) The method as claimed in claim 1, wherein the one or more body-parts movement tracking sensors comprises Inertial Motion Unit (IMU), an electroencephalogram (EEG) sensor, eye tracking sensor, gyroscope, and magnetometer. ### Claim 3 (depends on 1) The method as claimed in claim 1, wherein the one or more physiological signal sensors comprises Galvanic Skin Response sensor and Photoplethysmogram sensor. ### Claim 4 (depends on 1) The method as claimed in claim 1, further comprising training a model for determining the plurality of model parameters, wherein training the model in an offline mode comprises: performing a plurality of known tasks and capturing a training sensor data during said performing, wherein the plurality of known tasks comprises interacting with the interactive UI; and obtaining a plurality of training model parameters based on the training sensor data, the plurality of training model parameters facilitates in training the model. ### Claim 5 (depends on 1) The method as claimed in claim 1, wherein controlling the navigation at the interactive UI comprises one or more of: indicating highlighting at least one element on the interactive UI, generating an audio output at the interactive UI and generating a visual output at the interactive UI. ### Claim 6 (depends on 1) The method as claimed in claim 1, further comprising: monitoring the sensor data generated based on the interaction of the subject with the interactive UI to determine a consistent change in the interaction; and automatically updating, on determination of the consistent change, one or more model parameters from the plurality of model parameters based on the sensor data. ### Claim 7 (depends on 6) The method as claimed in claim 6, further comprising storing the updated one or more model parameters at a server to derive the adapted interactive UI. ### Claim 8 (depends on 6) The method as claimed in claim 6, further comprising adapting the interactive UI based on the updating of the one or more model parameters. ### Claim 9 (depends on 7) The method as claimed in claim 7, wherein adapting the interactive UI comprises adapting one or more elements of the interactive UI. ### Claim 10 (depends on 7) The method as claimed in claim 7, wherein adapting the interactive UI comprises adapting an orientation of a layout of the interactive UI. ### Claim 11 (depends on 1) The method as claimed in claim 1, wherein the interactive UI comprises an on-screen keyboard layout. ### Claim 12 A system for aiding communication, the system comprising: one or more memories storing instructions; and one or more hardware processors coupled to the one or more memories, wherein said one or more hardware processors are configured by said instructions to: capture, from a plurality of sensors, sensor data generated based on an interaction of a subject with an interactive UI, the plurality of sensors comprising one or more body-parts movement tracking sensors for tracking motion of body-parts of the subject and one or more physiological signal sensors for monitoring physiological signals generated from the subject during the interaction; determine, based on the sensor data, a plurality of model parameters indicative of characteristics of the subject related to the interaction; and control navigation at the interactive UI based on the plurality of model parameters. ### Claim 13 (depends on 12) The system as claimed in claim 12, wherein the one or more body-parts movement tracking sensors comprises Inertial Motion Unit (IMU), an electroencephalogram (EEG) sensor, eye tracking sensor, gyroscope, and magnetometer. ### Claim 14 (depends on 12) The system as claimed in claim 12, wherein the one or more physiological signal sensors comprises Galvanic Skin Response sensor and Photoplethysmogram sensor. ### Claim 15 (depends on 12) The system as claimed in claim 12, wherein the one or more hardware processors are further configured by the instructions to train a model for determining the plurality of model parameters, and wherein to train the model in an offline mode, the one or more hardware processors are further configured by the instructions to: perform a plurality of known tasks and capturing a training sensor data during said performing, wherein the plurality of known tasks comprises interacting with the interactive UI; and obtain a plurality of training model parameters based on the training sensor data, the plurality of training model parameters facilitates in training the model. ### Claim 16 (depends on 12) The system as claimed in claim 12, wherein to control the navigation at the interactive UI, the one or more hardware processors are further configured by the instructions to perform one or more of: indicate highlighting at least one element on the interactive UI, generate an audio output at the interactive UI and generate a visual output at the interactive UI. ### Claim 17 (depends on 12) The system as claimed in claim 12, wherein the one or more hardware processors are further configured by the instructions to: monitor the sensor data generated based on the interaction of the subject with the interactive UI to determine a consistent change in the interaction; and automatically update, on determination of the consistent change, one or more model parameters from the plurality of model parameters based on the sensor data. ### Claim 18 (depends on 17) The system as claimed in claim 17, wherein the one or more hardware processors are further configured by the instructions to: adapt the interactive UI based on the updating of the one or more model parameters; and store the updated one or more model parameters at a server to derive the adapted interactive UI. ### Claim 19 (depends on 18) The system as claimed in claim 18, wherein to adapt the interactive UI, the one or more hardware processors are further configured by the instructions to adapt one or more elements of the interactive UI. ### Claim 20 (depends on 18) The system as claimed in claim 18, wherein to adapt the interactive UI, the one or more hardware processors are further configured by the instructions to adapt an orientation of a layout of the interactive UI.