# US 10,441,443 B2 · Multi-modal neural interfacing for prosthetic devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,441,443 B2 - Document ID: US10441443B2 - Status: Granted - Kind: B2 - Application number: 15/273,039 - Filed: 2016-09-22 - Published: 2017-01-26 - Granted: 2019-10-15 - Earliest claimed: 2016-09-22 - Assignees: The Johns Hopkins University - Applicants: The Johns Hopkins University - Inventors: Stuart D. Harshbarger; James D. Beaty; R. Jacob Vogelstein; Nitish V. Thakor - Web page: https://patentgrep.com/patent/US10441443B2 ## CPC - A61F 2/68: Operating or control means - A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems} - A61F 2/54: Artificial arms or hands or parts thereof - A61F 2/72: Bioelectric control, e.g. myoelectric - B25J 9/16: Program controls (total factory control, i.e. centrally controlling a plurality of machines, G05B19/418) - A61F 2002/5058: {having means for restoring the perception of senses} - A61F 2002/5059: {the feeling of position} - A61F 2002/5061: {the sense of touch} - A61F 2002/5063: {the feeling of temperature, i.e. of warmth or cold} - A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) ## Abstract Methods and systems to interface between physiological devices and a prosthetic device, including to receive a plurality of types of physiological activity signals from a user, decode a user movement intent from each of the plurality of signals types, and fuse the movement intents into a joint decision to control moveable elements of the prosthetic device. ## Claims ### Claim 1 A method for controlling a prosthetic device, the method comprising: receiving, by a plurality of types of sensors, a plurality of types of physiological activity signals from a prosthetic device user, wherein the plurality of types of physiological activity signals include a combination of two or more of: a local field potential (LFP) signal, a unit activity (spike) signal, an epidural electrocorticography grid (ECoG) signal, an electromyography (EMG) signal, an electroencephalography (EEG) signal, and an electronystagmography (ENG) signal; determining, by a plurality of classifier modules, user movement states from each of the plurality of types of physiological activity signals, wherein each classifier module of the plurality of classifier modules is associated with a corresponding type of physiological activity signal; decoding, by a plurality of decoders, movement intents from each of the plurality of types of physiological activity signals and from one or more of the user movement states, wherein each decoder of the plurality of decoders is associated with a corresponding type of physiological activity signal; and fusing, by a fusion module, the movement intents into a joint decision to control moveable elements of the prosthetic device. ### Claim 2 (depends on 1) The method of claim 1, wherein: determining the user movement state for each of the plurality of types of physiological activity signals is performed for each of a plurality of groups of control (GOC) of the prosthetic device; decoding of the movement intent for each of the plurality of types of physiological activity signals is performed for each of a plurality of the GOC; fusing of the movement intents into the joint decision is performed for each of a plurality of the GOC; and wherein the method of claim 1 further includes generating a movement action from joint decisions of a plurality of the GOCs. ### Claim 3 (depends on 2) The method of claim 2, wherein: the prosthetic device includes a prosthetic arm and hand; and the groups of control include an upper arm group, a wrist group, a hand and finger group, and an endpoint group. ### Claim 4 (depends on 2) The method of claim 2, wherein the plurality of decoders comprise a decoder to process cortical signals, a decoder to process peripheral nerve signals, a decoder to process EMG signals, and a decoder to process prosthetic control signals, wherein the method further comprises: processing cortical signals using a cortical multimodal control unit (cMCU); processing peripheral nerve signals using a peripheral nerve multimodal control unit (pMCU); and processing EMG signals and conventional prosthetic controls (CPC) signals using a neural fusion unit (NFU) including the fusion module and one or more of the plurality of decoders. ### Claim 5 (depends on 1) The method of claim 1, wherein determining user movement states includes classifying user movement states as one of motionless, pre-movement, and peri-movement. ### Claim 6 (depends on 1) The method of claim 1, wherein the fusion module is configured to perform one or more of a decision function and data fusion. ### Claim 7 (depends on 1) The method of claim 1, wherein the fusion module is integrated within the decoder. ### Claim 8 (depends on 1) The method of claim 1, further including, for each of the plurality of types of physiological activity signals: pre-processing signals of the plurality of types of physiological activity signals and selectively directing subsets of the signals to one or more classifier modules to determine the user movement state. ### Claim 9 (depends on 8) The method of claim 8, wherein the pre-processing includes identifying signals as one of valid and invalid. ### Claim 10 (depends on 1) The method of claim 1, further including: receiving and incorporating sensory feedback from the prosthetic device into the joint decision, wherein the sensory feedback includes one or more of velocity, speed, force, direction, position, and temperature information. ### Claim 11 (depends on 1) The method of claim 1, wherein the plurality of types of physiological activity signals comprises a relatively non-invasive physiological sensors or a relatively invasive physiological sensor. ### Claim 12 (depends on 1) The method of claim 1, wherein: the decoding includes computing an unnormalized log posterior probability (ULPP) value for each of a plurality of classes of movement in accordance with Bayesian classifiers and determining the movement intent from the ULPP values; and the fusing includes generating the joint decision based at least in part of the ULPP values.