# US 12,178,632 B2 · Sensor fusion to validate sound-producing behaviors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,178,632 B2 - Document ID: US12178632B2 - Status: Granted - Kind: B2 - Application number: 17/455,152 - Filed: 2021-11-16 - Published: 2022-03-10 - Granted: 2024-12-31 - Earliest claimed: 2018-08-30 - Assignees: Biointellisense, Inc. - Applicants: Biointellisense, Inc. - Inventors: Eduardo Chan; James Mault; David Wang; Henry Leung; Sam Jones; Mark Ross - Web page: https://patentgrep.com/patent/US12178632B2 ## CPC - A61B 7/04: Electric stethoscopes - H04R 3/00: Circuits for transducers (arrangements for producing a reverberation or echo sound G10K15/08; amplifiers H03F) - A61B 5/02055: {Simultaneously evaluating both cardiovascular condition and temperature} - A61B 5/318: Heart-related electrical modalities, e.g. electrocardiography [ECG] - A61B 5/4809: {Sleep detection, i.e. determining whether a subject is asleep or not} - A61B 5/681: {Wristwatch-type devices} - A61B 5/7221: {Determining signal validity, reliability or quality (preventing, reducing or removing noise induced by motion artefacts A61B5/7207; noise originating from a therapeutic or surgical apparatus A61B5/7217)} - A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems} - A61B 5/7278: {Artificial waveform generation or derivation, e.g. synthesizing signals from measured signals} - A61B 5/7282: {Event detection, e.g. detecting unique waveforms indicative of a medical condition (cough events A61B5/0823; seizures A61B5/4094; sleep apnoea A61B5/4818)} - A61B 7/003: {Detecting lung or respiration noise} - A61B 7/008: {Detecting noise of gastric tract, e.g. caused by voiding} - G10L 25/66: for extracting parameters related to health condition (detecting or measuring for diagnostic purposes A61B5/00) - H04R 29/00: Monitoring arrangements; Testing arrangements {(for hearing aids H04R25/30; detection of loudspeaker connection H04R5/04; sound-field adaptation dependent on speaker detection H04S7/308)} - A61B 5/0261: {using optical means, e.g. infrared light} - A61B 5/0531: Measuring skin impedance - A61B 5/0823: {Detecting or evaluating cough events} - A61B 5/113: occurring during breathing - A61B 5/14551: {for measuring blood gases} - A61B 2562/0204: Acoustic sensors - A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches - G10L 25/51: for comparison or discrimination - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - H04R 1/04: Structural association of microphone with electric circuitry therefor (in electric hearing aids H04R25/00) - H04R 2420/07: Applications of wireless loudspeakers or wireless microphones - H04R 2499/11: Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract A method to measure sound-producing behaviors of a subject with a power- and bandwidth-limited electronic device that includes a processor includes measuring, by a microphone communicatively coupled to the processor, sound in a vicinity of the subject to generate an audio data signal that represents the sound. The method also includes measuring, by at least one second sensor communicatively coupled to the processor, at least one parameter other than sound to generate at least a second data signal that represents the at least one parameter other than sound. The method also includes detecting one or more sound-producing behaviors of the subject based on: both the audio data signal and the second data signal; or information derived from both the audio data signal and the second data signal. ## Claims ### Claim 1 A method to measure sound-producing behaviors of a subject with a power- and bandwidth-limited electronic device that includes a processor, the method comprising: measuring, by at least one sensor communicatively coupled to the processor, at least one parameter other than sound to generate a second data signal that represents the at least one parameter other than sound; detecting a first wake-up event occurring within the second data signal; in response to the detected first wake-up event, providing power to a microphone communicatively coupled to the processor for a first fixed window of time such that the microphone measures sound in a vicinity of the subject to generate a first audio data signal during the first fixed window of time; detecting a second wake-up event occurring within the second data signal; in response to the detected second wake-up event, providing power to the microphone for a second fixed window of time such that the microphone measures sound in the vicinity of the subject to generate a second audio data signal during the second fixed window of time; based on a signal quality of the first audio data signal being below threshold, discarding the first audio data signal; and detecting one or more sound-producing behaviors of the subject based on: both the second audio data signal and second data signal; or information derived from both the second audio data signal and the second data signal. ### Claim 2 (depends on 1) The method of claim 1, wherein measuring, by the at least one sensor, the at least one parameter other than sound to generate the second data signal comprises at least one of: measuring, by an accelerometer communicatively coupled to the processor, acceleration of a chest of the subject to generate an acceleration data signal that represents acceleration of the chest of the subject; measuring, by a gyro sensor communicatively coupled to the processor, angular velocity of at least a portion of the subject to generate an angular velocity data signal that represents angular velocity of the at least the portion of the subject; measuring, by a thermometer communicatively coupled to the processor, at least one of a skin temperature or a core body temperature of the subject to generate a temperature signal that represents at least one of the skin temperature or the core body temperature of the subject; measuring, by an oxygen saturation sensor communicatively coupled to the processor, blood oxygenation of the subject to generate an oxygen saturation signal of the oxygen saturation level of the subject; measuring, by a photoplethysmograph (PPG) sensor communicatively coupled to the processor, blood flow of the subject to generate a blood flow data signal that represents blood flow of the subject; measuring, by an electrocardiogram (ECG) sensor communicatively coupled to the processor, electrical activity of a heart of the subject to generate an ECG data signal that represents electrical activity of the heart; or measuring, by an electrodermal activity (EDA) sensor communicatively coupled to the processor, EDA of the subject to generate an EDA data signal that represents EDA of the subject. ### Claim 3 (depends on 1) The method of claim 1, wherein the detecting further includes: extracting, by the processor, a plurality of audio features from the second audio data signal; and sub-classifying each audio feature of the plurality of audio features as indicative of a corresponding type of sound-producing behavior. ### Claim 4 (depends on 3) The method of claim 3, wherein the corresponding types of sound-producing behaviors include at least one of sneezing, wheezing, shortness of breath, chewing, swallowing, masturbation, sex, coughing, vomiting, and shouting. ### Claim 5 (depends on 1) The method of claim 1, further comprising: receiving, from the subject, annotation input that confirms occurrence of one or more sound-producing behaviors; reporting, to a remote server, the annotation input, wherein the remote server is configured to receive annotation inputs and information about sound-producing behaviors from a plurality of subjects and to update an algorithm used in the detecting; and receiving, from the remote server, the updated algorithm. ### Claim 6 (depends on 1) The method of claim 1, wherein the detecting is based on the information derived from both the second audio data signal and the second data signal, the method further comprising: deriving the information from both the second audio data signal and the second data signal by converting the second audio data signal from a time domain into a frequency domain across a plurality of frames; and discarding one or more frames without identified sounds of interest. ### Claim 7 (depends on 1) The method of claim 1, wherein the microphone is oriented against a chest of the subject. ### Claim 8 (depends on 1) The method of claim 1, wherein the detecting is based on a third data signal from a third sensor monitoring an additional characteristic other than sound. ### Claim 9 (depends on 1) The method of claim 1, further comprising generating a reference signal for a plurality of sound-producing behaviors during an enrollment process. ### Claim 10 (depends on 1) The method of claim 1, wherein the detecting includes identifying a plurality of candidate sound-producing behaviors. ### Claim 11 A power- and bandwidth-limited electronic device, comprising: a processor; a first sensor configured to monitor a parameter other than sound coupled to the processor; a microphone coupled to the processor; a controllable power source; and a non-transitory storage media containing instructions that, when executed by the processor, cause the electronic device to perform operations, the operations comprising: receiving, from the first sensor, a second data signal that represents the parameter other than sound; detecting a first wake-up event occurring within the second data signal; in response to the first detected wake-up event, causing the controllable power source to provide power to the microphone for a first fixed window of time such that the microphone measures sound in a vicinity of a subject to generate a first audio data signal during the first fixed window of time; detecting a second wake-up event occurring within the second data signal; in response to the detected second wake-up event, causing the controllable power source to provide power to the microphone for a second fixed window of time such that the microphone measures sound in the vicinity of the subject to generate a second audio data signal during the second fixed window of time; based on a signal quality of the first audio data signal being below threshold, discarding the first audio data signal; and detecting one or more sound-producing behaviors of the subject based on: both the second audio data signal and second data signal; or information derived from both the second audio data signal and the second data signal. ### Claim 12 (depends on 11) The electronic device of claim 11, wherein receiving, from the first sensor, the second data signal comprises at least one of: measuring, by an accelerometer communicatively coupled to the processor, acceleration of a chest of the subject to generate an acceleration data signal that represents acceleration of the chest of the subject; measuring, by a gyro sensor communicatively coupled to the processor, angular velocity of at least a portion of the subject to generate an angular velocity data signal that represents angular velocity of the at least the portion of the subject; measuring, by a thermometer communicatively coupled to the processor, at least one of a skin temperature or a core body temperature of the subject to generate a temperature signal that represents at least one of the skin temperature or the core body temperature of the subject; measuring, by an oxygen saturation sensor communicatively coupled to the processor, blood oxygenation of the subject to generate an oxygen saturation signal of the oxygen saturation level of the subject; measuring, by a photoplethysmograph (PPG) sensor communicatively coupled to the processor, blood flow of the subject to generate a blood flow data signal that represents blood flow of the subject; measuring, by an electrocardiogram (ECG) sensor communicatively coupled to the processor, electrical activity of a heart of the subject to generate an ECG data signal that represents electrical activity of the heart; or measuring, by an electrodermal activity (EDA) sensor communicatively coupled to the processor, EDA of the subject to generate an EDA data signal that represents EDA of the subject. ### Claim 13 (depends on 11) The electronic device of claim 11, wherein the detecting further includes: extracting, by the processor, a plurality of audio features from the second audio data signal; and sub-classifying each audio feature of the plurality of audio features as indicative of a corresponding type of sound-producing behavior. ### Claim 14 (depends on 13) The electronic device of claim 13, wherein the corresponding types of sound-producing behaviors include at least one of sneezing, wheezing, shortness of breath, chewing, swallowing, masturbation, sex, coughing, vomiting, and shouting. ### Claim 15 (depends on 11) The electronic device of claim 11, wherein the operations further comprise: receiving, from the subject, annotation input that confirms occurrence of one or more sound-producing behaviors; reporting, to a remote server, the annotation input, wherein the remote server is configured to receive annotation inputs and information about sound-producing behaviors from a plurality of subjects and to update an algorithm used in the detecting; and receiving, from the remote server, the updated algorithm. ### Claim 16 (depends on 11) The electronic device of claim 11, wherein the detecting is based on the information derived from both the second audio data signal and the second data signal, the operations further comprising: deriving the information from both the second audio data signal and the second data signal by converting the second audio data signal from a time domain into a frequency domain across a plurality of frames; and discarding one or more frames without identified sounds of interest. ### Claim 17 (depends on 11) The electronic device of claim 11, wherein the microphone is oriented against a chest of the subject. ### Claim 18 (depends on 11) The electronic device of claim 11, further comprising a third sensor and wherein the detecting is based on a third data signal from the third sensor monitoring an additional characteristic other than sound. ### Claim 19 (depends on 11) The electronic device of claim 11, wherein the operations further comprise generating a reference signal for a plurality of sound-producing behaviors during an enrollment process. ### Claim 20 (depends on 11) The electronic device of claim 11, wherein the detecting includes identifying a plurality of candidate sound-producing behaviors.