# US 9,438,985 B2 · System and method of detecting a user's voice activity using an accelerometer > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,438,985 B2 - Document ID: US9438985B2 - Status: Granted - Kind: B2 - Application number: 13/631,716 - Filed: 2012-09-28 - Published: 2014-04-03 - Granted: 2016-09-06 - Earliest claimed: 2012-09-28 - Assignees: Apple Inc. - Applicants: Sorin V. Dusan; Esge B. Andersen; Aram Lindahl; Andrew P. Bright - Inventors: Sorin V. Dusan; Esge B. Andersen; Aram Lindahl; Andrew P. Bright - Web page: https://patentgrep.com/patent/US9438985B2 ## CPC - H04R 1/1083: {Reduction of ambient noise (active noise reduction per se G10K11/175; protective devices for the ear, e.g. providing acoustic protection A61F11/06)} - H04R 1/1016: {Earpieces of the intra-aural type} - H04R 1/406: {microphones} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - H04R 2201/403: Linear arrays of transducers - H04R 2410/01: Noise reduction using microphones having different directional characteristics - H04R 2410/05: Noise reduction with a separate noise microphone - H04R 2460/13: Hearing devices using bone conduction transducers - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract A method of detecting a user's voice activity in a headset with a microphone array is described herein. The method starts with a voice activity detector (VAD) generating a VAD output based on acoustic signals received from microphones included in a pair of earbuds and the microphone array included on a headset wire and data output by an accelerometer that is included in the pair of earbuds. A noise suppressor may then receive the acoustic signals from the microphone array and the VAD output and suppress the noise included in the acoustic signals received from the microphone array based on the VAD output. The method may also include steering one or more beamformers based on the VAD output. Other embodiments are also described. ## Claims ### Claim 1 A method of detecting a user's voice activity in a headset comprising: generating by a voice activity detector (VAD) a VAD output based on (i) acoustic signals received from at least one microphone included in a pair of earbuds and (ii) data output by at least one accelerometer that is included in the pair of earbuds, the at least one accelerometer to detect vibration of the user's vocal chords, wherein the headset includes the pair of earbuds, wherein the VAD generates a microphone VAD (VADm) output based on the acoustic signals and generates an accelerometer VAD (VADa) output based on the data output by the at least one accelerometer, wherein the VAD output is based on the VADm output and the VADa output, wherein generating the VAD output comprises: detecting speech included in the acoustic signals, and setting the VADm output to indicate that speech is detected in the acoustic signals, detecting the vibration of the user's vocal chords from the data output by the at least one accelerometer, and setting the VADa output to indicate that the user's voiced speech is detected, computing the coincidence of the VADm output being set to indicate detected speech in acoustic signals and the VADa output being set to indicate detected vibration of the user's vocal chords, and setting the VAD output to indicate that the user's voiced speech is detected if the coincidence is detected and setting the VAD output to indicate that the user's voiced speech is not detected if the coincidence is not detected. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one accelerometer is an accelerometer included in each of the earbuds. ### Claim 3 (depends on 1) The method of claim 1, wherein the at least one microphone included the pair of earbuds comprises: a front microphone and a rear microphone in each of the earbuds. ### Claim 4 (depends on 2) The method of claim 2, wherein generating the VAD output comprises: computing a power envelope of at least one of x, y, z signals generated by the at least one accelerometer; and setting the VADa output to indicate that the user's voiced speech is detected if the power envelope is greater than a threshold and setting the VADa output to indicate that the user's voiced speech is not detected if the power envelope is less than the threshold. ### Claim 5 (depends on 2) The method of claim 2, wherein generating the VAD output comprises: computing the normalized cross-correlation between any pair of x, y, z direction signals generated by the at least one accelerometer; setting the VADa output to indicate that the user's voiced speech is detected if normalized cross-correlation is greater than a threshold within a short delay range, and setting the VADa output to indicate that the user's voiced speech is not detected if the normalized cross-correlation is less than the threshold. ### Claim 6 (depends on 1) The method of claim 1, wherein generating the VAD output comprises: detecting unvoiced speech in the acoustic signals by: analyzing at least one of the acoustic signals; if an energy envelope in a high frequency band of the at least one of the acoustic signals is greater than a threshold, a VAD output for unvoiced speech (VADu) is set to indicate that unvoiced speech is detected; and setting the VAD output to indicate that the user's speech is detected if the voiced speech is detected or if the VADu is set to indicate that unvoiced speech is detected. ### Claim 7 (depends on 6) The method of claim 6, further comprising: receiving acoustic signals from a microphone array by a fixed beamformer, the microphone array is included on a headset wire, wherein the headset includes the headset wire; and steering the fixed beamformer in a direction of the user's mouth during a normal wearing position of the headset. ### Claim 8 (depends on 7) The method of claim 7, further comprising: receiving by a noise suppressor (i) a main speech signal from the fixed beamformer and (ii) the VAD output; and suppressing by the noise suppressor noise included in the main speech signal based on the VAD output. ### Claim 9 (depends on 6) The method of claim 6, further comprising: receiving acoustic signals from a microphone array by a source direction detector, the microphone array is included on a headset wire, wherein the headset includes the headset wire; detecting by the source direction detector the user's speech source based on the VAD output; adaptively steering a first beamformer in a direction of the detected user's speech source when the VAD output is set to indicate that the user's speech is detected, the first beamformer outputting a main speech signal. ### Claim 10 (depends on 9) The method of claim 9, wherein detecting by the source direction detector the user's speech source based on the VAD output comprises: determining a delay for a sound signal between microphones in the microphone array; and detecting the main acoustic source location using generalized cross correlation (GCC) or adaptive eigenvalue decomposition (AED). ### Claim 11 (depends on 9) The method of claim 9, detecting by the source direction detector the user's speech source based on the VAD output comprises: steering the first beamformer over a range of directions; and calculating a power of the first beamformer for each direction in the range of directions, wherein the user's speech source is detected as a direction in the range of directions having the highest power. ### Claim 12 (depends on 9) The method of claim 9, further comprising: adaptively steering a second beamformer with a null towards the user's speech source, wherein the second beamformer has a cardioid pattern, wherein the second beamformer outputs a signal representing environmental noise when the VAD output is set to indicate that the user's speech is not detected; receiving by a noise suppressor (i) a main speech signal from the first beamformer, (ii) the signal representing the environmental noise from the second beamformer, and (iii) the VAD output; and suppressing by the noise suppressor noise included in the main speech signal based on the signal representing the environmental noise and the VAD output. ### Claim 13 (depends on 9) The method of claim 9, further comprising: adaptively steering a second beamformer in a direction of strongest environmental noise location when the VAD output is set to indicate that the user's speech is not detected, wherein the second beamformer outputs a signal representing the strongest environmental noise; receiving by a noise suppressor (i) a main speech signal from the first beamformer, (ii) the signal representing the strongest environmental noise outputted from the second beamformer, and (iii) the VAD output; and suppressing by the noise suppressor noise included in the main speech signal based on the signal representing the strongest environmental noise and the VAD output. ### Claim 14 (depends on 9) The method of claim 9, further comprising: detecting by a second beamformer a direction of strongest environmental noise location when the VAD output is set to indicate that the user's speech is not detected; adaptively steering the nulls of the first beamformer in the direction of the strongest environmental noise location to output a main speech signal from the first beamformer; receiving by a noise suppressor (i) the main speech signal being output from the first beamformer, and (ii) the VAD output; and suppressing by the noise suppressor noise included in the main speech signal based on the VAD output. ### Claim 15 (depends on 1) The method of claim 1, wherein the at least one accelerometer has a sampling rate between 2000Hz to 6000Hz. ### Claim 16 (depends on 1) The method of claim 1, wherein the at least one accelerometer is tuned to be sensitive to a frequency band range that is below 3000Hz. ### Claim 17 A system detecting a user's voice activity comprising: a headset including a pair of earbuds, wherein the pair of earbuds includes at least one earbud microphone and at least one accelerometer to detect vibration of the user's vocal chords; a voice activity detector (VAD) coupled to the headset, the VAD to generate a VAD output based on (i) acoustic signals received from the at least one earbud microphone and (ii) data output by the at least one accelerometer, wherein the VAD generates a microphone VAD (VADm) output based on the acoustic signals and generates an accelerometer VAD (VADa) output based on the data output by the at least one accelerometer, wherein the VAD output is based on the VADm output and the VADa output, wherein the VAD generates the VAD output by: detecting speech included in the acoustic signals, and setting the VADm output to indicate that speech is detected in the acoustic signals, detecting the vibrations of the user's vocal chords from the data output by the at least one accelerometer, and setting the VADa output to indicate that the user's voiced speech is detected, computing the coincidence of the VADm output being set to indicate detected speech in acoustic signals and the VADa output being set to indicate detected vibrations of the user's vocal chords, and setting the VAD output to indicate that the user's voiced speech is detected if the coincidence is detected and setting the VAD output to indicate that the user's voiced speech is not detected if the coincidence is not detected; and a noise suppressor coupled to the headset and the VAD, the noise suppressor to suppress noise based on the VAD output. ### Claim 18 (depends on 17) The system of claim 17, wherein the at least one earbud microphone comprises a front microphone and a rear microphone in each of the earbuds. ### Claim 19 (depends on 17) The system of claim 17, wherein the VAD generates the VAD output by: computing a power envelope of at least one of x, y, z signals generated by the at least one accelerometer; and setting the VADa output to indicate that the user's voiced speech is detected if the power envelope is greater than a threshold and setting the VADa output to indicate that the user's voiced speech is not detected if the power envelope is less than the threshold. ### Claim 20 (depends on 17) The system of claim 17, wherein the VAD generates the VAD output by: computing the normalized cross-correlation between any pair of x, y, z direction signals generated by the at least one accelerometer; and setting the VADa output to indicate that the user's voiced speech is detected if normalized cross-correlation is greater than a threshold within a short delay range, and setting the VADa output to indicate that the user's voiced speech is not detected if the normalized cross-correlation is less than the threshold. ### Claim 21 (depends on 17) The system of claim 17, wherein generating the VAD output comprises: detecting unvoiced speech in the acoustic signals by: analyzing at least one of the acoustic signals; if an energy envelope in a high frequency band of the at least one of the acoustic signals is greater than a threshold, a VAD output for unvoiced speech (VADu) is set to indicate that unvoiced speech is detected; and setting the VAD output to indicate that the user's speech is detected if the voiced speech is detected or if the VADu is set to indicate that unvoiced speech is detected. ### Claim 22 (depends on 17) The system of claim 17, further comprising: a fixed beamformer receiving acoustic signals from a microphone array included on a headset wire, wherein the headset includes the headset wire, wherein the fixed beamformer is steered in a direction of the user's mouth during a normal wearing position of the headset to output a main speech signal. ### Claim 23 (depends on 22) The system of claim 22, wherein the noise suppressor suppresses the noise included in the main speech signal outputted by the fixed beamformer based on the VAD output. ### Claim 24 (depends on 17) The system of claim 17, further comprising: a source direction detector receiving acoustic signals from a microphone array included on a headset wire and detecting the user's speech source based on the VAD output, wherein the headset includes the headset wire; and a first beamformer being adaptively steered in a direction of the detected user's speech source when the VAD output is set to indicate that the user's voiced speech is detected, wherein the first beamformer outputs a main speech signal. ### Claim 25 (depends on 24) The system of claim 24, wherein the source direction detector detects the user's speech source based on the VAD output by: determining a delay for a sound signal between microphones in the microphone array; and detecting the main acoustic source location using generalized cross correlation (GCC) or adaptive eigenvalue decomposition (AED). ### Claim 26 (depends on 24) The system of claim 24, wherein the source direction detector detects the user's speech source based on the VAD output by: steering the first beamformer over a range of directions; and calculating a power of the first beamformer for each direction in the range of directions, wherein the user's speech source is detected as a direction in the range of directions having the highest power. ### Claim 27 (depends on 24) The system of claim 24, further comprising: a second beamformer being adaptively steered to direct a null of the second beamformer towards the user's speech source, wherein the second beamformer has a cardioid pattern, wherein the second beamformer outputs a signal representing environmental noise when the VAD output is set to indicate that the user's voiced speech is not detected, wherein the noise suppressor suppresses the noise included in the main speech signal based the signal representing environmental noise outputted from the second beamformer and the VAD output. ### Claim 28 (depends on 24) The system of claim 24, further comprising: a second beamformer being adaptively steered in a direction of strongest environmental noise location when the VAD output is set to indicate that the user's speech is not detected, wherein the second beamformer outputs a signal representing the strongest environmental noise, wherein the noise suppressor suppresses the noise included in the main speech signal based on the signal representing the strongest environmental noise outputted from the second beamformer and the VAD output. ### Claim 29 (depends on 24) The system of claim 24, further comprising: a second beamformer detecting a direction of strongest environmental noise location when the VAD output is set to indicate that the user's speech is not detected, wherein the nulls of the first beamformer are adaptively steered in the direction of the strongest environmental noise location. ### Claim 30 (depends on 24) The system of claim 24, wherein the VAD and the noise suppressor are included in an electronic device.