# US 12,182,674 B2 · Sound detection for electronic devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,674 B2 - Document ID: US12182674B2 - Status: Granted - Kind: B2 - Application number: 17/737,017 - Filed: 2022-05-04 - Published: 2022-12-08 - Granted: 2024-12-31 - Earliest claimed: 2021-06-06 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Jonathan Huang; Miquel Espi Marques; Carlos M. Avendano; Kevin M. Durand; David Findlay; Vasudha Kowtha; Daniel C. Klingler; Yichi Zhang - Web page: https://patentgrep.com/patent/US12182674B2 ## CPC - G06N 20/00: Machine learning - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G08B 1/08: using electric transmission {; transformation of alarm signals to electrical signals from a different medium, e.g. transmission of an electric alarm signal upon detection of an audible alarm signal} - G08B 21/18: Status alarms (G08B21/02 takes precedence) - G10L 15/063: {Training} - G10L 15/16: using artificial neural networks - H04R 1/08: Mouthpieces; {Microphones;} Attachments therefor - H04R 29/00: Monitoring arrangements; Testing arrangements {(for hearing aids H04R25/30; detection of loudspeaker connection H04R5/04; sound-field adaptation dependent on speaker detection H04S7/308)} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract The subject disclosure provides systems and methods for providing locally trained models for detecting individual sounds using electronic devices. Local detection of individual sounds with a detection model at an electronic device can be provided by obtaining training samples for the detection model with the electronic device, and generating additional negative and positive training samples based on the obtained training samples. A two-stage detection process may be provided, in which a trigger model at a device compares an audio input to a reference sound to trigger a detection model at the device. The detection of individual sounds with a detection model at an electronic device can also leverage audio capture capabilities of multiple devices in an acoustic scene to capture multiple concurrent training samples. ## Claims ### Claim 1 A method, comprising: obtaining, with an electronic device, a sound input; providing an audio signal based on the sound input to a first classifier at the electronic device; determining, with the first classifier, that the audio signal meets a similarity threshold based on a comparison of the audio signal with a device-generated class prototype; providing, based on the determining with the first classifier, the audio signal to a second classifier at the electronic device, the second classifier having been trained to detect a specific sound using at least one training sound sample captured by the electronic device and at least one device-generated training sound sample generated by the electronic device based on the at least one training sound sample; and generating, by the second classifier, an output indicating whether a specific sound is detected in the sound input. ### Claim 2 (depends on 1) The method of claim 1, further comprising generating an alert with the electronic device if the output of the second classifier indicates that the specific sound is detected. ### Claim 3 (depends on 1) The method of claim 1, wherein the at least one device-generated training sound sample includes: at least one negative training sample generated by the electronic device based on a first modification of the at least one training sound sample, and at least one positive training sample generated by the electronic device based on a second modification the at least one training sound sample. ### Claim 4 (depends on 3) The method of claim 3, further comprising training the second classifier at the electronic device by providing the at least one training sound sample, the at least one negative training sample, and the at least one positive training sample as inputs to the second classifier. ### Claim 5 (depends on 4) The method of claim 4, wherein the at least one training sound sample comprises a plurality of training sound samples, the method further comprising generating the device-generated class prototype at the electronic device, in part, by: segmenting of each of the plurality of training sound samples to generate a corresponding plurality of training segments; selecting a subset of the plurality of training segments based on a similarity of the training segments of the subset; and aligning the subset of the plurality of training segments to generate a set of aligned training segments. ### Claim 6 (depends on 5) The method of claim 5, further comprising obtaining at least a first one of the plurality of training sound samples using a microphone of the electronic device. ### Claim 7 (depends on 6) The method of claim 6, further comprising obtaining at least a second one of the plurality of training sound samples from another electronic device. ### Claim 8 (depends on 1) The method of claim 1, wherein the comparison of the audio signal with the device-generated class prototype comprises a determination of a cosine distance between a representation of the audio signal and the device-generated class prototype. ### Claim 9 An electronic device, comprising: a memory storing: a trigger model, a reference sound generated at the electronic device, and a detection model trained to detect a specific sound using at least one training sound sample captured by the electronic device and at least one device-generated training sound sample generated by the electronic device based on the at least one training sound sample; and one or more processors, wherein the one or more processors are configured to: obtain a sound input; trigger the detection model by comparing, with the trigger model, an audio signal based on the sound input with the reference sound; and determine whether the sound input includes the specific sound by providing the audio signal to the detection model. ### Claim 10 (depends on 9) The electronic device of claim 9, wherein the one or more processors are further configured to generate the reference sound based on a plurality of training inputs to the electronic device. ### Claim 11 (depends on 9) The electronic device of claim 9, further comprising an output component configured to generate a notification responsive to a determination, by the detection model, that the sound input includes the specific sound. ### Claim 12 (depends on 9) The electronic device of claim 9, further comprising a microphone configured to obtain the sound input. ### Claim 13 (depends on 12) The electronic device of claim 12, wherein the one or more processors are further configured to: obtain the at least one training sound sample using the microphone; and generate the reference sound based on the at least one training sound sample. ### Claim 14 (depends on 13) The electronic device of claim 13, wherein the one or more processors are further configured to: generate the at least one device-generated training sound sample by: generating at least one negative training sample by applying a first modification of the at least one training sound sample, and generate at least one positive training sample by applying a second modification to the at least one training sound sample. ### Claim 15 (depends on 14) The electronic device of claim 14, wherein the one or more processors are further configured to train the detection model by providing the at least one training sound sample, the at least one negative training sample, and the at least one positive training sample to the detection model as training inputs. ### Claim 16 (depends on 13) The electronic device of claim 13, wherein the one or more processors are further configured to obtain at least one additional training sound sample from another electronic device. ### Claim 17 An electronic device, comprising: a memory; and one or more processors configured to: obtain a sound input; provide an audio signal based on the sound input to a first classifier at the electronic device; determine, with the first classifier, that the audio signal meets a similarity threshold based on a comparison of the audio signal with a device-generated class prototype; provide, based on the determining with the first classifier, the audio signal to a second classifier at the electronic device, the second classifier having been trained to detect a specific sound using at least one training sound sample captured by the electronic device and at least one device-generated training sound sample generated by the electronic device based on the at least one training sound sample; and generate, by the second classifier, an output indicating whether the specific sound is detected in the sound input. ### Claim 18 (depends on 17) The electronic device of claim 17, wherein the one or more processors are further configured to generate an alert with the electronic device if the output of the second classifier indicates that the specific sound is detected. ### Claim 19 (depends on 17) The electronic device of claim 17, wherein the at least one device-generated training sound sample includes: at least one negative training sample generated by the electronic device based on a first modification of the at least one training sound sample, and at least one positive training sample generated by the electronic device based on a second modification the at least one training sound sample. ### Claim 20 (depends on 19) The electronic device of claim 19, wherein the one or more processors are further configured to train the second classifier at the electronic device by providing the at least one training sound sample, the at least one negative training sample, and the at least one positive training sample as inputs to the second classifier. ### Claim 21 (depends on 20) The electronic device of claim 20, wherein the at least one training sound sample comprises a plurality of training sound samples, and wherein the one or more processors are further configured to generate the device-generated class prototype at the electronic device, in part, by: segmenting of each of the plurality of training sound samples to generate a corresponding plurality of training segments; selecting a subset of the plurality of training segments based on a similarity of the training segments of the subset; and aligning the subset of the plurality of training segments to generate a set of aligned training segments. ### Claim 22 (depends on 21) The electronic device of claim 21, wherein the one or more processors are further configured to obtain at least a first one of the plurality of training sound samples using a microphone of the electronic device. ### Claim 23 (depends on 22) The electronic device of claim 22, wherein the one or more processors are further configured to obtain at least a second one of the plurality of training sound samples from another electronic device. ### Claim 24 (depends on 23) The electronic device of claim 23, wherein the electronic device is associated with an account and the other electronic device is also associated with the account. ### Claim 25 (depends on 17) The electronic device of claim 17, wherein the comparison of the audio signal with the device-generated class prototype comprises a determination of a cosine distance between a representation of the audio signal and the device-generated class prototype. ### Claim 26 (depends on 17) The electronic device of claim 17, wherein the first classifier comprises a trigger model and the second classifier comprises a detection model. ### Claim 27 (depends on 26) The electronic device of claim 26, wherein the one or more processors are further configured to provide, to the detection model along with the audio signal, a trigger signal to activate the detection model to generate the output indicating whether the specific sound is detected in the sound input. ### Claim 28 (depends on 17) The electronic device of claim 17, wherein the specific sound comprises a sound that has been previously enrolled for detection by the electronic device.