# US 12,014,732 B2 · Energy efficient custom deep learning circuits for always-on embedded applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,014,732 B2 - Document ID: US12014732B2 - Status: Granted - Kind: B2 - Application number: 17/275,484 - Filed: 2019-09-13 - Published: 2022-04-21 - Granted: 2024-06-18 - Earliest claimed: 2018-09-14 - Assignees: Aondevices, Inc. - Applicants: Aondevices, Inc. - Inventors: Mouna Elkhatib; Adil Benyassine - Web page: https://patentgrep.com/patent/US12014732B2 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06N 3/02: Neural networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/063: using electronic means - G10L 15/08: Speech classification or search - G10L 15/16: using artificial neural networks - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 25/30: using neural networks - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - H03M 1/12: Analogue/digital converters ({H03M1/001 – } H03M1/10 take precedence) - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H03M: CODING; DECODING; CODE CONVERSION IN GENERAL (using fluidic means F15C4/00; optical analogue/digital converters G02F7/00; coding, decoding or code conversion, specially adapted for particular applications, see the relevant subclasses, e.g. G01D, G01R, G06F, G06T, G09G, G10L, G11B, G11C, H04B, H04L, H04M, H04N; ciphering or deciphering for cryptography or other purposes involving the need for secrecy G09C) ## Abstract A system can include a circuit holistically customized for the detection of commands in an audio or video input signal to meet certain application-specific requirements. The circuit can have a neural network topology that is hardwired to perform detection based on application-specific detection criteria. The hardwired custom circuit can provide improved energy efficiencies compared to similar functionality carried out using software and generic hardware modules. The system can also include a sound change trigger module and perform non-voiced sound detection. ## Claims ### Claim 1 A system comprising: non-transitory computer-readable medium comprising a plurality of command instructions stored thereon; and a customized neural network detector comprising a fixed deep learning neural network hardware topology with a set of fixed weights tailored for system-specific functions and programmable weights providing limited adaptability, predetermined structural parameters optimized for low power consumption, fixed activation functions for specific network operations and customizable activation functions, the neural network detector hardwired to receive an audio signal and/or video signal, and hardwired to select, based at least in part on the audio signal and/or video signal, a command instruction of the plurality of command instructions independently within the system without external network data transmission. ### Claim 2 (depends on 1) The system of claim 1, further comprising: an electrical input configured to receive the audio signal and/or video signal and directly provide the audio signal and/or video signal to the neural network detector; and an application processor interface in communication with the neural network detector configured to transmit the selected command instruction, wherein the electrical input, neural network detector, and application processor interface are integrated in a single device. ### Claim 3 (depends on 2) The system of claim 2, wherein the single device is a single semiconductor chip. ### Claim 4 (depends on 2) The system of claim 2, wherein the single device is one of a standalone chip, a codec chip, a microphone, a speaker, an image capture device, and an application processor. ### Claim 5 (depends on 1) The system of claim 1, wherein the audio signal is an audio stream and/or the video signal is a video stream. ### Claim 6 (depends on 1) The system of claim 1, further comprising: an event detection circuit configured to receive an audio stream and/or video stream, detect an event based on the audio stream and/or video stream, and transmit the audio signal and/or video signal following the event to the neural network detector. ### Claim 7 (depends on 6) The system of claim 6, wherein the event detection circuit comprises a voice activity detection (VAD) module, wherein the VAD module is configured to receive the audio stream and detect a presence of a voice in the audio stream, and wherein the neural network detector is hardwired to receive the audio signal and is hardwired to select, based at least in part on the audio signal, the command instruction of the plurality of command instructions. ### Claim 8 (depends on 7) The system of claim 7, wherein the event detection circuit further comprises a sound profile change trigger module configured to detect a change in non-voiced sound in the audio stream. ### Claim 9 (depends on 8) The system of claim 8 wherein the event detection circuit further comprises a status logic module configured to provide an output state based at least in part on an output of the VAD module and an output of the sound profile change trigger module. ### Claim 10 (depends on 1) The system of claim 1, further comprising: a microphone interface; and a digital signal processing block in communication with the microphone interface and the neural network detector. ### Claim 11 (depends on 1) The system of claim 1, further comprising: a microphone interface; a digital signal processing block in communication with the microphone interface; and a multiplexer in communication with the digital signal processing block and the neural network detector. ### Claim 12 (depends on 1) The system of claim 1, wherein the neural network hardware topology comprises a hardware deep learning data path. ### Claim 13 (depends on 1) The system of claim 1, wherein the fixed deep learning neural network hardware topology includes a long short-term memory (LSTM) network for processing the audio signal, and is configured with a fixed number of layers optimized for audio processing and recognition. ### Claim 14 (depends on 1) The system of claim 1, wherein the fixed deep learning neural network hardware topology includes a convolutional neural network (CNN) for processing the video signal, and is configured with fixed filter sizes and layer configurations for real-time video processing tasks and reduced power consumption. ### Claim 15 (depends on 1) The system of claim 1, wherein the programmable weights are for specific neural network layers to provide adaptability in processing and maintaining power efficiency. ### Claim 16 A method comprising: determining a plurality of detection criteria; associating each of the detection criteria with a respective command instruction of a plurality of command instructions; and hardwiring a customized fixed neural network based on the detection criteria such that the neural network is hardwired to receive an audio signal and/or video signal and select a command instruction based on the received audio signal and/or video signal and based on the detection criteria independently without external network data transmission, the detection criteria including a set of fixed weights tailored for specific functions and programmable weights providing limited adaptability, fixed activation functions for specific network operations and customizable activation functions, with predetermined structural parameters optimized for low power consumption. ### Claim 17 (depends on 16) The method of claim 16, wherein hardwiring the neural network based on the detection criteria comprises hardwiring the neural network to include weighted paths, each weighted path associated with a weight. ### Claim 18 (depends on 16) The method of claim 16, further comprising: transmitting the selected command instruction to an application processor. ### Claim 19 (depends on 16) The method of claim 16, wherein the neural network comprises a convolutional neural network and/or a recurrent neural network. ### Claim 20 (depends on 16) The method of claim 16, further comprising: hardwiring the neural network to receive the audio signal such that the audio signal is an audio stream; and hardwiring the neural network to continuously monitor the audio stream to select the command instruction. ### Claim 21 (depends on 16) The method of claim 16, further comprising: configuring a voice activity detection (VAD) module to receive an audio stream and provide the audio signal to the neural network in response to detecting a voice in the audio stream, the audio signal being a portion of the audio stream. ### Claim 22 (depends on 16) The method of claim 16, further comprising: configuring a sound profile change trigger module to receive an audio stream and provide the audio signal to the neural network in response to detecting a change in a non-voiced sound in the audio stream, the audio signal being a portion of the audio stream.