# US 10,504,506 B2 · Neural network method and apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,506 B2 - Document ID: US10504506B2 - Status: Granted - Kind: B2 - Application number: 15/834,416 - Filed: 2017-12-07 - Published: 2019-02-14 - Granted: 2019-12-10 - Earliest claimed: 2017-08-14 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Sang Hyun Yoo - Web page: https://patentgrep.com/patent/US10504506B2 ## CPC - G10L 15/16: using artificial neural networks - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G10L 15/063: {Training} - G10L 15/20: Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech (G10L21/02 takes precedence) - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A neural network method and apparatus, the method including providing a voice signal to a main neural network and a sub-neural network, obtaining a scaling factor by implementing the sub-neural network configured to generate the scaling factor by interpreting the provided voice signal, determining a size of a future context, based on the scaling factor, to be considered by the main neural network configured to perform speech recognition, obtaining a result of a recognizing of the voice signal by implementing the main neural network with the determined size of the future context. ## Claims ### Claim 1 A processor implemented recognition method, comprising: providing a voice signal to a main neural network and a sub-neural network; obtaining a scaling factor by implementing the sub-neural network configured to generate the scaling factor by interpreting the provided voice signal; determining a size of a future context, based on the scaling factor, to be considered by the main neural network configured to perform speech recognition; and obtaining a result of a recognizing of the voice signal by implementing the main neural network with the determined size of the future context. ### Claim 2 (depends on 1) The recognition method of claim 1, wherein the main neural network includes a first layer having previous time connections and a second layer having future time connections. ### Claim 3 (depends on 1) The recognition method of claim 1, wherein the determining of the size of the future context includes determining sizes of respective future contexts for respective speech recognitions by the main neural network, the sizes of the respective future contexts being dynamically determined dependent on respective context interpretations of voice signals by the sub-neural network. ### Claim 4 (depends on 1) The recognition method of claim 1, wherein the providing of the voice signal to the main neural network includes providing plural audio frames or windows to the main neural network, and the providing of the voice signal to the sub-neural network includes providing less than all of the plural audio frames or windows to the sub-neural network, wherein the sub-neural network is configured to generate the scaling factor by interpreting the provided less than all of the plural audio frames or windows, and the main neural network performs the recognizing of the voice signal based on the plural audio frames or windows. ### Claim 5 (depends on 1) The recognition method of claim 1, wherein, for the obtaining of the scaling factor, the providing of the voice signal to the sub-neural network includes selecting between providing all of the plural audio frames or windows to the sub-neural network and providing less than all of the plural audio frames or windows to the sub-neural network. ### Claim 6 (depends on 1) The recognition method of claim 1, wherein the obtaining of the scaling factor comprises obtaining, using the sub-neural network, the scaling factor as indicating a time based significance of the future context to be considered by the main neural network when recognizing the voice signal. ### Claim 7 (depends on 6) The recognition method of claim 6, wherein the obtaining of the scaling factor comprises obtaining the scaling factor based a trained interpretation by the sub-neural network of a determined or interpreted level of noise included in the voice signal and/or a determined or interpreted accuracy of user pronunciation included in the voice signal. ### Claim 8 (depends on 6) The recognition method of claim 6, wherein the obtaining of the scaling factor includes generating the scaling factor with a relatively large number of components having values greater than a predetermined threshold based on a clarity of the voice signal being interpreted to be relatively low by the sub-neural network, such that the size of the future context increases as a number of components that are greater than the predetermined threshold increase and decreases as the number of the components that are greater than the predetermined threshold decrease. ### Claim 9 (depends on 1) The recognition method of claim 1, wherein the determining of the size of the future context includes: determining, based on the scaling factor, the size of the future context to be greater as a clarity of the voice signal is determined or interpreted to be relatively low by the sub-neural network than as the clarity of the voice signal is determined or interpreted to be relatively high by the sub-neural network, and determining, based on the scaling factor, the size of the future context to be smaller or zero as the clarity of the voice signal is interpreted to be relatively high by the sub-neural network than as the clarity of the voice signal is determined or interpreted to be relatively low by the sub-neural network. ### Claim 10 (depends on 1) The recognition method of claim 1, wherein the determining of the size of the future context includes determining the size of the future context by comparing values of components of the scaling factor to a predetermined threshold. ### Claim 11 (depends on 10) The recognition method of claim 10, wherein the determining of the size of the future context further includes determining the size of the future context based on a determined highest dimension component, among all components of the scaling factor, having a value greater than the predetermined threshold. ### Claim 12 (depends on 1) The recognition method of claim 1, wherein the obtaining of the result of recognizing of the voice signal comprises: adjusting, based on the determined size of the future context, an extent among plural future context for provision to a lookahead convolution layer of the main neural network; implementing the main neural network based on the adjusted extent; and obtaining the result of the recognizing of the voice signal from the lookahead convolution layer implemented based on the adjusted extent. ### Claim 13 (depends on 12) The recognition method of claim 12, wherein the adjusting of the extent among the plural future context for the provision to the lookahead convolution layer includes selectively, based on the determined size of the future context, providing between zero future context to the lookahead convolution layer, some future context but not all of the plural future context to the lookahead convolution layer, and all of the plural future context to the lookahead convolution layer. ### Claim 14 (depends on 1) The recognition method of claim 1, wherein the obtaining of the scaling factor by implementing the sub-neural network includes the sub-neural network generating the scaling factor from only a first window of plural windows of the voice signal, and the obtaining of the result of recognizing of the voice signal includes obtaining the recognition result with respect to each of the plural windows. ### Claim 15 (depends on 1) The recognition method of claim 1, wherein the main neural network is a unidirectional recurrent neural network (RNN) including a lookahead convolution layer. ### Claim 16 (depends on 1) The recognition method of claim 1, wherein the main neural network and the sub-neural network are neural networks having been trained simultaneously. ### Claim 17 (depends on 1) A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1. ### Claim 18 A processor implemented training method, comprising: obtaining a scaling factor from a sub-neural network into which a training input is input; determining a size of a future context to be considered by a main neural network based on the scaling factor; and training the main neural network and the sub-neural network such that the main neural network outputs a training output mapped to the training input in response to the training input being input into the main neural network to which the determined size of the future context is applied. ### Claim 19 (depends on 18) The training method of claim 18, wherein the training comprises training the main neural network and the sub-neural network simultaneously. ### Claim 20 A recognition apparatus, comprising: a processor configured to provide a voice signal to a main neural network and a sub-neural network, obtain a scaling factor by implementing the sub-neural network configured to generate the scaling factor by interpreting the provided voice signal, determine a size of a future context, based on the scaling factor, to be considered by the main neural network configured to perform speech recognition, and obtain a result of a recognizing of the voice signal by implementing the main neural network with the determined size of the future context. ### Claim 21 (depends on 20) The recognition apparatus of claim 20, further comprising a memory, the memory including instructions, that when executed by the processor, cause the processor to perform the providing of the voice signal, the obtaining of the scaling factor, the determining of the size of the future context, and the obtaining of the result of the recognizing. ### Claim 22 (depends on 20) The recognition apparatus of claim 20, further comprising a memory storing respective trained parameters of the main neural network and the sub-neural network, and the processor is further configured to apply the respective trained parameters when respectively implementing the main neural network and implementing the sub-neural network. ### Claim 23 (depends on 20) The recognition apparatus of claim 20, wherein, for the obtaining of the scaling factor, the processor is configured to obtain, using the sub-neural network, the scaling factor as indicating a time based significance of the future context to be considered by the main neural network when recognizing the voice signal. ### Claim 24 (depends on 23) The recognition apparatus of claim 23, wherein, for the obtaining of the scaling factor, the processor is configured to obtain the scaling factor based on a trained interpretation by the sub-neural network of a determined or interpreted level of noise included in the voice signal and/or a determined or interpreted accuracy of user pronunciation included in the voice signal. ### Claim 25 (depends on 20) The recognition apparatus of claim 20, wherein, for the determining of the size of the future context, the processor is configured to: determine, based on the scaling factor, the size of the future context to be greater as a clarity of the voice signal is determined or interpreted to be relatively low by the sub-neural network than as the clarity of the voice signal is determined or interpreted to be relatively high by the sub-neural network, and determine, based on the scaling factor, the size of the future context to be smaller or zero as the clarity of the voice signal is interpreted to be relatively high by the sub-neural network than as the clarity of the voice signal is determined or interpreted to be relatively low by the sub-neural network. ### Claim 26 (depends on 20) The recognition apparatus of claim 20, wherein, for the determining of the size of the future context, the processor is configured to determine the size of the future context by comparing values of components included in the scaling factor to a predetermined threshold. ### Claim 27 (depends on 20) The recognition apparatus of claim 20, wherein the recognition apparatus is an electronic device and further comprises a speech receiver configured to capture audio and the processor is configured to extract the voice signal from the captured audio, wherein the processor is further configured to control non-speech recognition functions of the recognition apparatus based on interpreted commands indicated by the obtained results of the recognizing. ### Claim 28 An electronic device, the electronic device comprising, comprising: a speech receiver configured to capture audio; a memory storing trained parameters of a main neural network and trained parameters of a sub-neural network; and a processor configured to apply the trained parameters of the sub-neural network to implement the sub-neural network to which a voice signal derived from the captured audio is input, and to apply the trained parameters of the main neural network to implement the main neural network, to recognize the voice signal, with dynamically adjusted future context consideration based on results of the implementation of the sub-neural network, wherein the main neural network includes a recurrent layer configured to consider past context. ### Claim 29 (depends on 28) The electronic device of claim 28, wherein the trained parameters of the sub-neural network configure the sub-neural network to interpret a clarity of the voice signal, with the dynamic adjusted future context consideration of the main neural network being performed by maximizing a size of the future context consideration of the main neural network when the sub-neural network interprets the clarity to be low and minimizing a size of the future context consideration of the main neural network when the sub-neural network interprets the clarity to be high. ### Claim 30 (depends on 28) The electronic device of claim 28, wherein the processor is further configured to indicate the result of the recognizing of the voice signal.