# US 12,183,324 B2 · Speech recognition and codec method and apparatus, electronic device and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,324 B2 - Document ID: US12183324B2 - Status: Granted - Kind: B2 - Application number: 17/738,651 - Filed: 2022-05-06 - Published: 2023-03-23 - Granted: 2024-12-31 - Earliest claimed: 2021-09-13 - Assignees: Beijing Baidu Netcom Science Technology Co., Ltd. - Applicants: Beijing Baidu Netcom Science Technology Co., Ltd. - Inventors: Xiaoyin Fu; Zhijie Chen; Mingxin Liang; Mingshun Yang; Lei Jia; Haifeng Wang - Web page: https://patentgrep.com/patent/US12183324B2 ## CPC - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/16: using artificial neural networks - G06F 16/683: using metadata automatically derived from the content - G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The present disclosure provides speech recognition and codec methods and apparatuses, an electronic device and a storage medium, and relates to the field of artificial intelligence such as intelligent speech, deep learning and natural language processing. The speech recognition method may include: acquiring an audio feature of to-be-recognized speech; encoding the audio feature to obtain an encoding feature; truncating the encoding feature to obtain continuous N feature fragments, N being a positive integer greater than one; and acquiring, for any one of the feature segments, corresponding historical feature abstraction information, encoding the feature segment in combination with the historical feature abstraction information, and decoding an encoding result to obtain a recognition result corresponding to the feature segment, wherein the historical feature abstraction information is information obtained by feature abstraction of recognized historical feature fragments. ## Claims ### Claim 1 A computer-implemented speech recognition method, comprising: acquiring an audio feature of to-be-recognized speech collected by a user equipment in real time; encoding the audio feature to obtain an encoding feature; truncating the encoding feature to obtain continuous N feature fragments with unequal lengths, N being a positive integer greater than one; acquiring, for any one of the feature segments, corresponding historical feature abstraction information, encoding the feature segment in combination with the historical feature abstraction information, and decoding an encoding result to obtain a recognition result corresponding to the feature segment, wherein the historical feature abstraction information is information obtained by feature abstraction of recognized historical feature fragments, wherein the encoding the feature segment in combination with the historical feature abstraction information comprises: encoding the feature segment in combination with the historical feature abstraction information by using a second encoder; and the decoding an encoding result comprises: decoding the encoding result by using a first decoder; wherein the historical feature abstraction information is generated according to output information of the decoder, wherein the second encoder comprises a P-layer convolution-augmented Conformer model, P being a positive integer; a number of the first decoder is Q, Q is a positive integer, and the Q first decoders are successively connected; and performing the following processing for any Conformer model: performing feature abstraction on a feature fragment currently processed in the Conformer model by using hidden layer features acquired from any first decoder, to obtain a feature vector with a predetermined length, and taking the feature vector as a historical feature vector of the currently processed feature fragment corresponding to the Conformer model, wherein the acquiring corresponding historical feature abstraction information comprises: for any Conformer model, splicing historical feature vectors of historical feature fragments corresponding to the Conformer model respectively, and taking a splicing result as the historical feature abstraction information. ### Claim 2 (depends on 1) The method according to claim 1, further comprising: performing convolutional downsampling on the audio feature, and encoding the downsampled audio feature. ### Claim 3 (depends on 1) The method according to claim 1, further comprising: determining peak information corresponding to the encoding feature in combination with the encoding of the audio feature and a Connectionist Temporal Classification (CTC) loss; wherein the truncating the encoding feature comprises: truncating the encoding feature according to the peak information. ### Claim 4 (depends on 3) The method according to claim 3, wherein the truncating the encoding feature according to the peak information comprises: taking an encoding feature located between each two adjacent peaks as a feature segment. ### Claim 5 (depends on 1) The method according to claim 1, wherein the encoding the audio feature comprises: encoding the audio feature by using a first encoder, the first encoder comprising an M-layer causal convolution-augmented Conformer model, M being a positive integer, the casual Conformer model integrating a time-masked attention model and a causal convolution model. ### Claim 6 A computer-implemented method for encoding and decoding, comprising: acquiring a feature corresponding to a to-be-processed object, the feature comprising at least a feature obtained according to historical output information of a decoder, wherein the output information comprises: hidden layer features outputted by the decoder, the acquiring a feature corresponding to a to-be-processed object comprises: acquiring historical feature abstraction information corresponding to the to-be-processed object, the historical feature abstraction information being information obtained by feature abstraction of historical processing objects; encoding the feature by using an encoder, to obtain an encoding result; and decoding the encoding result by using the decoder, to obtain a processing result corresponding to the to-be-processed object; performing feature abstraction on the to-be-processed object by using the hidden layer features acquired from the decoder, to obtain a feature vector with a predetermined length, and taking the feature vector as a historical feature vector corresponding to the to-be-processed object; wherein the acquiring historical feature abstraction information corresponding to the to-be-processed object comprises: splicing historical feature vectors corresponding to the historical processing objects respectively, and taking a splicing result as the historical feature abstraction information. ### Claim 7 (depends on 6) The method according to claim 6, wherein the to-be-processed object comprises: any one of continuous N feature fragments with unequal lengths obtained after an encoding feature corresponding to to-be-recognized speech is truncated, N being a positive integer greater than one, the encoding feature being a feature obtained after an audio feature of the to-be-recognized speech is encoded; and the processing result comprises: a recognition result corresponding to the feature fragment. ### Claim 8 An electronic device, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a speech recognition, wherein the speech recognition method comprises: acquiring an audio feature of to-be-recognized speech collected by a user equipment in real time; encoding the audio feature to obtain an encoding feature; truncating the encoding feature to obtain continuous N feature fragments with unequal lengths, N being a positive integer greater than one; acquiring, for any one of the feature segments, corresponding historical feature abstraction information, encoding the feature segment in combination with the historical feature abstraction information, and decoding an encoding result to obtain a recognition result corresponding to the feature segment, wherein the historical feature abstraction information is information obtained by feature abstraction of recognized historical feature fragments, wherein the encoding the feature segment in combination with the historical feature abstraction information comprises: encoding the feature segment in combination with the historical feature abstraction information by using a second encoder; and the decoding an encoding result comprises: decoding the encoding result by using a first decoder; wherein the historical feature abstraction information is generated according to output information of the decoder, wherein the second encoder comprises a P-layer convolution-augmented Conformer model, P being a positive integer; a number of the first decoder is Q, Q is a positive integer, and the Q first decoders are successively connected; and performing the following processing for any Conformer model: performing feature abstraction on a feature fragment currently processed in the Conformer model by using hidden layer features acquired from any first decoder, to obtain a feature vector with a predetermined length, and taking the feature vector as a historical feature vector of the currently processed feature fragment corresponding to the Conformer model, wherein the acquiring corresponding historical feature abstraction information comprises: for any Conformer model, splicing historical feature vectors of historical feature fragments corresponding to the Conformer model respectively, and taking a splicing result as the historical feature abstraction information. ### Claim 9 (depends on 8) The electronic device according to claim 8, further comprising: performing convolutional downsampling on the audio feature, and encoding the downsampled audio feature. ### Claim 10 (depends on 8) The electronic device according to claim 8, further comprising: determining peak information corresponding to the encoding feature in combination with the encoding of the audio feature and a CTC loss; and wherein the truncating the encoding feature comprises: truncating the encoding feature according to the peak information. ### Claim 11 (depends on 10) The electronic device according to claim 10, wherein the truncating the encoding feature according to the peak information comprises: taking an encoding feature located between each two adjacent peaks as a feature segment. ### Claim 12 (depends on 8) The electronic device according to claim 8, wherein the encoding the audio feature comprises: encoding the audio feature by using a first encoder, the first encoder comprising an M-layer causal convolution-augmented Conformer model, M being a positive integer, the casual Conformer model integrating a time-masked attention model and a causal convolution model. ### Claim 13 A non-transitory computer readable storage medium with computer instructions stored thereon, wherein the computer instructions are used for causing a speech recognition method, wherein the speech recognition method comprises: acquiring an audio feature of to-be-recognized speech; encoding the audio feature to obtain an encoding feature; truncating the encoding feature to obtain continuous N feature fragments with unequal lengths, N being a positive integer greater than one; acquiring, for any one of the feature segments, corresponding historical feature abstraction information, encoding the feature segment in combination with the historical feature abstraction information, and decoding an encoding result to obtain a recognition result corresponding to the feature segment, wherein the historical feature abstraction information is information obtained by feature abstraction of recognized historical feature fragments, wherein the encoding the feature segment in combination with the historical feature abstraction information comprises: encoding the feature segment in combination with the historical feature abstraction information by using a second encoder; and the decoding an encoding result comprises: decoding the encoding result by using a first decoder; wherein the historical feature abstraction information is generated according to output information of the decoder, wherein the second encoder comprises a P-layer convolution-augmented Conformer model, P being a positive integer; a number of the first decoder is Q, Q is a positive integer, and the Q first decoders are successively connected; and performing the following processing for any Conformer model: performing feature abstraction on a feature fragment currently processed in the Conformer model by using hidden layer features acquired from any first decoder, to obtain a feature vector with a predetermined length, and taking the feature vector as a historical feature vector of the currently processed feature fragment corresponding to the Conformer model, wherein the acquiring corresponding historical feature abstraction information comprises: for any Conformer model, splicing historical feature vectors of historical feature fragments corresponding to the Conformer model respectively, and taking a splicing result as the historical feature abstraction information.