# US 2018/0025729 A1 · Audio-Visual Speech Recognition with Scattering Operators > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0025729 A1 - Document ID: US20180025729A1 - Status: Published application - Kind: A1 - Application number: 15/639,149 - Filed: 2017-06-30 - Published: 2018-01-25 - Earliest claimed: 2015-08-25 - Assignees: Nuance Communications, Inc. - Applicants: Nuance Communications, Inc. - Inventors: Etienne Marcheret; Josef Vopicka; Vaibhava Goel - Web page: https://patentgrep.com/patent/US20180025729A1 ## CPC - G10L 15/25: using position of the lips, movement of the lips or face analysis - G06T 7/60: Analysis of geometric attributes - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G10L 15/16: using artificial neural networks - G10L 21/02: Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Aspects described herein are directed towards methods, computing devices, systems, and computer-readable media that apply scattering operations to extracted visual features of audiovisual input to generate predictions regarding the speech status of a subject. Visual scattering coefficients generated according to one or more aspects described herein may be used as input to a neural network operative to generate the predictions regarding the speech status of the subject. Predictions generated based on the visual features may be combined with predictions based on audio input associated with the visual features. In some embodiments, the extracted visual features may be combined with the audio input to generate a combined feature vector for use in generating predictions. ## Claims ### Claim 1 A method comprising: receiving, by a computing device, audiovisual input comprising audio input and video input associated with a subject; extracting, by the computing device, visual features from the video input; applying, by the computing device, a scattering operation to the extracted visual features to generate a vector of scattering coefficients; providing the vector of scattering coefficients as input to a first neural network for visual processing; providing the audio input to a second neural network for audio processing; combining, by the computing device, a first output of the first neural network with a second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input; providing the fused audiovisual feature vector to a third neural network for audiovisual processing; and generating, by the computing device and using the third neural network, a first prediction regarding a speech status of the subject based on the fused audiovisual feature vector. ### Claim 2 (depends on 1) The method of claim 1, wherein providing the vector of scattering coefficients as input to the first neural network for visual processing comprises: normalizing the vector of scattering coefficients to generate a first normalized vector of scattering coefficients; aggregating a plurality of normalized vectors of scattering coefficients including the first normalized vector of scattering coefficients to generate a set of aggregated visual feature vectors; and providing the set of aggregated visual feature vectors to the first neural network for visual input processing. ### Claim 3 (depends on 1) The method of claim 1, wherein the video input is sampled at a first frequency and the audio input is sampled at a different second frequency. ### Claim 4 (depends on 1) The method of claim 1, wherein the vector of scattering coefficients is in a first dimensional space, and wherein applying the scattering operation to the extracted visual features to generate the vector of scattering coefficients comprises: applying the scattering operation to the extracted visual features to generate a second vector of scattering coefficients in a second dimensional space; and projecting the second vector of scattering coefficients into the first dimensional space to generate the vector of scattering coefficients in the first dimensional space, wherein the second dimensional space is of a higher dimensionality than the first dimensional space. ### Claim 5 (depends on 1) The method of claim 1, wherein applying the scattering operation to the extracted visual features to generate a vector of scattering coefficients comprises generating first order scattering coefficients and second order scattering coefficients. ### Claim 6 (depends on 1) The method of claim 1, further comprising: generating, by the computing device and using the first neural network, a second prediction regarding the speech status of the subject based on the vector of scattering coefficients; and generating, by the computing device and using the second neural network, a third prediction regarding the speech status of the subject based on the audio input; and combining the first prediction, the second prediction, and the third prediction to generate a combined prediction regarding the speech status of the subject. ### Claim 7 (depends on 1) The method of claim 1, wherein the first prediction regarding the speech status of the subject is used to recognize speech content of the audiovisual input. ### Claim 8 (depends on 1) The method of claim 1, further comprising: determining, based on the first prediction regarding the speech status of the subject, a synchrony state of the audio input and the video input, wherein the synchrony state indicates whether the audio input is in-sync with the video input. ### Claim 9 (depends on 1) The method of claim 1, further comprising: determining, based on the first prediction regarding the speech status of the subject, whether speech in the audio input originates from a foreground source or a background source. ### Claim 10 One or more non-transitory computer readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising: receiving audiovisual input comprising audio input and video input associated with a subject; extracting visual features from the video input; applying a scattering operation to the extracted visual features to generate a vector of scattering coefficients in a first dimensional space; providing the vector of scattering coefficients as input to a first neural network for visual processing; providing the audio input to a second neural network for audio processing; combining a first output of the first neural network with a second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input; providing the fused audiovisual feature vector to a third neural network for audiovisual processing; and generating, using the third neural network, a prediction regarding a speech status of the subject based on the fused audiovisual feature vector. ### Claim 11 (depends on 10) The one or more non-transitory computer readable media of claim 10, wherein providing the vector of scattering coefficients as input to the first neural network for visual processing comprises: normalizing the vector of scattering coefficients to generate a first normalized vector of scattering coefficients; aggregating a plurality of normalized vectors of scattering coefficients including the first normalized vector of scattering coefficients to generate a set of aggregated visual feature vectors; and providing the set of aggregated visual feature vectors to the first neural network for visual input processing. ### Claim 12 (depends on 10) The one or more non-transitory computer readable media of claim 10, wherein the video input is sampled at a first frequency and the audio input is sampled at a different second frequency. ### Claim 13 (depends on 10) The one or more non-transitory computer readable media of claim 10, wherein applying the scattering operation to the extracted visual features to generate the vector of scattering coefficients comprises: applying the scattering operation to the extracted visual features to generate a second vector of scattering coefficients in a second dimensional space; and projecting the second vector of scattering coefficients into the first dimensional space to generate the vector of scattering coefficients in the first dimensional space, wherein the second dimensional space is of a higher dimensionality than the first dimensional space. ### Claim 14 (depends on 10) The one or more non-transitory computer readable media of claim 10, wherein the prediction regarding the speech status of the subject is used to recognize speech content of the audiovisual input. ### Claim 15 (depends on 10) The one or more non-transitory computer readable media of claim 10, wherein the prediction regarding the speech status of the subject is used to determine a synchrony state of the audio input and the video input, wherein the synchrony state indicates whether the audio input is in-sync with the video input. ### Claim 16 (depends on 10) The one or more non-transitory computer readable media of claim 10, wherein the prediction regarding the speech status of the subject is used to determine whether speech in the audio input originates from a foreground source or a background source. ### Claim 17 An apparatus comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the apparatus to: receive audiovisual input comprising audio input and video input associated with a subject; extract visual features from the video input; apply a scattering operation to the extracted visual features to generate a vector of scattering coefficients; provide the vector of scattering coefficients as input to a first neural network for visual processing; provide the audio input to a second neural network for audio processing; combine a first output of the first neural network with a second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input; provide the fused audiovisual feature vector to a third neural network for audiovisual processing; and generate, and using the third neural network, a first prediction regarding a speech status of the subject based on the fused audiovisual feature vector. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein the video input is sampled at a first frequency and the audio input is sampled at a different second frequency. ### Claim 19 (depends on 17) The apparatus of claim 17, wherein the vector of scattering coefficients is in a first dimensional space, and wherein the instructions, when executed by the one or more processors, further cause the apparatus to: generate, using the first neural network, a second prediction regarding the speech status of the subject based on the vector of scattering coefficients; and generate, using the second neural network, a third prediction regarding the speech status of the subject based on the audio input; and combine the first prediction, the second prediction, and the third prediction to generate a combined prediction regarding the speech status of the subject. ### Claim 20 (depends on 17) The apparatus of claim 17, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: determine, based on the first prediction regarding the speech status of the subject, whether speech in the audio input originates from a foreground source or a background source.