# US 2017/0256254 A1 · Modular deep learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0256254 A1
- Document ID: US20170256254A1
- Status: Published application
- Kind: A1
- Application number: 15/199,346
- Filed: 2016-06-30
- Published: 2017-09-07
- Earliest claimed: 2016-03-04
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Yan HUANG; Chaojun LIU; Kshitiz KUMAR; Kaustubh Prakash KALGAONKAR; Yifan GONG
- Web page: https://patentgrep.com/patent/US20170256254A1

## CPC

- G10L 15/16: using artificial neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/063: {Training}
- G10L 15/065: Adaptation
- G10L 15/183: using context dependencies, e.g. language models
- G10L 15/28: Constructional details of speech recognition systems
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The technology described herein uses a modular model to process speech. A deep learning based acoustic model comprises a stack of different types of neural network layers. The sub-modules of a deep learning based acoustic model can be used to represent distinct non-phonetic acoustic factors, such as accent origins (e.g. native, non-native), speech channels (e.g. mobile, bluetooth, desktop etc.), speech application scenario (e.g. voice search, short message dictation etc.), and speaker variation (e.g. individual speakers or clustered speakers), etc. The technology described herein uses certain sub-modules in a first context and a second group of sub-modules in a second context.

## Claims

### Claim 1

An automatic speech recognition (ASR) system comprising:
a computer-storage media storing an acoustic model configured to identify acoustic units within an acoustic information; and
the acoustic model (AM) comprising a modular deep learning model comprising multiple hidden layers, the multiple hidden layers including at least one modular layer and a shared layer, the at least one modular layer comprising multiple context-specific sub-modules, wherein only a first context-specific sub-module in the modular layer processes the acoustic information in a first context.

### Claim 2 (depends on 1)

The system of claim 1, comprising a control module that determines the first context of the acoustic information and activates the first context-specific sub-module.

### Claim 3 (depends on 1)

The system of claim 1, wherein the modular deep learning model comprises a gate that opens to allow output from a previous layer to be input to the first context-specific sub-module.

### Claim 4 (depends on 1)

The system of claim 1, wherein the first context is the acoustic information being recorded by a Bluetooth microphone and the first context-specific sub-module is trained using acoustic data captured on a Bluetooth microphone.

### Claim 5 (depends on 1)

The system of claim 1, wherein the first context is for the acoustic information having an sCluster value within a first range and the first context-specific sub-module is trained using acoustic data having sCluster values within the first range.

### Claim 6 (depends on 1)

The system of claim 1, wherein the first context is for the acoustic information having an iCluster value within a first range and the first context-specific sub-module is trained using acoustic data having iCluster values within the first range.

### Claim 7 (depends on 1)

The system of claim 1, wherein the modular deep learning model comprises multiple context-specific layers trained to different context classes.

### Claim 8

A method of labeling sounds using a modular deep learning model, the method comprising:
receiving acoustic information comprising speech from a user;
determining a context for the acoustic information;
converting the acoustic information into acoustic features;
activating a sub-module associated with the context to perform calculations when the modular deep learning model is processing the acoustic features;
inputting the acoustic features into the modular deep learning model; and
generating label data for the acoustic information by processing the acoustic features using the modular deep learning model.

### Claim 9 (depends on 8)

The method of claim 8, wherein the context is the acoustic information capturing a male voice.

### Claim 10 (depends on 8)

The method of claim 8, wherein the context is a sCluster value calculated for the acoustic information.

### Claim 11 (depends on 8)

The method of claim 8, wherein the context is an iCluster value calculated for the acoustic information.

### Claim 12 (depends on 8)

The method of claim 8, wherein the context is derived from either an external signal or the acoustic information.

### Claim 13 (depends on 8)

The method of claim 8, wherein the method further comprises determining an additional context for the acoustic information and activating an additional sub-module associated with the additional context to perform calculations when the modular deep learning model is processing the acoustic information.

### Claim 14 (depends on 8)

The method of claim 8, wherein the sub-module forms an entire hidden layer within the modular deep learning model.

### Claim 15 (depends on 8)

The method of claim 8, wherein the modular deep learning model comprises a shared layer that performs calculations in all implementations of the modular deep learning model regardless of a context.

### Claim 16

A computer-storage media comprising computer-executable instructions embodied thereon that when executed by a computing device cause the computer device to perform a method of labeling sounds using a modular deep learning model, the method comprising:
receiving acoustic information comprising speech from a user;
converting the acoustic information into acoustic features;
determining, for a first context class, a first context for the acoustic information;
selecting a first sub-module from a plurality of sub-modules associated with the first context class;
activating the first sub-module associated with the first context to perform calculations when the modular deep learning model is processing the acoustic features, wherein the non-selected sub-modules associated with the first context class do not perform calculations within the modular deep learning model when the first sub-module is activated;
inputting the acoustic features into the modular deep learning model; and
generating label data for the acoustic information by processing the acoustic features using the modular deep learning model.

### Claim 17 (depends on 16)

The media of claim 16, wherein the modular deep learning model comprises a shared layer that performs calculations in all implementations of the modular deep learning model regardless of a context.

### Claim 18 (depends on 8)

The method of claim 8, wherein the sub-module forms an entire hidden layer within the modular deep learning model.

### Claim 19 (depends on 8)

The method of claim 8, wherein the sub-module forms multiple hidden layers within the modular deep learning model.

### Claim 20 (depends on 8)

The method of claim 8, wherein the first context is a sCluster value calculated for the acoustic information.
