# US 2021/0082399 A1 · Aligning spike timing of models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0082399 A1
- Document ID: US20210082399A1
- Status: Published application
- Kind: A1
- Application number: 16/570,022
- Filed: 2019-09-13
- Published: 2021-03-18
- Earliest claimed: 2019-09-13
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Gakuto Kurata; Kartik Audhkhasi
- Web page: https://patentgrep.com/patent/US20210082399A1

## CPC

- G10L 15/063: {Training}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G10L 15/16: using artificial neural networks
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/34: Adaptation of a single recogniser for parallel processing, e.g. by use of multiple processors or cloud computing
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A technique for aligning spike timing of models is disclosed. A first model having a first architecture trained with a set of training samples is generated. Each training sample includes an input sequence of observations and an output sequence of symbols having different length from the input sequence. Then, one or more second models are trained with the trained first model by minimizing a guide loss jointly with a normal loss for each second model and a sequence recognition task is performed using the one or more second models. The guide loss evaluates dissimilarity in spike timing between the trained first model and each second model being trained.

## Claims

### Claim 1

A computer-implemented method for aligning spike timing of models, comprising:
generating a first model having a first architecture trained with a set of training samples each including an input sequence of observations and an output sequence of symbols having different length from the input sequence;
training one or more second models with the trained first model by minimizing a guide loss jointly with a normal loss for each second model, the guide loss evaluating dissimilarity in spike timing between the trained first model and each second model being trained; and
performing a sequence recognition task using the one or more second models.

### Claim 2 (depends on 1)

The method of claim 1, wherein the training of the one or more second models comprises:
preparing a mask using posterior distributions obtained from the trained first model by feeding an input sample thereto; and
applying the mask to outputs obtained for the input sample from each second model being trained to obtain masked posterior distributions, the masked posterior distributions at least partially determining the guide loss.

### Claim 3 (depends on 2)

The method of claim 2, wherein the mask is configured to pass at least an output value corresponding to a spike emitted from the trained first model at each time index, the spike representing an output symbol other than at least a blank symbol.

### Claim 4

The method of 2, wherein the mask has a value representing whether to pass or not pass an output value for each output symbol and each time index in a manner depending on a corresponding posterior distribution for each time index in the posterior distributions.

### Claim 5 (depends on 2)

The method of claim 2, wherein the mask has a factor representing degree of passing an output value for each output symbol and each time index in a manner depending on a corresponding probability in the posterior distributions.

### Claim 6 (depends on 1)

The method of claim 1, wherein at least one of the one or more second models trained with the first model is used for posterior fusion.

### Claim 7 (depends on 1)

The method of claim 1, wherein at least one of the one or more second models trained with the first model is used as a teacher model for knowledge distillation to train a student model, the student model having an architecture matched to the first architecture of the first model.

### Claim 8 (depends on 7)

The method of claim 7, wherein each second model has a second architecture that is different from the first architecture.

### Claim 9 (depends on 8)

The method of claim 8, wherein the first architecture is a unidirectional and the second architecture is a bidirectional.

### Claim 10 (depends on 1)

The method of claim 1, wherein at least one of the one or more second models trained with the first model is used for ROVER (Recognizer Output Voting Error Reduction).

### Claim 11 (depends on 1)

The method of claim 1, wherein each second model constitutes an end-to-end speech recognition model, each observation in the input sequence of the training sample represents an acoustic feature, and each symbol in the output sequence of the training sample represents at least one of a phone, a context dependent phone, a character, a word-piece, and a word.

### Claim 12 (depends on 1)

The method of claim 1, wherein the first model and the second model are alignment-free models.

### Claim 13 (depends on 1)

The method of claim 1, wherein the normal loss is CTC (Connectionist Temporal Classification) loss, the first model and the second model are CTC models and the guide loss evaluates the dissimilarity in the spike timing while ignoring at least blank symbols.

### Claim 14

A computer system for aligning spike timing of models, by executing program instructions, the computer system comprising:
a memory storing the program instructions;
a processing circuitry in communications with the memory for executing the program instructions, wherein the processing circuitry is configured to:
generate a first model having a first architecture trained with a set of training samples, wherein each training sample includes an input sequence of observations and an output sequence of symbols having different length from the input sequence;
train one or more second models with the trained first model by minimizing a guide loss jointly with a normal loss for each second model, wherein the guide loss evaluates dissimilarity in spike timing between the trained first model and each second model being trained; and
perform a sequence recognition task using the one or more second models.

### Claim 15 (depends on 14)

The computer system of claim 14, wherein the processing circuitry is configured to:
prepare a mask using posterior distributions obtained from the trained first model by feeding an input sample thereto; and
apply the mask to outputs obtained for the input sample from each second model being trained to obtain masked posterior distributions, wherein the masked posterior distributions at least partially determine the guide loss.

### Claim 16 (depends on 15)

The computer system of claim 15, wherein the mask is configured to pass at least an output value corresponding to a spike emitted from the trained first model at each time index, the spike representing an output symbol other than a blank symbol.

### Claim 17 (depends on 14)

The computer system of claim 14, wherein at least one of the one or more second models trained with the first model is used for at least one of a model for posterior fusion and a teacher model for knowledge distillation to train a student model, wherein the student model has an architecture matched to the first architecture of the first model.

### Claim 18

A computer program product for aligning spike timing of models, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
generating a first model having a first architecture trained with a set of training samples each including an input sequence of observations and an output sequence of symbols having different length from the input sequence;
training one or more second models with the trained first model by minimizing a guide loss jointly with a normal loss for each second model, the guide loss evaluating dissimilarity in spike timing between the trained first model and each second model being trained; and
performing a sequence recognition task using the one or more second models.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein the training of the one or more second models comprises:
preparing a mask using posterior distributions obtained from the trained first model by feeding an input sample thereto; and
applying the mask to outputs obtained for the input sample from each second model being trained to obtain masked posterior distributions, the masked posterior distributions at least partially determining the guide loss.

### Claim 20 (depends on 18)

The computer program product of claim 18, wherein at least one of the one or more second models trained with the first model is used a model for posterior fusion and a teacher model for knowledge distillation to train a student model, the student model having an architecture matched to the first architecture of the first model.
