Decoding method and apparatus in artificial neural network for speech recognition

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to:

• perform a first decoding task of decoding a feature including speech information and at least one token recognized up to a current point in time, using a multi-task trained shared decoding layer included in the artificial neural network, including adjusting parameters of the shared decoding layer from a second set of values to be a first set of values;
• perform a second decoding task of decoding the at least one token using the shared decoding layer, including adjusting the parameters of the shared decoding layer from the first set of values to be the second set of values;
• and
• determine an output token to be recognized subsequent to the at least one token based on a result of the first decoding task and a result of the second decoding task,
• wherein the first decoding task comprises a task of processing a relationship between a signal characteristic of the speech information and a language element,
• wherein the performing the second decoding task comprises a task of estimating a level of semantic correctness of the speech information.

A training method of an artificial neural network for speech recognition, the training method comprising:

• selecting at least one sample from a batch including a first training set including a pair of a speech and a text corresponding to the speech and a second training set including a text;
• in response to the at least one sample being selected from the first training set, training an encoding layer and a shared decoding layer that are included in the artificial neural network based on the sample;
• and
• in response to the at least one sample being selected from the second training set, training the shared decoding layer based on the sample,
• wherein, after the training the shared decoding layer, the shared decoding layer is configured to, during an inference phase, perform a first decoding task of decoding a feature including speech information and at least one token recognized up to a current point in time by adjusting parameters of the shared decoding layer from a second set of values to be a first set of values and perform a second decoding task of decoding the at least one token by adjusting the parameters of the shared decoding layer from the first set of values to the second set of values, and an output token to be recognized subsequent to the at least one token is determined based on a result of the first decoding task and a result of the second decoding task,
• wherein the first decoding task comprises a task of processing a relationship between a signal characteristic of the speech information and a language element, and
• wherein performing the second decoding task comprises a task of estimating a level of semantic correctness of the speech information.

A speech recognition apparatus comprising:

• an encoder configured to generate a feature from speech information;
• and
• a decoder configured to, using a multi-task trained shared decoding layer, perform a first decoding task of decoding the feature and at least one token recognized up to a current point in time by adjusting parameters of the shared decoding layer from a second set of values to be a first set of values, perform a second decoding task of decoding the at least one token recognized up to the current point by adjusting the parameters of the shared decoding layer from the first set of values to be the second set of values, and determine an output token to be recognized subsequent to the at least one token based on a result of the first decoding task and a result of the second decoding task,
• wherein the first decoding task comprises a task of processing a relationship between a signal characteristic of the speech information and a language element, and
• wherein the performing the second decoding task comprises a task of estimating a level of semantic correctness of the speech information.

A speech recognition apparatus comprising:

• an encoder configured to receive a speech feature vector corresponding to speech information and output a context vector;
• and
• a decoder configured to:
• receive the context vector;
• decode the context vector and a most recently recognized token using a multi-task trained shared decoding layer included in an artificial neural network to output a first result, including adjusting parameters of the shared decoding layer from a second set of values to be a first set of value;
• decode only the most recently recognized token using the shared decoding layer to output a second result, including adjusting the parameters of the shared decoding layer from the first set of values to be the second set of values;
• and
• output a current token based on the first result and the second result,
• wherein the first result related to a relationship between a signal characteristic of the speech information and a language element, and
• wherein the second result related to a level of semantic correctness of the speech information.