Efficient automatic punctuation with robust inference

A non-transitory computer readable medium having instructions stored thereon that, when executed by a computing system for automatically punctuating text, cause the computing system to perform operations comprising:

• (a) applying, by one or more computing devices, a text string to a first component of a non-recurrent neural network trained to generate one or more contextualized vectors representing a contextualized meaning for each word in the text string, and wherein the first component determines the contextualized vectors by processing each word in the text string in parallel with one another;
• (b) generating, by the first component, the one or more contextualized vectors;
• (c) applying, by the one or more computing devices, an output of the contextualized vectors by the first component to a second component of the non-recurrent neural network trained to generate a set of probability values for each word in the text string, wherein the set of probability values indicates a likelihood that a punctuation mark exists for each word in the text string, and wherein the second component determines the set of probability values by processing the contextualized vectors in parallel with one another;
• (d) generating, by a first linear layer of the second component, a first set of vectors representing a compressed representation for each of the contextualized vectors output by the first component, wherein the first set of vectors is generated based on multiplying each of the contextualized vectors output by the first component with a first set of tunable parameters trained to predict the punctuation mark;
• (e) concatenating, after the first linear layer, the first set of vectors into a further vector;
• (f) generating, by the second component, the set of probability values;
• (g) transmitting, by the one or more computing devices, the set of probability values to a text generation engine to generate a formatted text string based on the set of probability values, wherein the formatted text string includes the punctuation mark for each word of the text string, wherein the applying the text string to the first component, applying the contextualized vectors output, and the transmitting occur in real-time;
• (h) generating, by the text generation engine, the formatted text string.

A computing system for automatically punctuating text comprising:

• a control unit configured to:
• (a) apply a text string to a first component of a non-recurrent neural network trained to generate one or more contextualized vectors representing a contextualized meaning for each word in the text string, and wherein the first component determines the contextualized vectors by processing each word in the text string in parallel with one another;
• (b) generate, by the first component, the one or more contextualized vectors;
• (c) apply an output of the contextualized vectors by the first component to a second component of the non-recurrent neural network trained to generate a set of probability values for each word in the text string, wherein the set of probability values indicates a likelihood that a punctuation mark exists for each word in the text string, and wherein the second component determines the set of probability values by processing the contextualized vectors in parallel with one another;
• (d) generate a first set of vectors representing a compressed representation for each of the contextualized vectors output by the first component, wherein the first set of vectors is generated based on multiplying each of the contextualized vectors output by the first component with a first set of tunable parameters trained to predict the punctuation mark;
• (e) concatenate the first set of vectors into a further vector;
• (f) generate, by the second component, the set of probability values;
• and
• a communication unit, coupled to the control unit, configured to:
• (g) transmit the set of probability values to a text generation engine to generate a formatted text string based on the set of probability values, wherein the formatted text string includes the punctuation mark for each word of the text string, wherein the applying the text string to the first component, applying the contextualized vectors output, and the transmitting occur in real-time;
• and
• (h) generate, by the text generation engine, the formatted text string.