Wide and deep network for language detection using hash embeddings

A system comprising:

• one or more data processors;
• and
• one or more non-transitory computer readable media storing instructions which, when executed by the one or more data processors cause the one or more data processors to perform processing comprising:
• receiving a textual unit;
• generating a language prediction for the textual unit, the language prediction identifying a language associated with the textual unit, wherein generating the language prediction comprises:
• parsing the textual unit to obtain a sequence of n-grams of the textual unit;
• using an embedding layer to obtain an ordered plurality of embedding vectors for the sequence of n-grams, wherein the embedding layer comprises a trained model having a plurality of component vectors, and wherein the embedding layer is configured to obtain the ordered plurality of embedding vectors by, for each n-gram in the sequence of n-grams:
• obtaining a first hash value for the n-gram and a second hash value for the n-gram,
• selecting, based on the first hash value, a first component vector from among the plurality of component vectors,
• selecting, based on the second hash value, a second component vector from among the plurality of component vectors, and
• obtaining an embedding vector for the n-gram that is based on the first component vector and the second component vector;
• using a deep network to obtain an encoded vector that is based on the ordered plurality of embedding vectors, wherein the deep network comprises a convolutional neural network (CNN) layer and an attention layer, and wherein the attention layer is configured to assign weights to an output of the CNN layer;
• and
• using a classifier to obtain the language prediction for the textual unit based on the encoded vector;
• and
• generating a response for the textual unit in the language associated with the textual unit.

A computer-program product tangibly embodied in one or more non-transitory machine-readable media, including instructions configured to cause one or more data processors to perform processing comprising:

• receiving, by a digital assistant system, a textual unit;
• generating, by a language detection apparatus, a language prediction for the textual unit, the language prediction identifying a language associated with the textual unit, wherein the language detection apparatus comprises:
• a parser configured to parse the textual unit to obtain a sequence of n-grams of the textual unit;
• an embedding layer configured to obtain an ordered plurality of embedding vectors for the sequence of n-grams, wherein the embedding layer comprises a trained model having a plurality of component vectors, and wherein the embedding layer is configured to obtain the ordered plurality of embedding vectors by, for each n-gram in the sequence of n-grams:
• obtaining a first hash value for the n-gram and a second hash value for the n-gram,
• selecting, based on the first hash value, a first component vector from among the plurality of component vectors,
• selecting, based on the second hash value, a second component vector from among the plurality of component vectors, and
• obtaining an embedding vector for the n-gram that is based on the first component vector and the second component vector;
• a deep network configured to obtain an encoded vector that is based on the ordered plurality of embedding vectors, wherein the deep network comprises a convolutional neural network (CNN) layer and an attention layer, and wherein the attention layer is configured to assign weights to an output of the CNN layer;
• and
• a classifier configured to obtain the language prediction for the textual unit based on the encoded vector;
• and
• generating, by the digital assistant system, a response for the textual unit in the language associated with the textual unit.