Systems and methods for word segmentation based on a competing neural character language model

A computer-implemented method for word segmentation, the method comprising:

• receiving a plurality of characters for segmentation;
• converting, using an embedding model, a character of the plurality of characters into an embedding vector;
• feeding the embedding vector into a forward language model to retrieve a first output vector;
• feeding the embedding vector into a reverse language model to retrieve a second output vector;
• comparing the embedding vector to the first output vector and the second output vector;
• and
• segmenting the plurality of characters based on the comparison,
• wherein comparing the embedding vector to the first output vector and the second output vector comprises determining a reverse exponent of a Euclidean distance between the embedding vector and each of the first output vector and the second output vector.

A computer-implemented system for word segmentation, the system comprising:

• a memory storing instructions;
• and
• at least one processor configured to execute the instructions to:
• receive a plurality of characters for segmentation;
• convert, using an embedding model, a character of the plurality of characters into an embedding vector, wherein the embedding model comprises a projection layer and a softmax classifier layer;
• feed the embedding vector into a forward language model to retrieve a first output vector;
• feed the embedding vector into a reverse language model to retrieve a second output vector;
• compare the embedding vector to the first output vector and the second output vector;
• segment the plurality of characters based on the comparison;
• and
• repeat the steps of converting, feeding, comparing, and segmenting for each of the plurality of characters for segmentation until each character belongs to a word,
• wherein comparing the embedding vector to the first output vector and the second output vector comprises determining a reverse exponent of a Euclidean distance between the embedding vector and each of the first output vector and the second output vector.