Back to results
US 2022/0092261 A1Published application
Machine learned language modeling and identification
AssigneeSnap Inc.
InventorsVitor Rocha de Carvalho, Luis Carlos Dos Santos Marujo, Leonardo Ribas Machado das Neves
Life of this patent
Estimating the term…
Abstract
Systems, devices, media, and methods are presented for generating a language detection model of a language analysis system. The systems and methods access a set of messages including text elements and convert the set of messages into a set of training messages. The set of training messages are configured for training a language detection model. The systems and methods train a classifier based on the set of training messages. The classifier has a set of features representing word frequency, character frequency, and a character ratio. The systems and methods generate a language detection model based on the classifier and the set of features.
Claims20
A method comprising:
- accessing a set of messages comprising text and graphical text elements, the graphical text elements conveying an emotion;
- identifying a first subset of messages from the set of messages, the first subset of messages comprising unsuitable messages;
- identifying a second subset of messages from the set of messages, the second subset of messages comprising text representing an entity;
- identifying a third subset of messages from the set of messages, the third subset of messages comprising superfluous characters;
- removing the first subset of messages from the set of messages;
- identifying text in the second subset of messages representing a named entity;
- removing the identified text from the second subset of messages;
- normalizing the superfluous characters within the third subset of messages;
- incorporating the second and third subset of messages into a final set of messages;
- converting the final set of messages into a set of training messages, the set of training messages configured for training a language detection machine-learning model;
- training a classifier based on the set of training messages, the classifier having a set of features comprising a character ratio, wherein the character ratio is based on a number of non-American Standard Code for Information Interchange (ASCII) characters included in a message;
- and
- generating a language detection model based on the trained classifier and the set of features.
10 more elements
A system comprising:
- one or more processors;
- and
- a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
- accessing a set of messages comprising text and graphical text elements, the graphical text elements conveying an emotion;
- identifying a first subset of messages from the set of messages, the first subset of messages comprising unsuitable messages;
- identifying a second subset of messages from the set of messages, the second subset of messages comprising text representing an entity;
- identifying a third subset of messages from the set of messages, the third subset of messages comprising superfluous characters;
- removing the first subset of messages from the set of messages;
- identifying text in the second subset of messages representing a named entity;
- removing the identified text from the second subset of messages;
- normalizing the superfluous characters within the third subset of messages;
- incorporating the second and third subset of messages into a final set of messages;
- converting the final set of messages into a set of training messages, the set of training messages configured for training a language detection machine-learning model;
- training a classifier based on the set of training messages, the classifier having a set of features comprising a character ratio, wherein the character ratio is based on a number of non-American Standard Code for Information Interchange (ASCII) characters included in a message;
- and
- generating a language detection model based on the trained classifier and the set of features.
13 more elements
A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:
- accessing a set of messages comprising text and graphical text elements, the graphical text elements conveying an emotion;
- identifying a first subset of messages from the set of messages, the first subset of messages comprising unsuitable messages;
- identifying a second subset of messages from the set of messages, the second subset of messages comprising text representing an entity;
- identifying a third subset of messages from the set of messages, the third subset of messages comprising superfluous characters;
- removing the first subset of messages from the set of messages;
- identifying text in the second subset of messages representing a named entity;
- removing the identified text from the second subset of messages;
- normalizing the superfluous characters within the third subset of messages;
- incorporating the second and third subset of messages into a final set of messages;
- converting the final set of messages into a set of training messages, the set of training messages configured for training a language detection machine-learning model;
- training a classifier based on the set of training messages, the classifier having a set of features comprising a character ratio, wherein the character ratio is based on a number of non-American Standard Code for Information Interchange (ASCII) characters included in a message;
- and
- generating a language detection model based on the trained classifier and the set of features.
10 more elements
Dependencies are inferred from the claim text.