Emoji sanitization for natural language model processing

A method for emoji sanitization for natural language model processing, comprising:

• obtaining, by a device, a natural language input including an emoji;
• identifying, by the device, one or more appearance modifiers associated with the emoji;
• generating, by the device, a token associated with the emoji that removes the one or more appearance modifiers,
• wherein the token is associated with multiple emojis including the emoji, and
• wherein the token is a modified code associated with the emoji or is associated with a cluster that is associated with the multiple emojis;
• providing, by the device and to a natural language processing (NLP) model, the token associated with the emoji;
• and
• obtaining, by the device and from the NLP model, an output that indicates an interpretation of the natural language input based on providing the token to the NLP model.

A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:

• one or more instructions that, when executed by one or more processors of a device, cause the device to:
• obtain a natural language input including an emoji,
• wherein the emoji is associated with an appearance modifier;
• generate a model input associated with the emoji based on an identifier associated with the emoji,
• wherein the model input is associated with multiple emojis including the emoji,
• wherein the model input does not indicate the appearance modifier, and
• wherein the multiple emojis are associated with a semantic meaning cluster;
• provide, to a natural language processing (NLP) model, the model input associated with the emoji;
• and
• obtain, from the NLP model, an output that indicates a semantic meaning of the natural language input based on providing the model input to the NLP model.