Back to results
US 11,687,720 B2Granted
Named entity recognition visual context and caption data
AssigneeSnap Inc.
InventorsDi Lu, Leonardo Ribas Machado das Neves, Vitor Rocha de Carvalho, Ning Zhang
Life of this patent
Estimating the term…
Abstract
A caption of a multimodal message (e.g., social media post) can be identified as a named entity using an entity recognition system. The entity recognition system can use a visual attention based mechanism to generate a visual context representation from an image and caption. The system can use the visual context representation to identify one or more terms of the caption as a named entity.
Claims20
A method comprising:
- identifying, using one or more processors of a machine, a multimodal message that includes an image and a caption comprising words;
- generating, using an attention neural network, a visual context vector from the caption and the image, the visual context vector emphasizing portions of the caption based on objects depicted in the image;
- generating, using an entity recognition neural network, an indication that one or more words of the caption correspond to a named entity;
- integrating, using a modulation layer, the visual context vector into the entity recognition neural network for each word in the caption;
- and
- storing the one or more words as the named entity of the multimodal message.
3 more elements
A system comprising:
- one or more processors of a machine;
- and
- a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
- identifying a multimodal message that includes an image and a caption comprising words;
- generating, using an attention neural network, a visual context vector from the caption and the image, the visual context vector emphasizing portions of the caption based on objects depicted in the image;
- generating, using an entity recognition neural network, an indication that one or more of the words of the caption correspond to a named entity;
- integrating, using a modulation layer, the visual context vector into the entity recognition neural network for each word in the caption;
- and
- storing the one or more of the words as the named entity of the multimodal message.
6 more elements
A machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
- identifying a multimodal message that includes an image and a caption comprising words;
- generating, using an attention neural network, a visual context vector from the caption and the image, the visual context vector emphasizing portions of the caption based on objects depicted in the image;
- generating, using an entity recognition neural network, an indication that one or more of the words of the caption correspond to a named entity;
- integrating, using a modulation layer, the visual context vector into the entity recognition neural network for each word in the caption;
- and
- storing the one or more of the words as the named entity of the multimodal message.
3 more elements
Dependencies are inferred from the claim text.