Multimodal sentiment classification
AssigneeSnap Inc.
InventorsJianfei Yu, Luis Carlos Dos Santos Marujo, Venkata Satya Pradeep Karuturi, Leonardo Ribas Machado das Neves, Ning Xu, William Brendel
Life of this patent
Estimating the term…
Abstract
Sentiment classification can be implemented by an entity-level multimodal sentiment classification neural network. The neural network can include left, right, and target entity subnetworks. The neural network can further include an image network that generates representation data that is combined and weighted with data output by the left, right, and target entity subnetworks to output a sentiment classification for an entity included in a network post.
Claims20
A method comprising:
- identifying a multimodal message comprising an image and a plurality of terms, the plurality of terms comprising an entity term and non-entity terms;
- generating, using a convolutional neural network (CNN), an image representation from the image in the multimodal message;
- generating, using a first recurrent neural network, a non-entity terms representation from non-entity terms located to the right or left of the entity term;
- generating, using a second recurrent neural network, a target entity representation from the entity term;
- and
- generating a sentiment classification by combining the image representation, the non-entity terms representation, and the target entity representation.
A system comprising:
- one or more processors of a machine;
- and
- a memory storing instructions that, when executed by at least one processor among the one or more processors, causes the machine to perform operations comprising:
- identifying a multimodal message comprising an image and a plurality of terms, the plurality of terms comprising an entity term and non-entity terms;
- generating, using a convolutional neural network (CNN), an image representation from the image in the multimodal message;
- generating, using a first recurrent neural network, a non-entity terms representation from non-entity terms located to the right or left of the entity term;
- generating, using a second recurrent neural network, a target entity representation from the entity term;
- and
- generating a sentiment classification by combining the image representation, the non-entity terms representation, and the target entity representation.
A non-transitory machine-readable storage device embodying instructions that, when executed by a device, cause the device to perform operations comprising:
- identifying a multimodal message comprising an image and a plurality of terms, the plurality of terms comprising an entity term and non-entity terms;
- generating, using a convolutional neural network (CNN), an image representation from the image in the multimodal message;
- generating, using a first recurrent neural network, a non-entity terms representation from non-entity terms located to the right or left of the entity term;
- generating, using a second recurrent neural network, a target entity representation from the entity term;
- and
- generating a sentiment classification by combining the image representation, the non-entity terms representation, and the target entity representation.
Dependencies are inferred from the claim text.