Method and apparatus for positioning description statement in image, electronic device, and storage medium

An apparatus for positioning a description statement in an image, comprising:

• a memory storing processor-executable instructions;
• and
• a processor arranged to execute the stored processor-executable instructions to perform operations of:
• performing analysis processing on a to-be-analyzed description statement and a to-be-analyzed image to obtain a plurality of statement attention weights of the to-be-analyzed description statement and a plurality of image attention weights of the to-be-analyzed image;
• obtaining a plurality of first matching scores based on the plurality of statement attention weights and a subject feature, a location feature and a relationship feature of the to-be-analyzed image, wherein the to-be-analyzed image comprises a plurality of objects, a subject object is an object with a highest attention weight in the plurality of objects, the subject feature is a feature of the subject object, the location feature is a location feature of the plurality of objects, and the relationship feature is a relationship feature between the plurality of objects;
• obtaining a second matching score between the to-be-analyzed description statement and the to-be-analyzed image based on the plurality of first matching scores and the plurality of image attention weights;
• and
• determining a positioning result of the to-be-analyzed description statement in the to-be-analyzed image based on the second matching score,
• wherein the processor is arranged to execute the stored processor-executable instructions to further perform an operation of:
• obtaining the plurality of statement attention weights of the to-be-analyzed description statement and the plurality of image attention weights of the to-be-analyzed image by using a neural network,
• wherein the processor is arranged to execute the stored processor-executable instructions to further perform an operation of:
• before the obtaining the plurality of statement attention weights of the to-be-analyzed description statement and the plurality of image attention weights of the to-be-analyzed image by using a neural network, training the neural network by using a sample set, wherein the sample set comprises a plurality of positive sample pairs and a plurality of negative sample pairs,
• wherein each positive sample pair comprises a first sample image and a first sample description statement of the first sample image, and
• wherein each negative sample pair comprises the first sample image and a second sample description statement obtained after a word is removed from the first sample description statement, or comprises the first sample description statement and a second sample image obtained after a region with a highest image attention weight is removed from the first sample image.

A non-transitory computer readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform operations of a computer-implemented method for positioning a description statement in an image, the method comprising:

• performing analysis processing on a to-be-analyzed description statement and a to-be-analyzed image to obtain a plurality of statement attention weights of the to-be-analyzed description statement and a plurality of image attention weights of the to-be-analyzed image;
• obtaining a plurality of first matching scores based on the plurality of statement attention weights and a subject feature, a location feature and a relationship feature of the to-be-analyzed image, wherein the to-be-analyzed image comprises a plurality of objects, a subject object is an object with a highest attention weight in the plurality of objects, the subject feature is a feature of the subject object, the location feature is a location feature of the plurality of objects, and the relationship feature is a relationship feature between the plurality of objects;
• obtaining a second matching score between the to-be-analyzed description statement and the to-be-analyzed image based on the plurality of first matching scores and the plurality of image attention weights;
• and
• determining a positioning result of the to-be-analyzed description statement in the to-be-analyzed image based on the second matching score,
• wherein the method further comprises:
• obtaining the plurality of statement attention weights of the to-be-analyzed description statement and the plurality of image attention weights of the to-be-analyzed image by using a neural network,
• wherein the method further comprises:
• before the obtaining the plurality of statement attention weights of the to-be-analyzed description statement and the plurality of image attention weights of the to-be-analyzed image by using a neural network, training the neural network by using a sample set, wherein the sample set comprises a plurality of positive sample pairs and a plurality of negative sample pairs,
• wherein each positive sample pair comprises a first sample image and a first sample description statement of the first sample image, and
• wherein each negative sample pair comprises the first sample image and a second sample description statement obtained after a word is removed from the first sample description statement, or comprises the first sample description statement and a second sample image obtained after a region with a highest image attention weight is removed from the first sample image.