Weakly supervised natural language localization networks for video proposal prediction based on a text query

A method for weakly supervised natural language localization comprising:

• receiving, at one or more processors, an untrimmed video clip;
• receiving, at the one or more processors, a text query regarding an item of interest in the untrimmed video clip;
• generating, at the one or more processors, a plurality of video proposals from the untrimmed video clip, wherein each video proposal is a segment of the untrimmed video clip that is less than the full length of the video clip;
• transforming, for each of the plurality of video proposals, a visual feature of the respective video proposal to have a same dimension with a text feature of the text query;
• for each video proposal, generating, by applying a first softmax function over a last layer of one or more fully-connected layers of a first convolutional network, a corresponding consistency score that measures semantic consistency between the respective video proposal and the text query based on visual features encoded from the respective video proposal and text features encoded from the text query;
• generating, by one or more fully-connected layers of a second convolutional network and applying a second softmax function over the plurality of video proposals, a plurality of selection scores, each selection score based on a comparison of at least two of the video proposals;
• and
• generating, at the one or more processors, a prediction of which one of the plurality of video proposals is the best match for the text query based on the consistency scores and the selection scores, wherein the best match captures the item of interest and contains the least amount of extraneous content.

A non-transitory machine-readable medium comprising executable code which when executed by one or more processors associated with a computer are adapted to cause the one or more processors to perform a method comprising:

• receiving an untrimmed video clip;
• receiving a text query regarding an item of interest in the untrimmed video clip;
• generating a plurality of video proposals from the untrimmed video clip, wherein each video proposal is a segment of the untrimmed video clip that is less than the full length of the video clip;
• transforming, for each of the plurality of video proposals, a visual feature of the respective video proposal to have a same dimension with a text feature of the text query;
• for each video proposal, generating, by applying a first softmax function over a last layer of one or more fully-connected layers of a first convolutional network, a corresponding consistency score that measures semantic consistency between the respective video proposal and the text query based on visual features encoded from the respective video proposal and text features encoded from the text query;
• generating, by one or more fully-connected layers of a second convolutional network and applying a second softmax function over the plurality of video proposals, a plurality of selection scores, each selection score based on a comparison of at least two of the video proposals;
• and
• generating a prediction of which one of the plurality of video proposals is the best match for the text query based on the consistency scores and the selection scores, wherein the best match captures the item of interest and contains the least amount of extraneous content.