Spatio-temporal action and actor localization
AssigneeQUALCOMM Incorporated
InventorsKirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, Cornelis Gerardus Maria Snoek
Life of this patent
Estimating the term…
Abstract
A method of pixel-wise localization of an actor and an action in a sequence of frames includes receiving a natural language query describing the action and the actor. The method also includes receiving the sequence of frames. The method further includes localizing the action and the actor in the sequence of frames based on the natural language query.
Claims20
A method of pixel-wise localization of an actor and an action in a sequence of frames, comprising:
- generating a first set of filters based on a natural language query describing the action and the actor;
- generating a visual representation for each frame of the sequence of frames;
- generating a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;
- generating a second set of dynamic filters based on the natural language query and the response map;
- labeling pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame;
- and
- localizing the action and the actor in the sequence of frames based on the labeled pixels.
An apparatus for pixel-wise localization of an actor and an action in a sequence of frames, the apparatus comprising:
- means for generating a first set of filters based on a natural language query describing the action and the actor;
- means for generating a visual representation for each frame of the sequence of frames;
- means for generating a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;
- means for generating a second set of dynamic filters based on the natural language query and the response map;
- means for labeling pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame;
- and
- means for localizing the action and the actor in the sequence of frames based on the labeled pixels.
An apparatus for pixel-wise localization of an actor and an action in a sequence of frames, the apparatus comprising:
- a memory;
- and
- at least one processor coupled to the memory, the at least one processor configured:
- to generate a first set of filters based on a natural language query describing the action and the actor;
- to generate a visual representation for each frame of the sequence of frames;
- to generate a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;
- to generate a second set of dynamic filters based on the natural language query and the response map;
- to label pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame;
- and
- to localize the action and the actor in the sequence of frames based on the labeled pixels.
A non-transitory computer-readable medium having program code recorded thereon for pixel-wise localization of an actor and an action in a sequence of frames, the program code executed by a processor and comprising:
- program code to generate a first set of filters based on a natural language query describing the action and the actor;
- program code to generate a visual representation for each frame of the sequence of frames;
- program code to generate a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;
- program code to generate a second set of dynamic filters based on the natural language query and the response map;
- program code to label pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame;
- and
- program code to localize the action and the actor in the sequence of frames based on the labeled pixels.
Dependencies are inferred from the claim text.