Graph convolutional networks for video grounding
AssigneeInternational Business Machines Corporation
InventorsChuang Gan, Sijia Liu, Subhro Das, Dakuo Wang, Yang Zhang
Life of this patent
Estimating the term…
Abstract
Method and apparatus that includes receiving a query describing an aspect in a video, the video including a plurality of frames, identifying multiple proposals that potentially correspond to the query where each of the proposals includes a subset of the plurality of frames, ranking the proposals using a graph convolution network that identifies relationships between the proposals, and selecting, based on the ranking, one of the proposals as a video segment that correlates to the query.
Claims20
A method comprising:
- receiving a query describing an aspect in a video, the video comprising a plurality of frames;
- identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames;
- generating a graph based on the query and the multiple proposals;
- ranking the proposals using a graph convolution network (GCN) that identifies relationships between the proposals, wherein the graph is input into the graph convolution network;
- and
- selecting, based on the ranking, one of the proposals as a video segment that correlates to the query.
A system, comprising:
- a processor;
- and
- memory comprising a program, which when executed by the processor performs an operation, the operation comprising:
- receiving a query describing an aspect in a video, the video comprising a plurality of frames;
- identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames;
- generating a graph based on the query and the multiple proposals;
- ranking the proposals using a GCN that identifies relationships between the proposals, wherein the graph is input into the graph convolution network;
- and
- selecting, based on the ranking, one of the proposals as a video segment that correlates to the query.
A computer program product for identifying a video segment that correlates to a query, the computer program product comprising:
- a computer readable storage medium having computer readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation, the operation comprising:
- receiving the query, the query describing an aspect in a video comprising a plurality of frames;
- identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames;
- generating a graph based on the query and the multiple proposals;
- ranking the proposals using a GCN that identifies relationships between the proposals, wherein the graph is input into the graph convolution network;
- and
- selecting, based on the ranking, one of the proposals as the video segment that correlates to the query.
Dependencies are inferred from the claim text.