# US 11,442,986 B2 · Graph convolutional networks for video grounding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,986 B2 - Document ID: US11442986B2 - Status: Granted - Kind: B2 - Application number: 16/792,208 - Filed: 2020-02-15 - Published: 2021-08-19 - Granted: 2022-09-13 - Earliest claimed: 2020-02-15 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Chuang Gan; Sijia Liu; Subhro Das; Dakuo Wang; Yang Zhang - Web page: https://patentgrep.com/patent/US11442986B2 ## CPC - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06F 16/75: Clustering; Classification - G06F 16/7837: {using objects detected or recognised in the video content} - G06F 18/2178: {based on feedback of a supervisor} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs - G06N 3/09: Supervised learning - G06V 10/764: using classification, e.g. of video objects - G06V 10/7784: {based on feedback from supervisors} - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Method and apparatus that includes receiving a query describing an aspect in a video, the video including a plurality of frames, identifying multiple proposals that potentially correspond to the query where each of the proposals includes a subset of the plurality of frames, ranking the proposals using a graph convolution network that identifies relationships between the proposals, and selecting, based on the ranking, one of the proposals as a video segment that correlates to the query. ## Claims ### Claim 1 A method comprising: receiving a query describing an aspect in a video, the video comprising a plurality of frames; identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames; generating a graph based on the query and the multiple proposals; ranking the proposals using a graph convolution network (GCN) that identifies relationships between the proposals, wherein the graph is input into the graph convolution network; and selecting, based on the ranking, one of the proposals as a video segment that correlates to the query. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the graph comprises: identifying visual features in the proposals using a visual feature encoder; and generating query features from the query using a recurrent neural network (RNN). ### Claim 3 (depends on 2) The method of claim 2, wherein the graph comprises nodes and edges based on the visual features and the query features. ### Claim 4 (depends on 3) The method of claim 3, wherein ranking the proposals comprises: updating node features for the nodes in the graph; and calculating edge weights for the edges in the graph. ### Claim 5 (depends on 3) The method of claim 3, wherein ranking the proposals further comprises: performing node aggregation; and ranking the proposals based on the node aggregation and results from processing the graph using the GCN. ### Claim 6 (depends on 1) The method of claim 1, wherein at least two proposals of the multiple proposals comprise overlapping frames of the plurality of frames in the video. ### Claim 7 (depends on 1) The method of claim 1, wherein at least two proposals of the multiple proposals comprise subsets of the plurality of frames in the video that do not overlap. ### Claim 8 A system, comprising: a processor; and memory comprising a program, which when executed by the processor performs an operation, the operation comprising: receiving a query describing an aspect in a video, the video comprising a plurality of frames; identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames; generating a graph based on the query and the multiple proposals; ranking the proposals using a GCN that identifies relationships between the proposals, wherein the graph is input into the graph convolution network; and selecting, based on the ranking, one of the proposals as a video segment that correlates to the query. ### Claim 9 (depends on 8) The system of claim 8, wherein generating the graph comprises: identifying visual features in the proposals using a visual feature encoder; and generating query features from the query using a recurrent neural network (RNN). ### Claim 10 (depends on 9) The system of claim 9, wherein the graph comprises nodes and edges based on the visual features and the query features. ### Claim 11 (depends on 10) The system of claim 10, wherein ranking the proposals comprises: updating node features for the nodes in the graph; and calculating edge weights for the edges in the graph. ### Claim 12 (depends on 10) The system of claim 10, wherein ranking the proposals further comprises: performing node aggregation; and ranking the proposals based on the node aggregation and results from processing the graph using the GCN. ### Claim 13 (depends on 8) The system of claim 8, wherein at least two proposals of the multiple proposals comprise overlapping frames of the plurality of frames in the video. ### Claim 14 (depends on 8) The system of claim 8, wherein at least two proposals of the multiple proposals comprise subsets of the plurality of frames in the video that do not overlap. ### Claim 15 A computer program product for identifying a video segment that correlates to a query, the computer program product comprising: a computer readable storage medium having computer readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation, the operation comprising: receiving the query, the query describing an aspect in a video comprising a plurality of frames; identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames; generating a graph based on the query and the multiple proposals; ranking the proposals using a GCN that identifies relationships between the proposals, wherein the graph is input into the graph convolution network; and selecting, based on the ranking, one of the proposals as the video segment that correlates to the query. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein generating the graph comprises: identifying visual features in the proposals using a visual feature encoder; and generating query features from the query using a recurrent neural network (RNN). ### Claim 17 (depends on 16) The computer program product of claim 16, wherein the graph comprises nodes and edges based on the visual features and the query features. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein ranking the proposals comprises: updating node features for the nodes in the graph; and calculating edge weights for the edges in the graph. ### Claim 19 (depends on 17) The computer program product of claim 17, wherein ranking the proposals further comprises: performing node aggregation; and ranking the proposals based on the node aggregation and results from processing the graph using the GCN. ### Claim 20 (depends on 15) The computer program product of claim 15, wherein at least two proposals of the multiple proposals comprise overlapping frames of the plurality of frames in the video.