# US 10,503,978 B2 · Spatio-temporal interaction network for learning object interactions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,978 B2 - Document ID: US10503978B2 - Status: Granted - Kind: B2 - Application number: 15/978,738 - Filed: 2018-05-14 - Published: 2019-01-17 - Granted: 2019-12-10 - Earliest claimed: 2017-07-14 - Assignees: NEC Corporation - Applicants: NEC Laboratories America, Inc. - Inventors: Asim Kadav; Chih-Yao Ma; Iain Melvin; Hans Peter Graf - Web page: https://patentgrep.com/patent/US10503978B2 ## CPC - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 30/2504: {Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems and methods for improving video understanding tasks based on higher-order object interactions (HOIs) between object features are provided. A plurality of frames of a video are obtained. A coarse-grained feature representation is generated by generating an image feature for each of for each of a plurality of timesteps respectively corresponding to each of the frames and performing attention based on the image features. A fine-grained feature representation is generated by generating an object feature for each of the plurality of timesteps and generating the HOIs between the object features. The coarse-grained and the fine-grained feature representations are concatenated to generate a concatenated feature representation. ## Claims ### Claim 1 A computer-implemented method for improving video understanding tasks based on higher-order object interactions (HOIs) between object features, comprising: obtaining a plurality of frames of a video; generating a coarse-grained feature representation, including generating an image feature for each of a plurality of timesteps respectively corresponding to each of the frames, and performing attention based on the image features; generating a fine-grained feature representation, including generating an object feature for each of the plurality of timesteps, and generating the HOIs between the object features; and concatenating the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation. ### Claim 2 (depends on 1) The method of claim 1, further comprising making a prediction based on the concatenated feature representation to perform one or more operations. ### Claim 3 (depends on 2) The method of claim 2, wherein the one or more operations include at least one of action recognition and video captioning. ### Claim 4 (depends on 1) The method of claim 1, wherein the image features are generated using a convolutional neural network (CNN). ### Claim 5 (depends on 1) The method of claim 1, wherein performing attention based on the image features further includes performing Scaled Dot Product (SDP) attention. ### Claim 6 (depends on 1) The method of claim 1, wherein the coarse-grained feature representation includes attended context information. ### Claim 7 (depends on 6) The method of claim 6, wherein generating the coarse-grained feature representation further comprises generating the attended context information by mean-pooling weighted image representations. ### Claim 8 (depends on 1) The method of claim 1, wherein the object features are generated using a region proposal network (RPN). ### Claim 9 (depends on 1) The method of claim 1, wherein generating the HOIs further includes: generating projected object features respectively corresponding to the object features; computing attention weights based at least in part on the projected object features; performing attentive selection using a plurality of attentive selection components to generate attended object features each corresponding to a respective one of the attentive selection components; concatenating the attended object features to generate an attended object feature representation; and determining an HOI representation based on the attended object feature representation. ### Claim 10 (depends on 9) The method of claim 9, wherein the projected object features are generated using a plurality of multilayer perceptrons (MLPs). ### Claim 11 (depends on 9) The method of claim 9, wherein the attentive selection components implement dot product attention. ### Claim 12 (depends on 9) The method of claim 9, wherein determining the HOI representation further includes implementing a long short term memory (LSTM) cell, and wherein a last hidden state of the LSTM cell is selected as the HOI representation. ### Claim 13 A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for improving video understanding tasks based on higher-order object interactions (HOIs) between object features comprising: obtaining a plurality of frames of a video; generating a coarse-grained feature representation, including generating an image feature for each of a plurality of timesteps respectively corresponding to each of the frames, and performing attention based on the image features; generating a fine-grained feature representation, including generating an object feature for each of the plurality of timesteps, and generating the HOIs between the object features; and concatenating the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation. ### Claim 14 (depends on 13) The computer program product of claim 13, wherein the method further comprises making a prediction based on the concatenated feature representation to perform one or more operations, and wherein the one or more operations include at least one of action recognition and video captioning. ### Claim 15 (depends on 13) The computer program product of claim 13, wherein the image features are generated using a convolutional neural network (CNN). ### Claim 16 (depends on 13) The computer program product of claim 13, wherein performing attention based on the image features further includes performing Scaled Dot Product (SDP) attention. ### Claim 17 (depends on 13) The computer program product of claim 13, wherein the coarse-grained feature representation includes attended context information, and wherein generating the coarse-grained feature representation further comprises generating the attended context information by mean-pooling weighted image representations. ### Claim 18 (depends on 13) The computer program product of claim 13, wherein generating the HOIs further includes: generating projected object features respectively corresponding to the object features; computing attention weights based at least in part on the projected object features; performing attentive selection using a plurality of attentive selection components to generate attended object features each corresponding to a respective one of the attentive selection components; concatenating the attended object features to generate an attended object feature representation; and determining an HOI representation based on the attended object feature representation. ### Claim 19 (depends on 18) The computer program product of claim 18, wherein determining the HOI representation further includes implementing a long short term memory (LSTM) cell, and wherein a last hidden state of the LSTM cell is selected as the HOI representation. ### Claim 20 A system for improving video understanding tasks based on higher-order object interactions (HOIs) between object features, comprising: at least one processor operatively coupled to a memory, wherein the at least one processor is configured to: obtain a plurality of frames of a video; generate a coarse-grained feature representation by generating an image feature for each of a plurality of timesteps respectively corresponding to each of the frames, and performing attention based on the image features; generate a fine-grained feature representation by generating an object feature for each of the plurality of timesteps, and generating the HOIs between the object features; and concatenate the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation.