Iterative spatio-temporal action detection in video
AssigneeNVIDIA Corporation
InventorsXiaodong YANG, Xitong YANG, Fanyi XIAO, Ming-Yu LIU, Jan KAUTZ
Life of this patent
Estimating the term…
Abstract
Iterative prediction systems and methods for the task of action detection process an inputted sequence of video frames to generate an output of both action tubes and respective action labels, wherein the action tubes comprise a sequence of bounding boxes on each video frame. An iterative predictor processes large offsets between the bounding boxes and the ground-truth.
Claims20
An iterative prediction method for a task of action detection in video, comprising:
- processing an inputted sequence of video frames to generate an output of both action tubes and respective action classification labels, wherein each said action tube comprises a bounding box on respective video frames in a sequence of video frames,
- wherein the processing comprises iteratively processing, based on the inputted sequence of video frames that is incrementally expanded in successive iterations, large offsets between the bounding boxes and ground-truth.
A method for detecting actions in a video stream, comprising:
- in an initial iteration, processing a selected video clip from a plurality of video clips in the video stream and a set of anchors to generate a set of action tubes and a set of action classification labels;
- iteratively processing, in one or more subsequent iterations, respective other groups of video clips from the plurality of video clips by using one of the other groups of video clips to extend a set of video clips processed in a previous iteration, and, processing the extended set of video clips and a set of anchors determined based on a set of action tubes generated in the previous iteration to generate another set of action tubes and another set of action classification labels;
- and
- outputting said another set of action tubes and said another set of action labels.
The method according to claim 5, wherein said using one of the other groups of video clips to extend a set of video clips processed in a previous iteration comprises using respective video clips to extend the set of video clips on either end of the set of video clips. The method according to claim 6, wherein the extended set of video clips consists of a sequence of consecutive frames from the video stream.
An iterative prediction system configured to perform a task of action detection in video, comprising:
- a video processor configured to process an inputted sequence of video frames to generate an output of both action tubes and respective action classification labels, wherein each said action tube comprises a bounding box on respective video frames in a sequence of video frames,
- wherein the video processor comprises an iterative predictor that processes, based on the inputted sequence of video frames that is incrementally expanded in successive iterations, large offsets between the bounding boxes and ground-truth.
A system of one or more interconnected processors comprising:
- a backbone convolutional processing network configured to produce a spatial feature volume for each video frame;
- a region of interest pooling layer processing network that processes the spatial feature volumes for multiple video frames together with initial anchor tubes to produce local features corresponding to the initial anchor tubes;
- a global branch processing network configured to concatenate the region of interest pooled features from different video frames and pass a result through a convolutional layer;
- the convolutional layer producing global features describing contents in an entire anchor tube;
- and
- fully-connected processing layers that concatenate the local and global features and process them to produce a 4-dimensional regression coefficient vector for each anchor tube for action classification.
Dependencies are inferred from the claim text.