Iterative spatio-temporal action detection in video
AssigneeNVIDIA Corporation
InventorsXiaodong YANG, Ming-Yu LIU, Jan KAUTZ, Fanyi XIAO, Xitong YANG
Life of this patent
Estimating the term…
Abstract
Iterative prediction systems and methods for the task of action detection process an inputted sequence of video frames to generate an output of both action tubes and respective action labels, wherein the action tubes comprise a sequence of bounding boxes on each video frame. An iterative predictor processes large offsets between the bounding boxes and the ground-truth.
Claims14
A method for detecting actions in a video stream, comprising:
- in an initial iteration, processing a selected video clip from a plurality of video clips in the video stream and a set of anchors to generate a set of action tubes and a set of action classification labels;
- iteratively processing, in one or more subsequent iterations, respective other groups of video clips from the plurality of video clips by using one of the other groups of video clips to extend a set of video clips processed in a previous iteration, and, processing the extended set of video clips and a set of anchors determined based on a set of action tubes generated in the previous iteration to generate another set of action tubes and another set of action classification labels;
- and
- outputting said another set of action tubes and said another set of action labels.
A non-transitory computer-readable storage medium having stored therein instructions which, when executed by one or more processors of a processing system, causes the processing system to perform operations comprising:
- in an initial iteration, processing a selected video clip from a plurality of video clips in a video stream and a set of anchors to generate a set of action tubes and a set of action classification labels;
- iteratively processing, in one or more subsequent iterations, respective other groups of video clips from the plurality of video clips by using one of the other groups of video clips to extend a set of video clips processed in a previous iteration, and, processing the extended set of video clips and a set of anchors determined based on a set of action tubes generated in the previous iteration to generate another set of action tubes and another set of action classification labels;
- and
- outputting said another set of action tubes and said another set of action labels.
A processing system comprising a memory and one or more processors, wherein the one or more processors are configured to perform operations comprising:
- in an initial iteration, processing a selected video clip from a plurality of video clips in a video stream and a set of anchors to generate a set of action tubes and a set of action classification labels;
- iteratively processing, in one or more subsequent iterations, respective other groups of video clips from the plurality of video clips by using one of the other groups of video clips to extend a set of video clips processed in a previous iteration, and, processing the extended set of video clips and a set of anchors determined based on a set of action tubes generated in the previous iteration to generate another set of action tubes and another set of action classification labels;
- and
- outputting said another set of action tubes and said another set of action labels.
Dependencies are inferred from the claim text.