# US 2021/0241489 A1 · Iterative spatio-temporal action detection in video > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241489 A1 - Document ID: US20210241489A1 - Status: Published application - Kind: A1 - Application number: 17/237,728 - Filed: 2021-04-22 - Published: 2021-08-05 - Earliest claimed: 2017-10-04 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Xiaodong YANG; Ming-Yu LIU; Jan KAUTZ; Fanyi XIAO; Xitong YANG - Web page: https://patentgrep.com/patent/US20210241489A1 ## CPC - G06V 10/82: using neural networks - G06T 7/277: involving stochastic approaches, e.g. using Kalman filters - G06T 7/74: {involving reference images or patches} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/764: using classification, e.g. of video objects - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06T 2207/10016: Video; Image sequence - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/20076: Probabilistic image processing - G06T 2207/20084: Artificial neural networks [ANN] - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Iterative prediction systems and methods for the task of action detection process an inputted sequence of video frames to generate an output of both action tubes and respective action labels, wherein the action tubes comprise a sequence of bounding boxes on each video frame. An iterative predictor processes large offsets between the bounding boxes and the ground-truth. ## Claims ### Claim 1 A method for detecting actions in a video stream, comprising: in an initial iteration, processing a selected video clip from a plurality of video clips in the video stream and a set of anchors to generate a set of action tubes and a set of action classification labels; iteratively processing, in one or more subsequent iterations, respective other groups of video clips from the plurality of video clips by using one of the other groups of video clips to extend a set of video clips processed in a previous iteration, and, processing the extended set of video clips and a set of anchors determined based on a set of action tubes generated in the previous iteration to generate another set of action tubes and another set of action classification labels; and outputting said another set of action tubes and said another set of action labels. ### Claim 2 (depends on 1) The method according to claim 1, wherein said using one of the other groups of video clips to extend a set of video clips processed in a previous iteration comprises using respective video clips to extend the set of video clips on either end of the set of video clips. ### Claim 3 (depends on 2) The method according to claim 2, wherein the extended set of video clips consists of a sequence of consecutive frames from the video stream. ### Claim 4 (depends on 1) The method according to claim 1, wherein, in the first iteration and in each of the subsequent iterations, each anchor in the set of anchors inputted corresponds to a position of a respective anchor tube from the set of anchor tubes generated in the immediately preceding iteration. ### Claim 5 (depends on 4) The method according to claim 4, wherein the position of the respective anchor tube is determined in a video frame latest in time in the set of video clips processed in the preceding iteration. ### Claim 6 (depends on 1) The method according to claim 1, wherein the selected video clip is a short video clip comprising less than all video frames of the video stream. ### Claim 7 (depends on 6) The method according to claim 6, wherein the selected video clip includes less than 10 video frames. ### Claim 8 (depends on 1) The method according to claim 1, wherein the each video clip in the plurality of video clips consist of a same number of video frames. ### Claim 9 (depends on 1) The method according to claim 1, further comprising determining the set of anchors used in the first iteration from configuration information. ### Claim 10 (depends on 9) The method according to claim 9, wherein the set of anchors used in the first iteration is independent of features of the selected video clip. ### Claim 11 (depends on 10) The method according to claim 10, wherein the set of anchors used in the first iteration overlays an entire frame. ### Claim 12 (depends on 11) The method according to claim 11, wherein one of the set of anchors used in the first iteration overlays an entire frame. ### Claim 13 A non-transitory computer-readable storage medium having stored therein instructions which, when executed by one or more processors of a processing system, causes the processing system to perform operations comprising: in an initial iteration, processing a selected video clip from a plurality of video clips in a video stream and a set of anchors to generate a set of action tubes and a set of action classification labels; iteratively processing, in one or more subsequent iterations, respective other groups of video clips from the plurality of video clips by using one of the other groups of video clips to extend a set of video clips processed in a previous iteration, and, processing the extended set of video clips and a set of anchors determined based on a set of action tubes generated in the previous iteration to generate another set of action tubes and another set of action classification labels; and outputting said another set of action tubes and said another set of action labels. ### Claim 14 A processing system comprising a memory and one or more processors, wherein the one or more processors are configured to perform operations comprising: in an initial iteration, processing a selected video clip from a plurality of video clips in a video stream and a set of anchors to generate a set of action tubes and a set of action classification labels; iteratively processing, in one or more subsequent iterations, respective other groups of video clips from the plurality of video clips by using one of the other groups of video clips to extend a set of video clips processed in a previous iteration, and, processing the extended set of video clips and a set of anchors determined based on a set of action tubes generated in the previous iteration to generate another set of action tubes and another set of action classification labels; and outputting said another set of action tubes and said another set of action labels.