# US 12,192,543 B2 · Video frame action detection using gated history > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,543 B2 - Document ID: US12192543B2 - Status: Granted - Kind: B2 - Application number: 18/393,664 - Filed: 2023-12-21 - Published: 2024-07-18 - Granted: 2025-01-07 - Earliest claimed: 2022-06-03 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Gaurav Mittal; Ye Yu; Mei Chen; Junwen Chen - Web page: https://patentgrep.com/patent/US12192543B2 ## CPC - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06V 10/82: using neural networks - G06V 20/42: {of sport video content} - G06V 20/44: {Event detection} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Example solutions for video frame action detection use a gated history and include: receiving a video stream comprising a plurality of video frames; grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame; determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame; weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames; and based on at least the set of weighted historical video frames and the set of present video frames, generating an action prediction for the current video frame. ## Claims ### Claim 1 A system comprising: a processor; and a computer-readable medium storing instructions that are operative upon execution by the processor to: receive a video stream comprising a plurality of video frames; group the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame; based on at least the set of historical video frames and the set of present video frames, generate an action prediction for the current video frame; and perform background suppression, wherein the action prediction comprises a confidence and wherein performing the background suppression comprises: generating a loss function that weights low confidence video frames more heavily, with separate emphasis on action and background classes, for a classifier that generates the action prediction. ### Claim 2 (depends on 1) The system of claim 1, wherein the instructions are further operative to: determine a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame; and weight the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames, wherein generating the action prediction for the current video frame is based on at least the set of weighted historical video frames and the set of present video frames. ### Claim 3 (depends on 2) The system of claim 2, wherein determining the set of attention weights comprises: determining, for each video frame of the set of historical video frames, a position-guided gating score. ### Claim 4 (depends on 1) The system of claim 1, wherein the instructions are further operative to: based on at least the action prediction for the current video frame, generate an annotation for the current video frame; and display the current video frame subject to the annotation for the current video frame. ### Claim 5 (depends on 1) The system of claim 1, wherein the action prediction comprises a no action prediction or an action class prediction selected from a plurality of action classes. ### Claim 6 (depends on 1) The system of claim 1, wherein the instructions are further operative to: based on at least the set of historical video frames and the set of present video frames, generate a future action prediction for a video frame not yet observed. ### Claim 7 (depends on 6) The system of claim 6, wherein the future action prediction is based on at least a predicted trajectory of an autonomous driving vehicle. ### Claim 8 A computerized method comprising: receiving a video stream comprising a plurality of video frames; grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame; based on at least the set of historical video frames and the set of present video frames, generating an action prediction for the current video frame; and performing background suppression, wherein the action prediction comprises a confidence and wherein performing the background suppression comprises: generating a loss function that weights low confidence video frames more heavily, with separate emphasis on action and background classes, for a classifier that generates the action prediction. ### Claim 9 (depends on 8) The method of claim 8, further comprising: determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame; and weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames, wherein generating the action prediction for the current video frame is based on at least the set of weighted historical video frames and the set of present video frames. ### Claim 10 (depends on 9) The method of claim 9, wherein determining the set of attention weights comprises: determining, for each video frame of the set of historical video frames, a position-guided gating score. ### Claim 11 (depends on 8) The method of claim 8, further comprising: based on at least the action prediction for the current video frame, generating an annotation for the current video frame; and displaying the current video frame subject to the annotation for the current video frame. ### Claim 12 (depends on 8) The method of claim 8, wherein the action prediction comprises a no action prediction or an action class prediction selected from a plurality of action classes. ### Claim 13 (depends on 8) The method of claim 8, further comprising: based on at least the set of historical video frames and the set of present video frames, generating a future action prediction for a video frame not yet observed. ### Claim 14 (depends on 13) The method of claim 13, wherein the future action prediction is based on at least a predicted trajectory of an autonomous driving vehicle. ### Claim 15 One or more computer storage devices having computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising: receiving a video stream comprising a plurality of video frames; grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame; based on at least the set of historical video frames and the set of present video frames, generating an action prediction for the current video frame; and performing background suppression, wherein the action prediction comprises a confidence and wherein performing the background suppression comprises: generating a loss function that weights low confidence video frames more heavily, with separate emphasis on action and background classes, for a classifier that generates the action prediction. ### Claim 16 (depends on 15) The one or more computer storage devices of claim 15, wherein the operations further comprise: determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame; and weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames, wherein generating the action prediction for the current video frame is based on at least the set of weighted historical video frames and the set of present video frames. ### Claim 17 (depends on 16) The one or more computer storage devices of claim 16, wherein determining the set of attention weights comprises: determining, for each video frame of the set of historical video frames, a position-guided gating score. ### Claim 18 (depends on 16) The one or more computer storage devices of claim 16, wherein the operations further comprise: based on at least the action prediction for the current video frame, generating an annotation for the current video frame; and displaying the current video frame subject to the annotation for the current video frame. ### Claim 19 (depends on 15) The one or more computer storage devices of claim 15, wherein the action prediction comprises a no action prediction or an action class prediction selected from a plurality of action classes. ### Claim 20 (depends on 15) The one or more computer storage devices of claim 15, wherein the operations further comprise: based on at least the set of historical video frames and the set of present video frames, generating a future action prediction for a video frame not yet observed.