# US 12,112,538 B2 · Systems and methods for improved video understanding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,538 B2 - Document ID: US12112538B2 - Status: Granted - Kind: B2 - Application number: 17/370,522 - Filed: 2021-07-08 - Published: 2023-01-19 - Granted: 2024-10-08 - Earliest claimed: 2021-07-08 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Anurag Arnab; Mostafa Dehghani; Georg Heigold; Chen Sun; Mario Lucic; Cordelia Luise Schmid - Web page: https://patentgrep.com/patent/US12112538B2 ## CPC - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 20/00: Machine learning - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A computer-implemented method for classifying video data with improved accuracy includes obtaining, by a computing system comprising one or more computing devices, video data comprising a plurality of video frames; extracting, by the computing system, a plurality of video tokens from the video data, the plurality of video tokens comprising a representation of spatiotemporal information in the video data; providing, by the computing system, the plurality of video tokens as input to a video understanding model, the video understanding model comprising a video transformer encoder model; and receiving, by the computing system, a classification output from the video understanding model. ## Claims ### Claim 1 A computer-implemented method for classifying video data with improved accuracy, the method comprising: obtaining, by a computing system comprising one or more computing devices, video data comprising a plurality of video frames; extracting, by the computing system, a plurality of video tokens from the video data, the plurality of video tokens comprising a representation of spatiotemporal information in the video data, wherein extracting the plurality of video tokens from the video data comprises: extracting, by the computing system, a plurality of video tubelets from the video data, the plurality of video tublets respectively comprising a length and a width and spanning two or more video frames of the plurality of video frames; projecting, by the computing system, the plurality of video tubelets to a plurality of tensor representations of the plurality of video tubelets; and merging, by the computing system, the plurality of tensor representations along at least one dimension to produce the plurality of video tokens; providing, by the computing system, the plurality of video tokens as input to a video understanding model, the video understanding model comprising a video transformer encoder model; and receiving, by the computing system, a classification output from the video understanding model. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the video transformer encoder comprises a factorized encoder, the factorized encoder comprising a spatial transformer encoder and a temporal transformer encoder. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the spatial transformer encoder is configured to receive the plurality of video tokens and produce, in response to receipt of the plurality of video tokens, a plurality of temporal representations; and wherein the temporal transformer encoder is configured to receive the plurality of temporal representations and produce, in response to receipt of the plurality of temporal representations, a spatiotemporal representation of the video data, wherein the spatiotemporal representation is classified to produce the classification output. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the video transformer encoder model comprises a factorized self-attention mechanism, wherein the factorized self-attention mechanism comprises a first self-attention block configured to compute spatial self-attention among the plurality of video tokens from a same temporal index and a second self-attention block configured to compute temporal self-attention among the plurality of video tokens from a same spatial index. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the plurality of video tokens are reshaped prior to being input to the factorized self-attention mechanism. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the video transformer encoder model comprises a factorized dot-product attention mechanism, the factorized dot-product attention mechanism comprising a plurality of spatial attention heads configured to compute attention weights for each of the plurality of video tokens over a spatial dimension and a plurality of temporal attention heads configured to compute attention weights for each of the plurality of video tokens over a temporal dimension. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein outputs from the plurality of spatial attention heads and the plurality of temporal attention heads are combined by concatenation and linear projection. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the plurality of video tokens are single-dimensional. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the plurality of video tubelets are nonoverlapping. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein positional embeddings are added to the plurality of video tokens and input to the video understanding model. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the transformer encoder model comprises at least one normalization layer. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the transformer encoder model comprises at least one multi-layer perceptron layer. ### Claim 13 A computing system configured for classifying video data with improved accuracy, the computing system comprising: one or more processors; and one or more memory devices storing: a video understanding model comprising a video transformer encoder model; and one or more operations that, when implemented by the one or more processors, cause the one or more processors to perform operations, the operations comprising: obtaining video data comprising a plurality of video frames; extracting a plurality of video tokens from the video data, the plurality of video tokens comprising a representation of spatiotemporal information in the video data, wherein extracting the plurality of video tokens from the video data comprises: extracting, by the computing system, a plurality of video tubelets from the video data, the plurality of video tublets respectively comprising a length and a width and spanning two or more video frames of the plurality of video frames; projecting, by the computing system, the plurality of video tubelets to a plurality of tensor representations of the plurality of video tubelets; and merging, by the computing system, the plurality of tensor representations along at least one dimension to produce the plurality of video tokens; providing the plurality of video tokens as input to a video understanding model, the video understanding model comprising a video transformer encoder model; and receiving a classification output from the video understanding model. ### Claim 14 (depends on 13) The computing system of claim 13, wherein the video transformer encoder comprises a factorized encoder, the factorized encoder comprising a spatial transformer encoder and a temporal transformer encoder, wherein the spatial transformer encoder is configured to receive the plurality of video tokens and produce, in response to receipt of the plurality of video tokens, a plurality of temporal representations; and wherein the temporal transformer encoder is configured to receive the plurality of temporal representations and produce, in response to receipt of the plurality of temporal representations, a spatiotemporal representation of the video data, wherein the spatiotemporal representation is classified to produce the classification output. ### Claim 15 (depends on 13) The computing system of claim 13, wherein the video transformer encoder model comprises a factorized self-attention mechanism, wherein the factorized self-attention mechanism comprises a first self-attention block configured to compute spatial self-attention among the plurality of video tokens from a same temporal index and a second self-attention block configured to compute temporal self-attention among the plurality of video tokens from a same spatial index. ### Claim 16 (depends on 13) The computing system of claim 13, wherein the video transformer encoder model comprises a factorized dot-product attention mechanism, the factorized dot-product attention mechanism comprising a plurality of spatial attention heads configured to compute attention weights for each of the plurality of video tokens over a spatial dimension and a plurality of temporal attention heads configured to compute attention weights for each of the plurality of video tokens over a temporal dimension.