# US 12,003,734 B2 · Machine learning based flow determination for video coding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,734 B2 - Document ID: US12003734B2 - Status: Granted - Kind: B2 - Application number: 17/676,510 - Filed: 2022-02-21 - Published: 2022-08-25 - Granted: 2024-06-04 - Earliest claimed: 2021-02-25 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Ankitesh Kumar Singh; Hilmi Enes Egilmez; Muhammed Zeyd Coban; Marta Karczewicz - Web page: https://patentgrep.com/patent/US12003734B2 ## CPC - H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/172: the region being a picture, frame or field - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/513: Processing of motion vectors - H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters} - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and techniques are described herein for processing video data. In some aspects, a method can include obtain, by a machine learning system, input video data. The input video data includes one or more luminance components for a current frame. The method can include determining, by the machine learning system, motion information for the luminance component(s) of the current frame and motion information for one or more chrominance components of the current frame using the luminance component(s) for the current frame. In some cases, the method can include determining the motion information for the luminance component(s) based on the luma component(s) of the current frame and at least one reconstructed luma component of a previous frame. In some cases, the method can further include determining the motion information for the chrominance component(s) of the current frame using the motion information determined for the luminance component(s) of the current frame. ## Claims ### Claim 1 A method of processing video data, the method comprising: obtaining, by a machine learning system, a latent representation of at least one luminance component for a current frame, the latent representation of the at least one luminance component of the current frame being based on the at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame; determining, by the machine learning system, motion information for the at least one luminance component of the current frame using the latent representation of the at least one luminance component of the current frame; and determining, using the machine learning system, motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component for the current frame. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining, by the machine learning system using the motion information for the at least one luminance component of the current frame and the motion information for the one or more chrominance components of the current frame, a warping parameter for the at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame; and determining one or more inter-frame predictions for the current frame using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame. ### Claim 3 (depends on 2) The method of claim 2, wherein the one or more inter-frame predictions are determined at least in part by applying an interpolation operation using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame. ### Claim 4 (depends on 3) The method of claim 3, wherein the interpolation operation includes a trilinear interpolation operation. ### Claim 5 (depends on 2) The method of claim 2, wherein the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame include space-scale flow (SSF) warping parameters. ### Claim 6 (depends on 5) The method of claim 5, wherein the SSF warping parameters include learned scale-flow vectors. ### Claim 7 (depends on 1) The method of claim 1, wherein the motion information for the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system. ### Claim 8 (depends on 1) The method of claim 1, wherein the motion information for the one or more chrominance components of the current frame is determined at least in part by sampling the motion information determined for the at least one luminance component of the current frame. ### Claim 9 (depends on 1) The method of claim 1, wherein the current frame includes a video frame. ### Claim 10 (depends on 1) The method of claim 1, wherein the one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component. ### Claim 11 (depends on 1) The method of claim 1, wherein the current frame has a luminance-chrominance (YUV) format. ### Claim 12 (depends on 11) The method of claim 11, wherein the YUV format is a YUV 4:2:0 format. ### Claim 13 An apparatus for processing video data, comprising: at least one memory; and one or more processors coupled to the at least one memory, the one or more processors configured to: obtain, using a machine learning system, a latent representation of at least one luminance component for a current frame the latent representation of the at least one luminance component of the current frame being based on the at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame; determine, using the machine learning system, motion information for the at least one luminance component of the current frame using the latent representation of the at least one luminance component of the current frame; and determine, using the machine learning system, motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component for the current frame. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the one or more processors are configured to: determine, using the machine learning system based on the motion information for the at least one luminance component of the current frame and the motion information for the one or more chrominance components of the current frame, a warping parameter for the at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame; and determine one or more inter-frame predictions for the current frame using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the one or more inter-frame predictions are determined at least in part by applying an interpolation operation using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the interpolation operation includes a trilinear interpolation operation. ### Claim 17 (depends on 14) The apparatus of claim 14, wherein the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame include space-scale flow (SSF) warping parameters. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein the SSF warping parameters include learned scale-flow vectors. ### Claim 19 (depends on 13) The apparatus of claim 13, wherein the motion information for the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system. ### Claim 20 (depends on 13) The apparatus of claim 13, wherein, to determine the motion information for the one or more chrominance components of the current frame, the one or more processors are configured to sample the motion information determined for the at least one luminance component of the current frame. ### Claim 21 (depends on 13) The apparatus of claim 13, wherein the current frame includes a video frame. ### Claim 22 (depends on 13) The apparatus of claim 13, wherein the one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component. ### Claim 23 (depends on 13) The apparatus of claim 13, wherein the current frame has a luminance-chrominance (YUV) format. ### Claim 24 (depends on 23) The apparatus of claim 23, wherein the YUV format is a YUV 4:2:0 format. ### Claim 25 (depends on 13) The apparatus of claim 13, further comprising at least one camera configured to capture one or more frames. ### Claim 26 (depends on 13) The apparatus of claim 13, further comprising at least one display configured to display one or more frames. ### Claim 27 (depends on 13) The apparatus of claim 13, wherein the apparatus includes a mobile device. ### Claim 28 A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: obtain, using a machine learning system, a latent representation of at least one luminance component for a current frame, the latent representation of the at least one luminance component of the current frame being based on the at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame; determine, using the machine learning system, motion information for the at least one luminance component of the current frame using the latent representation of the at least one luminance component of the current frame; and determine, using the machine learning system, motion information for one or more chrominance components of the current frame using the motion information determined for the at least one luminance component for the current frame. ### Claim 29 (depends on 1) The method of claim 1, wherein the input video data does not include a latent representation of luminance component for a current frame. ### Claim 30 (depends on 13) The apparatus of claim 13, wherein the input video data does not include a latent representation of luminance component for a current frame.