# US 12,003,728 B2 · Methods and systems for temporal resampling for multi-task machine vision > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,728 B2 - Document ID: US12003728B2 - Status: Granted - Kind: B2 - Application number: 17/816,825 - Filed: 2022-08-02 - Published: 2024-02-08 - Granted: 2024-06-04 - Earliest claimed: 2022-08-02 - Assignees: Alibaba Innovation Private Limited - Applicants: Alibaba Innovation Private Limited - Inventors: Shurun Wang; Zhao Wang; Yan Ye; Shiqi Wang - Web page: https://patentgrep.com/patent/US12003728B2 ## CPC - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference - H04N 19/172: the region being a picture, frame or field - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method for temporal resampling for multi-task machine vision is provided. The method includes receiving a bitstream of a video sequence after temporal resampling; and constructing a target frame from the bitstream using a frame construction model. ## Claims ### Claim 1 A method for temporal resampling for multi-task machine vision, comprising: receiving a bitstream of a video sequence after temporal resampling; and constructing a target frame from the bitstream using a frame construction model, wherein an input of the frame construction model is a channel-wise concatenation of an average residual of the target frame and a prediction of the target frame, the average residual of the target frame is obtained based on resampled frames and a resampling factor. ### Claim 2 (depends on 1) The method according to claim 1, wherein the average residual is obtained using the resampling factor and a frame from the video sequence. ### Claim 3 (depends on 1) The method according to claim 1, wherein the prediction is obtained using the average residual. ### Claim 4 (depends on 1) The method according to claim 1, wherein the frame construction model is trained with feature maps, the feature maps being extracted from corresponding one or more machine analysis models for the target frame and the constructed frame of the target frame, respectively. ### Claim 5 A method for temporal resampling for multi-task machine vision, comprising: receiving a bitstream of a video sequence after temporal resampling; and constructing a target frame from the bitstream using a frame construction model; wherein the frame construction model is trained with feature maps, the feature maps being extracted from corresponding one or more machine analysis models for the target frame and the constructed frame of the target frame, respectively, and one or more loss functions of corresponding one or more machine analysis models are obtained with label information. ### Claim 6 (depends on 5) The method according to claim 5, wherein a total loss function is determined using contour loss, feature map distortion of the one or more feature maps, and the one or more loss functions. ### Claim 7 (depends on 6) The method according to claim 6, wherein the loss function of each of the one or more machine analysis model is determined by a mean squared difference between a machine analysis model feature map of the target frame and a machine analysis model feature map of the constructed frame of the machine analysis model. ### Claim 8 An apparatus for temporal resampling for multi-task machine vision, the apparatus comprising: a memory figured to store instructions; and one or more processors configured to execute the instructions to cause the apparatus to perform: receiving a bitstream of a video sequence after temporal resampling; and constructing a target frame from the bitstream using a frame construction model wherein an input of the frame construction model is a channel-wise concatenation of an average residual of the target frame and a prediction of the target frame, the average residual of the target frame is obtained based on resampled frames and a resampling factor. ### Claim 9 (depends on 8) The apparatus according to claim 8, wherein the average residual is obtained using the resampling factor and a frame from the video sequence. ### Claim 10 (depends on 8) The apparatus according to claim 8, wherein the prediction is obtained using the average residual. ### Claim 11 (depends on 8) The apparatus according to claim 8, wherein the frame construction model is trained with feature maps, the feature maps being extracted from corresponding one or more machine analysis models for the target frame and the constructed frame of the target frame, respectively. ### Claim 12 An apparatus for temporal resampling for multi-task machine vision, the apparatus comprising: a memory configured to store instructions; and one or more processors configured to execute the instructions to cause the apparatus to perform: receiving a bitstream of a video sequence after temporal resampling; and constructing a target frame from the bitstream using a frame construction model; wherein the frame construction model is trained with feature maps, the feature maps being extracted from corresponding one or more machine analysis models for the target frame and the constructed frame of the target frame, respectively; and one or more loss functions of corresponding one or more machine analysis models are obtained with label information. ### Claim 13 (depends on 12) The apparatus according to claim 12, wherein a total loss function is determined using contour loss, feature map distortion of the one or more feature maps, and the one or more loss functions. ### Claim 14 A method for temporal resampling for multi-task machine vision, comprising: receiving a video sequence; determining a moving complexity of the video sequence, wherein determining the moving complexity further comprises: calculating an average mean absolute difference (MAD) for a testing point with a range; sampling all the testing points across the video sequence equally with an interval; and obtaining the moving complexity of the video sequence based on the average MAD and the interval; and determining whether to perform temporal resampling based on the moving complexity. ### Claim 15 (depends on 14) The method according to claim 14, wherein determining whether to perform the temporal resampling based on the moving complexity further comprises: not performing the temporal resampling when the moving complexity is greater than or equal to a first threshold; or performing the temporal resampling when the moving complexity is less than a second threshold. ### Claim 16 (depends on 15) The method according to claim 15, when the moving complexity is less than the first threshold and greater than or equal to the second threshold, the method further comprises: determining a quantization parameter associated with the video sequence; and determining whether to perform the temporal resampling based on the quantization parameter. ### Claim 17 (depends on 16) The method according to claim 16, wherein determining whether to perform the temporal resampling based on the quantization parameter further comprises: performing the temporal resampling when the quantization parameter is equal to or greater than a preset value.