# US 11,445,168 B1 · Content-adaptive video sampling for cost-effective quality monitoring > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,168 B1 - Document ID: US11445168B1 - Status: Granted - Kind: B1 - Application number: 17/039,349 - Filed: 2020-09-30 - Granted: 2022-09-13 - Earliest claimed: 2020-09-30 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Hai Wei; Charles Benjamin Waggoner; Zongyi Liu; Srinivas Rajagopalan; Lei Li - Web page: https://patentgrep.com/patent/US11445168B1 ## CPC - H04N 17/00: Diagnosis, testing or measuring for television systems or their details - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06V 10/82: using neural networks - G06V 20/188: {Vegetation} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - H04N 19/102: characterised by the element, parameter or selection affected or controlled by the adaptive coding - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/142: Detection of scene cut or scene change - H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147) - H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction - H04N 19/172: the region being a picture, frame or field - H04N 19/176: the region being a block, e.g. a macroblock - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/30168: Image quality inspection - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques for content-adaptive video sampling for automated video quality monitoring are described. As one example, a computer-implemented method includes receiving a request to train a machine learning model on a training video file comprising at least one labeled defect, performing an encode on the training video file to generate one or more compression features for each compressed frame of the training video file, training the machine learning model to identify a proper subset of candidate defect frames of the training video file based at least in part on the one or more compression features for each compressed frame of the training video file and the at least one labeled defect, receiving an inference request for an input video file, performing an encode on the input video file to generate one or more compression features for each compressed frame of the input video file, generating, by the machine learning model, a proper subset of candidate defect frames of the input video file based at least in part on the one or more compression features for each compressed frame of the input video file, and determining a defect in the input video file based at least in part on the proper subset of candidate defect frames of the input video file. ## Claims ### Claim 1 A computer-implemented method comprising: receiving a request to train a machine learning model on a training video file comprising at least one labeled defect; performing an encode on the training video file to generate compressed frames of the training video file by a compression of the training video file, and one or more compression features from the compression of the training video file; training the machine learning model to identify a proper subset of candidate defect frames of the training video file based at least in part on the one or more compression features from the compression of the training video file and the at least one labeled defect; receiving an inference request for an input video file; performing an encode on the input video file to generate compressed frames of the input video file by a compression of the input video file, and one or more compression features from the compression of the input video file; generating, by the machine learning model, a proper subset of candidate defect frames of the input video file based at least in part on the one or more compression features from the compression of the input video file; and determining a defect in the input video file based at least in part on the proper subset of candidate defect frames of the input video file. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising triggering a remedial action for the input video file based at least in part on the defect. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising determining a global defect in the input video file based at least in part on the one or more compression features. ### Claim 4 A computer-implemented method comprising: receiving an inference request for an input video file; performing an encode on the input video file to generate one or more compressed frames of the input video file by a compression of the input video file, and one or more compression features from the compression of the input video file; generating, by a machine learning model, a proper subset of candidate defect frames of the input video file based at least in part on the one or more compression features from the compression of the input video file; and determining a defect in the input video file based at least in part on the proper subset of candidate defect frames of the input video file. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising triggering a remedial action for the input video file based at least in part on the defect. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the remedial action comprises re-encoding the input video file. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, wherein the remedial action comprises requesting another copy of the input video file from a content provider. ### Claim 8 (depends on 4) The computer-implemented method of claim 4, wherein the one or more compression features includes a frame complexity for each compressed frame of the input video file. ### Claim 9 (depends on 4) The computer-implemented method of claim 4, wherein the one or more compression features includes a frame-level encoding type for each compressed frame of the input video file. ### Claim 10 (depends on 4) The computer-implemented method of claim 4, wherein the one or more compression features includes a block-level encoding mode for each compressed frame of the input video file. ### Claim 11 (depends on 4) The computer-implemented method of claim 4, further comprising determining a global defect in the input video file based at least in part on the one or more compression features. ### Claim 12 (depends on 4) The computer-implemented method of claim 4, further comprising sending the one or more compressed frames of the input video file to a client device when the defect is below a defect threshold. ### Claim 13 (depends on 4) The computer-implemented method of claim 4, wherein the generating comprises the machine learning model outputting a frame identification value and a defect probability value for each of the proper subset of candidate defect frames of the input video file. ### Claim 14 (depends on 4) The computer-implemented method of claim 4, wherein the generating comprises the machine learning model outputting a first subset of candidate defect frames of the input video file for a first defect category and a second, different subset of candidate defect frames of the input video file for a second, different defect category. ### Claim 15 A system comprising: one or more electronic devices to implement a storage service to receive and store an input video file; and one or more electronic devices to implement a defect detection service, the defect detection service including instructions that upon execution cause the defect detection service to perform operations comprising: receiving an inference request for the input video file, performing an encode on the input video file to generate one or more compressed frames of the input video file by a compression of the input video file, and one or more compression features from the compression of the input video file, generating, by a machine learning model, a proper subset of candidate defect frames of the input video file based at least in part on the one or more compression features from the compression of the input video file, and determining a defect in the input video file based at least in part on the proper subset of candidate defect frames of the input video file. ### Claim 16 (depends on 15) The system of claim 15, wherein the instructions upon execution cause the defect detection service to further perform operations comprising triggering a remedial action for the input video file based at least in part on the defect. ### Claim 17 (depends on 15) The system of claim 15, wherein the instructions upon execution cause the defect detection service to further perform operations comprising determining a global defect in the input video file based at least in part on the one or more compression features. ### Claim 18 (depends on 15) The system of claim 15, wherein the instructions upon execution cause the defect detection service to further perform operations comprising sending the one or more compressed frames of the input video file to a client device when the defect is below a defect threshold. ### Claim 19 (depends on 15) The system of claim 15, wherein the instructions upon execution cause the defect detection service to perform operations wherein the generating comprises the machine learning model outputting a frame identification value and a defect probability value for each of the proper subset of candidate defect frames of the input video file. ### Claim 20 (depends on 15) The system of claim 15, wherein the instructions upon execution cause the defect detection service to perform operations wherein the generating comprises the machine learning model outputting a first subset of candidate defect frames of the input video file for a first defect category and a second, different subset of candidate defect frames of the input video file for a second, different defect category.