# US 9,171,577 B1 · Encoding and decoding selectively retrievable representations of video content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,171,577 B1 - Document ID: US9171577B1 - Status: Granted - Kind: B1 - Application number: 13/113,950 - Filed: 2011-05-23 - Granted: 2015-10-27 - Earliest claimed: 2003-04-25 - Assignees: GoPro, Inc. - Applicants: David A. Newman; Brian G. Schunck - Inventors: David A. Newman; Brian G. Schunck - Web page: https://patentgrep.com/patent/US9171577B1 ## CPC - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - G11B 20/00007: {Time or data compression or expansion (audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30)} - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - H04N 19/423: characterised by memory arrangements (H04N19/433 takes precedence) - H04N 19/63: using sub-band based transform, e.g. wavelets - G11B 2020/00072: {the compressed signal including a video signal} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract A system and method disposed to enable encoding, decoding and manipulation of digital video with substantially less processing load than would otherwise required. In particular, one disclosed method is directed to generating a compressed video data structure that is selectively decodable to a plurality of resolutions including the full resolution of the uncompressed stream. The desired number of data components and the content of the data components that make up the compressed video data, which determine the available video resolutions, are variable based upon the processing carried out and the resources available to decode and process the data components. During decoding, efficiency is substantially improved because only the data components necessary to generate a desired resolution are decoded. In variations, both temporal and spatial decoding are utilized to reduce frame rates, and hence, further reduce processor load. The system and method are particularly useful for real-time video editing applications. ## Claims ### Claim 1 A method for processing compressed video data, the method comprising: storing, in a storage structure, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including a base image component associated with a lowest resolution and one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution, and the base image component comprising the corresponding image at the lowest resolution; selecting a first display resolution at which to display a first frame of video based on a first processing load of a decoder; retrieving a first subset of the image components corresponding to the first frame of video from the storage structure selected based on the first display resolution, the first subset of image components including the base image component corresponding to the first frame of video; decoding the retrieved first subset of image components to generate a first modified frame of video comprising the image corresponding to the first frame of video at the first display resolution; displaying the first modified frame of video; in response to a change in processing load of the decoder to a second processing load, selecting a second display resolution at which to display a second frame of video based on the second processing load; retrieving a second subset of the image components corresponding to the second frame of video from the storage structure selected based on the second display resolution, the second subset of image components including the base image component corresponding to the second frame of video; decoding the retrieved second subset of image components to generate a second modified frame of video comprising the image corresponding to the second frame of video at the second display resolution; and displaying the second modified frame of video. ### Claim 2 (depends on 1) The method of claim 1, wherein retrieving a subset of image components comprises retrieving all image components from the storage structure which are associated with a resolution equal to or less than a selected display resolution. ### Claim 3 (depends on 1) The method of claim 1, wherein image components corresponding to a frame of video are stored in the storage structure consecutively based on an increase in the resolutions associated with the image components from the lowest resolution to the original resolution. ### Claim 4 (depends on 3) The method of claim 3, wherein retrieving a subset of image components comprises: identifying an image component associated with a selected display resolution; and retrieving image components from the storage structure consecutively from a base image component to the identified image component. ### Claim 5 (depends on 1) The method of claim 1, wherein the storage structure additionally comprises an image component index storing the locations of the image components in the storage structure, and wherein retrieving a subset of image components comprises: identifying one or more image components with associated resolutions equal to or less than a selected display resolution; identifying the one or more locations of the one or more identified image components in the storage structure using the using the image component index; and retrieving the one or more identified image components from the one or more identified locations in the storage structure. ### Claim 6 (depends on 1) The method of claim 1, wherein a retrieved subset of image components comprises all of the image components corresponding to a frame when a selected resolution comprises the original resolution of the image corresponding to the frame. ### Claim 7 (depends on 1) The method of claim 1, wherein a retrieved subset of image components comprises less than all of the image components corresponding to a frame when a selected resolution comprises a resolution less than the original resolution of the image corresponding to the frame. ### Claim 8 A system for processing compressed video data, the system comprising: a memory configured to store, in a storage structure, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including a base image component associated with a lowest resolution and one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution, and the base image component comprising the corresponding image at the lowest resolution; a decoder configured to, for each frame of video, decode a retrieved subset of image components corresponding to the frame of video to generate a modified frame of video comprising the image corresponding to the frame of video at a display resolution; a resolution module configured to: select a display resolution at which to display a first frame of video based on a processing load of the decoder; and in response to a change in processing load of the decoder to a second processing load, select a second display resolution based on the second processing load; a data component module configured to: retrieve a first subset of image components corresponding to the first frame of video from the storage structure selected based on the first display resolution, the first subset of image components including the base image component corresponding to the first frame of video; and retrieve a second subset of image components corresponding to the second frame of video from the storage structure selected based on the second display resolution, the second subset of image components including the base image component corresponding to the second frame of video; and a display configured to display modified frames of video. ### Claim 9 (depends on 8) The system of claim 8, wherein retrieving a subset of image components comprises retrieving all image components from the storage structure which are associated with a resolution equal to or less than a selected display resolution. ### Claim 10 (depends on 8) The system of claim 8, wherein image components corresponding to a frame of video are stored in the storage structure consecutively based on an increase in the resolutions associated with the image components from the lowest resolution to the original resolution. ### Claim 11 (depends on 10) The system of claim 10, wherein retrieving a subset of image components comprises: identifying an image component associated with a selected display resolution; and retrieving image components from the storage structure consecutively from a base image component to the identified image component. ### Claim 12 (depends on 8) The system of claim 8, wherein the storage structure additionally comprises an image component index storing the locations of the image components in the storage structure, and wherein retrieving a subset of image components comprises: identifying one or more image components with associated resolutions equal to or less than a selected display resolution; identifying the one or more locations of the one or more identified image components in the storage structure using the using the image component index; and retrieving the one or more identified image components from the one or more identified locations in the storage structure. ### Claim 13 (depends on 8) The system of claim 8, wherein a retrieved subset of image components comprises all of the image components corresponding to a frame when a selected resolution comprises the original resolution of the image corresponding to the frame. ### Claim 14 (depends on 8) The system of claim 8, wherein a retrieved subset of image components comprises less than all of the image components corresponding to a frame when a selected resolution comprises a resolution less than the original resolution of the image corresponding to the frame. ### Claim 15 A non-transitory computer-readable storage medium storing executable computer program instructions thereon for processing compressed video data that, in response to execution by a computing device, cause the computing device to perform operations comprising: storing, in a storage structure, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including a base image component associated with a lowest resolution and one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution, and the base image component comprising the corresponding image at the lowest resolution; selecting a first display resolution at which to display a first frame of video based on a first processing load of a decoder; retrieving a first subset of the image components corresponding to the first frame of video from the storage structure selected based on the first display resolution, the first subset of image components including the base image component corresponding to the first frame of video; decoding the retrieved first subset of image components to generate a first modified frame of video comprising the image corresponding to the first frame of video at the first display resolution; displaying the first modified frame of video; in response to a change in processing load of the decoder to a second processing load, selecting a second display resolution at which to display a second frame of video based on the second processing load; retrieving a second subset of the image components corresponding to the second frame of video from the storage structure selected based on the second display resolution, the second subset of image components including the base image component corresponding to the second frame of video; decoding the retrieved second subset of image components to generate a second modified frame of video comprising the image corresponding to the second frame of video at the second display resolution; and displaying the second modified frame of video. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein retrieving a subset of image components comprises retrieving all image components from the storage structure which are associated with a resolution equal to or less than a selected display resolution. ### Claim 17 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein image components corresponding to a frame of video are stored in the storage structure consecutively based on an increase in the resolutions associated with the image components from the lowest resolution to the original resolution. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein retrieving a subset of image components comprises: identifying an image component associated with a selected display resolution; and retrieving image components from the storage structure consecutively from a base image component to the identified image component. ### Claim 19 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the storage structure additionally comprises an image component index storing the locations of the image components in the storage structure, and wherein retrieving a subset of image components comprises: identifying one or more image components with associated resolutions equal to or less than a selected display resolution; identifying the one or more locations of the one or more identified data components in the storage structure using the using the image component index; and retrieving the one or more identified image components from the one or more identified locations in the storage structure. ### Claim 20 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein a retrieved subset of image components comprises all of the image components corresponding to a frame when a selected resolution comprises the original resolution of the image corresponding to the frame. ### Claim 21 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein a retrieved subset of image components comprises less than all of the image components corresponding to a frame when a selected resolution comprises a resolution less than the original resolution of the image corresponding to the frame.