# US 9,438,799 B2 · Compression and decoding of single sensor color image data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,438,799 B2 - Document ID: US9438799B2 - Status: Granted - Kind: B2 - Application number: 14/676,776 - Filed: 2015-04-01 - Published: 2015-10-01 - Granted: 2016-09-06 - Earliest claimed: 2006-03-22 - Assignees: GoPro, Inc. - Applicants: GoPro, Inc. - Inventors: David A. Newman - Web page: https://patentgrep.com/patent/US9438799B2 ## CPC - H04N 19/186: the unit being a colour or a chrominance component - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - H04N 1/648: {Transmitting or storing the primary (additive or subtractive) colour signals; Compression thereof (H04N1/642 - H04N1/646 take precedence)} - H04N 5/265: Mixing - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method is described to greatly improve the efficiency of and reduce the complexity of image compression when using single-sensor color imagers for video acquisition. The method in addition allows for this new image compression type to be compatible with existing video processing tools, improving the workflow for film and television production. ## Claims ### Claim 1 A method for previewing captured image data, the method comprising: capturing, by an image sensor of a camera, image data, the image data comprising a plurality of image planes; encoding, by an image processor, the captured image data into a plurality of encoded image planes, each encoded image plane representative of one or more of the image planes of the image data, such that all of the encoded image planes form an image at a first resolution and a subset of less than all of the encoded image planes form an image at a second resolution less than the first resolution; and providing, to an external display, the subset of the encoded image planes for a substantially real-time preview of the image at the second resolution, the external display configured to decode the subset of encoded image planes to obtain the image at the second resolution. ### Claim 2 (depends on 1) The method of claim 1, wherein the second resolution comprises a resolution that is approximately one-quarter the resolution of the first resolution. ### Claim 3 (depends on 1) The method of claim 1, wherein the plurality of encoded image planes comprises a first encoded image plane representative of a first color and a second encoded image plane representative of the first color, and wherein the subset of encoded image planes comprises only one of the first encoded image plane and the second encoded image plane. ### Claim 4 (depends on 1) The method of claim 1, wherein decoding the subset of encoded image planes comprises: decoding one or more of the subset of encoded image planes to produce a subset of the plurality of image planes comprising less than all of the plurality of image planes; and combining the subset of the plurality of image planes to produce the image at the second resolution. ### Claim 5 (depends on 1) The method of claim 1, wherein each pixel of the image at the second resolution comprises a red sub-pixel, a blue sub-pixel, and a green sub-pixel. ### Claim 6 (depends on 1) The method of claim 1, wherein each pixel of the image at the first resolution comprises a red sub-pixel, a blue sub-pixel, a first green sub-pixel, and a second green sub-pixel. ### Claim 7 (depends on 1) The method of claim 1, wherein the external display is configured to display the image at the second resolution. ### Claim 8 (depends on 1) The method of claim 1, further comprising: in response to the preview of the image at the second resolution, receiving an editing instruction from a user. ### Claim 9 A system for previewing captured image data, the system comprising: an image sensor configured to capture image data, the image data comprising a plurality of image planes; an image processor configured to encode the captured image data into a plurality of encoded image planes, each encoded image plane representative of one or more of the image planes of the image data, such that all of the encoded image planes form an image at a first resolution and a subset of less than all of the encoded image planes form an image at a second resolution less than the first resolution; an output to provide, to an external display, the subset of the encoded image planes for a substantially real-time preview of the image at the second resolution, the external display configured to decode the subset of encoded image planes to obtain the image at the second resolution. ### Claim 10 (depends on 9) The system of claim 9, wherein the second resolution comprises a resolution that is approximately one-quarter the resolution of the first resolution. ### Claim 11 (depends on 9) The system of claim 9, wherein the plurality of encoded image planes comprises a first encoded image plane representative of a first color and a second encoded image plane representative of the first color, and wherein the subset of encoded image planes comprises only one of the first encoded image plane and the second encoded image plane. ### Claim 12 (depends on 9) The system of claim 9, wherein decoding the subset of encoded image planes comprises: decoding one or more of the subset of encoded image planes to produce a subset of the plurality of image planes comprising less than all of the plurality of image planes; and combining the subset of the plurality of image planes to produce the image at the second resolution. ### Claim 13 (depends on 9) The system of claim 9, wherein each pixel of the image at the second resolution comprises a red sub-pixel, a blue sub-pixel, and a green sub-pixel. ### Claim 14 (depends on 9) The system of claim 9, wherein each pixel of the image at the first resolution comprises a red sub-pixel, a blue sub-pixel, a first green sub-pixel, and a second green sub-pixel. ### Claim 15 (depends on 9) The system of claim 9, wherein the external display is configured to display the image at the second resolution. ### Claim 16 (depends on 9) The system of claim 9, further comprising: an input configured to, in response to the preview of the image at the second resolution, receive an editing instruction from a user. ### Claim 17 (depends on 1) The method of claim 1, wherein the image data comprises a set of pixels, and wherein each of the plurality of image planes comprises a subset of the set of pixels. ### Claim 18 (depends on 17) The method of claim 17, wherein each of one or more of the plurality of image planes comprises approximately one quarter of the pixels in the set of pixels. ### Claim 19 (depends on 17) The method of claim 17, wherein each of one or more of the plurality of image planes comprises approximately one half of the pixels in the set of pixels. ### Claim 20 (depends on 17) The method of claim 17, wherein each of one or more of the plurality of image planes comprises pixels corresponding to one color. ### Claim 21 (depends on 17) The method of claim 17, wherein each of one or more of the plurality of image planes comprises pixels corresponding to two or more colors. ### Claim 22 (depends on 1) The method of claim 1, wherein the image data includes metadata describing camera settings for the camera. ### Claim 23 (depends on 8) The method of claim 8, wherein the received editing instruction is applied to the image at the first resolution. ### Claim 24 (depends on 1) The method of claim 1, wherein decoding the subset of encoded image planes to obtain the image at the second resolution comprises bypassing de-Bayer filtering. ### Claim 25 (depends on 1) The method of claim 1, wherein the second resolution comprises a resolution that is less than half the resolution of the first resolution. ### Claim 26 (depends on 1) The method of claim 1, wherein the image data comprises video image data. ### Claim 27 (depends on 26) The method of claim 26, wherein the plurality of encoded image planes comprises one or more first encoded image planes and one or more second encoded image planes, wherein each of the first encoded image planes includes more image data than each of the second encoded image planes. ### Claim 28 (depends on 27) The method of claim 27, wherein the subset of encoded image planes comprises only one of the one or more first encoded image planes. ### Claim 29 (depends on 27) The method of claim 27, wherein the subset of encoded image planes comprises only one of the one or more second encoded image planes. ### Claim 30 (depends on 1) The method of claim 1, wherein the external display is configured to display the image at the second resolution, and further comprising: in response to the preview of the image at the second resolution, receiving an editing instruction from a user. ### Claim 31 (depends on 30) The method of claim 30, wherein the received editing instruction is applied to the image at the first resolution. ### Claim 32 (depends on 30) The method of claim 30, wherein the image data includes metadata describing camera settings for the camera. ### Claim 33 (depends on 9) The system of claim 9, wherein the image data comprises a set of pixels, and wherein each of the plurality of image planes comprises a subset of the set of pixels. ### Claim 34 (depends on 33) The system of claim 33, wherein each of one or more of the plurality of image planes comprises approximately one quarter of the pixels in the set of pixels. ### Claim 35 (depends on 33) The system of claim 33, wherein each of one or more of the plurality of image planes comprises approximately one half of the pixels in the set of pixels. ### Claim 36 (depends on 33) The system of claim 33, wherein each of one or more of the plurality of image planes comprises pixels corresponding to one color. ### Claim 37 (depends on 33) The system of claim 33, wherein each of one or more of the plurality of image planes comprises pixels corresponding to two or more colors. ### Claim 38 (depends on 9) The system of claim 9, wherein the image data includes metadata describing settings for the system. ### Claim 39 (depends on 16) The system of claim 16, wherein the received editing instruction is applied to the image at the first resolution. ### Claim 40 (depends on 9) The system of claim 9, wherein decoding the subset of encoded image planes to obtain the image at the second resolution comprises bypassing de-Bayer filtering. ### Claim 41 (depends on 9) The system of claim 9, wherein the second resolution comprises a resolution that is less than half the resolution of the first resolution. ### Claim 42 (depends on 9) The system of claim 9, wherein the image data comprises video image data. ### Claim 43 (depends on 42) The system of claim 42, wherein the plurality of encoded image planes comprises one or more first encoded image planes and one or more second encoded image planes, wherein each of the first encoded image planes includes more image data than each of the second encoded image planes. ### Claim 44 (depends on 43) The system of claim 43, wherein the subset of encoded image planes comprises only one of the one or more first encoded image planes. ### Claim 45 (depends on 43) The system of claim 43, wherein the subset of encoded image planes comprises only one of the one or more second encoded image planes. ### Claim 46 (depends on 9) The system of claim 9, wherein the external display is configured to display the image at the second resolution, and further comprising: an input configured to, in response to the preview of the image at the second resolution, receive an editing instruction from a user. ### Claim 47 (depends on 46) The system of claim 46, wherein the received editing instruction is applied to the image at the first resolution. ### Claim 48 (depends on 46) The system of claim 46, wherein the image data includes metadata describing camera settings for the camera. ### Claim 49 A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a hardware processor, cause a camera system to perform steps comprising: capturing, by an image sensor of a camera system, image data, the image data comprising a plurality of image planes; encoding, by an image processor of a camera system, the captured image data into a plurality of encoded image planes, each encoded image plane representative of one or more of the image planes of the image data, such that all of the encoded image planes form an image at a first resolution and a subset of less than all of the encoded image planes form an image at a second resolution less than the first resolution; and providing, to an external display, the subset of the encoded image planes for a substantially real-time preview of the image at the second resolution, the external display configured to decode the subset of encoded image planes to obtain the image at the second resolution. ### Claim 50 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein the second resolution comprises a resolution that is approximately one-quarter the resolution of the first resolution. ### Claim 51 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein the plurality of encoded image planes comprises a first encoded image plane representative of a first color and a second encoded image plane representative of the first color, and wherein the subset of encoded image planes comprises only one of the first encoded image plane and the second encoded image plane. ### Claim 52 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein decoding the subset of encoded image planes comprises: decoding one or more of the subset of encoded image planes to produce a subset of the plurality of image planes comprising less than all of the plurality of image planes; and combining the subset of the plurality of image planes to produce the image at the second resolution. ### Claim 53 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein each pixel of the image at the second resolution comprises a red sub-pixel, a blue sub-pixel, and a green sub-pixel. ### Claim 54 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein each pixel of the image at the first resolution comprises a red sub-pixel, a blue sub-pixel, a first green sub-pixel, and a second green sub-pixel. ### Claim 55 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein the external display is configured to display the image at the second resolution. ### Claim 56 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein the instructions, when executed by the hardware processor, cause the camera system to perform further steps comprising: in response to the preview of the image at the second resolution, receiving an editing instruction from a user. ### Claim 57 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein the image data comprises a set of pixels, and wherein each of the plurality of image planes comprises a subset of the set of pixels. ### Claim 58 (depends on 57) The non-transitory computer-readable storage medium of claim 57, wherein each of one or more of the plurality of image planes comprises approximately one quarter of the pixels in the set of pixels. ### Claim 59 (depends on 57) The non-transitory computer-readable storage medium of claim 57, wherein each of one or more of the plurality of image planes comprises approximately one half of the pixels in the set of pixels. ### Claim 60 (depends on 57) The non-transitory computer-readable storage medium of claim 57, wherein each of one or more of the plurality of image planes comprises pixels corresponding to one color. ### Claim 61 (depends on 57) The non-transitory computer-readable storage medium of claim 57, wherein each of one or more of the plurality of image planes comprises pixels corresponding to two or more colors. ### Claim 62 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein the image data includes metadata describing camera settings for the camera system. ### Claim 63 (depends on 62) The non-transitory computer-readable storage medium of claim 62, wherein the received editing instruction is applied to the image at the first resolution. ### Claim 64 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein decoding the subset of encoded image planes to obtain the image at the second resolution comprises bypassing de-Bayer filtering. ### Claim 65 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein the second resolution comprises a resolution that is less than half the resolution of the first resolution. ### Claim 66 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein the image data comprises video image data. ### Claim 67 (depends on 66) The non-transitory computer-readable storage medium of claim 66, wherein the plurality of encoded image planes comprises one or more first encoded image planes and one or more second encoded image planes, wherein each of the first encoded image planes includes more image data than each of the second encoded image planes. ### Claim 68 (depends on 67) The non-transitory computer-readable storage medium of claim 67, wherein the subset of encoded image planes comprises only one of the one or more first encoded image planes. ### Claim 69 (depends on 67) The non-transitory computer-readable storage medium of claim 67, wherein the subset of encoded image planes comprises only one of the one or more second encoded image planes. ### Claim 70 (depends on 49) The non-transitory computer-readable storage medium of claim 49, wherein the external display is configured to display the image at the second resolution, and wherein the instructions, when executed by the hardware processor, cause the camera system to perform further steps comprising: in response to the preview of the image at the second resolution, receiving an editing instruction from a user. ### Claim 71 (depends on 70) The non-transitory computer-readable storage medium of claim 70, wherein the received editing instruction is applied to the image at the first resolution. ### Claim 72 (depends on 70) The non-transitory computer-readable storage medium of claim 70, wherein the image data includes metadata describing camera settings for the camera.