# US 11,443,403 B2 · Image and video processing using multiple pipelines > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,403 B2 - Document ID: US11443403B2 - Status: Granted - Kind: B2 - Application number: 17/097,399 - Filed: 2020-11-13 - Published: 2021-04-29 - Granted: 2022-09-13 - Earliest claimed: 2019-09-17 - Assignees: GoPro, Inc. - Applicants: GoPro, Inc. - Inventors: Etienne Terree; Jerome Labarthe; Romain Gounelle; Anandhakumar Chinnaiyan - Web page: https://patentgrep.com/patent/US11443403B2 ## CPC - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G06F 9/3867: {using instruction pipelines} - G06T 1/0007: {Image acquisition} - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 3/60: Rotation of whole images or parts thereof - G06T 5/70: Denoising; Smoothing - H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation} - H04N 19/436: using parallelised computational arrangements - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - H04N 23/51: Housings - H04N 23/631: {Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters} - H04N 23/6811: {based on the image signal} - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - H04N 23/815: {for controlling the resolution by using a single image} - G06T 1/60: Memory management - G09G 2340/06: Colour space transformation - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/172: the region being a picture, frame or field - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract An image captured using an image sensor is processed to determine control statistics and to produce an altered image which can be output for display or further processing at an image capture device, and the image is then processed according to the control statistics to produce a processed image which can be output for display or further processing at the image capture device or at another device. The altered image may have a lower resolution than the processed image. The image may be stored in a buffer. For example, the image may be retrieved from the buffer to produce the altered image, and again retrieved from the buffer to produce the processed image. The altered image may replace the image within the buffer. For example, the processed image may be produced to represent the altered image at a higher resolution. ## Claims ### Claim 1 An image capture device, comprising: an image sensor configured to capture an image at a first resolution; a memory configured to store instructions; and a processor configured to execute the instructions to: determine control statistics from the image; produce an altered image representative of the image at a second resolution that is lower than the first resolution; output the altered image for display or further processing; produce, according to the control statistics, a processed image representative of the image at the first resolution; and output the processed image for display or further processing. ### Claim 2 (depends on 1) The image capture device of claim 1, further comprising a display, wherein the instructions to output the altered image for display or further processing include instructions to: output the altered image to the display of the image capture device. ### Claim 3 (depends on 2) The image capture device of claim 2, wherein the altered image is output to the display of the image capture device before the processed image is output for display or further processing. ### Claim 4 (depends on 1) The image capture device of claim 1, further comprising a network interface, wherein the instructions to output the processed image for display or further processing include instructions to: encode the processed image; and transmit, using the network interface, the encoded processed image for decoding and display at a computing device. ### Claim 5 (depends on 4) The image capture device of claim 4, wherein the processed image is a video frame of a video streamed from the image capture device to the computing device. ### Claim 6 (depends on 1) The image capture device of claim 1, wherein the memory includes a buffer, and wherein the instructions include instructions to: store the image within the buffer after the image is captured; and retrieve the image from the buffer after the control statistics are determined. ### Claim 7 A method, comprising: obtaining an image captured using an image sensor; using one or more first processes to determine control statistics and to produce an altered image representing the image at a first resolution; outputting the altered image for display or further processing; using one or more second processes to produce a processed image representing the image at a second resolution according to the control statistics; and outputting the processed image for display or further processing. ### Claim 8 (depends on 7) The method of claim 7, further comprising: storing the image within a buffer after obtaining the image, wherein using the one or more first processes comprises retrieving the image from the buffer, and wherein using the one or more second processes comprises retrieving the image from the buffer. ### Claim 9 (depends on 8) The method of claim 8, wherein the processed image represents the altered image instead of the image, wherein using the one or more first processes comprises storing the altered image within the buffer to replace the image, and wherein using the one or more second processes comprises retrieving the altered image from the buffer. ### Claim 10 (depends on 8) The method of claim 8, wherein using the one or more first processes comprises storing the control statistics in the buffer, and wherein using the one or more second processes comprises retrieving the control statistics from the buffer. ### Claim 11 (depends on 7) The method of claim 7, wherein the image is a video frame of a timewarp video, wherein using the one or more first processes comprises: scaling a timing of the video frame to achieve a desired timewarp effect. ### Claim 12 (depends on 7) The method of claim 7, wherein the control statistics include information indicating motion elements of the image, and wherein using the one or more second processes comprises: performing stabilization correction against the altered image according to the information indicating the motion elements of the image. ### Claim 13 (depends on 7) The method of claim 7, wherein the image is an image of a burst image set, wherein using the one or more first processes comprises: determining quality measures for at least some of the images of the burst image set; and determining that the image has a highest quality measure of the quality measures. ### Claim 14 (depends on 7) The method of claim 7, wherein outputting the altered image for display or further processing comprises: outputting the altered image for display with zero shutter lag. ### Claim 15 An image processor, comprising: one or more first processing units configured to process an image to determine control statistics and to produce an altered image at a first resolution for display at an image capture device; and one or more second processing units configured to process the image using the control statistics to produce a processed image at a second resolution for further processing at the image capture device, wherein the second resolution is higher than the first resolution. ### Claim 16 (depends on 15) The image processor of claim 15, wherein, to process the image, the one or more first processing units are configured to determine whether the image is intended for display or encoding. ### Claim 17 (depends on 16) The image processor of claim 16, wherein the one or more first processing units are further configured to: responsive to a determination that the image is intended for encoding, perform motion estimation against at least a portion of the image. ### Claim 18 (depends on 15) The image processor of claim 15, wherein, to process the image, the one or more second processing units are configured to modify the image according to the control statistics. ### Claim 19 (depends on 18) The image processor of claim 18, wherein modifications to the image according to the control statistics include one or more of dead pixel correction, color lens shading, lateral chromatic processing, Bayer resizing, or scaling. ### Claim 20 (depends on 15) The image processor of claim 15, wherein the one or more second processing units are configured to produce the processed image based on the altered image instead of the image, wherein at least one of the one or more first processing units are configured to store the altered image in a buffer after the altered image is produced using the one or more first processing units, and wherein at least one of the one or more second processing units are configured to retrieve the altered image from the buffer before the altered image is processed using the one or more second processing units.