# US 12,015,856 B2 · Image processing for aperture size transition in a variable aperture (VA) camera > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,015,856 B2 - Document ID: US12015856B2 - Status: Granted - Kind: B2 - Application number: 17/809,712 - Filed: 2022-06-29 - Published: 2024-01-04 - Granted: 2024-06-18 - Earliest claimed: 2022-06-29 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Micha Galor Gluskin - Web page: https://patentgrep.com/patent/US12015856B2 ## CPC - H04N 23/71: Circuitry for evaluating the brightness variation - H04N 23/80: Camera processing pipelines; Components thereof - H04N 23/75: by influencing optical camera components - H04N 23/76: by influencing the image signals - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This disclosure provides systems, methods, and devices for image signal processing that support variable aperture (VA) camera operation. In a first aspect, a method of image processing includes receiving first image data from a variable aperture (VA) camera acquired at a first aperture size; determining a first set of output image frames based on the first image data; receiving second image data from the VA camera acquired during an aperture size transition; and determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames comprises adjusting a characteristic of the second image data based on the first image data. Other aspects and features are also claimed and described. ## Claims ### Claim 1 A method, comprising: receiving first image data, from a variable aperture (VA) camera, acquired at a first aperture size; determining a first set of output image frames based on the first image data; determining first image statistics for the first image data; receiving second image data, from the VA camera, acquired during an aperture size transition; determining second image statistics for the second image data; and determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames comprises adjusting a characteristic of the second image data based on the first image data, the first image statistics, and the second image statistics. ### Claim 2 (depends on 1) The method of claim 1, wherein adjusting the characteristic of the second image data comprises adjusting a brightness of the second image data based on a brightness of the first image data. ### Claim 3 (depends on 1) The method of claim 1, further comprising: processing the second image data in an image signal processor (ISP) before adjusting the characteristic of the second image data; and storing the second image data in a memory from the ISP before adjusting the characteristic of the second image data, wherein adjusting the characteristic of the second image data comprises: retrieving the second image data from the memory; modifying the second image data to adjust the characteristic of the second image data; and storing the second image data into the memory. ### Claim 4 (depends on 1) The method of claim 1, further comprising: commanding the aperture size transition before receiving the second image data; receiving third image data acquired after the aperture size transition; determining a third set of output image frames based on the third image data; and outputting the first set of output image frames, the second set of output image frames, and the third set of output image frames as a stream of output images. ### Claim 5 (depends on 4) The method of claim 4, further comprising: determining the aperture size transition is complete based on the adjusting the characteristic for an output image frame of the second set of output image frames being below a threshold adjustment. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the first image statistics for the first image data comprises: determining a plurality of statistical values for a corresponding plurality of portions of the first image data; and determining at least one outlier statistical value of the plurality of statistical values exceeds a threshold difference from a characteristic of the plurality of statistical values, wherein adjusting the characteristic of the second image data comprises: modifying the second image data based on the plurality of statistical values without the at least one outlier statistical value. ### Claim 7 (depends on 1) The method of claim 1, further comprising: determining an adjustment limit based on the first aperture size and a second aperture size after the aperture size transition, wherein adjusting the characteristic comprises: determining an adjustment to the characteristic of the second image data based on the second image data and the first image data; limiting the adjustment based on the adjustment limit; and modifying the second image data to adjust the characteristic of the second image data based on the adjustment after limiting the adjustment. ### Claim 8 (depends on 1) The method of claim 1, wherein adjusting the characteristic of the second image data comprises applying a rolling shutter correction (RSC) to a first output image frame of the second image data, the applying the RSC comprising: modifying a first portion of the second image data used for the first output image frame based on a difference in the characteristic between the first portion of the second image data and the first image data; and modifying a second portion of the second image data used for the first output image frame based on a difference in the characteristic between the second portion of the second image data and the first image data. ### Claim 9 (depends on 1) The method of claim 1, further comprising outputting the first set of output image frames and the second set of output image frames in a sequence of frames as a preview of an output of the VA camera. ### Claim 10 An apparatus, comprising: a memory storing processor-readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including: receiving first image data, from a variable aperture (VA) camera, acquired at a first aperture size; determining a first set of output image frames based on the first image data; determining first image statistics for the first image data; receiving second image data, from the VA camera, acquired during an aperture size transition; determining second image statistics for the second image data; and determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames comprises adjusting a characteristic of the second image data based on the first image data, the first image statistics, and the second image statistics. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein adjusting the characteristic of the second image data comprises adjusting a brightness of the second image data based on a brightness of the first image data. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein the at least one processor is further configured to execute the processor-readable code to cause the at least one processor to perform operations including: processing the second image data in an image signal processor (ISP) before adjusting the characteristic of the second image data; and storing the second image data in a memory from the ISP before adjusting the characteristic of the second image data, wherein adjusting the characteristic of the second image data comprises: retrieving the second image data from the memory; modifying the second image data to adjust the characteristic of the second image data; and storing the second image data into the memory. ### Claim 13 (depends on 10) The apparatus of claim 10, wherein the at least one processor is further configured to execute the processor-readable code to cause the at least one processor to perform operations including: commanding the aperture size transition before receiving the second image data; receiving third image data acquired after the aperture size transition; determining a third set of output image frames based on the third image data; and outputting the first set of output image frames, the second set of output image frames, and the third set of output image frames as a stream of output images. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the at least one processor is further configured to execute the processor-readable code to cause the at least one processor to perform operations including: determining the aperture size transition is complete based on the adjusting the characteristic for an output image frame of the second set of output image frames being below a threshold adjustment. ### Claim 15 (depends on 10) The apparatus of claim 10, wherein determining the first image statistics for the first image data comprises: determining a plurality of statistical values for a corresponding plurality of portions of the first image data; and determining at least one outlier statistical value of the plurality of statistical values exceeds a threshold difference from a characteristic of the plurality of statistical values, wherein adjusting the characteristic of the second image data comprises: modifying the second image data based on the plurality of statistical values without the at least one outlier statistical value. ### Claim 16 (depends on 10) The apparatus of claim 10, wherein the at least one processor is further configured to execute the processor-readable code to cause the at least one processor to perform operations including: determining an adjustment limit based on the first aperture size and a second aperture size after the aperture size transition, wherein adjusting the characteristic comprises: determining an adjustment to the characteristic of the second image data based on the second image data and the first image data; limiting the adjustment based on the adjustment limit; and modifying the second image data to adjust the characteristic of the second image data based on the adjustment after limiting the adjustment. ### Claim 17 (depends on 10) The apparatus of claim 10, wherein adjusting the characteristic of the second image data comprises applying a rolling shutter correction (RSC) to a first output image frame of the second image data, the applying the RSC comprising: modifying a first portion of the second image data used for the first output image frame based on a difference in the characteristic between the first portion of the second image data and the first image data; and modifying a second portion of the second image data used for the first output image frame based on a difference in the characteristic between the second portion of the second image data and the first image data. ### Claim 18 (depends on 10) The apparatus of claim 10, wherein the at least one processor is further configured to execute the processor-readable code to cause the at least one processor to perform operations including: further comprising outputting the first set of output image frames and the second set of output image frames in a sequence of frames as a preview of an output of the VA camera. ### Claim 19 A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving first image data from a variable aperture (VA) camera acquired at a first aperture size; determining a first set of output image frames based on the first image data; determining first image statistics for the first image data; receiving second image data from the VA camera acquired during an aperture size transition; determining second image statistics for the second image data; and determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames comprises adjusting a characteristic of the second image data based on the first image data, the first image statistics, and the second image statistics. ### Claim 20 (depends on 19) The non-transitory, computer-readable medium of claim 19, wherein adjusting the characteristic of the second image data comprises adjusting a brightness of the second image data based on a brightness of the first image data. ### Claim 21 (depends on 19) The non-transitory, computer-readable medium of claim 19, wherein the operations further include one or more operations of: processing the second image data in an image signal processor (ISP) before adjusting the characteristic of the second image data; and storing the second image data in a memory from the ISP before adjusting the characteristic of the second image data, wherein adjusting the characteristic of the second image data comprises: retrieving the second image data from the memory; modifying the second image data to adjust the characteristic of the second image data; and storing the second image data into the memory. ### Claim 22 (depends on 19) The non-transitory, computer-readable medium of claim 19, wherein the operations further include one or more operations of: commanding the aperture size transition before receiving the second image data; receiving third image data acquired after the aperture size transition; determining a third set of output image frames based on the third image data; and outputting the first set of output image frames, the second set of output image frames, and the third set of output image frames as a stream of output images. ### Claim 23 An apparatus, comprising: a camera comprising a variable aperture; a memory storing processor-readable code; and at least one processor coupled to the memory and to the camera, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations comprising: receiving first image data, from the camera, acquired at a first aperture size; determining a first set of output image frames based on the first image data; determining first image statistics for the first image data; receiving second image data, from the camera, acquired during an aperture size transition; determining second image statistics for the second image data; and determining a second set of output image frames based on the first image data and the second image data, wherein determining the second set of output image frames comprises adjusting a characteristic of the second image data based on the first image data, the first image statistics, and the second image statistics. ### Claim 24 (depends on 23) The apparatus of claim 23, wherein adjusting the characteristic of the second image data comprises adjusting a brightness of the second image data based on a brightness of the first image data. ### Claim 25 (depends on 23) The apparatus of claim 23, wherein the at least one processor comprises an image signal processor, and wherein the at least one processor is configured to perform operations comprising: processing the second image data in the image signal processor (ISP) before adjusting the characteristic of the second image data; and storing the second image data in the memory from the ISP before adjusting the characteristic of the second image data, wherein adjusting the characteristic of the second image data comprises: retrieving the second image data from the memory; modifying the second image data to adjust the characteristic of the second image data; and storing the second image data into the memory. ### Claim 26 (depends on 25) The apparatus of claim 25, wherein the adjusting the characteristic is performed by a second processor of the at least one processor different from the image signal processor (ISP). ### Claim 27 (depends on 23) The apparatus of claim 23, wherein the at least one processor is further configured to execute the processor-readable code to cause the at least one processor to perform operations including: commanding the aperture size transition before receiving the second image data; receiving third image data acquired after the aperture size transition; determining a third set of output image frames based on the third image data; and outputting the first set of output image frames, the second set of output image frames, and the third set of output image frames as a stream of output images.