# US 10,771,697 B2 · Still image stabilization/optical image stabilization synchronization in multi-camera image capture > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,697 B2 - Document ID: US10771697B2 - Status: Granted - Kind: B2 - Application number: 15/696,583 - Filed: 2017-09-06 - Published: 2018-03-08 - Granted: 2020-09-08 - Earliest claimed: 2016-09-06 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Paul M. Hubel; Marius Tico; Ting Chen - Web page: https://patentgrep.com/patent/US10771697B2 ## CPC - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/70: Denoising; Smoothing - G06T 7/0002: {Inspection of images, e.g. flaw detection} - H04N 23/6811: {based on the image signal} - H04N 23/6812: {based on additional sensors, e.g. acceleration sensors} - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - H04N 23/80: Camera processing pipelines; Components thereof - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - G06T 2207/10144: Varying exposure - G06T 2207/20221: Image fusion; Image merging - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - H04N 23/71: Circuitry for evaluating the brightness variation - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Techniques are disclosed for managing image capture and processing in a multi-camera imaging system. In such a system, a pair of cameras each may output a sequence of frames representing captured image data. The cameras' output may be synchronized to each other to cause synchronism in the image capture operations of the cameras. The system may assess image quality of frames output from the cameras and, based on the image quality, designate a pair of the frames to serve as a “reference frame pair.” Thus, one frame from the first camera and a paired frame from the second camera will be designated as the reference frame pair. The system may adjust each reference frame in the pair using other frames from their respective cameras. The reference frames also may be processed by other operations within the system, such as image fusion. ## Claims ### Claim 1 A method, comprising: assessing quality of a sequence of paired images output by a plurality of cameras, a first image in each pair output by a first camera and a second image in each pair output by a second camera; selecting a camera of the plurality of cameras based on a parameter of the camera; selecting a frame pair based on the assessed quality of frames from the selected camera; performing image correction on a first image from the selected pair using other images output from the first image's associated camera; and merging the corrected first image from the selected pair and a second image from the selected pair into a composite image. ### Claim 2 (depends on 1) The method of claim 1, further comprising, prior to the merging, performing image correction on the second image in the selected pair using other images output from the second image's associated camera. ### Claim 3 (depends on 1) The method of claim 1, wherein images output from one of the cameras are generated according to variable exposure times. ### Claim 4 (depends on 1) The method of claim 1, wherein images output from one of the cameras are generated according to a common exposure time. ### Claim 5 (depends on 1) The method of claim 1, wherein images output from one of the cameras are generated according to a predetermined exposure cadence. ### Claim 6 (depends on 1) The method of claim 1, wherein, when one of the images is generated according to an exposure time that exceeds a predetermined duration, the one image is disqualified from being selected. ### Claim 7 (depends on 1) The method of claim 1, wherein the quality assessment includes blur estimation. ### Claim 8 (depends on 1) The method of claim 1, wherein the quality assessment includes sharpness estimation. ### Claim 9 (depends on 1) The method of claim 1, wherein the quality assessment includes brightness estimation. ### Claim 10 (depends on 1) The method of claim 1, wherein the quality assessment is derived from motion sensor data associated with image capture events of the cameras. ### Claim 11 (depends on 1) The method of claim 1, wherein the merging comprises fusing content of the frames of the selected frame pair together. ### Claim 12 (depends on 1) The method of claim 1, wherein the merging comprises comparing content of the frames of the selected frame pair. ### Claim 13 (depends on 1) The method of claim 1, wherein the image correction is a denoising technique comprising: comparing the content of the first image in the selected pair with the other images output from the first image's associated camera to identify content in the first image in the selected pair that is likely noise; and removing the noise in the identified content. ### Claim 14 (depends on 1) The method of claim 1, wherein the image correction is a denoising technique comprising: comparing the content of the first image in the selected pair with the other images output from the first image's associated camera to select parameters of a denoising filter; and applying the denoising filter with the selected parameters to the first image in the selected pair. ### Claim 15 (depends on 1) The method of claim 1, wherein the image correction is a denoising technique comprising: upsamping the first frame by increasing the resolution of the first frame using content from the other images output from the first image's associated camera. ### Claim 16 (depends on 1) The method of claim 1, wherein the operational parameter is based on image stabilization assessments of the paired images. ### Claim 17 (depends on 16) The method of claim 16, wherein the first camera and second camera are mounted within the same device, and wherein the image stabilization assessments of the paired images are derived from motion sensor data for the device and associated with each pair of the images. ### Claim 18 (depends on 1) The method of claim 1, wherein the selecting of a frame pair is based on the assessed quality of only the images output by the selected camera and not based on the image output by the non-selected camera. ### Claim 19 (depends on 1) The method of claim 1, wherein the selecting of the frame pair is based on the similarity in the parameter between first camera and the second camera used when capturing the frame pair. ### Claim 20 (depends on 1) The method of claim 1, wherein the selecting of the frame pair is based on the similarity in an exposure time between first camera and the second camera used when capturing the frame pair. ### Claim 21 (depends on 1) The method of claim 1, wherein the parameter is a level of zoom. ### Claim 22 (depends on 1) The method of claim 1, wherein the parameter is an operator control. ### Claim 23 (depends on 1) The method of claim 1, wherein the selecting of a frame pair is based on the assessed quality of the images output by both the first and second cameras. ### Claim 24 Apparatus, comprising: a plurality of cameras having synchronized image capture operations; an image processor to: assess quality of a sequence of paired images output by the cameras; select a camera of the plurality of cameras based on a parameter of the selected camera; select a frame pair based on the assessed quality of the frames from the selected camera; perform image correction on a first image in the selected pair using other images output from the first image's associated camera; and merge the corrected first image from the selected pair and a second image from the selected pair into a composite image. ### Claim 25 (depends on 24) The apparatus of claim 24, wherein, prior to the merging, the image processor performs image correction on the second image in the selected pair using other images output from the second image's associated camera. ### Claim 26 (depends on 24) The apparatus of claim 24, wherein at least one of the cameras operate according to a variable exposure cadence. ### Claim 27 (depends on 24) The apparatus of claim 24, wherein at least one of the cameras operate according to a common exposure time. ### Claim 28 (depends on 24) The apparatus of claim 24, wherein, when one of the images is generated according to an exposure time that exceeds a predetermined duration, the image processor disqualifies the one image from being selected. ### Claim 29 (depends on 24) The apparatus of claim 24, wherein the quality assessment includes blur estimation. ### Claim 30 (depends on 24) The apparatus of claim 24, wherein the quality assessment includes sharpness estimation. ### Claim 31 (depends on 24) The apparatus of claim 24, wherein the quality assessment includes brightness estimation. ### Claim 32 (depends on 24) The apparatus of claim 24, further comprising a motion sensor, wherein the quality assessment applies motion sensor data associated with image capture events of the cameras. ### Claim 33 (depends on 24) The apparatus of claim 24, further comprising an ambient light sensor, wherein the quality assessment applies light sensor data associated with image capture events of the cameras. ### Claim 34 Non-transitory computer readable medium storing program instructions that, when executed by a processing device, cause the device to: assess quality of a sequence of image pairs output by synchronized cameras, a first image in each pair output by a first camera and a second image in each pair output by a second camera; select a camera of the plurality of cameras based on a parameter of the selected camera; select a frame pair based on the assessed quality of frames from the selected camera; perform image correction on the first image in the selected pair using other images output from the first image's associated camera; and merge the corrected first image from the selected pair and a second image from the selected pair into a composite image. ### Claim 35 (depends on 34) The medium of claim 34, wherein the program instructions further cause the processing device to, prior to the merging, perform image correction on the second image in the selected pair using other images output from the second image's associated camera.