# US 9,438,809 B2 · Methodology for generating high fidelity digital zoom for mobile phone cameras > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,438,809 B2 - Document ID: US9438809B2 - Status: Granted - Kind: B2 - Application number: 14/690,046 - Filed: 2015-04-17 - Published: 2016-01-28 - Granted: 2016-09-06 - Earliest claimed: 2014-07-25 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Hamid R. Sheikh; John W. Glotzbach; Osman G. Sezer - Web page: https://patentgrep.com/patent/US9438809B2 ## CPC - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038) - G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) - G06T 3/60: Rotation of whole images or parts thereof - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - H04N 5/262: Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects {; Cameras specially adapted for the electronic generation of special effects} - H04N 5/265: Mixing - H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes - H04N 23/6812: {based on additional sensors, e.g. acceleration sensors} - H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming - H04N 23/80: Camera processing pipelines; Components thereof - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - G06T 2200/12: involving antialiasing - G06T 2207/20024: Filtering details - G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering - G06T 2207/20212: Image combination - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A mobile device includes an embedded digital camera that is configured to capture a burst of N images. The mobile device includes processing circuitry comprising a registration module configured to, for each image within the burst of images: analyze an amount of warp of the image and generate a set of affine matrices indicating the amount of warp of the image. The processing circuitry includes a High Fidelity Interpolation block configured to, for each image within the burst of images: perform affine transformation using the set of affine matrices associated with the image, apply an aliasing-retaining interpolation filter, and implement rotation transformation and sub-pixel shifts, yielding an interpolated image. The processing circuitry includes a blending module configured to receive the N interpolated images and blend the N interpolated images into a single-blended image having a user-selected digital zoom ratio. ## Claims ### Claim 1 A non-transitory computer readable medium embodying a computer program, the computer program comprising computer readable program code that, when executed by processing circuitry, causes the processing circuitry to: capture, by a digital camera, a burst of N images; for each image within the burst of images: analyze an amount of warp of the image and generate a set of affine matrices indicating the amount of warp of the image, generate an interpolated image by: performing affine transformation using the set of affine matrices associated with the image, applying an aliasing-retaining interpolation filter, and implementing rotation transformation and sub-pixel shifts; and blend the N interpolated images into a single-blended image having a user-selected digital zoom ratio. ### Claim 2 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the computer program further comprises computer readable program code that, when executed by the processing circuitry, causes the processing circuitry to: automatically lock an auto-exposure, an auto-focus, and an auto-white/color balance of the digital camera while the burst of images is captured. ### Claim 3 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the computer program further comprises computer readable program code that, when executed by the processing circuitry, causes the processing circuitry to: generate and transmit a set of parameters to an image signal processor (ISP) of the digital camera, wherein the parameters are configured to cause the ISP to perform at least one of: reduced noise filtering, reduced pixel defect correction, and reduced interpolation aliasing cancellation. ### Claim 4 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the computer program further comprises computer readable program code that, when executed by the processing circuitry, causes the processing circuitry to: canceling aliasing by blending the N images into the single-blended image. ### Claim 5 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the computer program further comprises computer readable program code that, when executed by the processing circuitry, causes the processing circuitry to: sharpen the interpolated burst of images prior to blending the N images into the single-blended image. ### Claim 6 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the computer program further comprises computer readable program code that, when executed by the processing circuitry, causes the processing circuitry to: upscale the single-blended image by a remainder zoom ratio using a single-frame interpolation algorithm, wherein the remainder zoom ratio equals a user-selected zoom ratio divided by an upscaling zoom ratio to which the interpolated image is upscaled. ### Claim 7 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the computer program further comprises computer readable program code that, when executed by the processing circuitry, causes the processing circuitry to: select a reference frame from the burst of N images based on the reference frame having at least one of: least motion difference from other images in the burst of N images, a specified frame number, a highest sharpness. ### Claim 8 (depends on 1) The non-transitory computer readable medium of claim 1, wherein the computer program further comprises computer readable program code that, when executed by the processing circuitry, causes the processing circuitry to: receive the user-selected digital zoom ratio; based on a determination that the user-selected digital zoom ratio is above a threshold value, upscale the burst of images using a high fidelity digital zoom; based on a determination that the user-selected digital zoom ratio is above a threshold value, upscale the burst of images using a single-frame interpolation. ### Claim 9 A mobile device comprising: an embedded digital camera configured to capture a burst of N images; processing circuitry comprising: a registration module configured to, for each image within the burst of images: analyze an amount of warp of the image and generate a set of affine matrices indicating the amount of warp of the image; a High Fidelity (HiFi) Interpolation block configured to, for each image within the burst of images: perform affine transformation using the set of affine matrices associated with the image, apply an aliasing-retaining interpolation filter, and implement rotation transformation and sub-pixel shifts, yielding an interpolated image; and a blending module configured to receive the N interpolated images and blend the N interpolated images into a single-blended image having a user-selected digital zoom ratio. ### Claim 10 (depends on 9) The mobile device of claim 9, wherein the processing circuitry is configured to: automatically lock an auto-exposure, an auto-focus, and an auto-white/color balance of the embedded digital camera while the burst of images is captured. ### Claim 11 (depends on 9) The mobile device of claim 9, wherein the processing circuitry is configured to: generate and transmit a set of parameters to an image signal processor (ISP) of the embedded digital camera, where the parameters are configured to cause the ISP to perform at least one of: reduced noise filtering, reduced pixel defect correction, and reduced interpolation aliasing cancellation. ### Claim 12 (depends on 9) The mobile device of claim 9, wherein the blending module is configured to: cancel aliasing by blending the N images into the single-blended image. ### Claim 13 (depends on 9) The mobile device of claim 9, wherein the processing circuitry is configured to: sharpen the interpolated burst of images prior to blending the N images into the single-blended image. ### Claim 14 (depends on 9) The mobile device of claim 9, wherein the processing circuitry comprises: second-frame interpolation block configured to upscale the single-blended image by a remainder zoom ratio using a single-frame interpolation algorithm, wherein the remainder zoom ratio equals a user-selected zoom ratio divided by an upscaling zoom ratio of the HiFi Interpolation block. ### Claim 15 (depends on 9) The mobile device of claim 9, wherein the processing circuitry further comprises: a reference frame selection module configured to select a reference frame from the burst of N images based on the reference frame having at least one of: least motion difference from other images in the burst of N images, a specified frame number, a highest sharpness. ### Claim 16 (depends on 9) The mobile device of claim 9, wherein the processing circuitry is further configured to: receive the user-selected digital zoom ratio; based on a determination that the user-selected digital zoom ratio is above a threshold value, upscale the burst of images using a high fidelity digital zoom; based on a determination that the user-selected digital zoom ratio is above a threshold value, upscale the burst of images using a single-frame interpolation. ### Claim 17 A system comprising: processing circuitry configured to couple to a mobile device comprising an embedded digital camera configured to capture a burst of N images, the processing circuitry comprising: a registration module configured to, for each image within the burst of images: analyze an amount of warp of the image and generate a set of affine matrices indicating the amount of warp of the image; a High Fidelity (HiFi) Interpolation block configured to, for each image within the burst of images: perform affine transformation using the set of affine matrices associated with the image, apply an aliasing-retaining interpolation filter, and implement rotation transformation and sub-pixel shifts, yielding an interpolated image; and a blending module configured to receive the N interpolated images and blend the N interpolated images into a single-blended image having a user-selected digital zoom ratio. ### Claim 18 (depends on 17) The system of claim 17, wherein the processing circuitry is configured to: automatically lock an auto-exposure, an auto-focus, and an auto-white/color balance of the embedded digital camera while the burst of images is captured. ### Claim 19 (depends on 17) The system of claim 17, wherein the processing circuitry is configured to: generate and transmit a set of parameters to an image signal processor (ISP) of the embedded digital camera, where the parameters are configured to cause the ISP to perform at least one of: reduced noise filtering, reduced pixel defect correction, and reduced interpolation aliasing cancellation. ### Claim 20 (depends on 17) The system of claim 17, wherein the blending module is configured to: cancel aliasing by blending the N images into the single-blended image.