# US 12,010,440 B2 · Optical image stabilization movement to create a super-resolution image of a scene > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,010,440 B2 - Document ID: US12010440B2 - Status: Granted - Kind: B2 - Application number: 18/184,179 - Filed: 2023-03-15 - Published: 2023-07-13 - Granted: 2024-06-11 - Earliest claimed: 2018-08-08 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Yi Hung Chen; Chia-Kai Liang; Bartlomiej Maciej Wronski; Peyman Milanfar; Ignacio Garcia Dorado - Web page: https://patentgrep.com/patent/US12010440B2 ## CPC - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - G06T 3/4069: by subpixel displacements - H04N 23/6811: {based on the image signal} - H04N 23/6812: {based on additional sensors, e.g. acceleration sensors} - H04N 23/685: {performed by mechanical compensation} - H04N 23/687: {by shifting the lens or sensor position} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present disclosure describes systems and techniques directed to optical image stabilization movement to create a super-resolution image of a scene. The systems and techniques include a user device (102) introducing (502), through an optical image stabilization system (114), movement to one or more components of a camera system (112) of the user device (102). The user device (102) then captures (504) respective and multiple frames (306) of an image of a scene, where the respective and multiple frames (306) of the image of the scene have respective, sub-pixel offsets of the image of the scene across the multiple frames (306) as a result of the introduced movement to the one or more components of the camera system (112). The user device (102) performs (506), based on the respective, sub-pixel offsets of the image of the scene across the respective, multiple frames (306), super-resolution computations and creates (508) the super-resolution image of the scene based on the super-resolution computations. ## Claims ### Claim 1 A method to create a super-resolution image of a scene, the method performed by a user device and comprising: detecting that the user device has entered a viewfinder mode; introducing, based on detecting that the user device has entered the viewfinder mode and through an optical image stabilization system, movement to one or more components of a camera system; capturing respective and multiple frames of an image of the scene, the respective and multiple frames of the image of the scene having respective, sub-pixel offsets of the image of the scene across the multiple frames as a result of the introduced movement to the one or more components of the camera system; performing, based on the respective, sub-pixel offsets of the image of the scene across the multiple frames, super-resolution computations; and creating, based on the super-resolution computations, the super-resolution image of the scene. ### Claim 2 (depends on 1) The method as recited by claim 1, wherein introducing the movement to the one or more components of the camera system of the user device includes introducing an in-plane movement to the one or more components of the camera system, wherein the in-plane movement is a movement contained in a plane defined by two axes. ### Claim 3 (depends on 1) The method as recited by claim 1, wherein introducing the movement to the one or more components of the camera system of the user device includes introducing an out-of-plane movement to the one or more components of the camera system, wherein the out-of-plane movement is a movement outside a plane defined by two axes. ### Claim 4 (depends on 1) The method as recited by claim 1, wherein performing the super-resolution computations includes computing Gaussian radial basis function kernels and computing a robustness model. ### Claim 5 (depends on 1) The method of claim 1, wherein performing the super-resolution computations includes analyzing the influence of motion blur across the multiple frames. ### Claim 6 (depends on 1) The method of claim 1, wherein performing the super-resolution computations uses a frame-recurrent approach that uses, from the multiple frames, a previous low-resolution frame and a current low-resolution frame to create a current super-resolution image. ### Claim 7 (depends on 1) The method of claim 1, wherein additional or supplemental movement is introduced to the one or more components of the camera system based on a coverage score that is computed through a sampling of the multiple frames of the image of the scene. ### Claim 8 (depends on 1) The method as recited by claim 1, further comprising detecting a motion condition, wherein introducing the movement to one or more components of the camera system is further based on the motion condition. ### Claim 9 (depends on 8) The method as recited by claim 8, wherein the motion condition is a static motion condition, the method further comprising introducing, through the optical image stabilization system and to one or more components of the camera system, an unsynchronized movement. ### Claim 10 (depends on 8) The method as recited by claim 8, wherein the motion condition is a dynamic motion condition, the method further comprising introducing, through the optical image stabilization system and to one or more components of the camera system, a synchronized movement. ### Claim 11 (depends on 1) The method as recited by claim 1, wherein performing the super-resolution computations or creating the super-resolution image of the scene is performed at a remote server. ### Claim 12 (depends on 1) The method as recited by claim 1, wherein introducing the movement to one or more components of the camera system is further based on a coverage score of at least one frame of an image of the scene. ### Claim 13 A user device comprising: one or more processors; and a non-transitory computer-readable storage media storing instructions, that when executed by the one or more processors, perform functions comprising: detecting that the user device has entered a viewfinder mode; introducing, based on detecting that the user device has entered the viewfinder mode and through an optical image stabilization system, movement to one or more components of a camera system; capturing respective and multiple frames of an image of the scene, the respective and multiple frames of the image of the scene having respective, sub-pixel offsets of the image of the scene across the multiple frames as a result of the introduced movement to the one or more components of the camera system; performing, based on the respective, sub-pixel offsets of the image of the scene across the multiple frames, super-resolution computations; and creating, based on the super-resolution computations, a super-resolution image of a scene. ### Claim 14 (depends on 13) The user device of claim 13, wherein introducing the movement to the one or more components of the camera system of the user device includes introducing an in-plane movement to the one or more components of the camera system, wherein the in-plane movement is a movement contained in a plane defined by two axes. ### Claim 15 (depends on 13) The user device of claim 13, wherein introducing the movement to the one or more components of the camera system of the user device includes introducing an out-of-plane movement to the one or more components of the camera system, wherein the out-of-plane movement is a movement outside a plane defined by two axes. ### Claim 16 (depends on 13) The user device of claim 13, wherein performing the super-resolution computations includes computing Gaussian radial basis function kernels and computing a robustness model. ### Claim 17 (depends on 13) The user device of claim 13, wherein performing the super-resolution computations includes analyzing the influence of motion blur across the multiple frames. ### Claim 18 (depends on 13) The user device of claim 13, wherein performing the super-resolution computations uses a frame-recurrent approach that uses, from the multiple frames, a previous low-resolution frame and a current low-resolution frame to create a current super-resolution image. ### Claim 19 (depends on 13) The user device of claim 13, wherein additional or supplemental movement is introduced to the one or more components of the camera system based on a coverage score that is computed through a sampling of the multiple frames of the image of the scene. ### Claim 20 A non-transitory computer-readable storage media storing instructions, that when executed by one or more processors, perform functions comprising: detecting that a user device has entered a viewfinder mode; introducing, based on detecting that the user device has entered the viewfinder mode and through an optical image stabilization system, movement to one or more components of a camera system; capturing respective and multiple frames of an image of the scene, the respective and multiple frames of the image of the scene having respective, sub-pixel offsets of the image of the scene across the multiple frames as a result of the introduced movement to the one or more components of the camera system; performing, based on the respective, sub-pixel offsets of the image of the scene across the multiple frames, super-resolution computations; and creating, based on the super-resolution computations, a super-resolution image of a scene.