# US 10,375,310 B2 · Stabilizing video using transformation matrices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,375,310 B2 - Document ID: US10375310B2 - Status: Granted - Kind: B2 - Application number: 15/942,983 - Filed: 2018-04-02 - Published: 2018-08-09 - Granted: 2019-08-06 - Earliest claimed: 2015-10-14 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Wei Hong; Rodrigo Carceroni - Web page: https://patentgrep.com/patent/US10375310B2 ## CPC - H04N 23/68: for stable pick-up of the scene, e.g. compensating for camera body vibrations - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 5/10: using non-spatial domain filtering - G06T 5/20: using local operators - G06T 5/73: Deblurring; Sharpening - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming - H04N 23/80: Camera processing pipelines; Components thereof - G06T 2207/10016: Video; Image sequence - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In general, the subject matter can be embodied in methods, systems, and program products for identifying, by a computing system and using first and second frames of a video, a transformation that indicates movement of a camera with respect to the frames. The computing system generates a modified transformation so that the transformation is less representative of recent movement. The computing system uses the transformation and the modified transformation to generate a second transformation. The computing system identifies an anticipated distortion that would be present in a stabilized version of the second frame. The computing system determines an amount by which to reduce a stabilizing effect. The computing system applies the second transformation to the second frame to stabilize the second frame, where the stabilizing effect has been reduced based on the determined amount by which to reduce the stabilizing effect. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving, by a computing system, first and second frames of a video that was captured by a camera; generating, by the computing system and using the first and second frames of the video, a first mathematical transformation that indicates movement of the camera with respect to a scene captured by the video from when the first frame was captured to when the second frame was captured, including movement that began recently and movement that has been occurring over a longer term, the computing system generating the first mathematical transformation by identifying how features points that are present in the first frame and the second frame moved between the first frame and the second frame; identifying, by the computing system, movement of the camera with respect to the scene that has been occurring over the longer term; generating, by the computing system using the first mathematical transformation and the identified movement of the camera with respect to the scene that has been occurring over the longer term, a second mathematical transformation by a process that involves removing from the first mathematical transformation indications of the identified movement of the camera that has been occurring over the longer term, the second mathematical transformation being more representative of movement of the camera with respect to the scene that began recently and less representative of movement of the camera with respect to the scene that has been occurring over the longer term, in comparison to the first mathematical transformation; generating, by the computing system, a stabilized version of the second frame by applying the second mathematical transformation to the second frame to remove from the second frame movement that began recently and leave movement that has been occurring over the longer term, without taking into account movement of the camera with respect to the scene from future frames of the video. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the second frame is a frame of the video that immediately follows the first frame of the video. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the first mathematical transformation that indicates movement of the camera with respect to the scene comprises a homography transformation matrix. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein generating the stabilized version of the second frame includes zooming into a version of the second frame that was generated by applying the second mathematical transformation to the second frame. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising shifting a zoomed-in region of the stabilized version of the second frame horizontally or vertically to avoid the zoomed-in region of the stabilized version of second frame from presenting an invalid region. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein identifying the movement of the camera that has been occurring over the longer term includes analyzing frames of the video that the camera captured before the first and second frames of the video to identify movement that has been occurring over the longer term. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein identifying the movement of the camera that has been occurring over the longer term includes applying a lowpass filter to the first mathematical transformation based on analysis of frames of the video that the camera captured before the first and second frames of the video. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: identifying, by the computing system, an anticipated distortion that would be present in a stabilized version of the second frame resulting from application of the second mathematical transformation to the second frame, based on a difference between: (i) an amount of distortion in the horizontal direction resulting from application of the second mathematical transformation to the second frame, and (ii) an amount of distortion in the vertical direction resulting from application of the second mathematical transformation to the second frame; determining, by the computing system, an amount by which to reduce a stabilizing effect that results from application of the second mathematical transformation to the second frame, based on a degree to which the anticipated distortion exceeds an acceptable change in distortion that was calculated from distortion in multiple frames of the video that preceded the second frame, wherein generating the stabilized version of the second frame includes reducing a stabilizing effect of applying the second mathematical transformation to the second frame based on the determined amount by which to reduce the stabilizing effect. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising: calculating, by the computing system, an acceptable change in speed of movement of the camera based on analysis of a speed of movement of the camera between multiple frames of the video that preceded the second frame; and determining, by the computing system, an amount by which to reduce a stabilizing effect that results from application of the second mathematical transformation to the second frame, based on a determined speed of movement of the camera from the first frame to the second frame exceeding the acceptable change in speed of movement of the camera, wherein generating the stabilized version of the second frame includes reducing a stabilizing effect of applying the second mathematical transformation to the second frame based on the determined amount by which to reduce the stabilizing effect. ### Claim 10 One or more non-transitory computer-readable devices including instructions that, when executed by one or more processors, cause performance of operations that include: receiving, by a computing system, first and second frames of a video that was captured by a camera; generating, by the computing system and using the first and second frames of the video, a first mathematical transformation that indicates movement of the camera with respect to a scene captured by the video from when the first frame was captured to when the second frame was captured, including movement that began recently and movement that has been occurring over a longer term, the computing system generating the first mathematical transformation by identifying how features points that are present in the first frame and the second frame moved between the first frame and the second frame; identifying, by the computing system, movement of the camera with respect to the scene that has been occurring over the longer term; generating, by the computing system using the first mathematical transformation and the identified movement of the camera with respect to the scene that has been occurring over the longer term, a second mathematical transformation by a process that involves removing from the first mathematical transformation indications of the identified movement of the camera that has been occurring over the longer term, the second mathematical transformation being more representative of movement of the camera with respect to the scene that began recently and less representative of movement of the camera with respect to the scene that has been occurring over the longer term, in comparison to the first mathematical transformation; generating, by the computing system, a stabilized version of the second frame by applying the second mathematical transformation to the second frame to remove from the second frame movement that began recently and leave movement that has been occurring over the longer term, without taking into account movement of the camera with respect to the scene from future frames of the video. ### Claim 11 (depends on 10) The one or more non-transitory computer-readable devices of claim 10, wherein the second frame is a frame of the video that immediately follows the first frame of the video. ### Claim 12 (depends on 10) The one or more non-transitory computer-readable devices of claim 10, wherein the first mathematical transformation that indicates movement of the camera with respect to the scene comprises a homography transformation matrix. ### Claim 13 (depends on 10) The one or more non-transitory computer-readable devices of claim 10, wherein generating the stabilized version of the second frame includes zooming into a version of the second frame that was generated by applying the second mathematical transformation to the second frame. ### Claim 14 (depends on 13) The one or more non-transitory computer-readable devices of claim 13, wherein the operations further comprise shifting a zoomed-in region of the stabilized version of the second frame horizontally or vertically to avoid the zoomed-in region of the stabilized version of second frame from presenting an invalid region. ### Claim 15 (depends on 10) The one or more non-transitory computer-readable devices of claim 10, wherein identifying the movement of the camera that has been occurring over the longer term includes analyzing frames of the video that the camera captured before the first and second frames of the video to identify movement that has been occurring over the longer term. ### Claim 16 (depends on 10) The one or more non-transitory computer-readable devices of claim 10, wherein identifying the movement of the camera that has been occurring over the longer term includes applying a lowpass filter to the first mathematical transformation based on analysis of frames of the video that the camera captured before the first and second frames of the video. ### Claim 17 (depends on 10) The one or more non-transitory computer-readable devices of claim 10, wherein the operations further comprise: identifying, by the computing system, an anticipated distortion that would be present in a stabilized version of the second frame resulting from application of the second mathematical transformation to the second frame, based on a difference between: (i) an amount of distortion in the horizontal direction resulting from application of the second mathematical transformation to the second frame, and (ii) an amount of distortion in the vertical direction resulting from application of the second mathematical transformation to the second frame; determining, by the computing system, an amount by which to reduce a stabilizing effect that results from application of the second mathematical transformation to the second frame, based on a degree to which the anticipated distortion exceeds an acceptable change in distortion that was calculated from distortion in multiple frames of the video that preceded the second frame, wherein generating the stabilized version of the second frame includes reducing a stabilizing effect of applying the second mathematical transformation to the second frame based on the determined amount by which to reduce the stabilizing effect. ### Claim 18 (depends on 10) The one or more non-transitory computer-readable devices of claim 10, wherein the operations further comprise: calculating, by the computing system, an acceptable change in speed of movement of the camera based on analysis of a speed of movement of the camera between multiple frames of the video that preceded the second frame; and determining, by the computing system, an amount by which to reduce a stabilizing effect that results from application of the second mathematical transformation to the second frame, based on a determined speed of movement of the camera from the first frame to the second frame exceeding the acceptable change in speed of movement of the camera, wherein generating the stabilized version of the second frame includes reducing a stabilizing effect of applying the second mathematical transformation to the second frame based on the determined amount by which to reduce the stabilizing effect.