# US 7,746,382 B2 · Video processing system and method for automatic enhancement of digital video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 7,746,382 B2
- Document ID: US7746382B2
- Status: Granted
- Kind: B2
- Application number: 11/468,541
- Filed: 2006-08-30
- Published: 2006-12-28
- Granted: 2010-06-29
- Earliest claimed: 2002-06-28
- Assignees: Microsoft Corporation
- Inventors: Andreas Soupliotis; Padmanabhan Anandan
- Web page: https://patentgrep.com/patent/US7746382B2

## CPC

- H04N 23/682: {Vibration or motion blur correction}
- G06T 5/20: using local operators
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 5/73: Deblurring; Sharpening
- H04N 5/262: Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects {; Cameras specially adapted for the electronic generation of special effects}
- H04N 7/012: {Conversion between an interlaced and a progressive signal (for graphics images G09G2310/0229)}
- H04N 23/68: for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N 23/6811: {based on the image signal}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20032: Median filtering
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T 2207/20201: Motion blur correction
- H04N 5/145: {Movement estimation (for video coding H04N19/51)}
- H04N 5/21: Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The automated video enhancement system and method includes a system and method for automatically enhancing video. The automated video enhancement method uses frame-to-frame motion estimation as the basis of the video enhancement. Motion estimation includes the computation of global motion (such as camera motion) and the computation of local motion (such as pixel motion). The automated video enhancement method includes generating global alignment transforms, generating optic flow vectors, and using these global alignment transforms and optic flow vectors to enhance the video. The invention also includes video processing and enhancement techniques that use the frame-to-frame motion estimation. These techniques include a deinterlace process, a denoise process, and a warp stabilization process that performs both damped and locked stabilization.

## Claims

### Claim 1

A method for processing a video containing a video frame sequence, comprising:
using a computing device having a processor to perform the following:
performing global motion computation to using an affine transformation to generate global alignment affine transforms; and
performing warp stabilization on the video frame sequence using the global alignment affine transforms.

### Claim 2 (depends on 1)

The method as set forth in claim 1, wherein performing warp stabilization further comprises:
discovering any blurry frames within the video frame sequence;
discarding the blurry frames;
synthesizing new frames using the global alignment affine transforms and
replacing the discarded blurry frames with the new frames.

### Claim 3 (depends on 1)

The method as set forth in claim 1, further comprising:
using the global alignment affine transforms to determine moving pixels in a video frame of the video frame sequence; and
discarding the moving pixels within the video frame.

### Claim 4 (depends on 3)

The method as set forth in claim 3, further comprising filling in the discarded moving pixels within the video frame with pixels borrowed from frames neighboring the video frame.

### Claim 5 (depends on 1)

The method as set forth in claim 1, wherein the warped stabilization is a damped stabilization, and performing the warp stabilization further comprises:
determining a curve corresponding to an actual motion of a camera that captured the video;
applying a smoothing operation to the actual camera motion to yield an intended camera motion; and
determining the difference between the intended camera motion and the actual camera motion to recover camera jitter.

### Claim 6 (depends on 1)

The method as set forth in claim 1, wherein the warped stabilization is a locked stabilization, and performing the warp stabilization further comprises:
determining a video frame from the video frame sequence to use as a reference frame; and
canceling out moving pixels in a video frame discovered by the global alignment affine transforms.

### Claim 7 (depends on 6)

The method as set forth in claim 6, wherein the reference frame is a first frame of the video frame sequence.

### Claim 8 (depends on 6)

The method as set forth in claim 6, further comprising filling in the canceled pixels by building a mosaic.

### Claim 9 (depends on 8)

The method as set forth in claim 8, wherein building a mosaic further comprises using similar pixels from nearby video frames to fill in the canceled pixels.