# US 2021/0241422 A1 · Image processing for reducing artifacts caused by removal of scene elements from images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241422 A1
- Document ID: US20210241422A1
- Status: Published application
- Kind: A1
- Application number: 17/018,960
- Filed: 2020-09-11
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: Weta Digital Limited
- Applicants: Weta Digital Limited
- Inventors: Sebastian R. F. Burke, III; Pravin P. Bhat
- Web page: https://patentgrep.com/patent/US20210241422A1

## CPC

- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An image processor eliminates a character or object from a sequence of frames and then merges the resulting images with those of nearby frames, both preceding and succeeding, to synthesize the background of the sequence of frames.

## Claims

### Claim 1

A computer-implemented method, for processing images, comprising:
determining, for a current frame of a plurality of frames, a pixel of interest, the pixel of interest to be replaced;
obtaining, from the plurality of frames, a set of nearby frames;
identifying, from the set of nearby frames, a set of pixels corresponding to the pixel of interest, wherein candidate pixels of the set of pixels are determined based on a geometric alignment of the set of nearby frames with the current frame;
determining a distribution for the set of pixels;
as a result of determining the distribution, determining that the distribution is within a threshold;
as a result of determining that the distribution is within the threshold, determining a replacement pixel; and
generating a stitched image from the current frame, wherein the pixel of interest is replaced with the replacement pixel.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the distribution is weighted.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the distribution is Gaussian.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the distribution is uniform.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining, for a second image, a spatially-varying gain, wherein the spatially-varying gain varies over pixels of the second image.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the geometric alignment comprises processing a camera projection matrix for each frame.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the camera projection matrix is obtained from a LIDAR scan of a scene.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the geometric alignment comprises aligning using optical flow and warping the frames into position.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the geometric alignment comprises using projections based on the geometry and camera parameters.

### Claim at position 10

-20. (canceled)

### Claim 21

A computer system for processing digital video, comprising:
at least one processor; and
a computer-readable medium storing instructions, which when executed by the at least one processor, causes the computer system to:
(a) determine, for a current frame of a plurality of frames, a pixel of interest, the pixel of interest to be replaced;
(b) obtain, from the plurality of frames, a set of nearby frames;
(c) identify, from the set of nearby frames, a set of pixels corresponding to the pixel of interest;
(d) determine a distribution for the set of pixels;
(e) as a result of determining the distribution, determine that the distribution is within a threshold;
(f) as a result of determining that the distribution is within the threshold, determine a replacement pixel; and
(g) generate a stitched image from the current frame, wherein the pixel of interest is replaced with the replacement pixel.

### Claim 22

A non-transitory computer-readable storage medium storing instructions, which when executed by at least one processor of a computer system, causes the computer system to:
(a) determine, for a current frame of a plurality of frames, a pixel of interest, the pixel of interest to be replaced;
(b) obtain, from the plurality of frames, a set of nearby frames;
(c) identify, from the set of nearby frames, a set of pixels corresponding to the pixel of interest;
(d) determine a distribution for the set of pixels;
(e) as a result of determining the distribution, determine that the distribution is within a threshold;
(f) as a result of determining that the distribution is within the threshold, determine a replacement pixel; and
(g) generate a stitched image from the current frame, wherein the pixel of interest is replaced with the replacement pixel.

### Claim 23 (depends on 1)

A carrier medium carrying image data that includes pixel values generated according to the method of claim 1.