# US 11,113,790 B2 · Adding greater realism to a computer-generated image by smoothing jagged edges

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,790 B2
- Document ID: US11113790B2
- Status: Granted
- Kind: B2
- Application number: 16/363,927
- Filed: 2019-03-25
- Published: 2019-10-17
- Granted: 2021-09-07
- Earliest claimed: 2018-04-12
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Adam Christopher Marrs; Josef B. Spjut; Holger Heinrich Gruen; Morgan McGuire; Rahul Sathe
- Web page: https://patentgrep.com/patent/US11113790B2

## CPC

- G06T 5/70: Denoising; Smoothing
- G06T 15/005: {General purpose rendering architectures}
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/50: Depth or shape recovery
- G06T 15/06: Ray-tracing
- G06T 15/503: {Blending, e.g. for anti-aliasing}
- G06T 2200/28: involving image processing hardware
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T 2207/30168: Image quality inspection
- G06T 2210/08: Bandwidth reduction
- G06T 2210/21: Collision detection, intersection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

During the rendering of an image, specific pixels in the image are identified where antialiasing would be helpful. Antialiasing is then performed on these identified pixels, where anti-aliasing is a technique used to add greater realism to a digital image by smoothing jagged edges. This reduces a cost of performing antialiasing by reducing a number of pixels within an image on which antialiasing is performed.

## Claims

### Claim 1

A method comprising:
applying temporal antialiasing (TAA) to an image, wherein for each pixel of at least a subset of pixels, the TAA compares a historical value for the pixel in a previous frame with current values of neighboring pixels to the pixel in a current frame;
identifying failure pixels resulting from applying the TAA to the image, where each pixel of the at least a subset of pixels with the historical value that differs from the current values of the neighboring pixels by more than a predetermined amount is identified as a failure pixel; and
performing antialiasing on each of the failure pixels, utilizing a combination of ray tracing and fast approximate anti-aliasing (FXAA).

### Claim 2 (depends on 1)

The method of claim 1, wherein the failure pixels are identified utilizing results of applying the TAA to the image, utilizing intermediate data, or utilizing source geometry.

### Claim 3 (depends on 1)

The method of claim 1, wherein the image is created utilizing forward rendering or deferred rendering.

### Claim 4 (depends on 1)

The method of claim 1, wherein each pixel of the at least a subset of pixels having a depth value for a pixel that differs from depth values for neighboring pixels by more than a predetermined amount is identified as a failure pixel.

### Claim 5 (depends on 1)

The method of claim 1, wherein each pixel of the at least a subset of pixels having a luminance value for a pixel that differs from luminance values for neighboring pixels by more than a predetermined amount is identified as a failure pixel.

### Claim 6 (depends on 1)

The method of claim 1, wherein the failure pixels are marked using a sentinel value.

### Claim 7 (depends on 1)

The method of claim 1, wherein the failure pixels are included within a segmentation mask.

### Claim 8 (depends on 1)

The method of claim 1, wherein performing the antialiasing includes determining point-sampling visibility for a sub-pixel area within each of the failure pixels, utilizing GPU ray tracing.

### Claim 9 (depends on 1)

The method of claim 1, wherein results of performing the antialiasing on the failure pixels are combined with the results of applying the TAA to the image to create an output image.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
identifying one or more pixels for which no temporal information exists; and
performing antialiasing on the identified one or more pixels, utilizing fast approximate anti-aliasing (FXAA).

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
creating an output image by combining results of applying the TAA to the image, results of performing the antialiasing on the failure pixels, and results of performing the antialiasing on the one or more pixels for which no temporal information exists.

### Claim 12 (depends on 1)

The method of claim 1, wherein results of applying the TAA to the image includes a color buffer for a second set of pixels of the plurality of pixels, and wherein results of performing the antialiasing on the failure pixels includes textures for a first set of pixels of the plurality of pixels that include the failure pixels.

### Claim 13 (depends on 12)

The method of claim 12, wherein a portion of the color buffer associated with the failure pixels is discarded.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
creating an output image by blending the textures with the color buffer.

### Claim 15

A system comprising:
a processor that is configured to:
apply temporal antialiasing (TAA) to an image, wherein for each pixel of at least a subset of pixels, the TAA compares a historical value for the pixel in a previous frame with current values of neighboring pixels to the pixel in a current frame;
identify failure pixels resulting from applying the TAA to the image, where each pixel of the at least a subset of pixels with the historical value that differs from the current values of the neighboring pixels by more than a predetermined amount is identified as a failure pixel; and
perform antialiasing on each of the failure pixels, utilizing a combination of ray tracing and fast approximate anti-aliasing (FXAA); and
a memory that is configured to store results of the antialiasing.

### Claim 16 (depends on 15)

The system of claim 15, wherein the failure pixels are identified utilizing results of applying the TAA to the image, utilizing intermediate data, or utilizing source geometry.

### Claim 17 (depends on 15)

The system of claim 15, wherein the image is created utilizing forward rendering or deferred rendering.

### Claim 18 (depends on 15)

The system of claim 15, wherein each pixel of the at least a subset of pixels having a depth value for a pixel that differs from depth values for neighboring pixels by more than a predetermined amount is identified as a failure pixel.

### Claim 19 (depends on 15)

The system of claim 15, wherein each pixel of the at least a subset of pixels having a luminance value for a pixel that differs from luminance values for neighboring pixels by more than a predetermined amount is identified as a failure pixel.

### Claim 20 (depends on 15)

The system of claim 15, wherein the failure pixels are marked using a sentinel value.

### Claim 21 (depends on 15)

The system of claim 15, wherein the failure pixels are included within a segmentation mask.

### Claim 22

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, causes the processor to perform steps comprising:
applying temporal antialiasing (TAA) to an image, wherein for each pixel of at least a subset of pixels, the TAA compares a historical value for the pixel in a previous frame with current values of neighboring pixels to the pixel in a current frame;
identifying failure pixels resulting from applying the TAA to the image, where each pixel of the at least a subset of pixels with the historical value that differs from the current values of the neighboring pixels by more than a predetermined amount is identified as a failure pixel; and
performing antialiasing on each of the failure pixels, utilizing a combination of ray tracing and fast approximate anti-aliasing (FXAA).