# US 10,504,281 B2 · Tracking pixel lineage in variable rate shading > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,281 B2 - Document ID: US10504281B2 - Status: Granted - Kind: B2 - Application number: 15/638,865 - Filed: 2017-06-30 - Published: 2019-01-03 - Granted: 2019-12-10 - Earliest claimed: 2017-06-30 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Ivan Nevraev; Martin J. I. Fuller; Mark S. Grossman - Web page: https://patentgrep.com/patent/US10504281B2 ## CPC - G06T 15/80: Shading - G06T 11/40: Filling planar surfaces by adding surface attributes, e.g. adding colours or textures - G06T 15/005: {General purpose rendering architectures} - G06T 15/60: Shadow generation - G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract Methods and devices for performing variable rate shading are described. Invocation information and lineage information for each pixel of a plurality of pixels of a primitive are stored in an invocation buffer and a lineage buffer of a graphics processing unit. One or more deferred shading or post-processing operations are performed on the image based at least in part on the invocation information and the lineage information associated with each pixel of the plurality of pixels. ## Claims ### Claim 1 A method of variable rate shading, comprising: rasterizing, by a graphics processing unit (GPU), a primitive of an image based on one or more corresponding shading rates, wherein the primitive includes a plurality of pixels; generating, based on the rasterizing, metadata that identifies whether each pixel of the plurality of pixels is one of an anchor pixel, a broadcast pixel, or an orphan pixel, wherein each anchor pixel is a pixel for which a pixel shader was previously invoked, wherein each broadcast pixel includes shading information copied from a referenced anchor pixel, and wherein each orphan pixel includes shading information copied from a previously referenced anchor pixel that was overwritten; tracking, by the GPU, a shading state for each pixel of the plurality of pixels based on the metadata by generating invocation information stored in an invocation buffer and lineage information stored in a lineage buffer, wherein the invocation information includes information about the pixel shader to indicate whether a respective pixel is the anchor pixel, wherein the lineage information includes pointer information identifying one of the plurality of pixels from which shading information originated to indicate whether the respective pixel is the broadcast pixel, and wherein a combination of the invocation information and the lineage information indicates whether the respective pixel is the orphan pixel; and performing one or more deferred shading or post-processing operations on at least one pixel of the plurality of pixels of the image based at least in part on the shading state of the at least one pixel as tracked by the invocation information and the lineage information. ### Claim 2 (depends on 1) The method of claim 1, wherein performing the one or more deferred shading or post-processing operations on the image includes performing a reprojection, a temporal anti-aliasing, a deferred lighting operation, or a combination thereof. ### Claim 3 (depends on 1) The method of claim 1, further comprising: performing pixel shading on each anchor pixel and each orphan pixel identified based on the invocation information and the lineage information; and copying a result of the pixel shading for each broadcast pixel from a corresponding referenced anchor pixel identified in the lineage information. ### Claim 4 (depends on 3) The method of claim 3, further comprising: determining that at least one anchor pixel of the primitive is overwritten; and updating the lineage information to identify a respective broadcast pixel as a respective orphan pixel for any broadcast pixel of the primitive pointing to the at least one anchor pixel that is overwritten by performing the pixel shading. ### Claim 5 (depends on 1) The method of claim 1, wherein performing the one or more deferred shading or post-processing operations on the image includes performing a temporal anti-aliasing, wherein the temporal anti-aliasing further comprises: identifying one or more orphan pixels based on the invocation information and the lineage information; identifying whether any of the one or more orphan pixels have a respective valid temporal sample available; and copying a value of the respective valid temporal sample instead of performing deferred shading on the one or more orphan pixels. ### Claim 6 (depends on 1) The method of claim 1, wherein performing the one or more deferred shading or post-processing operations on the image further comprises: assigning a first weight to each anchor pixel, a second weight to each broadcast pixel, and a third weight to each orphan pixel, wherein the first weight is greater than the second weight and the first weight is greater than the third weight, and wherein the first weight, the second weight, and the third weight are assigned based on the invocation information and the lineage information; and performing the one or more deferred shading or post-processing operations on the image based at least on the first weight, the second weight, and the third weight, wherein the first weight, the second weight, and the third weight are assigned such that a subset of the plurality of pixels are blended with respective temporal samples during temporal anti-aliasing. ### Claim 7 (depends on 6) The method of claim 6, further comprising: determining whether the at least one pixel is a respective anchor pixel based at least on the invocation information of the pixel stored in the invocation buffer; or determining whether the at least one pixel is a respective orphan pixel based at least on the invocation information stored in the invocation buffer and the lineage information of the pixel stored in the lineage buffer. ### Claim 8 (depends on 1) The method of claim 1, further comprising: clearing, by the GPU, a render target associated with the plurality of pixels of the image. ### Claim 9 (depends on 1) The method of claim 1, wherein a size of the invocation buffer is one bit per pixel and a size of the lineage buffer is two bits per pixel. ### Claim 10 (depends on 1) The method of claim 1, wherein performing the one or more deferred shading or post-processing operations on the at least one pixel of the plurality of pixels further comprises performing on a minimum number of the plurality of pixels of the image as identified by the tracking of the shading state and generating of the invocation information and the lineage information. ### Claim 11 (depends on 1) The method of claim 1, further comprising: determining whether the at least one pixel is a respective anchor pixel based at least on the invocation information of the at least one pixel stored in the invocation buffer; determining whether the at least one pixel is a respective orphan pixel based at least on the invocation information stored in the invocation buffer and based on the lineage information of the at least one pixel stored in the lineage buffer; and wherein performing the one or more deferred shading or post-processing operations on the at least one pixel of the plurality of pixels further comprises performing on a subset of the plurality of pixels of the image determined based on the invocation information and the lineage information, wherein the subset of the plurality of pixels includes each anchor pixel and each orphan pixel. ### Claim 12 A computer device for variable rate shading, comprising: a memory; and a graphics processing unit (GPU) in communication with the memory, wherein the GPU is configured to: rasterize, by a graphics processing unit (GPU), a primitive of an image based on one or more corresponding shading rates, wherein the primitive includes a plurality of pixels; generate, based on the rasterization, metadata that identifies whether each pixel of the plurality of pixels is one of an anchor pixel, a broadcast pixel, or an orphan pixel, wherein each anchor pixel is a pixel for which a pixel shader was previously invoked, wherein each broadcast pixel includes shading information copied from a referenced anchor pixel, and wherein each orphan pixel includes shading information copied from a previously referenced anchor pixel that was overwritten; track, by the GPU, a shading state for each pixel of the plurality of pixels based on the metadata by generating invocation information stored in an invocation buffer and lineage information stored in a lineage buffer, wherein the invocation information includes information about the pixel shader to indicate whether a respective pixel is the anchor pixel, wherein the lineage information includes pointer information identifying one of the plurality of pixels from which shading information originated to indicate whether the respective pixel is the broadcast pixel, and wherein a combination of the invocation information and the lineage information indicates whether the respective pixel is the orphan pixel; and perform one or more deferred shading or post-processing operations on at least one pixel of the plurality of pixels of the image based at least in part on the shading state of the at least one pixel as tracked by the invocation information and the lineage information. ### Claim 13 (depends on 12) The computer device of claim 12, wherein the one or more deferred shading or post-processing operations on the image includes a reprojection, a temporal anti-aliasing, a deferred lighting operation, or a combination thereof. ### Claim 14 (depends on 12) The computer device of claim 12, wherein the GPU is further configured to: perform pixel shading on each anchor pixel and each orphan pixel identified based on the invocation information and the lineage information; and copy a result of the pixel shading for each broadcast pixel from a corresponding referenced anchor pixel identified in the lineage information. ### Claim 15 (depends on 14) The computer device of claim 14, wherein the GPU is further configured to: determine that at least one anchor pixel of the primitive is overwritten; and update the lineage information to identify a respective broadcast pixel as a respective orphan pixel for any broadcast pixel of the primitive pointing to the at least one anchor pixel that is overwritten by performing the pixel shading. ### Claim 16 (depends on 12) The computer device of claim 12, wherein the one or more deferred shading or post-processing operations on the image includes a temporal anti-aliasing, wherein the GPU is further configured to perform the temporal anti-aliasing to: identify one or more orphan pixels based on the invocation information and the lineage information; identify whether any of the one or more orphan pixels have a respective valid temporal sample available; and copy a value of the respective valid temporal sample instead of performing deferred shading on the one or more orphan pixels. ### Claim 17 (depends on 12) The computer device of claim 12, wherein the GPU is further configured to perform the one or more deferred shading or post-processing operations on the image to: assign a first weight to each anchor pixel, a second weight to each broadcast pixel, and a third weight to each orphan pixel, wherein the first weight is greater than the second weight and the first weight is greater than the third weight, and wherein the first weight, the second weight, and the third weight are assigned based on the invocation information and the lineage information; and perform the one or more deferred shading or post-processing operations on the image based at least on the first weight, the second weight, and the third weight, wherein the first weight, the second weight, and the third weight are assigned such that a subset of the plurality of pixels are blended with respective temporal samples during temporal anti-aliasing. ### Claim 18 (depends on 17) The computer device of claim 17, wherein the GPU is further configured to: determine whether the at least one pixel is a respective anchor pixel based at least on the invocation information of the pixel stored in the invocation buffer; or determine whether the the at least one pixel is a respective orphan pixel based at least on the invocation information stored in the invocation buffer and the lineage information of the pixel stored in the lineage buffer. ### Claim 19 (depends on 12) The computer device of claim 12, wherein the GPU is further configured to: clear, by the GPU, a render target associated with the plurality of pixels of the image. ### Claim 20 (depends on 12) The computer device of claim 12, wherein a size of the invocation buffer is one bit per pixel and a size of the lineage buffer is two bits per pixel. ### Claim 21 A computer-readable medium storing computer-executable instructions executable by a processor for variable rate shading, comprising instructions to: rasterize, by a graphics processing unit (GPU), a primitive of an image based on one or more corresponding shading rates, wherein the primitive includes a plurality of pixels; generate, based on the rasterization, metadata that identifies whether each pixel of the plurality of pixels is one of an anchor pixel, a broadcast pixel, or an orphan pixel, wherein each anchor pixel is a pixel for which a pixel shader was previously invoked, wherein each broadcast pixel includes shading information copied from a referenced anchor pixel, and wherein each orphan pixel includes shading information copied from a previously referenced anchor pixel that was overwritten; track, by the GPU, a shading state for each pixel of the plurality of pixels based on the metadata by generating invocation information stored in an invocation buffer and lineage information stored in a lineage buffer, wherein the invocation information includes information about the pixel shader to indicate whether a respective pixel is the anchor pixel, wherein the lineage information includes pointer information identifying one of the plurality of pixels from which shading information originated to indicate whether the respective pixel is the broadcast pixel, and wherein a combination of the invocation information and the lineage information indicates whether the respective pixel is the orphan pixel; and perform one or more deferred shading or post-processing operations on at least one pixel of the plurality of pixels of the image based at least in part on the shading state of the at least one pixel as tracked by the invocation information and the lineage information. ### Claim 22 (depends on 21) The computer-readable medium of claim 21, wherein the instructions to perform the one or more deferred shading or post-processing operations on the image include instructions to perform a reprojection, a temporal anti-aliasing, a deferred lighting operation, or a combination thereof.