# US 10,375,398 B2 · Lightfield compression for per-pixel, on-demand access by a graphics processing unit > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,375,398 B2 - Document ID: US10375398B2 - Status: Granted - Kind: B2 - Application number: 15/468,741 - Filed: 2017-03-24 - Published: 2018-09-27 - Granted: 2019-08-06 - Earliest claimed: 2017-03-24 - Assignees: Google Inc. - Applicants: Google Inc. - Inventors: Stuart Abercrombie - Web page: https://patentgrep.com/patent/US10375398B2 ## CPC - H04N 19/146: Data rate or code amount at the encoder output - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - H04N 19/127: Prioritisation of hardware or computational resources - H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability - H04N 19/182: the unit being a pixel - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/433: characterised by techniques for memory access - H04N 19/436: using parallelised computational arrangements - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/12: Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264 - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A first processing unit, such as a CPU, is configured to encode information that represents values of pixels in a block of an image in a lightfield dataset at a first bit rate that is selected from a plurality of bit rates and generate a block descriptor that includes information indicating locations of the encoded information for each pixel. A memory is configured to store the block descriptor and the encoded information in a format that is accessible by a second processing unit, such as a GPU, on a pixel-by-pixel basis. The second processing unit is configured to access, on the pixel-by-pixel basis, the encoded information representative of a value of a pixel in the block, decode the encoded information, and render the value of the pixel for presentation on a display. ## Claims ### Claim 1 A method, comprising: encoding, at a first processing unit of an acquisition and display system, information that represents values of pixels in a block of an image in a lightfield dataset at a first bit rate that is selected from a plurality of bit rates; generating, at the first processing unit, a block descriptor that includes information indicating locations of the encoded information for each pixel in a memory, wherein the block descriptor is configured to provide a field indicating wherein the block descriptor is configured to provide a field indicating whether encoded information is included in the block descriptor; and wherein a second processing unit of the acquisition and display system is configured to decompress the encoded information that represents values of pixels of the image in the lightfield dataset on a pixel-by-pixel basis using the block descriptor; and storing the block descriptor and the encoded information in the memory. ### Claim 2 (depends on 1) The method of claim 1, wherein storing the encoded information comprises storing the encoded information in a first bucket associated with the first bit rate, wherein the first bucket is one of a plurality of buckets associated with the plurality of bit rates. ### Claim 3 (depends on 2) The method of claim 2, wherein generating the block descriptor comprises generating the block descriptor including a first field having a value that indicates a starting location of the first bucket in the memory and a second field having a value that indicates a starting location of the encoded information in the first bucket. ### Claim 4 (depends on 1) The method of claim 1, wherein generating the block descriptor comprises generating the block descriptor that includes a first field, said first field being the field having a value that indicates that the encoded information is included in a second field of the block descriptor. ### Claim 5 (depends on 4) The method of claim 4, wherein generating the block descriptor includes generating the block descriptor having a color value in the second field that represents a color of all the pixels in the block. ### Claim 6 (depends on 1) The method of claim 1, wherein encoding the information further comprises: determining at least one motion vector that associates the pixels in the block with reference pixels in a reference image; determining residual values that are equal to a difference between the values of the pixels and reference values of the reference pixels; encoding the residual values of the pixels at the first bit rate; and storing the at least one motion vector and the encoded residual values of the pixels in the memory. ### Claim 7 (depends on 6) The method of claim 6, wherein generating the block descriptor comprises generating the block descriptor having a first value of a third field that indicates that the at least one motion vector and the encoded residual values of the pixels are stored in the memory. ### Claim 8 (depends on 1) The method of claim 1, wherein generating the block descriptor comprises generating the block descriptor having a second value of a third field that indicates that encoded values of the pixels are stored in the memory. ### Claim 9 A method, comprising: accessing, at a first processing unit of an acquisition and display system, a block descriptor that indicates locations of information representative of values of pixels in a block of an image in a lightfield dataset, wherein the information is encoded at a first bit rate that is selected from a plurality of bit rates and wherein the block descriptor is configured to provide a field indicating whether encoded information is included in the block descriptor; decoding, at the first processing unit, the encoded information representative of a value of a pixel in the block; and rendering, at the first processing unit, the value of the pixel for presentation on a display. ### Claim 10 (depends on 9) The method of claim 9, wherein accessing the block descriptor comprises accessing the block descriptor based on a set of four coordinates that identify the pixel in the block of the image. ### Claim 11 (depends on 9) The method of claim 9, wherein decoding the encoded information comprises determining that a value of a first field corresponding to the field of the block descriptor indicates that encoded information representative of the value of the pixel is stored in a second field of the block descriptor. ### Claim 12 (depends on 11) The method of claim 11, wherein decoding the encoded information comprises determining the value of the pixel based on the encoded information that is stored in the second field. ### Claim 13 (depends on 9) The method of claim 9, wherein decoding the encoded information comprises determining that the value of a first field of the block descriptor indicates that the encoded information is stored in a first bucket associated with the first bit rate, wherein the first bucket is one of a plurality of buckets associated with the plurality of bit rates. ### Claim 14 (depends on 13) The method of claim 13, further comprising: accessing a value of a third field of the block descriptor that indicates whether the encoded information representative of the value of the pixel comprises a motion vector and an encoded residual value of the pixel. ### Claim 15 (depends on 14) The method of claim 14, wherein decoding the encoded information comprises accessing the encoded information stored in the first bucket and decoding the encoded information based on the first bit rate to generate the value of the pixel in response to the value of the third field indicating that the encoded information representative of the value of the pixel comprises the value of the pixel and does not comprise the motion vector and the encoded residual value of the pixel. ### Claim 16 (depends on 14) The method of claim 14, wherein decoding the encoded information further comprises, in response to the value of the third field indicating that the encoded information representative of the value of the pixel comprises the motion vector and the encoded residual value of the pixel: accessing the motion vector that associates the pixel with a reference pixel in a reference image that is stored in a memory; decoding, based on the first bit rate, the encoded information representative of the residual value; and determining the value of the pixel by adding the residual value to a value of the reference pixel. ### Claim 17 An apparatus, comprising: a first processing unit of an acquisition and display system configured to encode information that represents values of pixels in a block of an image in a lightfield dataset at a first bit rate that is selected from a plurality of bit rates and generate a block descriptor that includes information indicating locations of the encoded information for each pixel wherein the block descriptor is configured to provide a field indicating whether encoded information is included in the block descriptor; and a memory configured to store the block descriptor and the encoded information in a format that is accessible by a second processing unit of the acquisition and display system, wherein the second processing unit is configured to decompress the encoded information that represents values of pixels of the image in the lightfield dataset on a pixel-by-pixel basis using the block descriptor. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein the memory is configured to store the encoded information in a first bucket associated with the first bit rate, wherein the first bucket is one of a plurality of buckets associated with the plurality of bit rates. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein the first processing unit is configured to generate the block descriptor including a first field having a value that indicates a starting location of the first bucket in the memory and a second field having a value that indicates a starting location of the encoded information in the first bucket. ### Claim 20 (depends on 17) The apparatus of claim 17, wherein the first processing unit is configured to generate the block descriptor that includes a first field having a value that indicates that the encoded information is included in a second field of the block descriptor. ### Claim 21 (depends on 20) The apparatus of claim 20, wherein the first processing unit is configured to generate the block descriptor having a color value in the second field that represents a color of all the pixels in the block. ### Claim 22 (depends on 17) The apparatus of claim 17, wherein the first processing unit is configured to: determine at least one motion vector that associates the pixels in the block with reference pixels in a reference image; determine residual values that are equal to a difference between the values of the pixels and reference values of the reference pixels; encode the residual values of the pixels at the first bit rate; and store the at least one motion vector and the encoded residual values of the pixels in the memory. ### Claim 23 (depends on 22) The apparatus of claim 22, wherein the first processing unit is configured to generate the block descriptor having a first value of a third field that indicates that the at least one motion vector and the encoded residual values of the pixels are stored in the memory. ### Claim 24 (depends on 17) The apparatus of claim 17, wherein the first processing unit is configured to generate the block descriptor having a second value of a third field that indicates that encoded values of the pixels are stored in the memory. ### Claim 25 An apparatus, comprising: a memory configured to store a block descriptor that indicates locations of information representative of values of pixels in a block of an image in a lightfield dataset, wherein the information is encoded at a first bit rate that is selected from a plurality of bit rates and wherein the block descriptor is configured to provide a field indicating whether encoded information is included in the block descriptor; and a first processing unit of an acquisition and display system is configured to access, on a pixel-by-pixel basis, the encoded information representative of a value of a pixel in the block, decode the encoded information, and render the value of the pixel for presentation on a display. ### Claim 26 (depends on 25) The apparatus of claim 25, wherein the first processing unit is configured to access the block descriptor based on a set of four coordinates that identify the pixel in the block of the image. ### Claim 27 (depends on 25) The apparatus of claim 25, wherein the first processing unit is configured to determine that a value of a first field corresponding to the field of the block descriptor indicates that encoded information representative of the value of the pixel is stored in a second field of the block descriptor. ### Claim 28 (depends on 27) The apparatus of claim 27, wherein the first processing unit is configured to determine the value of the pixel based on the encoded information that is stored in the second field. ### Claim 29 (depends on 25) The apparatus of claim 25, wherein the first processing unit is configured to determine that the value of a first field of the block descriptor indicates that the memory stores the encoded information in a first bucket associated with the first bit rate, wherein the first bucket is one of a plurality of buckets associated with the plurality of bit rates. ### Claim 30 (depends on 29) The apparatus of claim 29, wherein the first processing unit is configured to access a value of a third field of the block descriptor that indicates whether the encoded information representative of the value of the pixel comprises a motion vector and an encoded residual value of the pixel. ### Claim 31 (depends on 30) The apparatus of claim 30, wherein the first processing unit is configured to access the encoded information stored in the first bucket and decode the encoded information based on the first bit rate to generate the value of the pixel in response to the value of the third field indicating that the encoded information representative of the value of the pixel comprises the value of the pixel and does not comprise the motion vector and the encoded residual value of the pixel. ### Claim 32 (depends on 30) The apparatus of claim 30, wherein, in response to the value of the third field indicating that the encoded information representative of the value of the pixel comprises the motion vector and the encoded residual value of the pixel, the first processing unit is configured to: access the motion vector that associates the pixel with a reference pixel in a reference image that is stored in the memory; decode, based on the first bit rate, the encoded information representative of the residual value; and determine the value of the pixel by adding the residual value to a value of the reference pixel.