# US 10,319,114 B2 · Foveated compression of display streams

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,319,114 B2
- Document ID: US10319114B2
- Status: Granted
- Kind: B2
- Application number: 15/649,047
- Filed: 2017-07-13
- Published: 2019-01-17
- Granted: 2019-06-11
- Earliest claimed: 2017-07-13
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Behnam Bastani; Carlin Vieri; Haomiao Jiang
- Web page: https://patentgrep.com/patent/US10319114B2

## CPC

- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 3/147: using display panels
- G09G 5/393: Arrangements for updating the contents of the bit-mapped memory
- G09G 5/395: Arrangements specially adapted for transferring the contents of the bit-mapped memory to the screen (G09G5/399 takes precedence)
- G09G 5/397: Arrangements specially adapted for transferring the contents of two or more bit-mapped memories to the screen simultaneously, e.g. for mixing or overlay (G09G5/02 takes precedence)
- H04N 19/129: Scanning of coding units, e.g. zig-zag scan of transform coefficients or flexible macroblock ordering [FMO]
- H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147)
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/172: the region being a picture, frame or field
- H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks
- H04N 19/182: the unit being a pixel
- H04N 19/423: characterised by memory arrangements (H04N19/433 takes precedence)
- H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
- G09G 5/391: Resolution modifying circuits, e.g. variable screen formats
- G09G 2310/0297: Special arrangements with multiplexing or demultiplexing of display data in the drivers for data electrodes, in a pre-processing circuitry delivering display data to said drivers or in the matrix panel, e.g. multiplexing plural data signals to one D/A converter or demultiplexing the D/A converter output to multiple columns
- G09G 2340/02: Handling of images in compressed format, e.g. JPEG, MPEG
- G09G 2340/0407: Resolution change, inclusive of the use of different resolutions for different screen areas
- G09G 2340/10: Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
- G09G 2350/00: Solving problems of bandwidth in display systems
- G09G 2354/00: Aspects of interface with display user
- G09G 2360/18: Use of a frame buffer in a display terminal, inclusive of the display panel
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A processing unit is configured to render first pixels representative of a high-acuity region in the image and second pixels representative of a low-acuity region in the image. A shaper is configured to reorganize the first pixels based on at least one dimension of the low-acuity region. A multiplexer is configured to multiplex the reorganized first pixels and the second pixels to form a display stream. An encoder is configured to compress the display stream for transmission to a display. A decoder configured to decompress the display stream. A demultiplexer is configured to demultiplex the first pixels and the second pixels. Another processing unit is configured to blend the first pixels and the second pixels to form blended pixel values representative of the image for presentation on a screen.

## Claims

### Claim 1

A method comprising:
accessing first pixels from a first buffer of an electronic device, wherein values of the first pixels represent a high-acuity region in an image and wherein the first pixels are organized as a set of first pixel lines in the first buffer;
accessing second pixels from a second buffer of the electronic device, wherein values of the second pixels represent a low-acuity region in the image and wherein the second pixels are organized as a set of second pixel lines in the second buffer;
reorganizing the first pixels based on at least one dimension of the low-acuity region, wherein reorganizing the first pixels comprises concatenating subsets of the set of first pixel lines to form modified first pixel lines including a number of first pixels that is equal to a number of second pixels in the second pixel lines;
multiplexing the reorganized first pixels and the second pixels to form a display stream; and
compressing the display stream for transmission to a display.

### Claim 2 (depends on 1)

The method of claim 1, wherein multiplexing the reorganized first pixels and the second pixels comprises combining the modified first pixel lines and the second pixel lines on a line-by-line basis.

### Claim 3 (depends on 2)

The method of claim 2, wherein combining the modified first pixel lines and the second pixel lines on the line-by-line basis comprises concatenating the modified first pixel lines and the second pixel lines.

### Claim 4 (depends on 3)

The method of claim 3, wherein concatenating the modified first pixel lines and the second pixel lines comprises appending the modified first pixel lines above, below, to the left, or to the right of the second pixel lines.

### Claim 5 (depends on 2)

The method of claim 2, wherein combining the modified first pixel lines and the second pixel lines on the line-by-line basis comprises interlacing the modified first pixel lines and the second pixel lines on a line-by-line basis.

### Claim 6 (depends on 5)

The method of claim 5, wherein interlacing the modified first pixel lines and the second pixel lines comprises partitioning the modified first pixel lines and the second pixel lines into slices having a height that corresponds to a slice height used to compress the display stream.

### Claim 7 (depends on 6)

The method of claim 6, wherein interlacing the modified first pixel lines and the second pixel lines comprises interlacing a number s2 of slices of the modified first pixel lines with each slice of the second pixel lines, wherein s represents an upscaling factor that is applied to the low-acuity region.

### Claim 8 (depends on 5)

The method of claim 5, wherein interlacing the modified first pixel lines and the second pixel lines comprises statically interlacing the modified first pixel lines at a fixed position relative to the second pixel lines.

### Claim 9 (depends on 5)

The method of claim 5, wherein interlacing the modified first pixel lines and the second pixel lines comprises dynamically interlacing the modified first pixel lines at a variable position relative to the second pixel lines, wherein the variable position is determined based on relative positions of the high-acuity region and the low-acuity region in the image.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
decompressing the display stream;
demultiplexing the display stream to generate the reorganized first pixels and the second pixels; and
reconstructing an original configuration of the first pixels from the reorganized first pixels.

### Claim 11

A method comprising:
accessing first pixels from a first buffer of an electronic device, wherein values of the first pixels represent a high-acuity region in an image;
accessing second pixels from a second buffer of the electronic device, wherein values of the second pixels represent a low-acuity region in the image;
reorganizing the first pixels based on at least one dimension of the low-acuity region;
multiplexing the reorganized first pixels and the second pixels to form a display stream; and
compressing the display stream for transmission to a display, wherein compressing the display stream comprises compressing slices of the display stream on a line-by-line basis, wherein each slice of the display stream includes a number of lines that is an integer fraction of a total number of the modified first pixel lines.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
blending the original configuration of the first pixels and the second pixels to form blended pixel values representative of the image; and
displaying the blended pixel values on a screen.

### Claim 13

An apparatus comprising:
a processing unit configured to render first pixels representative of a high-acuity region in an image and second pixels representative of a low-acuity region in the image;
a first buffer configured to store values of the first pixels rendered by the processing unit, wherein the first pixels are organized as a set of first pixel lines in the first buffer;
a second buffer configured to store values of the second pixels rendered by the processing unit, wherein the second pixels are organized as a set of second pixel lines in the second buffer;
a shaper configured to reorganize the first pixels based on at least one dimension of the low-acuity region, wherein the shaper is configured to access the values of the first pixels from the first buffer and to access the values of the second pixels from the second buffer and configured to reorganized the first pixels by concatenating subsets of the set of first pixel lines to form modified first pixel lines including a number of first pixels that is equal to a number of second pixels in the second pixel lines;
a multiplexer configured to multiplex the reorganized first pixels and the second pixels to form a display stream; and
an encoder configured to compress the display stream for transmission to a display.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the multiplexer is configured to combine the modified first pixel lines and the second pixel lines on a line-by-line basis.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the multiplexer is configured to concatenate the modified first pixel lines and the second pixel lines.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the multiplexer is configured to append the modified first pixel lines above, below, to the left, or to the right of the second pixel lines.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein the multiplexer is configured to interlace the modified first pixel lines and the second pixel lines on a line-by-line basis.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein the multiplexer is configured to interlace the modified first pixel lines and the second pixel lines into slices having a height that corresponds to a slice height used to compress the display stream.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the multiplexer is configured to interlace a number s2 of slices of the modified first pixel lines with each slice of the second pixel lines, wherein s represents an upscaling factor that is applied to the low-acuity region.

### Claim 20 (depends on 17)

The apparatus of claim 17, wherein the multiplexer is configured to statically interlace the modified first pixel lines at a fixed position relative to the second pixel lines.

### Claim 21 (depends on 17)

The apparatus of claim 17, wherein the multiplexer is configured to dynamically interlace the modified first pixel lines at a variable position relative to the second pixel lines, wherein the variable position is determined based on relative positions of the high-acuity region and the low-acuity region in the image.

### Claim 22

An apparatus comprising:
a processing unit configured to render first pixels representative of a high-acuity region in an image and second pixels representative of a low-acuity region in the image;
a shaper configured to reorganize the first pixels based on at least one dimension of the low-acuity region;
a multiplexer configured to multiplex the reorganized first pixels and the second pixels to form a display stream; and
an encoder configured to compress the display stream for transmission to a display, wherein the encoder is configured to compress slices of the display stream on a line-by-line basis, wherein each slice of the display stream includes a number of lines that is an integer fraction of a total number of the modified first pixel lines.

### Claim 23

An apparatus comprising:
a decoder configured to decompress a display stream formed of first pixels having values representative of a high-acuity region in an image and second pixels having values representative of a low-acuity region in the image, wherein the first pixels are reorganized from an array having a first dimension to an array having a corresponding second dimension different than the first dimension prior to being compressed based on at least one dimension of the low-acuity region;
a demultiplexer configured to demultiplex the first pixels and the second pixels; and
a processing unit configured to blend the first pixels and the second pixels to form blended pixel values representative of the image, wherein the processing unit is configured to reconstruct the first pixels from the array having the second dimension into the array having the first dimension.

### Claim 24 (depends on 23)

The apparatus of claim 23, wherein the processing unit is configured to blend the first pixels in the array having the first dimension and the second pixels to form the blended pixel values representative of the image.

### Claim 25 (depends on 23)

The apparatus of claim 23, further comprising:
a first buffer configured to store values of the first pixels received from the demultiplexer, wherein the processing unit is configured to access the first pixels from the first buffer; and
a second buffer configured to store values of the second pixels received from the demultiplexer, wherein the processing unit is configured to access the second pixels from the second buffer.

### Claim 26 (depends on 23)

The apparatus of claim 23, further comprising:
a screen configured to present the image based on the blended pixel values.