# US 12,112,723 B2 · Method and apparatus for efficient application screen compression

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,723 B2
- Document ID: US12112723B2
- Status: Granted
- Kind: B2
- Application number: 17/833,777
- Filed: 2022-06-06
- Published: 2022-09-22
- Granted: 2024-10-08
- Earliest claimed: 2020-04-01
- Assignees: Alibaba Group Holding Limited
- Applicants: Alibaba Group Holding Limited
- Inventors: Tae Meon Bae; Sicheng Li; Yen-kuang Chen; Guanlin Wu; Shaolin Xie; Minghai Qin; Qinggang Zhou
- Web page: https://patentgrep.com/patent/US12112723B2

## CPC

- G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence)
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06T 9/002: {using neural networks}
- G09G 5/397: Arrangements specially adapted for transferring the contents of two or more bit-mapped memories to the screen simultaneously, e.g. for mixing or overlay (G09G5/02 takes precedence)
- H04N 19/27: involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]
- G09G 5/14: Display of multiple viewports
- G09G 2320/0613: The adjustment depending on the type of the information to be displayed
- G09G 2340/02: Handling of images in compressed format, e.g. JPEG, MPEG
- G09G 2340/10: Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
- G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
- G09G 2340/125: wherein one of the images is motion video
- G09G 2352/00: Parallel handling of streams of display data
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for of encoding an application screen comprises partitioning graphic data into a plurality of graphic layers and classifying each of the plurality of graphic layers as either a screen content (SC) or a non-screen content (non-SC) layer. The method further comprises classifying each of the plurality of graphic layers as either a screen content (SC) or a non-screen content (non-SC) layer. Further, the method comprises rendering and encoding the one or more SC layers using a first codec and the one or more non-SC layers using a second codec.

## Claims

### Claim 1

A computer-implemented method of encoding a screen frame of an application, the method comprising:
partitioning graphic data of the screen frame into a plurality of graphic layers;
classifying each of the plurality of graphic layers as either a screen content (SC) layer or a non-screen content (non-SC) layer based on layer depth of the each of the plurality of graphic layers;
rendering and encoding one or more SC layers of the plurality of graphic layers using a first codec and rendering and encoding one or more non-SC layers of the plurality of graphic layers using a second codec; and
transmitting the one or more SC layers and the one or more non-SC layers to a destination device, wherein the one or more SC layers and the one or more non-SC layers are operable to be decoded at the destination device, and wherein the one or more SC layers and the one or more non-SC layers are combined.

### Claim 2 (depends on 1)

The method of claim 1, wherein combination of the one or more SC layers and the one or more non-SC layers form a rendered video frame.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more SC layers and the one or more non-SC layers are combined using alpha blending.

### Claim 4 (depends on 1)

The method of claim 1, wherein the encoding the one or more SC layers is performed using encoding techniques including H.264, HEVC, and VC1.

### Claim 5 (depends on 1)

The method of claim 1, wherein the encoding the one or more non-SC layers is performed using encoding techniques including Portable Network Graphics (PNG), HEVC-SCC extension, and Quick-time animation.

### Claim 6 (depends on 1)

The method of claim 1, wherein the classifying comprises:
identifying the one or more SC layers and the one or more non-SC layers of the plurality of graphic layers using machine learning methods.

### Claim 7 (depends on 6)

The method of claim 6, wherein the machine learning methods are selected from a group including support vector machines, Bayesian machine learning, deep neural network, convolutional neural networks (CNNs), deep belief networks, deep learning, and artificial neural networks.

### Claim 8 (depends on 1)

The method of claim 1, wherein the classifying comprises:
identifying the one or more SC layers and the one or more non-SC layers of the plurality of graphic layers based on characteristics associated with respective layers of the plurality of graphic layers.

### Claim 9 (depends on 1)

The method of claim 1, wherein the classifying comprises:
identifying the one or more SC layers and the one or more non-SC layers of the plurality of graphic layers based on rendering commands associated with respective layers of the plurality of graphic layers.

### Claim 10

A system for performing a method of encoding a screen frame of an application, the system comprising:
a memory; and
a processing device communicatively coupled with the memory and the processing device configured to:
partition graphic data of the screen frame into a plurality of graphic layers;
classify each of the plurality of graphic layers as either a screen content (SC) layer or a non-screen content (non-SC) layer based on layer depth of the each of the plurality of graphic layers;
group one or more SC layers into a first group and group one or more non-SC layers into a second group;
render and encode the first group using a first codec to produce an encoded first group and render and encode the second group using a second codec to produce an encoded second group; and
transmit the encoded first group and the encoded second group to a destination device, wherein the encoded first group and the encoded second group are operable to be decoded at the destination device with respective decoders, and wherein the one or more SC layers and the one or more non-SC layers are combined to form a rendered video frame.

### Claim 11 (depends on 10)

The system of claim 10, where the classifying comprises:
identifying the one or more SC layers and the one or more non-SC layers of the plurality of graphic layers using machine learning methods.

### Claim 12 (depends on 11)

The system of claim 11, wherein the decoded first group and the decoded second group are combined using alpha blending.

### Claim 13 (depends on 10)

The system of claim 10, wherein the encoded second group is encoded using encoding techniques including H.264, HEVC, and VC1.

### Claim 14 (depends on 10)

The system of claim 10, wherein the encoded first group is encoded using encoding techniques including Portable Network Graphics (PNG), HEVC-SCC extension, and Quick-time animation.

### Claim 15

A non-transitory computer-readable storage medium having stored thereon, computer executable instructions that, if executed by a computer system cause the computer system to perform a method of encoding a screen frame of an application, the method comprising:
partitioning graphic data of the screen frame into a plurality of graphic layers;
classifying each of the plurality of graphic layers as either a screen content (SC) layer or a non-screen content (non-SC) layer based on layer depth of the each of the plurality of graphic layers;
rendering and encoding one or more non-SC layers of the plurality of graphic layers using a codec to produce an encoded one or more non-SC layers;
transmitting the encoded one or more non-SC layers to a destination device; and
streaming rendering commands associated with the one or more SC layers to the destination device, wherein the encoded one or more non-SC layers are operable to be decoded at the destination device, wherein the one or more SC layers are operable to be rendered at the destination device using the rendering commands, and wherein the rendered one or more SC layers and the decoded one or more non-SC layers are combined.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein combination of the decoded one or more non-SC layers and the rendered one or more SC layers form a rendered video frame at the destination device.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the rendering and encoding the one or more non-SC layers is performed using encoding techniques including H.264, HEVC, and VC1.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the decoded one or more non-SC layers and the rendered one or more SC layers are operable to be combined using alpha blending.

### Claim 19 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the rendering commands include Open Graphics Library (OpenGL) and DirectX commands.

### Claim 20 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the classifying comprises:
identifying the one or more SC layers and the one or more non-SC layers of the plurality of graphic layers using machine learning methods, wherein the machine learning methods are selected from a group including support vector machines, Bayesian machine learning, deep neural network, convolutional neural networks (CNNs), deep belief networks, deep learning, and artificial neural networks.