# US 12,118,639 B2 · Method and system for interactive graphics streaming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,639 B2
- Document ID: US12118639B2
- Status: Granted
- Kind: B2
- Application number: 17/146,828
- Filed: 2021-01-12
- Published: 2021-05-06
- Granted: 2024-10-15
- Earliest claimed: 2014-01-21
- Assignees: Mythical, Inc.
- Applicants: Mythical, Inc.
- Inventors: Adam Billyard
- Web page: https://patentgrep.com/patent/US12118639B2

## CPC

- H04L 65/60: Network streaming of media packets
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 1/60: Memory management
- H04L 65/80: Responding to QoS
- H04L 67/564: Enhancement of application control based on intercepted application data
- G06T 2200/16: involving adaptation to the client's capabilities
- G06T 2200/24: involving graphical user interfaces [GUIs]
- H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
- H04L 67/568: Storing data temporarily at an intermediate stage, e.g. caching
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present invention relates to a method of streaming interactive computer graphics from a server to a client device. The method includes the steps of: intercepting graphics instructions transmitted from an application destined for a graphical processing unit (GPU) at the server; processing the graphics instructions to generate graphics data at the server; generating index information for, at least, some of the graphics data at the server; transmitting the index information in place of the graphics data to a client device; extracting corresponding graphics data stored at the client device utilizing the index information; and rendering computer graphics at a graphical processing unit (GPU) at the client device using the corresponding graphics data. A system for streaming interactive computer graphics is also disclosed.

## Claims

### Claim 1

A method of streaming interactive computer graphics from a server to a client device, wherein the client device includes local storage and a client-side graphical processing unit (GPU), the method including:
a) intercepting, at the server, graphics instructions transmitted from an application destined for a graphical processing unit (GPU) at the server, wherein the application is being executed at the server;
b) processing the graphics instructions to generate graphics data at the server, wherein the graphics data includes a vertex buffer, wherein the vertex buffer is divided into blocks, and wherein the vertex buffer is further divided into stripes corresponding to different vertex fields;
c) generating index information for at least some of the graphics data at the server, including particular index information that indexes one or more portions of the vertex buffer that have been modified due to execution of one or more of the graphics instructions, wherein the index information is generated for a run of blocks in the vertex buffer that have been modified, and wherein individual values of the index information are generated for individual ones of the stripes that have been modified;
d) transmitting the index information in place of the graphics data from the server to the client device, wherein the transmitted index information includes the particular index information that indexes the one or more portions of the vertex buffer that have been modified;
e) extracting, by the client device, corresponding graphics data that corresponds to the graphics data, wherein the corresponding graphics data is stored in the local storage, and wherein extracting the corresponding graphics data utilizes the transmitted index information; and
f) rendering computer graphics by the client-side graphical processing unit (GPU) using the corresponding graphics data and the transmitted index information.

### Claim 2 (depends on 1)

The method of claim 1, wherein the graphics data includes one or more from the set of graphics state, static resources, and dynamic resources.

### Claim 3 (depends on 1)

The method of claim 1, wherein objects within the graphics data are hashed to generate the index information.

### Claim 4 (depends on 1)

The method of claim 1, wherein the index information is generated for a trimmed run of blocks that have been modified such that the run extends from a first modified bit within a first block in the trimmed run to a last modified bit within a last block in the run.

### Claim 5 (depends on 1)

The method of claim 1, further including:
synchronizing the graphics data between the server and the client.

### Claim 6 (depends on 5)

The method of claim 5, wherein, when the graphics data includes textures, only data for the textures that are used are synchronized.

### Claim 7 (depends on 5)

The method of claim 5, wherein a profile of the client device determines the synchronization of graphics data.

### Claim 8 (depends on 7)

The method of claim 7, wherein the profile of the client device determines the synchronization of graphics data by assigning lower resolution graphics for higher resolution graphics at the client device.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
displaying the rendered computer graphics on a display at the client device.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
receiving, by the client device, user input in response to the displayed computer graphics; and
transmitting the user input back to the application that is being executed on the server.

### Claim 11 (depends on 10)

The method of claim 10, wherein the user input is transmitted, at least in part, using User Datagram Protocol (UDP).

### Claim 12 (depends on 11)

The method of claim 11, wherein State Transition Events are synthesized on the server.

### Claim 13 (depends on 1)

The method of claim 1, wherein the application is selected for execution at the server by a user from a plurality of applications.

### Claim 14 (depends on 1)

A client device configured for use with the method of claim 1.

### Claim 15

A system for streaming interactive computer graphics, including:
a server configured for:
(i) executing an application,
(ii) intercepting graphics instructions transmitted from the application destined for a graphical processing unit (GPU) at the server,
(iii) processing the graphics instructions to generate graphics data, wherein the graphics data includes a vertex buffer, wherein the vertex buffer is divided into blocks, and wherein the vertex buffer is further divided into stripes corresponding to different vertex fields, wherein the different vertex fields include vertex color, vertex texture coordinate, and vertex position,
(iv) generating index information for at least some of the graphics data at the server, including particular index information that indexes one or more portions of the vertex buffer that have been modified due to execution of one or more of the graphics instructions, wherein the index information is generated for a run of blocks in the vertex buffer that are dirty, and wherein individual values of the index information are generated for individual ones of the stripes that have been modified, and
(v) transmitting the index information in place of the graphics data from the server to a client device, wherein the transmitted index information includes the particular index information that indexes the one or more portions of the vertex buffer that have been modified; and
the client device configured for:
(i) extracting corresponding graphics data that corresponds to the graphics data, wherein the corresponding graphics data is stored in local storage at the client device, and wherein extracting the corresponding graphics data utilizes the transmitted index information; and
(ii) rendering computer graphics at a client-side graphical processing unit (GPU) using the corresponding graphics data and the transmitted index information.