# US 10,482,561 B1 · Interaction monitoring for virtualized graphics processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,561 B1
- Document ID: US10482561B1
- Status: Granted
- Kind: B1
- Application number: 15/404,131
- Filed: 2017-01-11
- Granted: 2019-11-19
- Earliest claimed: 2017-01-11
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Malcolm Featonby; Yuxuan Liu; Umesh Chandani; John Merrill Phillips, Jr.; Adithya Bhat; Douglas Cotton Kurtz; Mihir Sadruddin Surani
- Web page: https://patentgrep.com/patent/US10482561B1

## CPC

- H04L 41/069: using logs of notifications; Post-processing of notifications
- G06F 9/45533: {Hypervisors; Virtual machine monitors}
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- H04L 41/0686: Additional information in the notification, e.g. enhancement of specific meta-data
- H04L 41/40: using virtualisation of network functions or resources, e.g. SDN or NFV entities
- H04L 67/30: Profiles
- G06T 2200/16: involving adaptation to the client's capabilities
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/025: for remote control or remote monitoring of applications
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
- H04L 67/56: Provisioning of proxy services (store-and-forward switching systems in data switching networks H04L12/54)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods, systems, and computer-readable media for interaction monitoring for virtualized graphics processing are disclosed. Execution of an application is initiated on a virtual compute instance that is implemented using CPU and memory resources of a server. Instruction calls are produced by the execution of the application and sent from the server to a graphics server over a network. The graphics server comprises a physical GPU, and a virtual GPU is implemented using the physical GPU and attached to the virtual compute instance. GPU output is generated at the graphics server based at least in part on execution of the instruction calls using the virtual GPU. A log of interactions between the application and the virtual GPU is stored. The interactions comprise the instruction calls sent to the graphics server and responses to the instruction calls sent to the virtual compute instance.

## Claims

### Claim 1

A system, comprising:
a server comprising at least one processor and a memory, wherein a virtual compute instance is implemented using the server, and wherein the virtual compute instance is provided by a multi-tenant provider network; and
a graphics server comprising a physical graphics processing unit (GPU), wherein the graphics server is accessible to the server over a network, wherein a virtual GPU is implemented using the physical GPU and attached to the virtual compute instance over the network, and wherein the virtual GPU is provided by the multi-tenant provider network; and
wherein the server is configured to send, over the network to the graphics server, a plurality of instruction calls produced by execution of an application using the virtual compute instance; and
wherein the graphics server is configured to:
generate GPU output based at least in part on execution of the one or more instruction calls using the virtual GPU; and
store a log of interactions between the application and the virtual GPU, wherein the interactions in the log comprise the one or more instruction calls sent to the graphics server and response information resulting from the execution of the one or more instruction calls using the virtual GPU.

### Claim 2 (depends on 1)

The system as recited in claim 1, further comprising:
a profiling service configured to:
perform programmatic analysis of the log of interactions; and
based at least in part on the programmatic analysis, generate one or more analytics associated with use of the virtual GPU by the application.

### Claim 3 (depends on 1)

The system as recited in claim 1, further comprising:
a profiling service configured to:
perform programmatic analysis of the log of interactions; and
based at least in part on the programmatic analysis, generate a recommendation of one or more modifications to the application.

### Claim 4 (depends on 1)

The system as recited in claim 1, further comprising:
a profiling service configured to:
replay at least a portion of the instruction calls in the log of interactions.

### Claim 5

A computer-implemented method, comprising:
receiving, from a server by a graphics server over a network, one or more instruction calls produced by execution of an application on a virtual compute instance, wherein the graphics server comprises a physical graphics processing unit (GPU), and wherein a virtual GPU is implemented using the physical GPU and attached to the virtual compute instance over the network;
generating GPU output at the graphics server based at least in part on execution of the one or more instruction calls using the virtual GPU;
generating a log of interactions between the application and the virtual GPU, wherein the interactions in the log comprise the one or more instruction calls sent to the graphics server and response information resulting from the execution of the one or more instruction calls using the virtual GPU.

### Claim 6 (depends on 5)

The method as recited in claim 5, further comprising:
based at least in part on the log of interactions, generating a profile associated with use of the virtual GPU by the application.

### Claim 7 (depends on 5)

The method as recited in claim 5, further comprising:
performing programmatic analysis of the log of interactions; and
based at least in part on the programmatic analysis, generating a recommendation of one or more modifications to the application.

### Claim 8 (depends on 5)

The method as recited in claim 5, further comprising:
replaying at least a portion of the instruction calls in the log of interactions.

### Claim 9 (depends on 5)

The method as recited in claim 5, further comprising:
using an additional virtual GPU associated with one or more different hardware characteristics than the virtual GPU, replaying at least a portion of the instruction calls in the log of interactions; and
generating a report indicating one or more differences between the execution of the one or more instruction calls using the virtual GPU and execution of the one or more instruction calls using the additional virtual GPU.

### Claim 10 (depends on 5)

The method as recited in claim 5, further comprising:
storing the log of interactions using an external storage service in one or more storage locations associated with a vendor of the application.

### Claim 11 (depends on 5)

The method as recited in claim 5, wherein the interactions in the log comprise one or more responses to the one or more instruction calls, wherein the one or more responses are sent from the graphics server to the virtual compute instance.

### Claim 12

A non-transitory computer-readable storage medium storing program instructions computer-executable to perform:
receiving, from a server by a graphics server over a network, a plurality of instruction calls produced by execution of an application executing in a virtual compute instance, wherein the graphics server comprises a physical graphics processing unit (GPU), wherein a virtual GPU is implemented using the physical GPU and attached to the virtual compute instance over the network, and wherein the graphics server and the server are provided by a multi-tenant provider network;
generating GPU output at the graphics server based at least in part on execution of the instruction calls using the virtual GPU; and
storing a log of interactions between the application and the virtual GPU, wherein the interactions in the log comprise the instruction calls sent to the graphics server and response information resulting from execution of the instruction calls using the virtual GPU.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium as recited in claim 12, wherein the program instructions are further computer-executable to perform:
based at least in part on the log of interactions, generating one or more analytics associated with use of the virtual GPU by the application.

### Claim 14 (depends on 12)

The non-transitory computer-readable storage medium as recited in claim 12, wherein the program instructions are further computer-executable to perform:
based at least in part on the log of interactions, generating a profile of application programming interface (API) commands associated with the instruction calls.

### Claim 15 (depends on 12)

The non-transitory computer-readable storage medium as recited in claim 12, wherein the program instructions are further computer-executable to perform:
performing programmatic analysis of the log of interactions; and
based at least in part on the programmatic analysis, generating a recommendation of one or more modifications to the application.

### Claim 16 (depends on 12)

The non-transitory computer-readable storage medium as recited in claim 12, wherein the program instructions are further computer-executable to perform:
replaying at least a portion of the instruction calls in the log of interactions.

### Claim 17 (depends on 12)

The non-transitory computer-readable storage medium as recited in claim 12, wherein the program instructions are further computer-executable to perform:
using an additional virtual GPU associated with one or more different hardware characteristics than the virtual GPU, replaying at least a portion of the instruction calls in the log of interactions; and
generating a report indicating one or more differences between the execution of the one or more instruction calls using the virtual GPU and execution of the one or more instruction calls using the additional virtual GPU.

### Claim 18 (depends on 12)

The non-transitory computer-readable storage medium as recited in claim 12, wherein the program instructions are further computer-executable to perform:
storing the log of interactions using an external storage service in one or more storage locations associated with a vendor of the application.

### Claim 19 (depends on 12)

The non-transitory computer-readable storage medium as recited in claim 12, wherein the interactions in the log comprise one or more responses to the one or more instruction calls, wherein the one or more responses are sent from the graphics server to the virtual compute instance.