# US 2015/0103894 A1 · Systems and methods to limit lag between a client and a server for remote computing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0103894 A1
- Document ID: US20150103894A1
- Status: Published application
- Kind: A1
- Application number: 14/054,532
- Filed: 2013-10-15
- Published: 2015-04-16
- Earliest claimed: 2013-10-15
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Franck DIARD
- Web page: https://patentgrep.com/patent/US20150103894A1

## CPC

- H04N 19/188: {the unit being a video data packet, e.g. a network abstraction layer [NAL] unit}
- H04L 47/10: Flow control; Congestion control
- H04L 65/80: Responding to QoS
- H04N 19/127: Prioritisation of hardware or computational resources
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 21/23805: {Controlling the feeding rate to the network, e.g. by controlling the video pump}
- H04N 21/2401: {Monitoring of the client buffer}
- H04N 21/4424: {Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used (error monitoring in general G06F11/30; arrangements for monitoring conditions of receiving stations in broadcast systems H04H60/32; diagnosis, testing or measuring for television receivers H04N17/04)}
- H04N 21/6373: for rate control {, e.g. request to the server to modify its transmission rate (flow control in packet networks H04L47/10)}
- H04L 47/38: by adapting coding or compression rate
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Novel solutions are described herein for providing a consistent quality of service, latency-wise, for remote processing by managing the process queues in a processing server and temporarily pausing frame production and delivery to limit the lag experienced by a user in a client device. The claimed embodiments limit the latency (lag) experienced by a user by preventing the production rate of rendered frames at the server from significantly outperforming the decoding and display of the received frames in the client device and avoiding the resultant lag.

## Claims

### Claim 1

A method comprising:
reading, from a network input socket of a server device, a plurality of data packets from a remote client device via a network connection communicatively coupling the network input socket of the server device with the remote client device;
rendering a plurality of images based on the plurality of data packets;
encoding the plurality of images into a plurality of compressed video files;
sending the plurality of compressed video files to the remote client device through the network connection;
monitoring a second network input socket for a reception of a token from the remote client device for each of the plurality of compressed video files received by the remote client device;
determining that a token in response to a compressed video file has not been received; and
managing the reading whenever a number of compressed video files have been sent without receiving a token from the remote client device exceeds a threshold number.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the reading is performed by a first processing thread executing in the server device.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the encoding is performed by a second processing thread executing in the server device.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the monitoring, the determining, and the managing are performed by a third processing thread executing in the server device.

### Claim 5 (depends on 1)

The method according to claim 1, further comprising creating, in the server device, an array of operating system events, wherein each event of the array of operating system events comprises either a signaled event or a non-signaled event.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the array of operating system events comprises a ring buffer of operating system events.

### Claim 7 (depends on 5)

The method according to claim 5, wherein each data packet of the plurality of data packets received from the remote client device is stored in a network input buffer, further wherein each element of the array of operating system events has a handle corresponding to an operating system resource.

### Claim 8 (depends on 7)

The method according to claim 7, wherein reading the plurality of data packets comprises:
referencing an element in the array of operating system events comprising a handle corresponding to an operating system resource;
if the element in the array of operating system events comprises a signaled event,
performing the reading of the plurality of data packets; and
incrementing an index of the array of operating system events;
otherwise, pausing the reading of the plurality of data packets.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the array of operating system events is initialized such that a first n number of elements of the array comprises signaled events, wherein n corresponds to the threshold number.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the third processing thread iterates through the array of operating system events incrementally for each received token from the remote client device.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the managing comprises:
for each compressed video file sent to the remote client device,
referencing the array of operating system events at a framerate lag control index, the framerate lag control index corresponding to the current index+n;
setting the framerate lag control element as unsignaled if no token has been received from the remote client device in response to the compressed video file sent;
setting the element in the array of operating system events at the current index as signaled when a token is received from the remote client device in response to the compressed video file sent; and
incrementing the framerate lag control index.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the rendering the plurality of graphical images is performed by a graphical processing unit (GPU) comprised in the server device.

### Claim 13 (depends on 1)

The method according to claim 1, wherein the threshold number is dynamically configurable.

### Claim 14 (depends on 1)

The method according to claim 1, wherein the server device comprises a computing device with a processor and a graphics processing unit.

### Claim 15 (depends on 1)

The method according to claim 1, wherein the client device is communicatively coupled to a display, and is comprised from one of the following:
a desktop computer;
a laptop computer;
a set-top box;
a tablet computer;
a mobile phone device.
a video game console device.

### Claim 16 (depends on 1)

The method according to claim 1, wherein the plurality of data packets from the remote client device comprise a plurality of user input actions received in the client device via a user input device.

### Claim 17 (depends on 16)

The method according to claim 16, wherein the user input device comprises at least one of the following group of devices:
a mouse;
a keyboard;
a stylus;
a touch screen device;
a remote control device.

### Claim 18 (depends on 1)

The method according to claim 1, wherein the compressed video file comprises an h.264 video file.

### Claim 19

A system, comprising:
a client device;
a display device coupled to the client device and operable to display a plurality of images from the client device;
a user input device, coupled to the client device and operable to submit user input to the client device;
a server, remotely positioned from client device;
a network connection communicatively coupling the server to the client device;
a processor, comprised in the server and operable to execute:
a first processing thread configured to receive a plurality of data packets from the client device;
a second processing thread configured to send a plurality of rendered images based on the plurality of data packets to the client device;
a third processing thread configured receive a corresponding acknowledgement from the client device and to manage the first and second processing threads such that the first processing thread is prevented from receiving additional data packets from the plurality of data packets so long as a number of plurality of rendered images sent to the client device without receiving a corresponding acknowledgement packet exceeds a threshold value.

### Claim 20 (depends on 19)

The system according to claim 19, further comprising an array of events comprised in the processor, wherein the first and second processing threads are managed by the third processing thread by setting a value of a plurality of elements comprising the array of events to one of: 1) a signaled event; or 2) an unsignaled event.

### Claim 21

A computer readable medium containing program instructions embodied therein for causing a computing system to implement frame lag control, the program instructions comprising.
instructions to read, from a network input socket of a server device, a plurality of data packets from a remote client device via a network connection communicatively coupling the network input socket of the server device with the remote client device;
instructions to render a plurality of images based on the plurality of data packets;
instructions to encode the plurality of images into a plurality of compressed video files;
instructions to send the plurality of compressed video files to the remote client device through the network connection;
instructions to monitor a second network input socket for a reception of a token from the remote client device for each of the plurality of compressed video files received by the remote client device;
instructions to determine when a token in response to a compressed video file has not been received; and
instructions to manage the reading whenever a number of compressed video files have been sent without receiving a token from the remote client device exceeds a threshold number.

### Claim 22

A method comprising:
receiving, in a client device, a compressed video file;
decoding the compressed video file into a decoded video file;
converting the decoded video file into a color space;
generating a bitmap for the color space;
displaying the bitmap in a display device communicatively coupled to the client device; and
sending a token to a remote server, the token being sent through a network socket communicatively coupling the remote server with the client device.
