# US 2016/0366228 A1 · Sending application input commands over a network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0366228 A1
- Document ID: US20160366228A1
- Status: Published application
- Kind: A1
- Application number: 15/245,638
- Filed: 2016-08-24
- Published: 2016-12-15
- Earliest claimed: 2010-12-15
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: ADAM J. OVERTON; BRIAN D. FISHER; ISAAC J. SHEPARD; JASON C. JENKS
- Web page: https://patentgrep.com/patent/US20160366228A1

## CPC

- H04L 67/141: Setup of application sessions (admission control or resource allocation in data switching networks H04L47/70)
- A63F 13/23: for interfacing with the game device, e.g. specific interfaces between game controller and console
- A63F 13/30: Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
- A63F 13/31: Communication aspects specific to video games, e.g. between several handheld game devices at close range
- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06F 9/452: {Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- H04L 65/61: for supporting one-way streaming services, e.g. Internet radio
- H04N 21/2393: {involving handling client requests (scheduling or organising the servicing of application requests H04L67/60)}
- H04N 21/4781: {Games}
- H04N 21/6175: {involving transmission via Internet (broadcast-related systems characterised by the transmission system being the Internet H04H60/82)}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Disclosed are various embodiments that facilitate sending input commands to an application over a network that may have variable latency characteristics. A first computing device sends a request to initiate a remote session with the application being executed by a second computing device. Upon initiation of the remote session, the first computing device receives application output data associated with the application for display via the first computing device. The first computing device may capture an input command associated with a video frame of the application output data being displayed. The input command is transmitted to the second computing device. To account for latency characteristics associated with the network, the second computing device provides the input command to the application after a delay.

## Claims

### Claim 1

A system, comprising:
a first computing device; and
a first application executable in the first computing device, wherein, when executed, the first application causes the first computing device to at least:
initiate a remote session over a network with a second application being executed in a hosted environment by at least one second computing device;
receive a video stream associated with a video signal being generated by the second application;
render the video stream on a display associated with the first computing device;
capture a first input command associated with a first input and a second input command associated with a second input; and
transmit application input data comprising the first input command and the second input command to the at least one second computing device, the second input command being provided to the second application after a delay based at least in part on a latency characteristic of the network.

### Claim 2 (depends on 1)

The system of claim 1, wherein the delay preserves a relative temporal relationship between the first input command and the second input command.

### Claim 3 (depends on 1)

The system of claim 1, wherein the first input command is associated with a first video frame of the video stream and the second input command is associated with a second video frame of the video stream, the first video frame being displayed when the first input command is generated and the second video frame being displayed when the second input command is generated.

### Claim 4 (depends on 1)

The system of claim 1, wherein the network has a variable amount of latency.

### Claim 5 (depends on 1)

The system of claim 1, wherein the delay is based at least in part a comparison of a first time period and a second time period, the first time period being between when the first input command was generated by the first computing device and when the second input command was generated by the first computing device and the second time period being between when the first input command is received by the at least one second computing device and when the second input command is received by the at least one second computing device.

### Claim 6 (depends on 1)

The system of claim 1, wherein the first input command and the second input command are transmitted to the at least one second computing device in a batch.

### Claim 7

A method, comprising:
initiating, by a first computing device, a remote session with an application being executed in a hosted environment by at least one second computing device;
rendering, by the first computing device, an application video output generated by the application on a display, the application video output being received from the at least one second device over a network;
receiving, by the first computing device, an input command from an input device, the input command corresponding to a video frame of the application video output being displayed when the input command is generated; and
transmitting, by the first computing device, the input command and a video frame identifier to the at least one second computing device, the input command being provided to the application via the at least one second computing device after a delay that accounts for a latency characteristic of the network.

### Claim 8 (depends on 7)

The method of claim 7, further comprising receiving another input command from the input device, the other input command corresponding to another video frame of the application video output, and the input command and the other input command being transmitted to the at least one second computing device in a batch.

### Claim 9 (depends on 7)

The method of claim 7, wherein the delay is further determined based at least in part on an input command type associated with the input command.

### Claim 10 (depends on 7)

The method of claim 7, wherein the network has a variable amount of latency.

### Claim 11 (depends on 7)

The method of claim 7, further comprising encoding, via the first computing device, the input command prior to transmitting to the at least one second computing device.

### Claim 12 (depends on 7)

The method of claim 7, wherein the input command is associated with a video frame generated by the application, the video frame having been displayed by the first computing device relative to the input command being generated.

### Claim 13

A non-transitory computer-readable medium embodying a program executable in a first computing device, wherein, when executed, the program causes the first computing device to at least:
render one or more frames of a video stream on a display associated with the first computing device, the video stream being generated by an application being executed in a hosted environment by at least one second computing device;
capture a plurality of input commands via one or more input devices, individual input commands of the plurality of input commands corresponding to a respective interaction by a user with the application;
generate application input data comprising the individual input commands; and
transmit the application input data including the individual input commands over a network to the at least one second computing device in response to determining that the individual input commands are ready to be transmitted.

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein the video stream is included in application output data received from the at least one computing device.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the application output data further comprises force feedback data for the one or more input devices.

### Claim 16 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the application output data further comprises an audio stream multiplexed with the video stream.

### Claim 17 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein the individual input commands are transmitted to the at least one second computing device at a predefined interval.

### Claim 18 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein the application input data includes the plurality of input commands and a corresponding timestamp for the individual input commands that indicates when a respective input command was generated by the first computing device.

### Claim 19 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein the application input data includes the individual input commands and a corresponding video frame identifier that correlates a particular video frame with a respective input command, the particular video frame being rendered on the display when the respective input command was generated.

### Claim 20 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein the input command is provided to the application via the at least one second computing device after a delay in order to preserve a relative temporal relationship between the input command and another input command.