# US 12,113,680 B2 · Reinforcement learning for jitter buffer control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,113,680 B2
- Document ID: US12113680B2
- Status: Granted
- Kind: B2
- Application number: 18/091,992
- Filed: 2022-12-30
- Published: 2023-05-04
- Granted: 2024-10-08
- Earliest claimed: 2020-02-13
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Xiulian Peng; Vinod Prakash; Xiangyu Kong; Sriram Srinivasan; Yan Lu
- Web page: https://patentgrep.com/patent/US12113680B2

## CPC

- H04L 41/16: using machine learning or artificial intelligence
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/092: Reinforcement learning
- G06N 20/00: Machine learning
- H04L 41/145: {involving simulating, designing, planning or modelling of a network}
- H04L 43/0817: by checking functioning
- H04L 43/087: Jitter
- H04L 47/283: in response to processing delays, e.g. caused by jitter or round trip time [RTT]
- G06F 2218/12: Classification; Matching
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed in some examples are methods, systems, and machine-readable mediums which determine jitter buffer delay by inputting jitter buffer and currently observed network status information to a machine learned model that is trained using a reinforcement learning (RL) method. The model maps these inputs to an action to compress, stretch, or hold the jitter buffer delay, which is used by a recipient computing device to optimize the jitter buffer delay. The model may be trained using a simulator that uses network traces of past real streaming sessions (e.g., communication sessions) of users. By training the model through reinforcement learning, the model learns to make better decisions through reinforcement in the form of reward signals that reflect the performance of each decision.

## Claims

### Claim 1

A device for controlling jitter-buffer delay in a media streaming session over a network, the device comprising:
a computer processor;
a memory, storing instructions, which when executed by the computer processor causes the computer processor to perform operations comprising:
identifying a jitter buffer state of a jitter buffer, the jitter buffer storing media frames of a media streaming session, the jitter buffer state comprising a current jitter buffer delay, current received frames in the jitter buffer, total delay of a current media frame of the media frames, and an immediately previous action;
identifying a network delay of the network;
determining an action for a media frame of media data in the jitter buffer based upon the jitter buffer state, the network delay, and a reward; and
determining a playback duration of a next media frame based upon the action.

### Claim 2 (depends on 1)

The device of claim 1, wherein the operations of determining the action for the media frame in the jitter buffer based upon the jitter buffer state and the network delay comprises determining the action for the media frame based upon past jitter buffer states, past network states, and past actions.

### Claim 3 (depends on 1)

The device of claim 1, wherein the jitter buffer state further comprises, whether the media frame is concealed or not, and whether the media frame is newly received.

### Claim 4 (depends on 1)

The device of claim 1, wherein the operations of determining the action for the media frame in the jitter buffer based upon the jitter buffer state and the network delay comprises determining the action by using a model trained using past network traces of previous media streaming sessions.

### Claim 5 (depends on 4)

The device of claim 4, wherein the operations further comprise:
training the model by:
simulating a jitter buffer using the past network traces;
producing a training action using the model for the simulated jitter buffer;
producing an estimated value of the training action using a second model; and
modifying the model based upon the estimated value and the reward.

### Claim 6 (depends on 5)

The device of claim 5, wherein the model and second model share a neural network layer.

### Claim 7 (depends on 4)

The device of claim 4, wherein the model comprises at least two layers wherein at least one layer includes a leaky rectified linear unit activation function.

### Claim 8 (depends on 4)

The device of claim 4, wherein the model comprises:
a first layer that comprises a leaky rectified linear unit (ReLu) activation function;
a second layer that comprises a leaky ReLu activation function;
a third layer that comprises a gated recurrent unit (GRU);
a fourth layer that comprises a leaky ReLu activation function; and
a fifth layer implementing a soft-max function.

### Claim 9 (depends on 1)

The device of claim 1, wherein the operations further comprise playing back the media frame at the playback duration.

### Claim 10 (depends on 1)

The device of claim 1, wherein the operations of determining the action for the media frame based upon the jitter buffer state and the network delay comprises using a machine-learning model, and wherein the operations further comprise:
producing an estimated value of the action using a second model; and
modifying the model based upon the estimated value and the reward.

### Claim 11 (depends on 1)

The device of claim 1, wherein the media frame comprises audio data, video data, or both audio and video data.

### Claim 12

A method for controlling jitter-buffer delay in a media streaming session over a network, the method comprising:
identifying a jitter buffer state of a jitter buffer, the jitter buffer storing media frames of a media streaming session, the jitter buffer state comprising a current jitter buffer delay, current received frames in the jitter buffer, total delay of a current media frame of the media frames, and an immediately previous action;
identifying a network delay of the network;
determining an action for a media frame of media data in the jitter buffer based upon the jitter buffer state, the network delay, and a reward; and
determining a playback duration of a next media frame based upon the action.

### Claim 13 (depends on 12)

The method of claim 12, wherein determining the action for the media frame in the jitter buffer based upon the jitter buffer state and the network delay comprises determining the action for the media frame based upon past jitter buffer states, past network states, and past actions.

### Claim 14 (depends on 12)

The method of claim 12, wherein the jitter buffer state further comprises whether the media frame is concealed or not and whether the media frame is newly received.

### Claim 15 (depends on 12)

The method of claim 12, wherein determining the action for the media frame in the jitter buffer based upon the jitter buffer state and the network delay comprises determining the action by using a model trained using past network traces of previous media streaming sessions.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
training the model by:
simulating a jitter buffer using the past network traces;
producing a training action using the model for the simulated jitter buffer;
producing an estimated value of the training action using a second model; and
modifying the model based upon the estimated value and the reward.

### Claim 17 (depends on 12)

The method of claim 12, wherein determining the action for the media frame in the jitter buffer based upon the jitter buffer state and the network delay comprises using a machine-learned model, and wherein the method further comprises:
producing an estimated value of the action using a second model; and
modifying the model based upon the estimated value and the reward.

### Claim 18 (depends on 12)

The method of claim 12, wherein the media frame comprises audio data, video data, or both audio and video data.

### Claim 19

A device for controlling jitter-buffer delay in a media streaming session over a network, the device comprising:
means for identifying a jitter buffer state of a jitter buffer, the jitter buffer storing media frames of a media streaming session, the jitter buffer state comprising a current jitter buffer delay, current received frames in the jitter buffer, total delay of a current media frame of the media frames, and an immediately previous action;
means for identifying a network delay of the network;
means for determining an action for a media frame of media data in the jitter buffer based upon the jitter buffer state, the network delay, and a reward; and
means for determining a playback duration of a next media frame based upon the action.

### Claim 20 (depends on 19)

The device of claim 19, wherein the means for determining the action for the media frame in the jitter buffer based upon the jitter buffer state and the network delay comprises means for determining the action for the media frame based upon past jitter buffer states, past network states, and past actions.