# US 9,521,057 B2 · Adaptive audio stream with latency compensation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,521,057 B2
- Document ID: US9521057B2
- Status: Granted
- Kind: B2
- Application number: 14/581,321
- Filed: 2014-12-23
- Published: 2016-04-14
- Granted: 2016-12-13
- Earliest claimed: 2014-10-14
- Assignees: Amazon Technologies, Inc.
- Applicants: Biba Systems, Inc.
- Inventors: Michael Klingbeil; Adam Marks
- Web page: https://patentgrep.com/patent/US9521057B2

## CPC

- H04L 43/087: Jitter
- H04L 47/2416: Real-time traffic
- H04L 47/283: in response to processing delays, e.g. caused by jitter or round trip time [RTT]
- H04L 47/30: in combination with information about buffer occupancy at either end or at transit nodes
- H04L 65/403: Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15)
- H04L 65/4038: with floor control
- H04L 65/764: {at the destination (reformatting of additional data in video clients H04N21/4355)}
- H04L 65/765: {intermediate}
- H04N 7/15: Conference systems
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A latency compensating adaptive jitter buffer method is implemented in an audio client running on a user device or in an audio server to adaptively adjust the size of a jitter buffer to optimize latency while minimizing packet loss during audio signal transmission. In some embodiments, the jitter buffer is kept to a nominal size for low latency. In response to a queuing delay event being detected, audio production is temporarily stopped and the size of the jitter buffer is temporarily increased to receive all incoming audio packets up to a certain refill level. The method then resumes audio production using accelerated playback to reduce the jitter buffer size back to the nominal size.

## Claims

### Claim 1

A method in a receive process of an audio client being executed on a user device or an audio conferencing server for receiving audio data packets, the method comprising:
setting a jitter buffer to a nominal size for low latency;
receiving incoming audio data packets;
storing the audio data packets into the jitter buffer;
producing audio playback using the audio data packets from the jitter buffer at a nominal playback rate;
monitoring the incoming audio data packets for a queuing delay event;
in response to a queuing delay event being detected, stopping production of audio playback;
increasing the size of the jitter buffer to greater than the nominal size to receive and store incoming audio data packets;
storing audio data packets in the jitter buffer up to a refill level greater than the nominal size;
in response to the jitter buffer having stored therein audio data packets reaching the refill level, resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate; and
in response to the jitter buffer size returning to the nominal size, continuing to receive incoming audio data packets and producing audio playback of the audio data packets from the jitter buffer at the nominal playback rate.

### Claim 2 (depends on 1)

The method of claim 1, wherein setting a jitter buffer to a nominal size for low latency comprises:
setting the jitter buffer to the nominal size as determined by the history of the jitter buffer size and a target jitter buffer size for absorbing packet queuing delay variations.

### Claim 3 (depends on 1)

The method of claim 1, wherein monitoring the incoming audio data packets for a queuing delay event comprises:
monitoring the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event.

### Claim 4 (depends on 1)

The method of claim 1, wherein resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate comprises:
resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate using an audio playback acceleration technique selected from sample rate conversion, asynchronous granular overlap add, pitch synchronous overlap add (PSOLA), phase vocoder, and sinusoidal modeling.

### Claim 5 (depends on 1)

The method of claim 1, wherein resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate comprises:
resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate by discarding accumulated audio data packets having a voice activity under a given amplitude threshold.

### Claim 6 (depends on 1)

The method of claim 1, wherein resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate comprises:
resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate by discarding accumulated audio data packets that are silent.

### Claim 7 (depends on 1)

The method of claim 1, wherein monitoring the incoming audio data packets for a queuing delay event comprises:
determining the jitter buffer has become empty;
monitoring the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event; and
measuring a first time duration of the jitter buffer becoming empty until an audio data packet has arrived.

### Claim 8 (depends on 7)

The method of claim 7, wherein storing audio data packets in the jitter buffer up to a refill level greater than the nominal size comprises:
storing audio data packets in the jitter buffer up to a refill level corresponding to a buffer duration equal to the first time duration multiplied by a refill factor, the refill factor being 1 or greater.

### Claim 9

A cloud-based audio conferencing system for processing audio signals of a conference call, comprising:
a processor configured to execute a receive process for receiving audio data packets, the receive process being configured to:
set a jitter buffer to a nominal size for low latency;
receive incoming audio data packets;
store the audio data packets into the jitter buffer;
produce audio playback using the audio data packets from the jitter buffer at a nominal playback rate;
monitor the incoming audio data packets for a queuing delay event;
in response to a queuing delay event being detected, stop production of audio playback;
increase the size of the jitter buffer to greater than the nominal size to receive and store incoming audio data packets;
store audio data packets in the jitter buffer up to a refill level greater than the nominal size;
in response to the jitter buffer having stored therein audio data packets reaching the refill level, resume production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate; and
in response to the jitter buffer size returning to the nominal size, continue to receive incoming audio data packets and producing audio playback of the audio data packets from the jitter buffer at the nominal playback rate; and
a memory coupled to the processor and configured to provide the processor with instructions.

### Claim 10 (depends on 9)

The system of claim 9, wherein the processor is configured to set the jitter buffer to the nominal size as determined by the history of the jitter buffer size and a target jitter buffer size for absorbing packet queuing delay variations.

### Claim 11 (depends on 9)

The system of claim 9, wherein the processor is configured to monitor the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event.

### Claim 12 (depends on 9)

The system of claim 9, wherein the processor is configured to resume production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate by discarding accumulated audio data packets having a voice activity under a given amplitude threshold.

### Claim 13 (depends on 9)

The system of claim 9, wherein the processor is configured to:
determine the jitter buffer has become empty;
monitor the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event; and
measure a first time duration of the jitter buffer becoming empty until an audio data packet has arrived.

### Claim 14 (depends on 13)

The system of claim 13, wherein the processor is configured to store audio data packets in the jitter buffer up to a refill level corresponding to a buffer duration equal to the first time duration multiplied by a refill factor, the refill factor being 1 or greater.

### Claim 15

A device for processing audio signals in an audio communication over a data network, comprising:
a processor configured to execute a receive process for receiving audio data packets, the receive process being configured to:
set a jitter buffer to a nominal size for low latency;
receive incoming audio data packets;
store the audio data packets into the jitter buffer;
produce audio playback using the audio data packets from the jitter buffer at a nominal playback rate;
monitor the incoming audio data packets for a queuing delay event;
in response to a queuing delay event being detected, stop production of audio playback;
increase the size of the jitter buffer to greater than the nominal size to receive and store incoming audio data packets;
store audio data packets in the jitter buffer up to a refill level greater than the nominal size;
in response to the jitter buffer having stored therein audio data packets reaching the refill level, resume production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate; and
in response to the jitter buffer size returning to the nominal size, continue to receive incoming audio data packets and producing audio playback of the audio data packets from the jitter buffer at the nominal playback rate; and
a memory coupled to the processor and configured to provide the processor with instructions.

### Claim 16 (depends on 15)

The device of claim 15, wherein the processor is configured to set the jitter buffer to the nominal size as determined by the history of the jitter buffer size and a target jitter buffer size for absorbing packet queuing delay variations.

### Claim 17 (depends on 15)

The device of claim 15, wherein the processor is configured to monitor the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event.

### Claim 18 (depends on 15)

The device of claim 15, wherein the processor is configured to resume production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate by discarding accumulated audio data packets having a voice activity under a given amplitude threshold.

### Claim 19 (depends on 15)

The device of claim 15, wherein the processor is configured to:
determine the jitter buffer has become empty;
monitor the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event; and
measure a first time duration of the jitter buffer becoming empty until an audio data packet has arrived.

### Claim 20 (depends on 19)

The device of claim 19, wherein the processor is configured to store audio data packets in the jitter buffer up to a refill level corresponding to a buffer duration equal to the first time duration multiplied by a refill factor, the refill factor being 1 or greater.