# US 2019/0173927 A1 · Extended selection and alignment of video segments for adaptive streaming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0173927 A1
- Document ID: US20190173927A1
- Status: Published application
- Kind: A1
- Application number: 16/266,402
- Filed: 2019-02-04
- Published: 2019-06-06
- Earliest claimed: 2016-01-29
- Assignees: Roku, Inc.
- Applicants: Roku, Inc.
- Inventors: Francis DEGRANGE
- Web page: https://patentgrep.com/patent/US20190173927A1

## CPC

- H04L 65/61: for supporting one-way streaming services, e.g. Internet radio
- G06F 13/1673: {using buffers}
- G06F 13/4068: {Electrical coupling}
- H04L 65/70: Media network packetisation
- H04N 21/23406: {involving management of server-side video buffer}
- H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement}
- H04N 21/23439: {for generating different versions}
- H04N 21/23605: {Creation or processing of packetized elementary streams [PES]}
- H04N 21/8455: {involving pointers to the content, e.g. pointers to the I-frames of the video stream}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N 21/8547: involving timestamps for synchronizing content
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Disclosed herein are system, apparatus, article of manufacture, method, and/or computer program product embodiments for extended selection and alignment of video segments for adaptive streaming. An embodiment operates by receiving variant segments, the variant segments including a first and a second variant segment having a first bitrate, and third and fourth variant segments having a second bitrate, for decoding and playback, generating a first, second, and third packetized elementary stream (PES) collections based on the first, second, and third variant segments. The embodiment further operate by storing the generated first, second, and third PES collections in a processing buffer, and determining, based on a maximum tolerance offset, a first and a second out-of-tolerance splicing locations and associated first and second out-of-tolerance offsets between the first PES collection and the second PES collection and between the first PES collection and the third PES collection.

## Claims

### Claim 1

A system comprising:
at least one processor; and
a memory operatively coupled to the at least one processor, the processor configured to:
receive one or more variant segments, the one or more variant segments including a first and second variant segments having a first bitrate, and a third and fourth variant segments having a second bitrate, for decoding and playback,
generate a first, second, and third packetized elementary stream (PES) collections based on the first, second, and third variant segments,
store the generated first, second, and third PES collections in a processing buffer, and
determine, based on a maximum tolerance offset, a first and a second out-of-tolerance splicing locations and associated first and second out-of-tolerance offsets between the first PES collection and the second PES collection and between the first PES collection and the third PES collection.

### Claim 2 (depends on 1)

The system of claim 1, wherein the first variant segment is temporally prior and adjacent to the second variant segment, the fourth variant segment is temporally prior and adjacent to the third variant segment, and the first bitrate is different than the second bitrate.

### Claim 3 (depends on 2)

The system of claim 2, wherein the at least one processor is further configured to:
request a fifth variant segment having the second bitrate, in response to the determination of the second out-of-tolerance splicing location,
receive the fifth variant segment, wherein the fifth variant segment is temporally prior and adjacent to the fourth variant segment,
generate a fifth PES collection based on the fifth variant segment, and
store the fifth PES collection in the processing buffer.

### Claim 4 (depends on 4)

The system of claim 4, wherein the at least one processor is further configured to:
realign the processing buffer based on PTS associated with one or more PES collections stored in the processing buffer, wherein the processing buffer is configured as a first-in-first-out (FIFO) queue.

### Claim 5 (depends on 5)

The system of claim 5, wherein the realign of the process buffer further configures the at least one processor to:
move the one or more PES collections stored in the processing buffer, such that the PES collections stored in the processing buffer are ordered in increasing PTS order, and
remove the second PES collection stored in the processing buffer based on a bitrate transition associated with the third in-tolerance splicing location.

### Claim 6 (depends on 1)

The system of claim 1, wherein the maximum tolerance offset is plus or minus one frame for a given frame rate determined based on parameter information associated with a key frame.

### Claim 7

A computer implemented method, comprising:
receiving one or more variant segments, the one or more variant segments including a first and second variant segments having a first bitrate, and a third and fourth variant segments having a second bitrate, for decoding and playback,
generating a first, second, and third packetized elementary stream (PES) collections based on the first, second, and third variant segments,
storing the generated first, second, and third PES collections in a processing buffer, and
determining, based on a maximum tolerance offset, a first and a second out-of-tolerance splicing locations and associated first and second out-of-tolerance offsets between the first PES collection and the second PES collection and between the first PES collection and the third PES collection.

### Claim 8 (depends on 8)

The computer implemented method of claim 8, wherein the first variant segment is temporally prior and adjacent to the second variant segment, the fourth variant segment is temporally prior and adjacent to the third variant segment, and the first bitrate is different than the second bitrate.

### Claim 9 (depends on 9)

The computer implemented method of claim 9, further comprising:

### Claim 10 (depends on 3)

The system of claim 3, wherein the at least one processor is further configured to:
determine, based on the maximum tolerance offset, a third in-tolerance splicing location between the first and the fifth PES collections, the first and the fourth PES collections, and/or the second and the fifth PES collections, and
splice the first and the fifth PES collections, the first and the fourth PES collections, or the second and the fifth PES collections based on the third in-tolerance splicing location.
requesting a fifth variant segment having the second bitrate, in response to the determination of the second out-of-tolerance splicing location;
receiving the fifth variant segment, wherein the fifth variant segment is temporally prior and adjacent to the fourth variant segment;
generating a fifth PES collection based on the fifth variant segment; and
storing the fifth PES collection in the processing buffer.

### Claim 11 (depends on 3)

The system of claim 3, wherein the at least one processor is further configured to:
determine, based on the maximum tolerance offset, a third in-tolerance splicing location between the first and the fifth PES collections, the first and the fourth PES collections, and/or the second and the fifth PES collections, and
splice the first and the fifth PES collections, the first and the fourth PES collections, or the second and the fifth PES collections based on the third in-tolerance splicing location.

### Claim 12 (depends on 11)

The computer implemented method of claim 11, wherein the determining the third in-tolerance splicing location further comprising:
generating a first key frames collection based on the first PES collection, wherein each key frames collection element of the first key frames collection includes a key frame, associated parameter information, and a corresponding presentation timestamp (PTS);
generating a fifth key frames collection based on the fifth PES collection, wherein each key frames collection element of the fifth key frames collection includes a key frame, associated parameter information, and a corresponding PTS;
determining whether a PTS of the first key frames collection substantially matches a PTS of the fifth key frames collection; and
generating the third in-tolerance splicing location, in response to determining that the PTS of the first key frames collection substantially matches the PTS of the fifth key frames collection.

### Claim 13 (depends on 11)

The computer implemented method of claim 11, wherein the determining the third in-tolerance splicing location further comprising:
generating a first key frames collection based on the first PES collection, wherein each key frames collection element of the first key frames collection includes a key frame, associated parameter information, and a corresponding presentation timestamp (PTS);
generating a fourth key frames collection based on the fourth PES collection, wherein each key frames collection element of the fourth key frames collection includes a key frame, associated parameter information, and a corresponding PTS;
determining whether a PTS of the first key frames collection substantially matches a PTS of the fourth key frames collection; and
generating the third in-tolerance splicing location, in response to determining that the PTS of the first key frames collection substantially matches the PTS of the fourth key frames collection.

### Claim 14 (depends on 11)

The computer implemented method of claim 11, wherein the determining the third in-tolerance splicing location further comprising
generating a second key frames collection based on the second PES collection, wherein each key frames collection element of the second key frames collection includes a key frame, associated parameter information, and a corresponding presentation timestamp (PTS);
generating a fifth key frames collection based on the fifth PES collection, wherein each key frames collection element of the fifth key frames collection includes a key frame, associated parameter information, and a corresponding PTS;
determining whether a PTS of the second key frames collection substantially matches a PTS of the fifth key frames collection; and
generating the third in-tolerance splicing location, in response to determining that the PTS of the second key frames collection substantially matches the PTS of the fifth key frames collection.

### Claim 15

A tangible computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:
receiving one or more variant segments, the one or more variant segments including a first and second variant segments having a first bitrate, and a third and fourth variant segments having a second bitrate, for decoding and playback,
generating a first, second, and third packetized elementary stream (PES) collections based on the first, second, and third variant segments,
storing the generated first, second, and third PES collections in a processing buffer, and
determining, based on a maximum tolerance offset, a first and a second out-of-tolerance splicing locations and associated first and second out-of-tolerance offsets between the first PES collection and the second PES collection and between the first PES collection and the third PES collection.

### Claim 16 (depends on 15)

The computer-readable device of claim 15, wherein the first variant segment is temporally prior and adjacent to the second variant segment, the fourth variant segment is temporally prior and adjacent to the third variant segment, and the first bitrate is different than the second bitrate.

### Claim 17 (depends on 16)

The computer-readable device of claim 16, wherein the operations further comprising:
requesting a fifth variant segment having the second bitrate, in response to the determination of the second out-of-tolerance splicing location;
receiving the fifth variant segment, wherein the fifth variant segment is temporally prior and adjacent to the fourth variant segment;
generating a fifth PES collection based on the fifth variant segment; and
storing the fifth PES collection in the processing buffer.

### Claim 18 (depends on 17)

The computer-readable device of claim 17, wherein the operations further comprising:
determining, based on the maximum tolerance offset, a third, fourth, and fifth out-tolerance splicing locations and associated third, fourth, and fifth out-of-tolerance offsets between the first and the fifth PES collections, the first and the fourth PES collections, and the second and the fifth PES collections, and
selecting an out-of-tolerance splicing location among the first, second, third, fourth, and fifth tolerance splicing locations that has a lowest out-of-tolerance offset.

### Claim 19 (depends on 18)

The computer-readable device of claim 18, wherein the operations further comprising:
splicing one or more PES collections stored in the processing buffer, based on the third in-tolerance splicing location or the selected out-of-tolerance splicing location; and
realigning the processing buffer based on PTS associated with the one or more stored PES collections, wherein the processing buffer is configured as a first-in-first-out (FIFO) queue.

### Claim 20 (depends on 19)

The computer-readable device of claim 19, wherein the operation of splicing the one or more PES collections further comprising:
moving the one or more PES collections stored in the processing buffer, such that the PES collections stored in the processing buffer are ordered in increasing PTS order; and
removing the second PES collection stored in the processing buffer based on a bitrate transition associated with the third in-tolerance splicing location or a bitrate transition associated with the selected out-of-tolerance splicing location.
