# US 2018/0324502 A1 · Selection and alignment of video segments for adaptive streaming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0324502 A1
- Document ID: US20180324502A1
- Status: Published application
- Kind: A1
- Application number: 16/036,043
- Filed: 2018-07-16
- Published: 2018-11-08
- Earliest claimed: 2016-01-29
- Assignees: Roku, Inc.
- Applicants: Roku, Inc.
- Inventors: Francis DEGRANGE
- Web page: https://patentgrep.com/patent/US20180324502A1

## CPC

- H04N 21/64769: {for rate control (flow control in packet networks H04L47/10)}
- H04N 21/23406: {involving management of server-side video buffer}
- H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement}
- H04N 21/23439: {for generating different versions}
- H04N 21/23605: {Creation or processing of packetized elementary streams [PES]}
- H04N 21/2362: Generation or processing of Service Information [SI]
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Disclosed herein are system, apparatus, article of manufacture, method, and/or computer program product embodiments for selection and alignment of video segments for adaptive streaming. An embodiment operates by receiving one or more variant segments, the one or more variant segments including a first variant segment having a first bitrate and a second variant segment having a second bitrate for decoding and playback and generating a first packetized elementary stream (PES) collection based on the first variant segment and a second PES collection based on the second variant segment. The embodiment further operates by storing the generated first PES collection and second PES collection in a processing buffer; and determining a first out-of-tolerance splicing location and associated first out-of-tolerance offset between the first PES collection and the second PES collection based on a maximum tolerance offset.

## Claims

### Claim 1

A device comprising:
at least one processor; and
a memory operatively coupled to the at least one processor, the at least one process configured to:
receive one or more variant segments for decoding and playback, the one or more variant segments including a first variant segment having a first bitrate and second and third variant segments having a second bitrate,
generate corresponding first, second, and third packetized elementary stream (PES) collections based on at least the first, second, and third variant segments,
store the generated first, second, and third PES collections in a processing buffer, and
determine a second in-tolerance splicing location between the first PES collection and the third PES collection based on at least a tolerance offset, in response to a determination of a first out-of-tolerance splicing location between the first PES collection and the second PES location.

### Claim 2 (depends on 1)

The device of claim 1, wherein the at least one processor is further configured to:
request the third variant segment having the second bitrate, in response to the determination of the first out-of-tolerance splicing location, wherein the received third variant segment is temporally prior and adjacent to the second variant segment.

### Claim 3 (depends on 2)

The device of claim 2, wherein the second in-tolerance splicing location is determined based on at least a determination of whether a substantial match exists between a presentation timestamp (PTS) associated with the first PES collection and a PTS associated with the third PES collection within the tolerance offset.

### Claim 4 (depends on 3)

The device of claim 3, wherein the at least one processor is further configured to:
splice the first PES collection and the third PES collection based on the second in-tolerance splicing location.

### Claim 5 (depends on 4)

The device of claim 4, wherein the at least one processor is further configured to:
realign the processing buffer based on a PTS associated with one or more stored PES collections, wherein the processing buffer is configured as a first-in-first-out (FIFO) queue.

### Claim 6 (depends on 5)

The device of claim 5, wherein to realign the processing buffer, the at least one processor is further configured to:
move the third PES collection stored in the processing buffer to a location between the first PES collection and the second PES collection, such that the PES collections stored in the processing buffer are ordered in increasing PTS order.

### Claim 7 (depends on 1)

The device of claim 1, wherein the tolerance offset is plus or minus one frame for a given frame rate determined based on parameter information associated with a key frame.

### Claim 8

A computer implemented method, comprising:
receiving one or more variant segments for decoding and playback, the one or more variant segments including a first variant segment having a first bitrate and second and third variant segments having a second bitrate;
generating corresponding first, second, and third packetized elementary stream (PES) collections based on at least the first, second, and third variant segments;
storing the generated first, second, and third PES collections in a processing buffer; and
determining a second in-tolerance splicing location between the first PES collection and the third PES collection based on at least a tolerance offset, in response to a determination of a first out-of-tolerance splicing location between the first PES collection and the second PES location.

### Claim 9 (depends on 8)

The computer implemented method of claim 8, further comprising:
requesting the third variant segment having the second bitrate, in response to the determination of the first out-of-tolerance splicing location, wherein the third variant segment is temporally prior and adjacent to the second variant segment.

### Claim 10 (depends on 9)

The computer implemented method of claim 9, wherein the determining of the second in-tolerance splicing location further comprises:
generating a first key frames collection based on the first PES collection, wherein each key frames collection element of the first key frames collection includes a key frame, associated parameter information, and a corresponding PTS; and
generating a third key frames collection based on the third PES collection, wherein each key frames collection element of the second key frames collection includes a key frame, associated parameter information, and a corresponding PTS.

### Claim 11 (depends on 10)

The computer implemented method of claim 10, wherein the determining the second in-tolerance splicing location further comprises:
determining whether a PTS of the first key frames collection substantially matches a PTS of the third key frames collection; and
generating the second in-tolerance splicing location, in response to determining that the PTS of the first key frames collection substantially matches the PTS of the third key frames collection.

### Claim 12 (depends on 8)

The computer implemented method of claim 8, wherein the second in-tolerance splicing location includes a first reference to the first PES collection and a corresponding first PTS and a second reference to the third PES collection and a corresponding second PTS.

### Claim 13 (depends on 10)

The computer implemented method of claim 10, wherein the associated parameter information comprises video resolution information for indicating resolution of video encoded in a PES collection for playback, aspect ratio information for indicating the aspect ratio information of the video encoded in the PES collection, and/or frame rate information for indicating the frame rate of the video encoded in the PES collection.

### Claim 14 (depends on 11)

The computer implemented method of claim 11, wherein the PTS of the first key frames collection substantially matches the PTS of the third key frames collection, when the PTS of the first key frames collection equals the PTS of the third key frames collection or the PTS of the first key frames collection is within the tolerance offset of the PTS of the third key frames collection.

### Claim 15

A tangible computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:
receiving one or more variant segments for decoding and playback, the one or more variant segments including a first variant segment having a first bitrate and second and third variant segments having a second bitrate;
generating corresponding first, second and third packetized elementary stream (PES) collections based on at least the first, second, and third, variant segments;
storing the generated first, second, and third PES collections in a processing buffer; and
determining a second in-tolerance splicing location between the first PES collection and the third PES collection based on at least a tolerance offset in response to a determination of a first out-of-tolerance splicing location between the first PES collection and the second PES location.

### Claim 16 (depends on 15)

The computer-readable device of claim 15, wherein the operations further comprise:
requesting a third variant segment having the second bitrate, in response to the determination of the first out-of-tolerance splicing location, wherein the third variant segment is temporally prior and adjacent to the second variant segment.

### Claim 17 (depends on 16)

The computer-readable device of claim 16, wherein the second in-tolerance splicing location includes a first reference to the first PES collection and a corresponding first presentation timestamp (PTS) and a second reference to the third PES collection and a corresponding second PTS.

### Claim 18 (depends on 17)

The computer-readable device of claim 17, wherein the operation of splicing the first PES collection and the third PES collection further comprises:
identifying one or more first PES collection elements from the first referenced first PES collection having an associated PTS that matches or is temporally subsequent to the first PTS; and
removing the one or more identified first PES collection elements from the first referenced first PES collection.

### Claim 19 (depends on 18)

The computer-readable device of claim 18, wherein the operation of splicing the first PES collection and the third PES collection further comprises:
identifying one or more third PES collection elements from the second referenced third PES collection having an associated PTS that is temporally prior to the second PTS; and
removing the one or more identified third PES collection elements from the second referenced third PES collection.

### Claim 20 (depends on 15)

The computer-readable device of claim 15, wherein the first out-of-tolerance splicing location includes a first PTS and a second PTS that have a lowest calculated time difference among all calculated time differences between PTSes of the first PES collection and PTSes of the second PES collection.
