# US 10,484,694 B2 · On-device multiplexing of streaming media content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,694 B2 - Document ID: US10484694B2 - Status: Granted - Kind: B2 - Application number: 14/685,558 - Filed: 2015-04-13 - Published: 2015-08-06 - Granted: 2019-11-19 - Earliest claimed: 2008-12-22 - Assignees: Netflix, Inc. - Applicants: Netflix, Inc - Inventors: David Randall Ronca; Chung-Ping Wu; Yung-Hsiao Lai - Web page: https://patentgrep.com/patent/US10484694B2 ## CPC - H04N 19/167: Position within a video image, e.g. region of interest [ROI] - H04L 65/60: Network streaming of media packets - H04L 65/70: Media network packetisation - H04N 19/115: Selection of the code volume for a coding unit prior to coding - H04N 19/164: Feedback from the receiver or from the transmission channel - H04N 19/177: the unit being a group of pictures [GOP] - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/61: in combination with predictive coding - H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement} - H04N 21/23439: {for generating different versions} - H04N 21/23611: {Insertion of stuffing data into a multiplex stream, e.g. to obtain a constant bitrate (synchronisation arrangements in time-division multiplex systems using bit stuffing for systems with different or fluctuating information rates H04J3/073)} - H04N 21/2368: Multiplexing of audio and video streams - H04N 21/26258: {for generating a list of items to be played back in a given order, e.g. playlist, or scheduling item distribution according to such list (retrieval of multimedia data based on playlists G06F16/40)} - H04N 21/2662: Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities - H04N 21/4325: {by playing back content from the storage medium (reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79)} - H04N 21/4346: {involving stuffing data, e.g. packets or bytes (synchronisation arrangements in time-division multiplex systems with different or fluctuating information rates H04J3/073)} - H04N 21/44016: {involving splicing one content stream with another content stream, e.g. for substituting a video clip} - H04N 21/44029: {for generating different versions} - H04N 21/4621: {Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen (arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Techniques are disclosed for multiplexing a dynamic bit-rate video stream with an audio stream received by a client device in a manner that allows the resulting multiplexed stream to be played back without disruption, despite dynamic changes in the bit rate of the video stream that may occur. A content server may stream both a video stream and an audio stream to a client device for playback. The client device may multiplex the video and audio streams prior to them being presented to a playback engine for decoding and playback to a user. ## Claims ### Claim 1 A computer-implemented method, comprising: identifying within each portion of video data included in a first video encoding of a media file one or more insertion points where a sub-portion of video data is multiplexed with a sub-portion of audio data, wherein the first video encoding comprises the media file encoded at a first bit rate; generating a first file header for the first video encoding that includes an indication of a position of each insertion point within the first video encoding; generating a second file header for an audio encoding of the media file that includes an indication of a position of each point within the audio encoding where a sub-portion of audio data is multiplexed with a sub-portion of video data; and storing the first video encoding and the audio encoding file for transmission to the client device, wherein the client device is configured to multiplex transmitted sub-portions of video data with transmitted sub-portions of audio data based on the indications of the positions of the insertions points included in the first file header and the indications of the positions of the points included in the second file header. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising padding at least one portion of video data included in the first video encoding to align the at least one portion with a continuity count boundary. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the first video encoding comprises a sequence of one or more groups of pictures (GOPs). ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein each GOP is encapsulated in a sequence of MPEG-2 transport stream packets, and wherein padding the at least one portion of video data to the at least one portion with a continuity count boundary comprises padding the at least one portion of video data with video filler packets such that each GOP begins with an MPEG-packet having a continuity count of 0 and ends with a packet having a continuity count of 15. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: aligning at least one portion of video data included in the first video encoding to an encryption block boundary; and aligning at least one portion of audio data included in the audio encoding to an encryption block boundary. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein aligning the at least one portion of video data comprises at least one of adding null padding packets to the at least one portion of video data and borrowing packets from a subsequent portion of video data, and wherein aligning the at least one portion of audio data comprises at least one of adding null padding packets to the at least one portion of audio data and borrowing packets from a subsequent portion of audio data. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, wherein the encryption block boundary comprises an AACS encryption block boundary. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving a request to stream the media file from the client device; transmitting the first file header for the first video encoding to the client device; transmitting the second file header for the audio encoding to the client device; and in response to one or more subsequent requests from the client device, transmitting one or more sub-portions of video data from the first video encoding to the client device, and transmitting one or more sub-portions of audio data from the audio encoding to the client device. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the first file header further includes a field indicating the size of the file header, a byte offset indicating where the video data included in the first video encoding begins, and an index indicating a position of each portion of video data included in the first video encoding as well as the indications of the positions of the insertion point within the first video encoding. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the second file header further includes a byte or packet offset for each portion of audio data included in the audio encoding. ### Claim 11 A non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to encode a media file by performing the steps of: generating a first file header for a first video encoding of a media file that includes an indication of a position of each insertion point within the first video encoding where a sub-portion of video data is multiplexed with a sub-portion of audio data; generating a second file header for an audio encoding of the media file that includes an indication of a position of each point within the audio encoding where a sub-portion of audio data is multiplexed with a sub-portion of video data; and storing the first video encoding and the audio encoding file for transmission to the client device, wherein the client device is configured to multiplex transmitted sub-portions of video data with transmitted sub-portions of audio data based on the indications of the positions of the insertions points included in the first file header. ### Claim 12 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the first video encoding includes a plurality of portions of video data, and further comprising padding at least one portion of video data to align the at least one portion with a continuity count boundary. ### Claim 13 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the first video encoding comprises a sequence of one or more groups of pictures (GOPs). ### Claim 14 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein each GOP is encapsulated in a sequence of MPEG-2 transport stream packets, and wherein padding the at least one portion of video data to the at least one portion with a continuity count boundary comprises padding the at least one portion of video data with video filler packets such that each GOP begins with an MPEG-packet having a continuity count of 0 and ends with a packet having a continuity count of 15. ### Claim 15 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the first video encoding includes a plurality of portions of video data, and further comprising: aligning at least one portion of video data included in the first video encoding to an encryption block boundary; and aligning at least one portion of audio data included in the audio encoding to an encryption block boundary. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein aligning the at least one portion of video data comprises at least one of adding null padding packets to the at least one portion of video data and borrowing packets from a subsequent portion of video data, and wherein aligning the at least one portion of audio data comprises at least one of adding null padding packets to the at least one portion of audio data and borrowing packets from a subsequent portion of audio data. ### Claim 17 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the encryption block boundary comprises an AACS encryption block boundary. ### Claim 18 (depends on 11) The non-transitory computer-readable storage medium of claim 11, further comprising: receiving a request to stream the media file from the client device; transmitting the first file header for the first video encoding to the client device; transmitting the second file header for the audio encoding to the client device; and in response to one or more subsequent requests from the client device, transmitting one or more sub-portions of video data from the first video encoding to the client device, and transmitting one or more sub-portions of audio data from the audio encoding to the client device. ### Claim 19 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the first file header further includes a field indicating the size of the file header, a byte offset indicating where the video data included in the first video encoding begins, and an index indicating a position of each portion of video data included in the first video encoding as well as the indications of the positions of the insertion point within the first video encoding. ### Claim 20 A server machine, comprising: a memory that includes an encoding tool; a storage element that includes a first video encoding of a media file and an audio encoding of the media file for transmission to a client device, wherein the first video encoding includes a plurality of portions of video data, and the audio encoding includes a plurality of portions of audio data; and a processor coupled to the memory and the storage element, wherein, when executed by the processor, the encoding tool configures the processor to: generate a first file header for a first video encoding of a media file that includes an indication of a position of each insertion point within the first video encoding where a sub-portion of video data is multiplexed with a sub-portion of audio data, and generate a second file header for an audio encoding of the media file that includes an indication of a position of each point within the audio encoding where a sub-portion of audio data is multiplexed with a sub-portion of video data, wherein the client device is configured to multiplex transmitted sub-portions of video data with transmitted sub-portions of audio data based on the indications of the positions of the insertions points included in the first file header.