# US 9,883,136 B2 · System and method of outputting multi-lingual audio and associated audio from a single container > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,883,136 B2 - Document ID: US9883136B2 - Status: Granted - Kind: B2 - Application number: 14/787,744 - Filed: 2014-04-08 - Published: 2016-03-17 - Granted: 2018-01-30 - Earliest claimed: 2013-04-30 - Assignees: Dolby Laboratories Licensing Corporation - Applicants: Dolby Laboratories Licensing Corporation - Inventors: Hugues A. De Saint Salvy; Gregory John McGarry - Web page: https://patentgrep.com/patent/US9883136B2 ## CPC - H04N 5/91: Television signal processing therefor - G11B 27/3027: {used signal is digitally coded} - H04N 21/4302: {Content synchronisation processes, e.g. decoder synchronisation} - H04N 21/8106: {involving special audio data, e.g. different tracks for different languages} - H04N 21/85406: {involving a specific file format, e.g. MP4 format} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract A method of storing and outputting associated audio. The associated audio and the main audio are stored in, and output from, a single multimedia container. In this manner, when the main language is changed, the associated audio automatically changes with minimal audio artifacts. ## Claims ### Claim 1 A method of outputting audio, comprising: progressively downloading a plurality of main audio data sets from a first server and a plurality of associated audio data sets from a second server, wherein the plurality of main audio data sets and the plurality of associated audio data sets are not interleaved on the first server and the second server; storing a multimedia container, wherein the multimedia container includes: the plurality of main audio data sets and the plurality of associated audio data sets, wherein the plurality of main audio data sets and the plurality of associated audio data sets are in a plurality of languages and data elements of the plurality of main audio data sets are interleaved with data elements of the plurality of associated audio data sets in the multimedia container; and synchronization information that synchronizes one of the plurality of main audio data sets and one of the plurality of associated audio data sets; receiving first selection information, wherein the first selection information corresponds to a first language of the plurality of languages; outputting a first audio data stream from the multimedia container after a defined amount of the multimedia container has been progressively downloaded and as the multimedia container is being progressively downloaded, wherein the first audio data stream corresponds to a first main audio data set of the plurality of main audio data sets in the first language; and outputting a second audio data stream from the multimedia container concurrently with outputting the first audio data stream according to the synchronization information, wherein the second audio data stream corresponds to a first associated audio data set of the plurality of associated audio data sets in the first language. ### Claim 2 (depends on 1) The method of claim 1 further comprising: receiving second selection information, wherein the second selection information relates to the plurality of associated audio data sets; receiving third selection information, wherein the third selection information corresponds to a second selected language of the plurality of languages; outputting a third audio data stream instead of the first audio data stream from the multimedia container, wherein the third audio data stream corresponds to a second selected main audio data set of the plurality of main audio data sets in the second selected language; and automatically outputting a fourth audio data stream instead of the second audio data stream from the multimedia container, wherein the fourth audio data stream corresponds to a second selected associated data set of the plurality of associated audio data sets in the second selected language. ### Claim 3 (depends on 2) The method of claim 2, wherein automatically outputting the fourth audio data stream comprises: automatically outputting the fourth audio data stream from the multimedia container concurrently with outputting the third audio data stream. ### Claim 4 (depends on 2) The method of claim 2, further comprising: concurrently reading the data elements of the second selected main audio data set and the data elements of the second selected associated audio data set. ### Claim 5 (depends on 2) The method of claim 2, wherein automatically outputting the fourth audio data stream comprises: automatically outputting the fourth audio data stream from the multimedia container concurrently with outputting the third audio data stream according to the synchronization information. ### Claim 6 (depends on 2) The method of claim 2, further comprising: concurrently reading data elements of the second selected main audio data set and data elements of the second selected associated audio data set according to the synchronization information. ### Claim 7 (depends on 2) The method of claim 2, wherein the multimedia container includes metadata that relates to the plurality of main audio data sets and the plurality of associated audio data sets, further comprising: outputting the second audio data stream from the multimedia container concurrently with outputting the first audio data stream, in accordance with the metadata. ### Claim 8 (depends on 2) The method of claim 2, wherein the plurality of associated audio data sets includes a director's commentary data set, further comprising: receiving fourth selection information, wherein the fourth selection information corresponds to the director's commentary data set; outputting a fifth audio data stream instead of the fourth audio data stream from the multimedia container, wherein the fifth audio data stream corresponds to the director's commentary data set. ### Claim 9 (depends on 1) The method of claim 1, further comprising: concurrently reading the data elements of the first main audio data set and the data elements of the first associated audio data set. ### Claim 10 (depends on 1) The method of claim 1, further comprising: concurrently reading data elements of the first main audio data set and data elements of the first associated audio data set according to the synchronization information. ### Claim 11 (depends on 1) The method of claim 1, wherein one of the plurality of associated audio data sets is associated with one of the plurality of main audio data sets, and wherein a number of channels of the one of the plurality of associated audio data sets is not greater than a number of channels of the one of the plurality of main audio data sets. ### Claim 12 (depends on 1) The method of claim 1, wherein the multimedia container includes metadata that relates to the plurality of main audio data sets and the plurality of associated audio data sets. ### Claim 13 (depends on 1) The method of claim 1, wherein the multimedia container includes metadata that relates to the plurality of main audio data sets and the plurality of associated audio data sets, wherein the metadata includes the synchronization information. ### Claim 14 (depends on 1) The method of claim 1, wherein the multimedia container includes metadata that relates to the plurality of main audio data sets and the plurality of associated audio data sets, further comprising: interleaving data elements of the plurality of main audio data sets with data elements of the plurality of associated audio data sets in the multimedia container, in accordance with the metadata. ### Claim 15 (depends on 1) The method of claim 1, wherein the plurality of associated audio data sets include a plurality of audio description data sets. ### Claim 16 (depends on 1) The method of claim 1, wherein playback of a video stream is paused when switching from the first audio data stream to the third audio data stream requires more than a defined period. ### Claim 17 (depends on 1) The method of claim 1, wherein switching from the second audio data stream to the fourth audio data stream has smooth switching without audio artifacts. ### Claim 18 (depends on 1) The method of claim 1, wherein the data elements of the plurality of main audio data sets and the data elements of the plurality of associated audio data sets are interleaved in the multimedia container according to a physical arrangement wherein a portion of the data elements of the plurality of main audio data sets and a corresponding portion of the data elements of the plurality of associated audio data sets that are to be output synchronously are interleaved before a next portion of video data. ### Claim 19 An apparatus for processing a multimedia container, the apparatus comprising: a processor configured to progressively download a plurality of main audio data sets from a first server and a plurality of associated audio data sets from a second server, wherein the plurality of main audio data sets and the plurality of associated audio data sets are not interleaved on the first server and the second server; a storage device that is coupled to the processor and that is configured to store a multimedia container, wherein the multimedia container includes the plurality of main audio data sets, the plurality of associated audio data sets and synchronization information, wherein the plurality of main audio data sets and the plurality of associated audio data sets are in a plurality of languages and data elements of the plurality of main audio data sets are interleaved with data elements of the plurality of associated audio data sets in the multimedia container, and wherein the synchronization information synchronizes one of the plurality of main audio data sets and one of the plurality of associated audio data sets; and an audio decoder that outputs a first and second audio data stream from the multimedia container, concurrently, according to the synchronization information, after a defined amount of the multimedia container has been progressively downloaded and as the multimedia container is being progressively downloaded, wherein the first audio data stream corresponds to a main audio data set of the plurality of main audio data sets in a first language, and the second audio data stream corresponds to an associated audio data set of the plurality of associated audio data sets in the first language. ### Claim 20 A computer program stored in a non-transitory computer readable medium for controlling an apparatus including a processor and a storage device for outputting multi-lingual audio and associated audio, to execute processing comprising: progressively downloading a plurality of main audio data sets from a first server and a plurality of associated audio data sets from a second server, wherein the plurality of main audio data sets and the plurality of associated audio data sets are not interleaved on the first server and the second server; storing a multimedia container, wherein the multimedia container includes: the plurality of main audio data sets and the plurality of associated audio data sets, wherein the plurality of main audio data sets and the plurality of associated audio data sets are in a plurality of languages and data elements of the plurality of main audio data sets are interleaved with data elements of the plurality of associated audio data sets in the multimedia container; and synchronization information that synchronizes one of the plurality of main audio data sets and one of the plurality of associated audio data sets; receiving first selection information, wherein the first selection information corresponds to a first language of the plurality of languages; and outputting a first audio data stream from the multimedia container after a defined amount of the multimedia container has been progressively downloaded and as the multimedia container is being progressively downloaded, wherein the first audio data stream corresponds to a first main audio data set of the plurality of main audio data sets in the first language; and outputting a second audio data stream from the multimedia container concurrently with outputting the first audio data stream according to the synchronization information, wherein the second audio data stream corresponds to a first associated audio data set of the plurality of associated audio data sets in the first language.