# US 10,484,447 B2 · Methods and apparatus for communicating delay information and minimizing delays > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,447 B2 - Document ID: US10484447B2 - Status: Granted - Kind: B2 - Application number: 14/582,770 - Filed: 2014-12-24 - Published: 2016-06-30 - Granted: 2019-11-19 - Earliest claimed: 2014-12-24 - Assignees: Ribbon Communications Operating Company, Inc. - Applicants: Ribbon Communications Operating Company, Inc. - Inventors: Tolga Asveren - Web page: https://patentgrep.com/patent/US10484447B2 ## CPC - H04L 65/65: Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP] - H04L 65/764: {at the destination (reformatting of additional data in video clients H04N21/4355)} - H04L 65/80: Responding to QoS - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Features relating to communicating delay information and minimizing aggregate processing delays corresponding to content streams, e.g., audio and video streams, in a multi-media communication session while still achieving stream synchronization, are described. Some embodiments are well suited for systems where video and audio streams are transcoded by different entities. In some embodiments downstream entities, e.g., transcoder and/or receiving end device, are informed about the delay introduced to at least one stream by one or more upstream devices, e.g., a transcoder. The downstream entity synchronizes the received content streams by adding artificial latency as needed to the relevant one of the streams based on the received delay information thereby avoiding unnecessary introduction of artificial delays at every transcoding node in the path. Received delay information for a content stream may indicate one or more individual transcoding delays introduced into the stream or an aggregate of transcoding delays introduced into the stream. ## Claims ### Claim 1 A method of operating a playback device, the method comprising: receiving, at interface circuitry of the playback device, a first content stream which was subject to transcoding by a first transcoder device prior to receipt, said first content stream including first content; receiving, at the interface circuitry of the playback device, a second content stream, said second content stream including second content, said first content and said second content being different, said first content being video content and said second content being audio content; receiving, at the interface circuitry of the playback device, from the first transcoder device information indicating a first transcoding delay introduced into said first content stream prior to receipt by the playback device; generating, at the playback device, a stream delay difference between said first content stream and said second content stream based on said received first transcoding delay; and delaying, by the playback device, the playback of the second content stream by an amount equal to the generated stream delay difference wherein said playback device outputs synchronized video and audio content; wherein generating, at the playback device, a stream delay difference between said first content stream and said second content stream includes: operating circuitry included in said playback device to generate, using at least the first transcoding delay, a difference between latency corresponding to said first content stream and latency corresponding to said second content stream, said generated difference being said stream delay difference; wherein said operating circuitry in said playback device to generate a difference between latency corresponding to said first content stream and latency corresponding to the second content stream includes: generating a first aggregate stream latency from the received first transcoding delay and one or more additional latencies corresponding to said first content stream, said one or more additional latencies including a third transcoding delay introduced into the first content stream by an additional transcoder device located between the source of the first content stream and the first transcoder device; generating a second aggregate stream latency from a plurality of latencies corresponding to said second content stream; and subtracting said second aggregate stream latency from said first aggregate stream latency when said first aggregate stream latency is larger than said second aggregate stream latency. ### Claim 2 (depends on 1) The method of claim 1, wherein said received information indicating a first transcoding delay introduced into said first content stream is communicated to the playback device in a Session Initiation Protocol message header corresponding to the first content stream. ### Claim 3 (depends on 1) The method of claim 1, further comprising: receiving, by the Input/Output interface circuitry of the playback device, information indicating the one or more additional latencies corresponding to the first content stream; and receiving, by the Input/Output interface circuitry of the playback device, information indicating the plurality of latencies corresponding to the second content stream. ### Claim 4 (depends on 3) The method of claim 3, wherein said information indicating one or more additional latencies corresponding to the first content stream is communicated by one of a first SDP (Session Description Protocol) attribute, a first SIP message header, a SIP parameter, or a first signaling port used to indicate latency information corresponding to the first content stream. ### Claim 5 (depends on 1) The method of claim 1, wherein said first and said second content streams are Real-time Transport Protocol content streams. ### Claim 6 A playback device comprising: interface circuitry configured to: receive a first content stream which was subject to transcoding by a first transcoder device prior to receipt, said first content stream including first content; and receive a second content stream, said second content stream including second content, said first and said second content being different, said first content being video content and said second content being audio content; receive from the first transcoder device information indicating a first transcoding delay introduced into said first content stream prior to receipt by the playback device; a processor configured to: generate a first aggregate stream latency from the first transcoding delay and one or more additional latencies corresponding to said first content stream; generate a second aggregate stream latency from a plurality of latencies corresponding to said second content stream; generate a stream delay difference between said first content stream and said second content stream based on said received first transcoding delay; and delay the playback of the second content stream by an amount equal to the generated stream delay difference outputting synchronized video and audio content; and wherein said processor is further configured to subtract said second aggregate stream latency from said first aggregate stream latency to generate said stream delay difference when said first aggregate stream latency is larger than said second aggregate stream latency. ### Claim 7 (depends on 6) The playback device of claim 6, wherein said information indicating the first transcoding delay introduced into said first content stream is communicated to the playback device as a Session Description Protocol message attribute. ### Claim 8 (depends on 6) The playback device of claim 6, wherein said first and said second content streams are Real-time Transport Protocol content streams. ### Claim 9 (depends on 6) The playback device of claim 6, wherein said one or more additional latencies corresponding to said first content stream include at least one additional latency introduced into said first content stream by a second transcoder prior to the introduction of the first transcoding delay introduced into the first content stream by the first transcoder; and wherein said plurality of latencies corresponding to said second content stream include a latency introduced into the second content stream by a second transcoding device prior to receipt of the second content stream by the playback device. ### Claim 10 A method of communicating and processing content streams corresponding to a multi-media communications session, the method comprising: performing, at a first transcoder device, a transcoding operation on a first content stream which is part of said multi-media communications session which also includes a second content stream; communicating information to a downstream playback device indicating a delay introduced into the first content stream by said transcoding operation performed on the first content stream, said first content stream including first content and said second content stream including second content, said first and said second content being different, said first content being audio content and said second content being video content; and communicating additional information to said downstream playback device indicating one or more latencies introduced into said first content stream, said one or more latencies having been introduced into said first content stream by one or more devices upstream of said first transcoder device; generating, by the downstream playback device, a first aggregate stream latency based on the information indicating a delay introduced into the first content stream by said transcoding operation performed on the first content stream and the one or more latencies introduced into said first content stream by said one or more devices upstream of said first transcoder device; generating, by the downstream playback device, a second aggregate stream latency from a plurality of latencies corresponding to said second content stream; generating a stream delay difference between said first content stream and said second content stream based on said generated first aggregate stream latency and said generated second aggregate stream latency; and when said second aggregate stream latency is larger than said first aggregate stream latency: (i) subtracting, by the downstream playback device, said first aggregate stream latency from said second aggregate stream latency to generate said stream delay difference, and (ii) delaying, by the downstream playback device, the playback of the first content stream by an amount equal to the generated stream delay difference outputting synchronized video and audio content; and when said first aggregate stream latency is larger than or equal to said second aggregate stream latency: (i) subtracting, by said downstream playback device, said second aggregate stream latency from said first aggregate stream latency to generate said stream delay difference, and (ii) delaying, by said downstream playback device, the playback of the second content stream by an amount equal to the generated stream delay difference outputting synchronized video and audio content. ### Claim 11 (depends on 10) The method of claim 10, further comprising: performing, at a second transcoder device, a transcoding operation on the second content stream; communicating information to said downstream playback device indicating a latency introduced into the second content stream by said transcoding operation performed on the second content stream; and wherein said plurality of latencies corresponding to said second content stream include the latency introduced into the second content stream by said transcoding operation performed on the second content stream. ### Claim 12 (depends on 11) The method of claim 11, wherein said second transcoder device is a separate device from said first transcoder device and wherein the latency introduced into the second content stream by the second transcoder device is different from the delay introduced into the first content stream by the first transcoder device. ### Claim 13 (depends on 11) The method of claim 11 further comprising: wherein said first and said second content streams are Real-time Transport Protocol content streams. ### Claim 14 (depends on 11) The method of claim 11, wherein said information indicating a delay introduced into the first content stream by said transcoding operation performed on the first content stream is included in one of the following: a first Session Description Protocol (SDP) attribute, a first Session Initiation Protocol (SIP) message header, or a SIP parameter; and wherein said information indicating a latency introduced into the second content stream by said transcoding operation performed on the second content stream is included in one of the following: a second SDP attribute, a second SIP message header, or a second SIP parameter. ### Claim 15 (depends on 10) The method of claim 10, wherein said information indicating a delay introduced into the first content stream by said transcoding operation performed on the first content stream is included in one of the following: a first Session Description Protocol (SDP) attribute, a first Session Initiation Protocol (SIP) message header, a SIP parameter, or a first transcoding delay notification message. ### Claim 16 A non-transitory computer readable medium including a first set of computer executable instructions which when executed by a processor of a playback device cause the playback device to: receive a first content stream which was subject to transcoding by a first transcoder device prior to receipt, said first content stream including first content; receive a second content stream, said second content stream including second content, said first and said second content being different, said first content being video content and said second content being audio content; receive from the first transcoder device information indicating a first transcoding delay introduced into said first content stream prior to receipt by the playback device; generate a first aggregate stream latency from the first transcoding delay and one or more additional latencies corresponding to said first content stream; generate a second aggregate stream latency from a plurality of latencies corresponding to said second content stream; generate a stream delay difference between said first content stream and said second content stream based on said generated first aggregate stream latency and said generated second aggregate stream latency; and when said second aggregate stream latency is larger than said first aggregate stream latency: (i) subtract said first aggregate stream latency from said second aggregate stream latency to generate said stream delay difference, and (ii) delay the playback of the first content stream by an amount equal to said generated stream delay difference outputting synchronized video and audio content; and when said first aggregate stream latency is larger than or equal to said second aggregate stream latency: (i) subtract said second aggregate stream latency from said first aggregate stream latency to generate said stream delay difference, and (ii) delay the playback of the second content stream by an amount equal to said generated stream delay difference outputting synchronized video and audio content. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, wherein said one or more additional latencies corresponding to said first content stream include at least one additional latency introduced into said first content stream by a second transcoder prior to the introduction of the first transcoding delay introduced into the first content stream by the first transcoder. ### Claim 18 (depends on 16) The non-transitory computer readable medium of claim 16, wherein said plurality of latencies corresponding to said second content stream include a latency introduced into the second content stream by a second transcoding device prior to receipt of the second content stream by the playback device. ### Claim 19 (depends on 16) The non-transitory computer readable medium of claim 16, wherein said received information indicating a first transcoding delay introduced into said first content stream is communicated to the playback device in a Session Initiation Protocol message header corresponding to the first content stream. ### Claim 20 (depends on 16) The non-transitory computer readable medium of claim 16, wherein said first and said second content streams are Real-time Transport Protocol content streams.