# US 2014/0192870 A1 · System And Method For Providing Error Resilience, Random Access And Rate Control In Scalable Video Communications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0192870 A1 - Document ID: US20140192870A1 - Status: Published application - Kind: A1 - Application number: 14/166,640 - Filed: 2014-01-28 - Published: 2014-07-10 - Earliest claimed: 2006-03-03 - Assignees: Vidyo, Inc. - Applicants: Vidyo, Inc. - Inventors: Alexandros Eleftheriadis; Danny Hong; Ofer Shapiro; Thomas Wiegand - Web page: https://patentgrep.com/patent/US20140192870A1 ## CPC - H04N 7/15: Conference systems - H04N 19/29: involving scalability at the object level, e.g. video object layer [VOL] - H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) - H04N 19/31: in the temporal domain - H04N 19/33: in the spatial domain - H04N 19/36: Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N 19/593: involving spatial prediction techniques - H04N 19/65: using error resilience - H04N 19/89: involving methods or arrangements for detection of transmission errors at the decoder - H04N 19/895: in combination with error concealment - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and methods for error resilient transmission, rate control, and random access in video communication systems that use scalable video coding are provided. Error resilience is obtained by using information from low resolution layers to conceal or compensate loss of high resolution layer information. The same mechanism is used for rate control by selectively eliminating high resolution layer information from transmitted signals, which elimination can be compensated at the receiver using information from low resolution layers. Further, random access or switching between low and high resolutions is also achieved by using information from low resolution layers to compensate for high resolution spatial layer packets that may have not been received prior to the switching time. ## Claims ### Claim 1 A digital video decoding system, the system comprising: a decoder configured to decode a received digital video signal, which is coded in a scalable video coding format supporting temporal scalability and at least one of spatial and quality scalability, wherein the scalable video coding format for spatial scalability includes a base spatial and at least one spatial enhancement layer, for quality scalability includes a base quality layer and at least one quality enhancement layer, and for temporal scalability includes a base temporal layer and at least one temporal enhancement layer, wherein the base temporal layers and enhancement temporal layers are interlinked by a threaded picture prediction structure for at least one of the spatial or quality scalability layers, wherein, for decoding a picture at a target spatial or quality layer higher than the corresponding base layer, the decoder is configured to use coded information from a spatial or quality layer of said picture lower than the target layer in the threaded prediction structure when a portion of the target layer's coded information is lost or not available. ### Claim 2 (depends on 1) The system of claim 1, wherein the digital video decoding system is disposed in a receiving endpoint, the system further comprising: a linking communication network; a conferencing server linked to the receiving endpoint and at least one transmitting endpoint by at least one communication channel each over the communication network, and at least one endpoint that transmits the coded digital video that is coded in the scalable video coding format, wherein the conferencing server is configured to selectively eliminate portions of input video signals received from transmitting endpoints that correspond to layers higher than the base spatial or quality layer, prior to creating an output video signal that is forwarded to the receiving endpoint. ### Claim 3 (depends on 2) The system of claim 2 wherein the conferencing server linked to the receiving endpoint and at least one transmitting endpoint is one of: a Transcoding Multipoint Control Unit using cascaded decoding and encoding; a Switching Multipoint Control Unit by selecting which input to transmit as output; a Scalable Video Communication Server using selective multiplexing; and a Compositing Scalable Video Communication Server using selective multiplexing and bitstream-level compositing. ### Claim 4 (depends on 2) The system of claim 2 wherein an encoder of the at least one transmitting endpoint is configured to encode transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“R”) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure so that the decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder, and wherein the server selectively eliminates portions of the input video signals received from transmitting endpoints that correspond to layers higher than the base spatial or quality layer in non-R frames only, prior to creating the output video signal that is forwarded to the receiving endpoint. ### Claim 5 (depends on 1) The system of claim 1, wherein the digital video decoding system is disposed in a receiving endpoint, the system further comprising: a transmitting endpoint that transmits coded digital video using a scalable video coding format; a communication network that links the transmitting endpoint with the receiving endpoint, wherein the transmitting endpoint is configured to selectively not transmit portions of its input video signal that correspond to layers higher than the base spatial or quality layer prior to creating an output video signal that is transmitted to the at least one receiving endpoint ### Claim 6 (depends on 5) The system of claim 5 wherein an encoder of the transmitting endpoint is configured to encode transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“R”) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder, and wherein the encoder selectively does not transmit to the at least one receiving endpoint portions of its input video signal that correspond to layers higher than the base spatial or quality layer in non-R frames only. ### Claim 7 (depends on 1) The system of claim 1, wherein the decoder is configured to display decoded output pictures at a desired spatial resolution that falls in between an immediately lower and an immediately higher spatial layer provided by the coded video signal. ### Claim 8 (depends on 1) The system of claim 1, wherein the decoder is further configured to operate a decoding loop of the immediately higher spatial layer at a desired spatial resolution by scaling all coded data of the immediately higher spatial layer to the desired spatial resolution, and wherein resultant drift is eliminated by using at least one of: periodic intra pictures; periodic use of intra base layer mode; and full resolution decoding of at least the lowest temporal layer of the immediately higher spatial layer. ### Claim 9 (depends on 1) The system of claim 1, wherein the scalable video coding format includes at least one of: periodic intra pictures, periodic intra macroblocks, and threaded picture prediction, in order to avoid drift when the target layer's coded information that is lost or is not available corresponds to the base temporal layer. ### Claim 10 (depends on 1) The system of claim 1, where the scalable video coding format is based on hybrid coding, the format comprising H.264, VC-1 or AVS standards, wherein the coded information from a spatial or quality layer lower than the target layer used by the decoder when some or all of the target layer's coded information is lost or is not available comprises at least one of: motion vector data, appropriately scaled for the target layer's resolution; coded prediction error difference, upsampled to the target layer's resolution; and intra data, upsampled to the target layer's resolution, and wherein the decoder is further configured to use the target layer's decoded pictures as references in the decoding process in order to construct decoded output pictures, rather than the lower layer decoded reference pictures. ### Claim 11 (depends on 1) The system of claim 1, wherein the decoder is further configured to operate at least one decoding loop for spatial or quality layers higher than the target spatial or quality layer for at least the base temporal layer, so that when the decoder switches target layers it can immediately display decoded pictures at the new target layer resolution. ### Claim 12 A method for decoding a digital video signal, comprising: receiving the digital video signal at a decoder, the digital video signal being coded in a scalable video coding format supporting temporal scalability and at least one of spatial and quality scalability, wherein the scalable video coding format for spatial scalability includes a base spatial and at least one spatial enhancement layer, for quality scalability includes a base quality layer and at least one quality enhancement layer, and for temporal scalability includes a base temporal layer and at least one temporal enhancement layer, wherein the base temporal layers and enhancement temporal layers are interlinked by a threaded picture prediction structure for at least one of the spatial or quality scalability layers; and decoding a picture at a target spatial or quality layer higher than the corresponding base layer using coded information from a spatial or quality layer of said picture lower than the target layer in the threaded prediction structure when a portion of the target layer's coded information is lost or not available. ### Claim 13 (depends on 12) The method of claim 12, wherein the decoder is disposed in a receiving endpoint in a linking communication network, wherein a conferencing server is linked to the receiving endpoint and at least one transmitting endpoint by at least one communication channel each over the communication network, and wherein the at least one transmitting endpoint transmits the coded digital video that is coded in the scalable video coding format, the method further comprising, at the conferencing server, selectively eliminating portions of input video signals received from transmitting endpoints that correspond to layers higher than the base spatial or quality layer, prior to creating an output video signal that is forwarded to the receiving endpoint. ### Claim 14 (depends on 13) The method of claim 13 wherein the conferencing server linked to the receiving endpoint and at least one transmitting endpoint is one of: a Transcoding Multipoint Control Unit using cascaded decoding and encoding; a Switching Multipoint Control Unit by selecting which input to transmit as output; a Scalable Video Communication Server using selective multiplexing; and a Compositing Scalable Video Communication Server using selective multiplexing and bitstream-level compositing. ### Claim 15 (depends on 13) The method of claim 13, further comprising, at an encoder of the at least one transmitting endpoint, encoding transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“R”) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure so that the decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder, and wherein the server selectively eliminates portions of the input video signals received from transmitting endpoints that correspond to layers higher than the base spatial or quality layer in non-R frames only, prior to creating the output video signal that is forwarded to the receiving endpoint. ### Claim 16 (depends on 12) The method of claim 12, wherein a transmitting endpoint transmits coded digital video using a scalable video coding format; wherein a communication network links the transmitting endpoint with the receiving endpoint, the method further comprising, at the transmitting endpoint, selectively not transmitting portions of its input video signal that correspond to layers higher than the base spatial or quality layer, prior to creating an output video signal that is transmitted to the at least one receiving endpoint. ### Claim 17 (depends on 16) The method of claim 16, further comprising, at the transmitting endpoint encoding transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“R”) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with an encoder, and wherein the encoder selectively does not transmit to the at least one receiving endpoint portions of its input video signal that correspond to layers higher than the base spatial or quality layer in non-R frames only. ### Claim 18 (depends on 12) The method of claim 12, further comprising, at the decoder, displaying decoded output pictures at a desired spatial resolution that falls in between an immediately lower and an immediately higher spatial layer provided by the coded video signal. ### Claim 19 (depends on 12) The method of claim 12, further comprising, at the decoder, operating a decoding loop of the immediately higher spatial layer at a desired spatial resolution by scaling all coded data of the immediately higher spatial layer to the desired spatial resolution, and wherein resultant drift is eliminated by using at least one of: periodic intra pictures; periodic use of intra base layer mode; and full resolution decoding of at least the lowest temporal layer of the immediately higher spatial layer. ### Claim 20 (depends on 12) The method of claim 12, wherein the scalable video coding format includes at least one of: periodic intra pictures, periodic intra macroblocks, and threaded picture prediction, in order to avoid drift when the target layer's coded information that is lost or is not available corresponds to the base temporal layer. ### Claim 21 (depends on 12) The method of claim 12, where the scalable video coding format is based on hybrid coding, the format comprising H.264, VC-1 or AVS standards, wherein the coded information from a spatial or quality layer lower than the target layer used by the decoder when some or all of the target layer's coded information is lost or is not available comprises at least one of: motion vector data, appropriately scaled for the target layer's resolution; coded prediction error difference, upsampled to the target layer's resolution; and intra data, upsampled to the target layer's resolution, the method further comprising, at the decoder using the target layer's decoded pictures as references in the decoding process in order to construct decoded output pictures, rather than the lower layer decoded reference pictures. ### Claim 22 (depends on 12) The method of claim 12 further comprising, at the decoder operating at least one decoding loop for spatial or quality layers higher than the target spatial or quality layer for at least the base temporal layer, so that when the decoder switches target layers it can immediately display decoded pictures at the new target layer resolution. ### Claim 23 A non-transitory computer readable medium comprising a set of executable instructions to direct a processor to decode a digital video signal, by: receiving the digital video signal at a decoder, the digital video signal being coded in a scalable video coding format supporting temporal scalability and at least one of spatial and quality scalability, wherein the scalable video coding format for spatial scalability includes a base spatial and at least one spatial enhancement layer, for quality scalability includes a base quality layer and at least one quality enhancement layer, and for temporal scalability includes a base temporal layer and at least one temporal enhancement layer, wherein the base temporal layers and enhancement temporal layers are interlinked by a threaded picture prediction structure for at least one of the spatial or quality scalability layers; and decoding a picture at a target spatial or quality layer higher than the corresponding base layer using coded information from a spatial or quality layer of said picture lower than the target layer in the threaded prediction structure when a portion of the target layer's coded information is lost or not available. ### Claim 24 (depends on 23) The non-transitory computer readable medium of claim 23, wherein the decoder is disposed in a receiving endpoint in a linking communication network, wherein a conferencing server is linked to the receiving endpoint and at least one transmitting endpoint by at least one communication channel each over the communication network, and wherein the at least one transmitting endpoint transmits the coded digital video that is coded in the scalable video coding format, further comprising executable instructions to direct the processor to, at the conferencing server, selectively eliminating portions of input video signals received from transmitting endpoints that correspond to layers higher than the base spatial or quality layer, prior to creating an output video signal that is forwarded to the receiving endpoint. ### Claim 25 (depends on 24) The non-transitory computer readable medium of claim 24, wherein the conferencing server linked to the receiving endpoint and at least one transmitting endpoint is one of: a Transcoding Multipoint Control Unit using cascaded decoding and encoding; a Switching Multipoint Control Unit by selecting which input to transmit as output; a Scalable Video Communication Server using selective multiplexing; and a Compositing Scalable Video Communication Server using selective multiplexing and bitstream-level compositing. ### Claim 26 (depends on 24) The non-transitory computer readable medium of claim 24, further comprising executable instructions to direct the processor to, at an encoder of the at least one transmitting endpoint, encoding transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“R”) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure so that the decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with the encoder, and wherein the server selectively eliminates portions of the input video signals received from transmitting endpoints that correspond to layers higher than the base spatial or quality layer in non-R frames only, prior to creating the output video signal that is forwarded to the receiving endpoint. ### Claim 27 (depends on 23) The non-transitory computer readable medium of claim 23, wherein a transmitting endpoint transmits coded digital video using a scalable video coding format; wherein a communication network links the transmitting endpoint with the receiving endpoint, further comprising executable instructions to direct the processor to, at the transmitting endpoint, selectively not transmitting portions of its input video signal that correspond to layers higher than the base spatial or quality layer, prior to creating an output video signal that is transmitted to the at least one receiving endpoint. ### Claim 28 (depends on 27) The non-transitory computer readable medium of claim 27, further comprising executable instructions to direct the processor to, at the transmitting endpoint encoding transmitted media as frames in a threaded coding structure having a number of different temporal levels, wherein a subset of the frames (“R”) is particularly selected for reliable transport and includes at least the frames of the lowest temporal layer in the threaded coding structure and such that the decoder can decode at least a portion of received media based on a reliably received frame of the type R after packet loss or error and thereafter is synchronized with an encoder, and wherein the encoder selectively does not transmit to the at least one receiving endpoint portions of its input video signal that correspond to layers higher than the base spatial or quality layer in non-R frames only. ### Claim 29 (depends on 23) The non-transitory computer readable medium of claim 23, further comprising executable instructions to direct the processor to, at the decoder, displaying decoded output pictures at a desired spatial resolution that falls in between an immediately lower and an immediately higher spatial layer provided by the coded video signal. ### Claim 30 (depends on 23) The non-transitory computer readable medium of claim 23, further comprising executable instructions to direct the processor to, at the decoder, operating a decoding loop of the immediately higher spatial layer at a desired spatial resolution by scaling all coded data of the immediately higher spatial layer to the desired spatial resolution, and wherein resultant drift is eliminated by using at least one of: periodic intra pictures; periodic use of intra base layer mode; and full resolution decoding of at least the lowest temporal layer of the immediately higher spatial layer. ### Claim 31 (depends on 23) The non-transitory computer readable medium of claim 23, wherein the scalable video coding format includes at least one of: periodic intra pictures, periodic intra macroblocks, and threaded picture prediction, in order to avoid drift when the target layer's coded information that is lost or is not available corresponds to the base temporal layer. ### Claim 32 (depends on 23) The non-transitory computer readable medium of claim 23, where the scalable video coding format is based on hybrid coding, the format comprising H.264, VC-1 or AVS standards, wherein the coded information from a spatial or quality layer lower than the target layer used by the decoder when some or all of the target layer's coded information is lost or is not available comprises at least one of: motion vector data, appropriately scaled for the target layer's resolution; coded prediction error difference, upsampled to the target layer's resolution; and intra data, upsampled to the target layer's resolution, further comprising executable instructions to direct the processor to, at the decoder using the target layer's decoded pictures as references in the decoding process in order to construct decoded output pictures, rather than the lower layer decoded reference pictures. ### Claim 33 (depends on 23) The non-transitory computer readable medium of claim 23, further comprising executable instructions to direct the processor to, at the decoder operating at least one decoding loop for spatial or quality layers higher than the target spatial or quality layer for at least the base temporal layer, so that when the decoder switches target layers it can immediately display decoded pictures at the new target layer resolution.