# US 10,123,027 B2 · Method and apparatus for video coding and decoding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,123,027 B2 - Document ID: US10123027B2 - Status: Granted - Kind: B2 - Application number: 14/583,371 - Filed: 2014-12-26 - Published: 2015-10-22 - Granted: 2018-11-06 - Earliest claimed: 2014-01-07 - Assignees: Nokia Technologies Oy - Applicants: Nokia Technologies Oy - Inventors: Miska Matias Hannuksela; Kemal Ugur - Web page: https://patentgrep.com/patent/US10123027B2 ## CPC - H04N 19/187: the unit being a scalable video layer - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) - H04N 19/463: by compressing encoding parameters before transmission - H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Various methods, apparatuses and computer program products for video encoding and decoding are provided. In some embodiments, to indicate a composition of pictures of different time instants, some usability information may be embedded to the video bitstream indicating the intended display behavior when more than one layer is used and associated display behavior using this information. More specifically, the embedded information may indicate how the layers need to be composed in the video bitstream. The information may indicate if the intended display behavior is to combine (overlay) base layer and enhancement layer decoded pictures. The information may indicate that the base and enhancement layer decoded pictures are allowed to be overlaid. In addition or alternatively, the information may indicate that the enhancement layer picture should not be combined with base layer picture during display. ## Claims ### Claim 1 A method comprising: determining an intended displaying behavior of at least two scalability layers of a bitstream, the at least two scalability layers comprising pictures; encoding, into the bitstream, a description of the bitstream, and an indication, which indicates the intended displaying behavior, when the at least two scalability layers are output layers for decoding the bitstream; and performing at least one of the following: (i) encoding pictures into the bitstream, the bitstream comprising the at least two scalability layers, (ii) or encapsulating the bitstream into a container file, wherein the encoded pictures at least partially represent a same scene and the indication of the intended displaying behavior indicates an intent for one or more of overlaying pictures of the at least two scalability layers, or displaying the pictures of the at least two scalability layers separately, wherein pictures are associated with access units, and in the absence of a picture at the output layer within an access unit, the indication indicates that the intended displaying behavior is: combining pictures at output layers that are present in an access unit and at least one of the following: another picture of the output layer that is not present in the access unit; an interpolated picture that is obtained by interpolating other pictures in the output layer, or a synthesized picture that is obtained by depth-image-based rendering, and wherein in an instance in which the indication indicates that the intended displaying behavior is overlaying pictures of the at least two scalability layers, the method further comprises at least one of the following: (i) encoding a picture of the first output layer; or (ii) encoding a picture of the second output layer. ### Claim 2 (depends on 1) The method according to claim 1 comprising: encoding, into the bitstream, another indication, which indicates that the at least two scalability layers are the output layers of an output layer set. ### Claim 3 (depends on 1) The method according to claim 1, wherein a media presentation description of adaptive streaming comprises the description of the bitstream. ### Claim 4 (depends on 1) The method according to claim 1, wherein the another picture of the output layer that is not present in the access unit is one of the previous picture in output order, the closest picture in picture order count, or an indicated picture. ### Claim 5 (depends on 1) The method according to claim 1 comprising: providing, for the indication regarding the intended displaying behavior, information as to which layers of the at least two scalability layers the indication is applicable. ### Claim 6 A method comprising: decoding pictures from a bitstream, the bitstream comprising at least two scalability layers and pictures associated with access units; decoding from the bitstream, a container file encapsulating the bitstream, a description of the bitstream, and an indication, which indicates an intended displaying behavior, when the at least two scalability layers are output layers; and displaying the at least two scalability layers according to the indication, wherein the decoded pictures at least partially represent a same scene and the indication of the intended displaying behavior indicates an intent for one or more of overlaying pictures of the at least two scalability layers, or displaying the pictures of the at least two scalability layers separately, wherein, in the absence of a picture at the output layer within an access unit, and in response to the indication, the method comprises: combining pictures at output layers that are present in an access unit and at least one of the following: another picture of the output layer that is not present in the access unit; an interpolated picture that is obtained by interpolating other pictures in the output layer; or a synthesized picture that is obtained by depth-image-based rendering, and wherein the method further comprises determining on the basis of the indication that the intended displaying behavior is overlaying pictures of the at least two scalability layers, and at least one of the following: (i) decoding a picture of the first output layer and displaying the decoded picture of the first output layer; or (ii) decoding a picture of the second output layer and displaying the decoded picture of the second output layer. ### Claim 7 (depends on 6) The method according to claim 6 comprising: decoding, from the bitstream, another indication, which indicates that the at least two scalability layers are the output layers of an output layer set. ### Claim 8 (depends on 6) The method according to claim 6, wherein a media presentation description of adaptive streaming comprises the description of the bitstream. ### Claim 9 An apparatus comprising at least one processor and at least one memory, said at least one memory stored with code thereon, which when executed by said at least one processor, causes an apparatus to perform at least the following: determine an intended displaying behavior of at least two scalability layers of a bitstream, the at least two scalability layers comprising pictures; encode, into the bitstream, a description of the bitstream, and an indication, which indicates the intended displaying behavior, when the at least two scalability layers are output layers for decoding the bitstream; and perform at least one of the following: (i) encode pictures into the bitstream, the bitstream comprising at least two scalability layers or (ii) encapsulate the bitstream into a container file, wherein the encoded pictures at least partially represent a same scene and the indication indicates an intent for one or more of overlaying pictures of the at least two scalability layers, or displaying the pictures of the at least two scalability layers separately, wherein pictures are associated with access units, and in the absence of a picture at the output layer within an access unit, the indication indicates that the intended displaying behavior is: combining pictures at output layers that are present in an access unit and at least one of the following: another picture of the output layer that is not present in the access unit; an interpolated picture that is obtained by interpolating other pictures in the output layer, or a synthesized picture that is obtained by depth-image-based rendering, wherein in an instance in which the indication indicates that the intended displaying behavior is overlaying pictures of the at least two scalability layers, the at least one memory stored with code thereon, when executed by said at least one processor, further causes the apparatus to perform at least one of the following: (i) encode a picture of the first output layer; or (ii) encode a picture of the second output layer. ### Claim 10 An apparatus comprising at least one processor and at least one memory, said at least one memory stored with code thereon, which when executed by said at least one processor, causes an apparatus to perform at least the following: decode pictures from a bitstream, the bitstream comprising at least two scalability layers and pictures associated with access units, wherein one of the at least two layers is an output layer and another of the at least two layers is an alternative layer; decode from the bitstream, a container file encapsulating the bitstream, a description of the bitstream, and an indication, which indicates an intended displaying behavior, when the at least two scalability layers are output layers of an output layer set; and output a picture of the output layer, a picture of the alternative layer, or a combined picture, wherein the decoded pictures at least partially represent a same scene and the indication of the intended displaying behavior indicates an intent for one or more of overlaying pictures of the at least two scalability layers, or displaying the pictures of the at least two scalability layers separately, wherein, in the absence of a picture at the output layer within an access unit, and in response to the indication, the apparatus is further caused to perform the following: combining pictures at output layers that are present in an access unit and at least one of the following: another picture of the output layer that is not present in the access unit; an interpolated picture that is obtained by interpolating other pictures in the output layer; or a synthesized picture that is obtained by depth-image-based rendering, wherein said at least one memory stored with code thereon, when executed by said at least one processor, further causes the apparatus to determine on the basis of the indication that the intended displaying behavior is overlaying pictures of the at least two scalability layers, and at least one of the following: (i) decode a picture of the first output layer and display the decoded picture of the first output layer; or (ii) decode a picture of the second output layer and display the decoded picture of the second output layer. ### Claim 11 (depends on 10) The apparatus according to claim 10, said at least one memory stored with code thereon, which when executed by said at least one processor, causes the apparatus to perform at least the following: decode, from the bitstream, another indication, which indicates that the at least two scalability layers are output layers of the output layer set. ### Claim 12 (depends on 10) The apparatus according to claim 10 wherein a media presentation description of adaptive streaming comprises the description of the bitstream. ### Claim 13 A computer program product embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to: determine an intended displaying behavior of at least two scalability layers of a bitstream, the at least two scalability layers comprising pictures; encode, into the bitstream, a description of the bitstream, and an indication, which indicates the intended displaying behavior, when the at least two scalability layers are output layers in an output layer set for decoding the bitstream; and perform at least one of the following: (i) encode pictures into the bitstream, the bitstream comprising at least two scalability layers or (ii) encapsulate the bitstream into a container file, wherein the encoded pictures at least partially represent a same scene and the indication of the intended displaying behavior indicates an intent for one or more of overlaying pictures of the at least two scalability layers, or displaying the pictures of the at least two scalability layers separately, wherein pictures are associated with access units, and in the absence of a picture at the output layer within an access unit, the indication indicates that the intended displaying behavior is: combining pictures at output layers that are present in an access unit and at least one of the following: another picture of the output layer that is not present in the access unit; an interpolated picture that is obtained by interpolating other pictures in the output layer, or a synthesized picture that is obtained by depth-image-based rendering, wherein in an instance in which the indication indicates that the intended displaying behavior is overlaying pictures of the at least two scalability layers, the computer program code is configured to, when executed on the at least one processor, further cause the apparatus or the system to perform at least one of the following: (i) encode a picture of the first output layer; or (ii) encode a picture of the second output layer. ### Claim 14 A computer program product stored on a non-transitory computer readable medium comprising at least one processor and at least one memory, said at least one memory stored with code thereon, which when executed by said at least one processor, causes an apparatus or a system to perform at least the following: decode pictures from a bitstream, the bitstream comprising at least two scalability layers and pictures associated with access units, wherein one of the at least two layers is an output layer and another of the at least two layers is an alternative layer; decode from the bitstream, a container file encapsulating the bitstream, a description of the bitstream, and an indication, which indicates an intended displaying behavior, when the at least two scalability layers are output layers in an output layer set; and output a picture of the output layer, a picture of the alternative layer, or a combined picture, wherein the decoded pictures at least partially represent a same scene and the indication of the intended displaying behavior indicates an intent for one or more of overlaying pictures of the at least two scalability layers, or displaying the pictures of the at least two scalability layers separately, wherein, in the absence of a picture at the output layer within an access unit, and in response to the indication, the apparatus or the system is further caused to perform the following: combining pictures at output layers that are present in an access unit and at least one of the following: another picture of the output layer that is not present in the access unit; an interpolated picture that is obtained by interpolating other pictures in the output layer; or a synthesized picture that is obtained by depth-image-based rendering, wherein the computer program code is configured to, when executed on the at least one processor, further cause the apparatus or the system to determine on the basis of the indication that the intended displaying behavior is overlaying pictures of the at least two scalability layers, and at least one of the following: (i) decode a picture of the first output layer and display the decoded picture of the first output layer; or (ii) decode a picture of the second output layer and display the decoded picture of the second output layer.