# US 9,521,393 B2 · Non-nested SEI messages in video coding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,521,393 B2 - Document ID: US9521393B2 - Status: Granted - Kind: B2 - Application number: 14/036,873 - Filed: 2013-09-25 - Published: 2014-07-10 - Granted: 2016-12-13 - Earliest claimed: 2013-01-07 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Ye-Kui Wang; Ying Chen; Adarsh Krishnan Ramasubramonian - Web page: https://patentgrep.com/patent/US9521393B2 ## CPC - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A device obtains, from a bitstream that includes an encoded representation of the video data, a non-nested Supplemental Enhancement Information (SEI) message that is not nested within another SEI message in the bitstream. Furthermore, the device determines a layer of the bitstream to which the non-nested SEI message is applicable. The non-nested SEI message is applicable to layers for which video coding layer (VCL) network abstraction layer (NAL) units of the bitstream have layer identifiers equal to a layer identifier of a SEI NAL unit that encapsulates the non-nested SEI message. A temporal identifier of the SEI NAL unit is equal to a temporal identifier of an access unit containing the SEI NAL unit. Furthermore, the device processes, based in part on one or more syntax elements in the non-nested SEI message, video data of the layer of the bitstream to which the non-nested SEI message is applicable. ## Claims ### Claim 1 A method of processing video data, the method comprising: obtaining, from a bitstream of coded video data that includes a sequence of network abstraction layer (NAL) units, a non-nested Supplemental Enhancement Information (SEI) message, wherein the non-nested SEI message is not nested within a scalable nesting SEI message in the bitstream and the non-nested SEI message is not a buffering period SEI message, a picture timing SEI message, or a decoding unit information SEI message; determining a layer of the bitstream to which the non-nested SEI message is applicable, wherein the layer to which the non-nested SEI message is applicable is a layer for which video coding layer (VCL) NAL units of the bitstream have layer identifiers equal to a layer identifier specified in a SEI NAL unit that encapsulates the non-nested SEI message, and wherein a temporal identifier specified in the SEI NAL unit is required to be equal to a temporal identifier of an access unit containing the SEI NAL unit; and processing, based in part on data in the non-nested SEI message, video data of the layer of the bitstream to which the non-nested SEI message is applicable. ### Claim 2 (depends on 1) The method of claim 1, wherein: the SEI NAL unit includes a NAL unit header and a payload, the NAL unit header of the SEI NAL unit includes at least a first syntax element and a second syntax element, the first syntax element specifies the layer identifier specified in the SEI NAL unit, the second syntax element specifies the temporal identifier specified in the SEI NAL unit, and the payload of the SEI NAL unit includes the non-nested SEI message. ### Claim 3 (depends on 1) The method of claim 1, wherein the non-nested SEI message is one of: a pan-scan rectangle SEI message that includes data associated with displaying at a different picture aspect ratio than a picture aspect ratio of output pictures, a filler payload SEI message that includes data for adjusting a bit rate to meet specific constraints, a recovery point SEI message that includes information for clean random access or gradual decoding refresh, a scene information SEI message that includes information associated with scene changes and transitions, a picture snapshot SEI message that includes an indication to label an associated decoded picture as a still-image snapshot of video content, a progressive refinement segment start SEI message that includes information associated with a start of a segment of consecutive pictures that represent a progressive refinement of quality of a picture rather than a moving scene, a progressive refinement segment end SEI message that includes information associated with an end of the segment of consecutive pictures, a film grain characteristics SEI message that includes information associated with synthesizing film grain effects, a post filter hint SEI message that includes information associated with suggested post-filter coefficients or correlation information for post-filter design, a tone mapping information SEI message that includes information associated with remapping to another color space than that used or assumed in encoding, a frame packing arrangement SEI message that includes information associated with packing of stereoscopic video into the bitstream, a display orientation SEI message that includes information that specifies flipping and/or rotation to be applied to the output pictures when the output pictures are displayed, a structure of pictures information SEI message that includes information that describes temporal and inter prediction structure of the bitstream, a temporal sub-layer zero index SEI message that includes information associated with detecting when coded pictures with temporal identifiers equal to 0 are missing, or a region refresh information SEI message that includes information that indicates whether slice segments to which the non-nested SEI message applies belong to a refreshed region of a current picture. ### Claim 4 (depends on 3) The method of claim 3, wherein the method further comprises determining, based on one or more syntax elements of the non-nested SEI message, a payload type variable for the non-nested SEI message, wherein: the payload type variable for the non-nested SEI message indicates a payload type of the non-nested SEI message, and the payload type variable for the non-nested SEI message is equal to 2, 3, 6, 9, 15, 16, 17, 19, 22, 23, 45, 47, 128, 131, or 134. ### Claim 5 (depends on 1) The method of claim 1, wherein: the non-nested SEI message is a first non-nested SEI message, and the method further comprises: obtaining, from the bitstream, a second non-nested buffering period SEI message that indicates initial delays for hypothetical reference decoder (HRD) operations; when the second non-nested buffering period SEI message is applicable to TargetOp, selecting, as an HRD initialization point, an access unit associated with the second non-nested buffering period SEI message, wherein TargetOp is the set of layer identifiers in a bitstream-to-decode, the bitstream-to-decode being the bitstream or a sub-stream of the bitstream; and initializing a HRD at the selected access unit. ### Claim 6 (depends on 1) The method of claim 1, wherein: the non-nested SEI message is a first non-nested SEI message, and the method further comprises: obtaining, from the bitstream, a second non-nested SEI message, the second non-nested SEI message being a buffering period SEI message that indicates initial delays for HRD operations, a picture timing SEI message that indicates picture output times and picture/sub-picture removal times for HRD operations, or a decoding unit SEI message that indicates sub-picture removal times for HRD operations; and for each respective access unit in a bitstream-to-decode: selecting the second non-nested SEI message for the respective access unit when the second non-nested SEI message is associated with the access unit and the second non-nested SEI message is applicable to the set of layer identifiers in the bitstream-to-decode, wherein the bitstream-to-decode is the bitstream or a sub-bitstream of the bitstream; and using one or more syntax elements of the second non-nested SEI message in an HRD operation with regard to the respective access unit. ### Claim 7 (depends on 1) The method of claim 1, wherein: the non-nested SEI message is a first non-nested SEI message, and the method further comprises: obtaining, from the bitstream, a second non-nested SEI message, the second non-nested SEI message being a buffering period SEI message that indicates initial delays for HRD operations, a picture timing SEI message that indicates picture output times and picture/sub-picture removal times for HRD operations, or a decoding unit SEI message that indicates sub-picture removal times for HRD operations; determining an operation point applicable to the second non-nested SEI message, wherein a highest temporal identifier specified in all NAL units of the operation point applicable to the second non-nested SEI message is equal to a temporal identifier specified in a SEI NAL unit that encapsulates the second non-nested SEI message, and wherein a set of layer identifiers of all the NAL units of the operation point applicable to the second non-nested SEI message contains all integer values in the range of 0 to the layer identifier specified in the SEI NAL unit that encapsulates the second non-nested SEI message; and performing, based in part on values of one or more syntax elements of the second non-nested SEI message, an HRD operation with regard to the operation point applicable to the second non-nested SEI message. ### Claim 8 A video processing device comprising one or more processors configured to: obtain, from a bitstream of coded video data that includes a sequence of network abstraction layer (NAL) units, a non-nested Supplemental Enhancement Information (SEI) message, wherein the non-nested SEI message is not nested within a scalable nesting SEI message in the bitstream and the non-nested SEI message is not a buffering period SEI message, a picture timing SEI message, or a decoding unit information SEI message; determine a layer of the bitstream to which the non-nested SEI message is applicable, wherein the layer to which the non-nested SEI message is applicable is a layer for which video coding layer (VCL) NAL units of the bitstream have layer identifiers equal to a layer identifier specified in a SEI NAL unit that encapsulates the non-nested SEI message, and wherein a temporal identifier specified in the SEI NAL unit is required to be equal to a temporal identifier of an access unit containing the SEI NAL unit; and process, based in part on data in the non-nested SEI message, video data of the layer of the bitstream to which the non-nested SEI message is applicable. ### Claim 9 (depends on 8) The video processing device of claim 8, wherein: the SEI NAL unit includes a NAL unit header and a payload, the NAL unit header of the SEI NAL unit includes at least a first syntax element and a second syntax element, the first syntax element specifies the layer identifier specified in the SEI NAL unit, the second syntax element specifies the temporal identifier specified in the SEI NAL unit, and the payload of the SEI NAL unit includes the non-nested SEI message. ### Claim 10 (depends on 8) The video processing device of claim 8, wherein the non-nested SEI message is one of: a pan-scan rectangle SEI message that includes data associated with displaying at a different picture aspect ratio than a picture aspect ratio of output pictures, a filler payload SEI message that includes data for adjusting a bit rate to meet specific constraints, a recovery point SEI message that includes information for clean random access or gradual decoding refresh, a scene information SEI message that includes information associated with scene changes and transitions, a picture snapshot SEI message that includes an indication to label an associated decoded picture as a still-image snapshot of video content, a progressive refinement segment start SEI message that includes information associated with a start of a segment of consecutive pictures that represent a progressive refinement of quality of a picture rather than a moving scene, a progressive refinement segment end SEI message that includes information associated with an end of the segment of consecutive pictures, a film grain characteristics SEI message that includes information associated with synthesizing film grain effects, a post filter hint SEI message that includes information associated with suggested post-filter coefficients or correlation information for post-filter design, a tone mapping information SEI message that includes information associated with remapping to another color space than that used or assumed in encoding, a frame packing arrangement SEI message that includes information associated with packing of stereoscopic video into the bitstream, a display orientation SEI message that includes information that specifies flipping and/or rotation to be applied to the output pictures when the output pictures are displayed, a structure of pictures information SEI message that includes information that describes temporal and inter prediction structure of the bitstream, a temporal sub-layer zero index SEI message that includes information associated with detecting when coded pictures with temporal identifiers equal to 0 are missing, or a region refresh information SEI message that includes information that indicates whether slice segments to which the non-nested SEI message applies belong to a refreshed region of a current picture. ### Claim 11 (depends on 10) The video processing device of claim 10, wherein: the one or more processors are configured to determine, based on one or more syntax elements of the non-nested SEI message, a payload type variable for the non-nested SEI message, wherein the payload type variable for the non-nested SEI message indicates a payload type of the non-nested SEI message, and the payload type variable for the non-nested SEI message is equal to 2, 3, 6, 9, 15, 16, 17, 19, 22, 23, 45, 47, 128, 131, or 134. ### Claim 12 (depends on 8) The video processing device of claim 8, wherein: the non-nested SEI message is a first non-nested SEI message, and the one or more processors are further configured to: obtain, from the bitstream, a second non-nested buffering period SEI message that indicates initial delays for Hypothetical Reference Decoder (HRD) operations; when the second non-nested buffering period SEI message is applicable to TargetOp, select, as an BIRD initialization point, an access unit associated with the second non-nested buffering period SEI message, wherein TargetOp is the set of layer identifiers in a bitstream-to-decode, the bitstream-to-decode being the bitstream or a sub-stream of the bitstream; and initialize a HRD at the selected access unit. ### Claim 13 (depends on 8) The video processing device of claim 8, wherein: the non-nested SEI message is a first non-nested SEI message, and the one or more processors are further configured to: obtain, from the bitstream, a second non-nested SEI message, the second non-nested SEI message being a buffering period SEI message that indicates initial delays for HRD operations, a picture timing SEI message that indicates picture output times and picture/sub-picture removal times for HRD operations, or a decoding unit SEI message that indicates sub-picture removal times for HRD operations, and for each respective access unit in a bitstream-to-decode: select the second non-nested SEI message for the respective access unit when the second non-nested SEI message is associated with the access unit and the second non-nested SEI message is applicable to the set of layer identifiers in the bitstream-to-decode, wherein the bitstream-to-decode is the bitstream or a sub-bitstream of the bitstream; and use one or more syntax elements of the second non-nested SEI message in an HRD operation with regard to the respective access unit. ### Claim 14 (depends on 8) The video processing device of claim 8, wherein: the non-nested SEI message is a first non-nested SEI message, and the one or more processors are further configured to: obtain, from the bitstream, a second non-nested SEI message, the second non-nested SEI message being a buffering period SEI message that indicates initial delays for HRD operations, a picture timing SEI message that indicates picture output times and picture/sub-picture removal times for HRD operations, or a decoding unit SEI message that indicates sub-picture removal times for HRD operations; determine an operation point applicable to the second non-nested SEI message, wherein a highest temporal identifier specified in all NAL units of the operation point applicable to the second non-nested SEI message is equal to a temporal identifier specified in a SEI NAL unit that encapsulates the second non-nested SEI message, and wherein a set of layer identifiers of all the NAL units of the operation point applicable to the second non-nested SEI message contains all integer values in the range of 0 to the layer identifier specified in the SEI NAL unit that encapsulates the second non-nested SEI message; and perform, based in part on values of one or more syntax elements of the second non-nested SEI message, an HRD operation with regard to the operation point applicable to the second non-nested SEI message. ### Claim 15 (depends on 8) The video processing device of claim 8, wherein the one or more processors are configured to decode, based in part on the one or more syntax elements in the non-nested SEI message, the video data of the layer of the bitstream to which the non-nested SEI message is applicable. ### Claim 16 A video processing device comprising: means for obtaining, from a bitstream of coded video data that includes a sequence of network abstraction layer (NAL) units, a non-nested Supplemental Enhancement Information (SEI) message, wherein the non-nested SEI message is not nested within a scalable nesting SEI message in the bitstream and the non-nested SEI message is not a buffering period SEI message, a picture timing SEI message, or a decoding unit information SEI message; means for determining a layer of the bitstream to which the non-nested SEI message is applicable, wherein the layer to which the non-nested SEI message is applicable is a layer for which video coding layer (VCL) NAL units of the bitstream have layer identifiers equal to a layer identifier specified in a SEI NAL unit that encapsulates the non-nested SEI message, and wherein a temporal identifier specified in the SEI NAL unit is required to be equal to a temporal identifier of an access unit containing the SEI NAL unit; and means for processing, based in part on data in the non-nested SEI message, video data of the layer of the bitstream to which the non-nested SEI message is applicable. ### Claim 17 (depends on 16) The video processing device of claim 16, wherein: the SEI NAL unit includes a NAL unit header and a payload, the NAL unit header of the SEI NAL unit includes at least a first syntax element and a second syntax element, the first syntax element specifies the layer identifier specified in the SEI NAL unit, the second syntax element specifies the temporal identifier specified in the SEI NAL unit, and the payload of the SEI NAL unit includes the non-nested SEI message. ### Claim 18 A computer-readable storage medium having instructions stored thereon that, when executed, configure a video processing device to: obtain, from a bitstream of coded video data that includes a sequence of network abstraction layer (NAL) units, a non-nested Supplemental Enhancement Information (SEI) message, wherein the non-nested SEI message is not nested within a scalable nesting SEI message in the bitstream and the non-nested SEI message is not a buffering period SEI message, a picture timing SEI message, or a decoding unit information SEI message; determine a layer of the bitstream to which the non-nested SEI message is applicable, wherein the layer to which the non-nested SEI message is applicable is a layer for which video coding layer (VCL) NAL units of the bitstream have layer identifiers equal to a layer identifier specified in a SEI NAL unit that encapsulates the non-nested SEI message, and wherein a temporal identifier specified in the SEI NAL unit is required to be equal to a temporal identifier of an access unit containing the SEI NAL unit; and process, based in part on data in the non-nested SEI message, video data of the layer of the bitstream to which the non-nested SEI message is applicable. ### Claim 19 (depends on 18) The computer-readable storage medium of claim 18, wherein: the SEI NAL unit includes a NAL unit header and a payload, the NAL unit header of the SEI NAL unit includes at least a first syntax element and a second syntax element, the first syntax element specifies the layer identifier specified in the SEI NAL unit, the second syntax element specifies the temporal identifier specified in the SEI NAL unit, and the payload of the SEI NAL unit includes the non-nested SEI message.