Scalable nested SEI message handling in video sub-bitstream extraction process

An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

• perform a conversion between a video including one or more layers and a bitstream of the video according to a rule,
• wherein the rule specifies, in a process of sub-bitstream extraction to output an output sub-bitstream, whether to generate non-scalable-nested supplementary enhancement information (SEI) messages by extracting scalable-nested SEI messages from a scalable nesting SEI message is based on a first flag indicating whether the scalable nesting SEI message applies to specific output layer sets (OLSs) and a second flag indicating whether the scalable nesting SEI message applies to all subpictures or only to specific subpictures,
• wherein the scalable nesting SEI message is contained in a first SEI network abstraction layer (NAL) unit contained in the output sub-bitstream, and
• wherein in a case that the non-scalable-nested SEI messages are generated, a second SEI NAL unit is generated, and the scalable-nested SEI messages are extracted from the scalable nesting SEI message and are included in the second SEI NAL unit as the non-scalable-nested SEI messages.

A non-transitory computer-readable storage medium storing instructions that cause a processor to:

• perform a conversion between a video including one or more layers and a bitstream of the video according to a rule,
• wherein the rule specifies, in a process of sub-bitstream extraction to output an output sub-bitstream, whether to generate non-scalable-nested supplementary enhancement information (SEI) messages by extracting scalable-nested SEI messages from a scalable nesting SEI message is based on a first flag indicating whether the scalable nesting SEI message applies to specific output layer sets (OLSs) and a second flag indicating whether the scalable nesting SEI message applies to all subpictures or only to specific subpictures,
• wherein the scalable nesting SEI message is contained in a first SEI network abstraction layer (NAL) unit contained in the output sub-bitstream, and
• wherein in a case that the non-scalable-nested SEI messages are generated, a second SEI NAL unit is generated, and the scalable-nested SEI messages are extracted from the scalable nesting SEI message and are included in the second SEI NAL unit as the non-scalable-nested SEI messages.

A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:

• generating the bitstream of the video including one or more layers according to a rule,
• wherein the rule specifies, in a process of sub-bitstream extraction to output an output sub-bitstream, whether to generate non-scalable-nested supplementary enhancement information (SEI) messages by extracting scalable-nested SEI messages from a scalable nesting SEI message is based on a first flag indicating whether the scalable nesting SEI message applies to specific output layer sets (OLSs) and a second flag indicating whether the scalable nesting SEI message applies to all subpictures or only to specific subpictures,
• wherein the scalable nesting SEI message is contained in a first SEI network abstraction layer (NAL) unit contained in the output sub-bitstream, and
• wherein in a case that the non-scalable-nested SEI messages are generated, a second SEI NAL unit is generated, and the scalable-nested SEI messages are extracted from the scalable nesting SEI message and are included in the second SEI NAL unit as the non-scalable-nested SEI messages.