# US 11,451,795 B2 · Supplemental enhancement information including confidence level and mixed content information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,795 B2 - Document ID: US11451795B2 - Status: Granted - Kind: B2 - Application number: 17/382,517 - Filed: 2021-07-22 - Published: 2021-11-18 - Granted: 2022-09-20 - Earliest claimed: 2012-09-30 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Gary J. Sullivan; Yongjun Wu - Web page: https://patentgrep.com/patent/US11451795B2 ## CPC - H04N 19/172: the region being a picture, frame or field - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/16: for a given display mode, e.g. for interlaced or progressive display mode - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This application relates to video encoding and decoding, and specifically to tools and techniques for using and providing supplemental enhancement information in bitstreams. Among other things, the detailed description presents innovations for bitstreams having supplemental enhancement information (SEI). In particular embodiments, the SEI message includes picture source data (e.g., data indicating whether the associated picture is a progressive scan picture or an interlaced scan picture and/or data indicating whether the associated picture is a duplicate picture). The SEI message can also express a confidence level of the encoder's relative confidence in the accuracy of this picture source data. A decoder can use the confidence level indication to determine whether the decoder should separately identify the picture as progressive or interlaced and/or a duplicate picture or honor the picture source scanning information in the SEI as it is. ## Claims ### Claim 1 A method performed by an encoder device, the method comprising: determining a first flag and a second flag for identifying source scan type of pictures in a sequence, the first flag and the second flag collectively and exclusively indicating one of the following unique states for the sequence of pictures: a state indicating that the source scan type of the pictures in the sequence is interlaced, a state indicating that the source scan type of the pictures in the sequence is progressive, a state indicating that the source scan type of the pictures in the sequence is unknown, and a state indicating that the source scan type is independently indicated for each picture of the pictures in the sequence by a value of a picture-level syntax element that is to be signaled as part of an SEI message or to be inferred, wherein the first flag indicates whether the source scan type of the pictures is interlaced and the second flag is different from the first flag, is a separate syntax element from the first flag, and indicates whether the source scan type of the pictures is progressive; outputting the first flag and the second flag as part of encoded data in a bitstream or bitstream portion; determining, for a given picture of the pictures in the sequence, the value of the picture-level syntax element that indicates the source scan type of the given picture, the picture-level syntax element indicating one of the following states: a state indicating that the source scan type of the given picture is interlaced, a state indicating that the source scan type of the given picture is progressive, and a state indicating that the source scan type of the given picture is unknown, wherein, if the picture-level syntax element is not present in the encoded data in the bitstream or bitstream portion, the value of the picture-level syntax element is to be inferred according to rules that include: if the first flag and the second flag indicate that the source scan type of the pictures in the sequence is progressive, inferring the value of the picture-level syntax element to indicate that the source scan type of the given picture is progressive; if the first flag and the second flag indicate that the source scan type of the pictures in the sequence is interlaced, inferring the value of the picture-level syntax element to indicate that the source scan type of the given picture is interlaced; and otherwise, inferring the value of the picture-level syntax element to indicate that the source scan type of the given picture is unknown; if the picture-level syntax element is to be signaled, outputting the picture-level syntax element as part of the encoded data in the bitstream or bitstream portion; encoding the given picture; and outputting, as part of the encoded data in the bitstream or bitstream portion, the encoded given picture. ### Claim 2 (depends on 1) The method of claim 1, wherein the encoded data in the bitstream or bitstream portion further comprises a duplicate picture flag indicating whether one or more of the pictures are duplicate pictures. ### Claim 3 (depends on 1) The method of claim 1, wherein the first flag and the second flag are part of information for a profile, level, and tier in the bitstream or bitstream portion. ### Claim 4 (depends on 1) The method of claim 1, wherein the SEI message is a picture-timing SEI message. ### Claim 5 (depends on 1) The method of claim 1, wherein: if the first flag is 0 and the second flag is 0, the source scan type of the pictures in the sequence is unknown; if the first flag is 0 and the second flag is 1, the source scan type of the pictures in the sequence is progressive; if the first flag is 1 and the second flag is 0, the source scan type of the pictures in the sequence is interlaced; and if the first flag is 1 and the second flag is 1, the source scan type is independently indicated for each picture of the pictures in the sequence by the value of the picture-level syntax element for that picture. ### Claim 6 (depends on 5) The method of claim 5, wherein the picture-level syntax element for the given picture is not present in the encoded data, and wherein: if the first flag is 0 and the second flag is 0, the source scan type of the given picture is set to be unknown; if the first flag is 0 and the second flag is 1, the source scan type of the given picture is set to be progressive; and if the first flag is 1 and the second flag is 0, the source scan type of the given picture is set to be interlaced. ### Claim 7 (depends on 1) The method of claim 1, further comprising: determining the source scan type of the pictures in the sequence, including analyzing source scan characteristics of the pictures in the sequence. ### Claim 8 (depends on 1) The method of claim 1, further comprising: determining the source scan type of the given picture, including analyzing source scan characteristics of the given picture. ### Claim 9 A tangible computer-readable media storing encoded data for a sequence of pictures in a bitstream or bitstream portion, the encoded data in the bitstream or bitstream portion including a first flag and a second flag for identifying source scan type of the pictures in the sequence, the first flag and the second flag collectively and exclusively indicating one of the following unique states for the sequence of pictures: a state indicating that the source scan type of the pictures in the sequence is interlaced, a state indicating that the source scan type of the pictures in the sequence is progressive, a state indicating that the source scan type of the pictures in the sequence is unknown, and a state indicating that the source scan type is independently indicated for each picture of the pictures in the sequence by a value of a picture-level syntax element that is to be signaled as part of an SEI message or to be inferred, wherein the first flag indicates whether the source scan type of the pictures is interlaced and the second flag is different from the first flag, is a separate syntax element from the first flag, and indicates whether the source scan type of the pictures is progressive, the encoded data being organized to facilitate decoding by operations that include: determining, for a given picture of the pictures in the sequence, the value of the picture-level syntax element that indicates the source scan type of the given picture, the value of the picture-level syntax element indicating one of the following states: a state indicating that the source scan type of the given picture is interlaced, a state indicating that the source scan type of the given picture is progressive, and a state indicating that the source scan type of the given picture is unknown, wherein the determining the value of the picture-level syntax element includes: determining whether or not the picture-level syntax element is present in the encoded data; if the picture-level syntax element is present in the encoded data, setting the value of the picture-level syntax element according to the picture-level syntax element in the encoded data; and otherwise, the picture-level syntax element not being present in the encoded data, inferring the value of the picture-level syntax element, including: if the first flag and the second flag indicate that the source scan type of the pictures in the sequence is progressive, inferring the value of the picture-level syntax element to indicate that the source scan type of the given picture is progressive; if the first flag and the second flag indicate that the source scan type of the pictures in the sequence is interlaced, inferring the value of the picture-level syntax element to indicate that the source scan type of the given picture is interlaced; and otherwise, inferring the value of the picture-level syntax element to indicate that the source scan type of the given picture is unknown; decoding the given picture; and processing the given picture in accordance with the source scan type of the given picture. ### Claim 10 (depends on 9) The tangible computer-readable media of claim 9, wherein the encoded data in the bitstream or bitstream portion further comprises a duplicate picture flag indicating whether one or more of the pictures are duplicate pictures. ### Claim 11 (depends on 9) The tangible computer-readable media of claim 9, wherein the first flag and the second flag are part of information for a profile, level, and tier in the bitstream or bitstream portion. ### Claim 12 (depends on 9) The tangible computer-readable media of claim 9, wherein the SEI message is a picture-timing SEI message. ### Claim 13 (depends on 9) The tangible computer-readable media of claim 9, wherein: if the first flag is 0 and the second flag is 0, the source scan type of the pictures in the sequence is unknown; if the first flag is 0 and the second flag is 1, the source scan type of the pictures in the sequence is progressive; if the first flag is 1 and the second flag is 0, the source scan type of the pictures in the sequence is interlaced; and if the first flag is 1 and the second flag is 1, the source scan type is independently indicated for each picture of the pictures in the sequence by the value of the picture-level syntax element for that picture. ### Claim 14 (depends on 13) The tangible computer-readable media of claim 13, wherein the picture-level syntax element for the given picture is not present in the encoded data, and wherein: if the first flag is 0 and the second flag is 0, the source scan type of the given picture is set to be unknown; if the first flag is 0 and the second flag is 1, the source scan type of the given picture is set to be progressive; and if the first flag is 1 and the second flag is 0, the source scan type of the given picture is set to be interlaced. ### Claim 15 A computer system comprising: a buffer, implemented using memory, configured to receive encoded data for a sequence of pictures in a bitstream or bitstream portion, the encoded data in the bitstream or bitstream portion including a first flag and a second flag for identifying source scan type of the pictures in the sequence, the first flag and the second flag collectively and exclusively indicating one of the following unique states for the sequence of pictures: a state indicating that the source scan type of the pictures in the sequence is interlaced, a state indicating that the source scan type of the pictures in the sequence is progressive, a state indicating that the source scan type of the pictures in the sequence is unknown, and a state indicating that the source scan type is independently indicated for each picture of the pictures in the sequence by a value of a picture-level syntax element that is to be signaled as part of an SEI message or to be inferred, wherein the first flag indicates whether the source scan type of the pictures is interlaced and the second flag is different from the first flag, is a separate syntax element from the first flag, and indicates whether the source scan type of the pictures is progressive; and a video decoder, implemented using one or more processors, configured to perform operations comprising: determining, for a given picture of the pictures in the sequence, the value of the picture-level syntax element that indicates the source scan type of the given picture, the value of the picture-level syntax element indicating one of the following states: a state indicating that the source scan type of the given picture is interlaced, a state indicating that the source scan type of the given picture is progressive, and a state indicating that the source scan type of the given picture is unknown, wherein the determining the value of the picture-level syntax element includes: determining that the picture-level syntax element is not present in the encoded data; and inferring the value of the picture-level syntax element, including: if the first flag and the second flag indicate that the source scan type of the pictures in the sequence is progressive, inferring the value of the picture-level syntax element to indicate that the source scan type of the given picture is progressive; if the first flag and the second flag indicate that the source scan type of the pictures in the sequence is interlaced, inferring the value of the picture-level syntax element to indicate that the source scan type of the given picture is interlaced; and otherwise, inferring the value of the picture-level syntax element to indicate that the source scan type of the given picture is unknown; decoding the given picture; and processing the given picture in accordance with the source scan type of the given picture. ### Claim 16 (depends on 15) The computer system of claim 15, wherein the encoded data in the bitstream or bitstream portion further comprises a duplicate picture flag indicating whether one or more of the pictures are duplicate pictures. ### Claim 17 (depends on 15) The computer system of claim 15, wherein the first flag and the second flag are part of information for a profile, level, and tier in the bitstream or bitstream portion. ### Claim 18 (depends on 15) The computer system of claim 15, wherein the SEI message is a picture-timing SEI message. ### Claim 19 (depends on 15) The computer system of claim 15, wherein: if the first flag is 0 and the second flag is 0, the source scan type of the pictures in the sequence is unknown; if the first flag is 0 and the second flag is 1, the source scan type of the pictures in the sequence is progressive; if the first flag is 1 and the second flag is 0, the source scan type of the pictures in the sequence is interlaced; and if the first flag is 1 and the second flag is 1, the source scan type is independently indicated for each picture of the pictures in the sequence by the value of the picture-level syntax element for that picture. ### Claim 20 (depends on 19) The computer system of claim 19, wherein: if the first flag is 0 and the second flag is 0, the source scan type of the given picture is set to be unknown; if the first flag is 0 and the second flag is 1, the source scan type of the given picture is set to be progressive; and if the first flag is 1 and the second flag is 0, the source scan type of the given picture is set to be interlaced.