# US 10,484,716 B2 · Video data stream concept > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,716 B2 - Document ID: US10484716B2 - Status: Granted - Kind: B2 - Application number: 15/928,742 - Filed: 2018-03-22 - Published: 2018-08-02 - Granted: 2019-11-19 - Earliest claimed: 2012-06-29 - Assignees: Ge Video Compression, LLC - Applicants: Ge Video Compression, LLC - Inventors: Thomas Schierl; Valeri George; Anastasia Henkel; Detlev Marpe; Karsten Grueneberg; Robert Skupin - Web page: https://patentgrep.com/patent/US10484716B2 ## CPC - H04N 19/167: Position within a video image, e.g. region of interest [ROI] - H04L 47/10: Flow control; Congestion control - H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks - H04N 19/188: {the unit being a video data packet, e.g. a network abstraction layer [NAL] unit} - H04N 19/423: characterised by memory arrangements (H04N19/433 takes precedence) - H04N 19/436: using parallelised computational arrangements - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/55: Motion estimation with spatial constraints, e.g. at image or region borders - H04N 19/67: involving unequal error protection [UEP], i.e. providing protection according to the importance of the data - H04N 19/68: involving the insertion of resynchronisation markers into the bitstream - H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards - H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding - H04N 21/234327: {by decomposing into layers, e.g. base layer and one or more enhancement layers} - H04N 21/4621: {Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen (arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10)} - H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region - G06F 15/173: using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake - H04L 12/56: {Packet switching systems} - H04L 12/66: Arrangements for connecting between networks having differing types of switching systems, e.g. gateways - H04L 47/31: by tagging of packets, e.g. using discard eligibility [DE] bits - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Decoder retrieval timing information, ROI information and tile identification information are conveyed within a video data stream at a level which allows for an easy access by network entities such as MANEs or decoder. In order to reach such a level, information of such types are conveyed within a video data stream by way of packets interspersed into packets of access units of a video data stream, In accordance with an embodiment, the interspersed packets are of a removable packet type, i.e. the removal of these interspersed packets maintains the decoder's ability to completely recover the video content conveyed via the video data stream. ## Claims ### Claim 1 An apparatus for encoding video content into a data stream, comprising: an encoder configured for, based on the video content, encoding sub-portions of the video content into a sequence of packets organized into a plurality of access units, wherein each of the plurality of access units relates to a picture of the video content and includes one or more decoding units and each of the decoding units includes at least one payload packet, and each of the sub-portions corresponds to a slice, the encoding including entropy encoding of slices across slice boundaries in accordance with a wavefront parallel processing technique, the slices including an independent slice and a dependent slice of the video content; and interspersing, in each of the plurality of access units, one or more timing control packets corresponding to the one or more decoding units, respectively, wherein each timing control packets is indicative of a decoder buffer retrieval time by which content of the corresponding decoding unit is to be retrieved from a buffer, wherein each packet of the sequence of packets includes a packet type field in a packet header of the respective packet, the packet type field for the at least one payload packet being different than the packet type field for the one or more timing control packets. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the at least one payload packet is of a non-removable packet type and each of the one or more timing control packets is of a removable packet type. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the sequence of packets includes Supplemental Enhancement Information (SEI) packets of different types, each of the SEI packets includes an SEI packet type field, wherein content of an SEI packet type field of a timing control packet differs from that of a different type of SEI packet. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein the at least one payload packet includes encoded information of a sub-portion of a picture of the video content. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein different types of packets in the sequence of packets are arranged in accordance with a pre-determined organization, and an organization of the packets in the sequence of packets can be determined in accordance with the pre-determined organization when one or more removable packets are removed from the sequence of packets. ### Claim 6 (depends on 3) The apparatus of claim 3, wherein the different type of SEI packet includes a Region of Interest (ROI) packet. ### Claim 7 An apparatus for decoding a data stream to reconstruct video content, comprising: a decoder configured for extracting, from the data stream, a sequence of packets, representing sub-portions of the video content and organized into a plurality of access units, wherein each of the plurality of access units relates to a picture of the video content and includes one or more decoding units and each of the decoding units includes at least one payload packet, and each of the sub-portions corresponds to a slice, the extracting including entropy decoding of slices across slice boundaries in accordance with a wavefront parallel processing technique, the slices including an independent slice and a dependent slice of the video content; identifying, in each of the plurality of access units, one or more timing control packets corresponding to the one or more decoding units, respectively, and interspersed in the access unit, wherein each of the timing control packets is indicative of a decoder buffer retrieval time by which content of the corresponding decoding unit is to be retrieved from a buffer; retrieving, from the buffer, the content of each decoding unit in accordance with a decoder buffer retrieval time associated with the decoding unit, wherein each packet of the sequence of packets includes a packet type field in a packet header of the respective packet, the packet type field for the at least one payload packet being different than the packet type field for the one or more timing control packets; and decoding the one or more decoding units using predictive decoding to reconstruct the video content. ### Claim 8 (depends on 7) The apparatus of claim 7, wherein the at least one payload packet is of a non-removable packet type and each of the one or more timing control packets is of a removable packet type. ### Claim 9 (depends on 7) The apparatus of claim 7, wherein the sequence of packets includes Supplemental Enhancement Information (SEI) packets of different types, each of the SEI packets includes an SEI packet type field, wherein content of an SEI packet type field of a timing control packet differs from that of a different type of SEI packet. ### Claim 10 (depends on 7) The apparatus of claim 7, wherein the at least one payload packet includes encoded information of a sub-portion of a picture of the video content. ### Claim 11 (depends on 7) The apparatus of claim 7, wherein different types of packets in the sequence of packets are arranged in accordance with a pre-determined organization, and an organization of the packets in the sequence of packets can be determined in accordance with the pre-determined organization when one or more removable packets are removed from the sequence of packets. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein the different type of SEI packet includes a Region of Interest (ROI) packet. ### Claim 13 A method for decoding a data stream to reconstruct video content, comprising: extracting, from the data stream, a sequence of packets, representing sub-portions of the video content and organized into a plurality of access units, wherein each of the plurality of access units relates to a picture of the video content and includes one or more decoding units and each of the decoding units includes at least one payload packet, and each of the sub-portions corresponds to a slice, the extracting including entropy decoding of slices across slice boundaries in accordance with a wavefront parallel processing technique, the slices including an independent slice and a dependent slice of the video content; identifying, in each of the plurality of access units, one or more timing control packets corresponding to the one or more decoding units, respectively, and interspersed in the access unit, wherein each of the timing control packets is indicative of a decoder buffer retrieval time by which content of the corresponding decoding unit is to be retrieved from a buffer; retrieving, from the buffer, the content of each decoding unit in accordance with a decoder buffer retrieval time associated with the decoding unit, wherein each packet of the sequence of packets includes a packet type field in a packet header of the respective packet, the packet type field for the at least one payload packet being different than the packet type field for the one or more timing control packets; and decoding the one or more decoding units using predictive decoding to reconstruct the video content. ### Claim 14 (depends on 13) The method of claim 13, wherein the at least one payload packet is of a non-removable packet type and each of the one or more timing control packets is of a removable packet type. ### Claim 15 (depends on 13) The method of claim 13, wherein the sequence of packets includes Supplemental Enhancement Information (SEI) packets of different types, each of the SEI packets includes an SEI packet type field, wherein content of an SEI packet type field of a timing control packet differs from that of a different type of SEI packet. ### Claim 16 (depends on 13) The method of claim 13, wherein the at least one payload packet includes encoded information of a sub-portion of a picture of the video content. ### Claim 17 (depends on 13) The method of claim 13, wherein different types of packets in the sequence of packets are arranged in accordance with a pre-determined organization, and an organization of the packets in the sequence of packets can be determined in accordance with the pre-determined organization when one or more removable packets are removed from the sequence of packets. ### Claim 18 (depends on 15) The method of claim 15, wherein the different type of SEI packet includes a Region of Interest (ROI) packet. ### Claim 19 A machine readable non-transitory medium for storing data associated with video content, comprising: a data stream stored in the non-transitory machine readable medium, the data stream comprising a sequence of packets representing sub-portions of the video content, wherein each of the sub-portions corresponds to a slice, and slices are encoded into the data stream as the sequence of packets based on entropy encoding of the slices across slice boundaries in accordance with a wavefront parallel processing technique, the slices including an independent slice and a dependent slice of the video content, and the sequence of packets is organized into a plurality of access units, wherein each of the plurality of access units relates to a picture of the video content and includes one or more decoding units, each of the decoding units includes at least one payload packet, each of the plurality of access units has one or more timing control packets interspersed therein and corresponding to the one or more decoding units, respectively, wherein each of the timing control packets is indicative of a decoder buffer retrieval time by which content of the corresponding decoding unit is to be retrieved from a buffer, and each packet of the sequence of packets includes a packet type field in a packet header of the respective packet, the packet type field for the at least one payload packet being different than the packet type field for the one or more timing control packets. ### Claim 20 (depends on 19) The machine readable non-transitory medium of claim 19, wherein the at least one payload packet is of a non-removable packet type. ### Claim 21 (depends on 19) The machine readable non-transitory medium of claim 19, wherein the sequence of packets includes Supplemental Enhancement Information (SEI) packets of different types, each of the SEI packets includes an SEI packet type field, wherein content of an SEI packet type field of a timing control packet differs from that of a different type of SEI packet. ### Claim 22 (depends on 19) The machine readable non-transitory medium of claim 19, wherein the at least one payload packet includes encoded information of a sub-portion of a picture of the video content. ### Claim 23 (depends on 19) The machine readable non-transitory medium of claim 19, wherein different types of packets in the sequence of packets are arranged in accordance with a pre-determined organization. ### Claim 24 (depends on 21) The machine readable non-transitory medium of claim 21, wherein the different type of SEI packet includes a Region of Interest (ROI) packet. ### Claim 25 (depends on 23) The machine readable non-transitory medium of claim 23, wherein an organization of the packets in the sequence of packets can be determined in accordance with the pre-determined organization when one or more removable packets are removed from the sequence of packets. ### Claim 26 (depends on 20) The machine readable non-transitory medium of claim 20, wherein each of the one or more timing control packets is of a removable packet type.