# US 12,764,353 B2 · Broadcast streaming of panoramic video for interactive clients

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,764,353 B2
- Document ID: US12764353B2
- Status: Granted
- Kind: B2
- Application number: 18/824,986
- Filed: 2024-09-05
- Published: 2024-12-26
- Granted: 2026-10-13
- Earliest claimed: 2016-05-26
- Assignees: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
- Applicants: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
- Inventors: Karsten Grüneberg; Serhan Gül; Cornelius Hellge; Yago Sánchez; Thomas Schierl; Robert Skupin
- Web page: https://patentgrep.com/patent/US12764353B2

## CPC

- H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/2362: Generation or processing of Service Information [SI]
- H04N 21/2365: Multiplexing of several video streams
- H04N 21/4345: {Extraction or processing of SI, e.g. extracting service information from an MPEG stream}
- H04N 21/4347: {Demultiplexing of several video streams}
- H04N 21/4402: involving reformatting operations of video signals for household redistribution, storage or real-time display {(details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764)}
- H04N 21/8451: {using Advanced Video Coding [AVC]}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

For transmitting a panorama video having a resolution higher than a decoder can decode, at a transmitter side, encoded data coding different (groups of) spatial segments of a video picture of a video stream are packetized into separate substreams, to obtain a group of separate substreams. At a receiver side, from the group of separate substreams a proper subset of the separate substreams is extracted and combined to a data stream containing encoded data coding respectively a proper subset of the spatial segments or groups of subsequent spatial segments of the video picture of the video stream. Thus, a decoder may decode only a subregion of the video picture of the video stream, the subregion being defined by the spatial segments or groups of spatial segments coded in the encoded data contained in the data stream. A concept allowing sending a substream or a partial data stream pertaining merely a segment of an original picture in a manner allowing easier treatment at reception side is also described.

## Claims

### Claim 1

A stream multiplexer, comprising:
a reception interface for receiving encoded data for each of at least two different spatial segments or different groups of subsequent spatial segments of a video picture of a video stream encoded thereinto; and
a data stream former configured to packetize the encoded data for each of the at least two different spatial segments or different groups of subsequent spatial segments into separate elementary streams, and to provide the separate elementary streams at an output,
wherein the at least two different spatial segments or different groups of subsequent spatial segments of the video picture of the video stream are encoded such that the encoded data comprises, for each video picture, at least one slice;
wherein the data stream former is configured to packetize the slice or slices for each spatial segment or group of spatial segments into one separate elementary stream;
wherein the data stream former is configured to, for the slices of at least one of the spatial segments or groups of subsequent spatial segments,
packetize the slices of the at least one of the spatial segments or groups of subsequent spatial segments into the separate elementary stream; and
provide a further separate stream comprising suitable slice headers, which are, relative to the slices of the at least one of the spatial segments or groups of subsequent spatial segments, modified with respect to its picture position or picture size.

### Claim 2 (depends on 1)

The stream multiplexer according to claim 1, wherein the at least two different spatial segments or different groups of subsequent spatial segments are encoded, such that the encoded data for each of the at least two different spatial segments or different groups of subsequent spatial segments contained in a respective separate elementary stream of the separate elementary streams references only to encoded data comprised in the same separate elementary stream.

### Claim 3 (depends on 1)

The stream multiplexer according to claim 1, wherein the data stream former is configured to packetize the slices of at least one of the spatial segments or groups of subsequent spatial segments into the separate elementary stream with removing a slice header thereof;
wherein the data stream former is configured to provide a further separate stream with parameter sets for each of at least one spatial subdivision of the video pictures of the video stream into subregions.

### Claim 4 (depends on 1)

The stream multiplexer according to claim 1, wherein the at least two different spatial segments or different groups of subsequent spatial segments are encoded into the video stream at one slice per video picture;
wherein the data stream former is configured to packetize, for each video picture, a portion of the slice having encoded thereinto a respective one of the spatial segments or groups of spatial segments into a separate elementary stream without, for at least one of the at least two spatial segments or groups of spatial segments, a slice header of the slice.

### Claim 5 (depends on 4)

The stream multiplexer according to claim 4, wherein the data stream former is configured to provide a further separate stream comprising the slice header or a modified version thereof modified relative to the removed slice header, with respect to its picture position or picture size.

### Claim 6 (depends on 4)

The stream multiplexer according to claim 4, wherein the data stream former is configured to provide a further separate stream with parameter sets for each of at least one spatial subdivision of the video pictures of the video stream into subregions.

### Claim 7 (depends on 1)

The stream multiplexer according to claim 1, wherein the data stream former is configured to provide the parameter sets and the slice header or the modified version thereof in the same further separate elementary stream.

### Claim 8 (depends on 1)

The stream multiplexer according to claim 1, wherein the data stream former is configured to signal the presence of the further elementary stream in a descriptor.

### Claim 9 (depends on 1)

The stream multiplexer according to claim 1, wherein the data stream former is configured to generate a elementary stream descriptor assigning a unique elementary stream identity to each of the separate elementary streams.

### Claim 10 (depends on 9)

The stream multiplexer according to claim 9, wherein the data stream former is configured to generate a subregion descriptor signaling, for each of at least one spatial subdivision of the video pictures of the video stream into subregions, one set of elementary stream identities for each subregion.

### Claim 11 (depends on 1)

The stream multiplexer according to claim 1, wherein the stream multiplexer is configured to signal at least one out of two stream types;
wherein a first stream type signals that a combination of a proper subset of the separate elementary streams corresponding to one of at least one spatial subdivision of the video picture of the video stream into subregions results in a standard conformant data stream;
wherein a second stream type signals that a combination of a proper subset of the separate elementary streams corresponding to one of at least one spatial subdivision of the video picture of the video stream into subregions results in a data stream that needs a further processing, to acquire a standard conformant version of data stream.

### Claim 12 (depends on 11)

The stream multiplexer according to claim 11, wherein the at least two different spatial segments or different groups of subsequent spatial segments of the video picture of the video stream are encoded such that the encoded data comprises at least two slices;
wherein the encoded data comprises a signaling information indicating whether a coding constraint is fulfilled or wherein the data stream former is configured to determine whether the coding constraint is fulfilled;
wherein the coding constraint is fulfilled when removing at least one slice header of the at least two slices while maintaining a slice header of the first slice of the at least two slices with respect to a coding order and concatenating the at least two slices or a proper subset of the at least two slices using the first slice header results in a standard conformant data stream;
wherein the stream multiplexer is configured to signal at least one out of the two stream types in dependence on the fulfillment of the coding constraint.

### Claim 13

A stream demultiplexer, comprising:
a data stream former configured to selectively extract at least two separate elementary streams from a group of separate elementary streams, the at least two separate elementary streams comprising encoded data coding different spatial segments or different groups of subsequent spatial segments of a video picture of a video stream, wherein the data stream former is configured to combine the at least two separate elementary streams to a data stream comprising the encoded data coding the different spatial segments or different groups of subsequent spatial segments of the video picture of the video stream, wherein the different spatial segments or different groups of subsequent spatial segments of the video picture of the video stream are encoded such that the encoded data comprises, for each spatial segment or group of spatial segments, at least one slice, wherein the slice or slices for each spatial segment or group of spatial segments are packetized into one separate elementary stream, wherein the slices of at least one of the spatial segments or groups of subsequent spatial segments are packetized into the separate elementary stream, wherein a further separate elementary stream comprises suitable slice headers which are, relative to the slices of the at least one of the spatial segments or groups of subsequent spatial segments, modified with respect to its picture position or picture size; and
an output interface configured to provide the data stream;
wherein the data stream former is configured to combine the at least two separate elementary streams to a HEVC standard conformant data stream,
wherein the data stream former is configured to modify a header information of the at least two separate elementary streams or to add a header information to the at least two separate elementary streams based on the suitable slice headers comprised in the further separate sub stream, to obtain the HEVC standard conformant data stream.

### Claim 14 (depends on 13)

The stream demultiplexer according to claim 13, wherein the group of separate elementary streams comprises a plurality of separate elementary streams, the separate elementary streams comprising encoded data coding different spatial segments or different groups of subsequent spatial segments of a video picture of the video stream;
wherein the data stream former is configured to selectively extract a subset of the plurality of separate elementary streams of the group of separate elementary streams.

### Claim 15 (depends on 14)

The stream demultiplexer according to claim 14, wherein the data stream former is configured to extract from the group of separate elementary streams a subregion descriptor signaling, for each of at least one spatial subdivision of the video pictures of the video stream into subregions, one set of elementary stream identities for each subregion, wherein the data stream former is configured to select the subset of elementary streams to be extracted from the encoded video stream using the subregion descriptor.

### Claim 16 (depends on 15)

The stream demultiplexer according to claim 15, wherein the data stream former is configured to extract from the group of separate elementary streams a elementary stream identifier assigning a unique elementary stream identity to each of the separate elementary streams, wherein the data stream former is configured to localize the subset of elementary streams to be extracted from the group of separate elementary streams using the elementary stream descriptor.

### Claim 17 (depends on 13)

The stream demultiplexer according to claim 13, wherein the stream demultiplexer is configured to identify at least one out of at least two stream types;
wherein a first stream type indicates that a combination of a proper subset of the separate elementary streams corresponding to one of at least one spatial subdivision of the video picture of the video stream into subregions results in a standard conformant data stream;
wherein a second stream type indicates that a combination of a proper subset of the separate elementary streams corresponding to one of at least one spatial subdivision of the video picture of the video stream into subregions results in a data stream that needs a further processing, to acquire a standard conformant version of data stream;
wherein the stream demultiplexer comprises a data stream processor configured to further process the data stream in dependence on the identified stream type using a processing information comprised in at least one elementary stream of the group of separate elementary streams, to acquire a standard conformant version of the data stream.

### Claim 18 (depends on 13)

The stream demultiplexer according to claim 13, wherein
the group of separate elementary streams is comprised by a broadcast transport stream and comprises
a plurality of separate elementary streams, the separate elementary streams comprising encoded data coding different spatial segments or different groups of subsequent spatial segments of the video stream; and
a program map table; and
the stream demultiplexer is configured to derive, for each of the plurality of separate elementary streams, a stream identifier from the program map table and distinguish each of plurality of separate elementary streams in the broadcast transport stream using the respective stream identifier.

### Claim 19 (depends on 18)

The stream demultiplexer according to claim 18, wherein the stream demultiplexer is configured to derive a predetermined packet identifier from a program association table conveyed within packets of packet identifier zero in the broadcast transport stream and to derive the program map table from packets of the broadcast transport stream comprising the predetermined packet identifier.

### Claim 20 (depends on 18)

The stream demultiplexer according to claim 18, wherein
the program map table uniquely associates each stream identifier with a respective packet identifier and the stream demultiplexer is configured to depacketize each of the plurality of separate elementary streams from packets of the broadcast transport stream having the packet identifier associated with the stream identifier of the respective separate elementary stream, or
the stream demultiplexer is configured to depacketize a sequence of NAL units from packets of the broadcast transport stream which comprise a packet identifier indicated in the program map table, and to associate each NAL unit with one of the plurality of separate elementary streams depending on a stream identifier indicated in an adaptation field of the packets of the broadcast transport stream.

### Claim 21 (depends on 18)

The stream demultiplexer according to claim 18, wherein the stream demultiplexer is configured to read from the program map table information on a spatial subdivision of the video into the segments and to derive the stream identifiers of the plurality of separate elementary streams inherently from the spatial subdivision by using a mapping from the segments of the spatial subdivision onto the stream identifier.

### Claim 22 (depends on 18)

The stream demultiplexer according to claim 18, wherein the stream demultiplexer is configured to read from the program map table information on a spatial subdivision of the video into the segments and to derive the stream identifiers of the plurality of separate elementary streams so that the stream identifiers of the plurality of separate elementary streams order the segments within the spatial subdivisions along a raster scan.

### Claim 23 (depends on 22)

The stream demultiplexer according to claim 22, wherein one or more of the elementary stream descriptors comprise the information on which one or more separate elementary streams of the plurality of separate elementary streams are to be extracted, along with the indexed separate elementary stream, as the at least two separate elementary streams from the group of separate elementary streams in form of a set of stream identifier offsets each indicating an offset relative to the stream identifier of the indexed separate elementary stream.

### Claim 24 (depends on 18)

The stream demultiplexer according to claim 18, wherein the stream demultiplexer is configured to read elementary stream descriptors from the program map table or an adaptation field of packets carrying the group of separate elementary streams, each of the elementary stream descriptors indexing one of the plurality of separate elementary streams, by the stream identifier of the one separate elementary stream, and comprising information on which one or more separate elementary streams of the plurality of separate elementary streams, along with the indexed separate elementary stream, form a coded representation of a subregion extractible as the at least two separate elementary streams from the group of separate elementary streams, the subregion being composed of the spatial segments or groups of subsequent spatial segments of the one or more separate elementary streams forming, along with the indexed separate elementary stream, a coded representation.

### Claim 25 (depends on 24)

The stream demultiplexer according to claim 24, wherein the stream demultiplexer is configured to read from the program map table information a subregion descriptor indicating one or more spatial subdivisions of the video into subregions so that each subregion is a set of the spatial segments or groups of subsequent spatial segments of one or more separate elementary streams, and, per spatial subdivision of the video into subregions, a size of the subregions, wherein the at least two separate elementary streams selectively extracted from the group of separate elementary streams together comprise encoded data coding different spatial segments or different groups of subsequent spatial segments which form one of the subregions of one of the one or more spatial subdivisions of the video.

### Claim 26 (depends on 13)

The stream demultiplexer according to claim 13, wherein
the group of separate elementary streams further comprises one or more elementary streams comprising slice headers and/or parameter sets stripped off from, or dedicated for modifying or replacing slice headers and/or parameter sets of, the plurality of separate elementary streams.

### Claim 27 (depends on 26)

The stream demultiplexer according to claim 26, wherein one or more of the elementary stream descriptors comprise the information on which one or more separate elementary streams of the plurality of separate elementary streams are to be extracted, along with the indexed separate elementary stream, as the at least two separate elementary streams from the group of separate elementary streams in form of a reference index into a list of sets of stream identifier offsets each indicating an offset relative to the stream identifier of the indexed separate elementary stream.

### Claim 28 (depends on 13)

The stream demultiplexer according to claim 13, wherein
the group of separate elementary streams comprises a subgroup of one or more separate elementary streams
each coding a respective spatial segment or a respective group of subsequent spatial segments,
each comprising a sequence of NAL units, and
the sequence of NAL units being composed of
a first set of one or more NAL units which forms a standard conformant version of a data stream representing the respective spatial segment or the respective group of subsequent spatial segments of the respective elementary stream, and
a second set of one or more NAL units which are of one of a set of one or more predetermined NAL unit types for ignorance of the respective NAL unit by a legacy decoder.

### Claim 29 (depends on 28)

The stream demultiplexer according to claim 28, wherein the second set of one or more NAL units is arranged in the sequence of NAL units, each NAL unit of the second set
indicating to a non-legacy decoder that an immediately succeeding NAL unit of the first set or a portion thereof is to be discarded from the sequence of NAL units along with the respective NAL unit of the second set and/or
comprising a payload section carrying a NAL unit to be inserted into the sequence of NAL units in replacement of the respective NAL unit of the second set.

### Claim 30 (depends on 28)

The stream demultiplexer according to claim 28, configured to, if any of the subgroup of separate elementary streams is among the extracted at least two separate elementary streams, for each NAL unit of the second set
discard an immediately succeeding NAL unit of the first set or a portion thereof from the sequence of NAL units along with the respective NAL unit of the second set and/or
insert a NAL unit carried in a payload section of the respective NAL unit of the second set into the sequence of NAL units in replacement of the respective NAL unit of the second set.

### Claim 31

A method for stream multiplexing, the method comprising:
receiving encoded data for each of at least two different spatial segments or different groups of subsequent spatial segments of video pictures of a video stream; and
packetizing the encoded data for each of the at least two different spatial segments or different groups of spatial segments into separate elementary streams, wherein the at least two different spatial segments or different groups of subsequent spatial segments of the video picture of the video stream are encoded such that the encoded data comprises, for each spatial segment or group of spatial segments, at least one slice, wherein the slice or slices for each spatial segment or group of spatial segments are packetized into one separate elementary stream,
wherein the slices of at least one of the spatial segments or groups of subsequent spatial segments are packetized into the separate elementary stream and the method further comprises
providing a further separate stream comprising suitable slice headers which are, relative to the slices of the at least one of the spatial segments or groups of subsequent spatial segments, modified with respect to its picture position or picture size.

### Claim 32

A method for stream demultiplexing, the method comprising:
selectively extracting at least two separate elementary streams from a group of separate elementary streams, the at least two separate elementary streams comprising encoded data coding different spatial segments or different groups of subsequent spatial segments of a video picture of the video stream, wherein the different spatial segments or different groups of subsequent spatial segments of the video picture of the video stream are encoded such that the encoded data comprises, for each spatial segment or group of spatial segments, at least one slice, wherein the slice or slices for each spatial segment or group of spatial segments are packetized into one separate elementary stream, wherein the slices of at least one of the spatial segments or groups of subsequent spatial segments are packetized into the separate elementary stream, wherein a further separate elementary stream comprises suitable slice headers which are, relative to the slices of the at least one of the spatial segments or groups of subsequent spatial segments, modified with respect to its picture position or picture size; and
combining the at least two separate elementary streams to a data stream comprising the encoded data coding the different spatial segments or different groups of subsequent spatial segments of a video picture of the video stream, wherein the at least two separate elementary streams are combined to a HEVC standard conformant data stream, wherein a header information of the at least two separate elementary streams is modified or a header information is added to the at least two separate elementary streams based on the suitable slice headers comprised in the further separate sub stream, to obtain the HEVC standard conformant data stream.