# US 10,873,733 B2 · Methods and apparatus for deriving composite tracks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,873,733 B2 - Document ID: US10873733B2 - Status: Granted - Kind: B2 - Application number: 16/014,817 - Filed: 2018-06-21 - Published: 2018-12-27 - Granted: 2020-12-22 - Earliest claimed: 2017-06-23 - Assignees: Mediatek Inc. - Applicants: MediaTek Inc. - Inventors: Xin Wang; Lulin Chen; Shuai Zhao - Web page: https://patentgrep.com/patent/US10873733B2 ## CPC - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 13/156: Mixing image signals - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - H04N 13/178: Metadata, e.g. disparity information - H04N 19/00: Methods or arrangements for coding, decoding, compressing or decompressing digital video signals - H04N 19/39: involving multiple description coding [MDC], i.e. with separate layers being structured as independently decodable descriptions of input picture data - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The techniques described herein relate to methods, apparatus, and computer readable media configured to process (including encoding and decoding) composition layouts. The video data includes a plurality of encoded two-dimensional sub-picture tracks associated with a viewport and a composition operation to compose the tracks. The composition operation comprises data indicative of a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the tracks into a canvas associated with the viewport, and a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. The plurality of two-dimensional tracks are composed into the canvas according to the composition, comprising determining two or more of the composed two-dimensional sub-picture tracks comprise the composition layout, and adjusting the composition based on the composition layout operation to compensate for the composition layout. ## Claims ### Claim 1 An encoding method for encoding a composition operation for a plurality of two-dimensional sub-picture tracks, the method comprising: encoding video data in a hierarchical track structure, comprising: receiving the video data including the plurality of two-dimensional sub-picture tracks associated with a viewport, wherein the plurality of two-dimensional sub-picture tracks are at a first level of the hierarchical track structure; and encoding a composition operation for composing the plurality of two-dimensional sub-picture tracks for the viewport, wherein the composition operation is associated with a composite track at a second level in the hierarchical track structure that is above the first level of the plurality of two-dimensional sub-picture tracks, and comprises data indicative of: a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the plurality of two-dimensional sub-picture tracks into a canvas associated with the viewport, wherein the canvas comprises a plurality of portions and each of the plurality of two-dimensional sub-picture tracks comprises video data for at least an associated portion of the plurality of portions; and a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas; and providing the received video data and the encoded composition operation. ### Claim 2 (depends on 1) The encoding method of claim 1, wherein the composition layout comprises a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, or both. ### Claim 3 (depends on 1) The encoding method of claim 1, wherein encoding the composition layout operation of the composition operation comprises encoding one or more of a background color, a background image, or a background video to use to fill a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. ### Claim 4 (depends on 1) The encoding method of claim 1, wherein encoding the composition layout operation of the composition operation comprises encoding blending data to use to blend an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. ### Claim 5 (depends on 1) The encoding method of claim 1, wherein encoding the composition of the composition operation comprises selecting the composition from the group consisting of: a track overlay composition specifying an overlay operation and an order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; a track grid composition specifying a grid order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; and a track matrix composition specifying an order and a matrix used to overlay each of the plurality of two-dimensional sub-picture tracks on the canvas. ### Claim 6 A decoding method for decoding video data, the method comprising: receiving: (a) video data including a plurality of two-dimensional sub-picture tracks associated with a viewport, wherein the plurality of two-dimensional sub-picture tracks are at a first level of a hierarchical track structure; and (b) a composition operation to compose the plurality of two-dimensional sub-picture tracks for the viewport, wherein the composition operation is associated with a composite track at a second level in the hierarchical track structure that is above the first level of the plurality of two-dimensional sub-picture tracks, and comprises data indicative of: a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the plurality of two-dimensional sub-picture tracks into a canvas associated with the viewport, wherein the canvas comprises a plurality of portions and each of the plurality of two-dimensional sub-picture tracks comprises video data for at least an associated portion of the plurality of portions; and a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two- dimensional sub-picture tracks composed on the canvas; and composing the plurality of two-dimensional sub-picture tracks into the canvas according to the composition, comprising: determining two or more of the composed two-dimensional sub-picture tracks comprise the composition layout; and adjusting the composition based on the composition layout operation to compensate for the composition layout. ### Claim 7 (depends on 6) The decoding method of claim 6, wherein the composition layout comprises a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, or both. ### Claim 8 (depends on 6) The decoding method of claim 6, wherein decoding the composition layout operation of the composition operation comprises: decoding one or more of a background color, a background image, or a background video; and composing the plurality of two-dimensional sub-picture tracks comprises filling a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. ### Claim 9 (depends on 6) The decoding method of claim 6, wherein decoding the composition layout operation of the composition operation comprises: decoding blending data; and composing the plurality of two-dimensional sub-picture tracks comprises blending an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. ### Claim 10 (depends on 6) The decoding method of claim 6, wherein decoding the composition of the composition operation comprises decoding the composition from the group consisting of: a track overlay composition specifying an overlay operation and an order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; a track grid composition specifying a grid order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; and a track matrix composition specifying an order and a matrix used to overlay each of the plurality of two-dimensional sub-picture tracks on the canvas. ### Claim 11 An apparatus configured to decode video data, the apparatus comprising a processor in communication with memory, the processor being configured to execute instructions stored in the memory that cause the processor to: receive: (a) video data including a plurality of two-dimensional sub-picture tracks associated with a viewport, wherein the plurality of two-dimensional sub-picture tracks are at a first level of a hierarchical track structure; and (b) a composition operation to compose the plurality of two-dimensional sub-picture tracks for the viewport, wherein the composition operation is associated with a composite track at a second level in the hierarchical track structure that is above the first level of the plurality of two-dimensional sub-picture tracks, and comprises data indicative of: a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the plurality of two-dimensional sub-picture tracks into a canvas associated with the viewport, wherein the canvas comprises a plurality of portions and each of the plurality of two-dimensional sub-picture tracks comprises video data for at least an associated portion of the plurality of portions; and a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two- dimensional sub-picture tracks composed on the canvas; and compose the plurality of two-dimensional sub-picture tracks into the canvas according to the composition, comprising: determining two or more of the composed two-dimensional sub-picture tracks comprise the composition layout; and adjusting the composition based on the composition layout operation to compensate for the composition layout. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein the composition layout comprises a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, or both. ### Claim 13 (depends on 11) The apparatus of claim 11, wherein decoding the composition layout operation of the composition operation comprises: decoding one or more of a background color, a background image, or a background video; and composing the plurality of two-dimensional sub-picture tracks comprises filling a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. ### Claim 14 (depends on 11) The apparatus of claim 11, wherein decoding the composition layout operation of the composition operation comprises: decoding blending data; and composing the plurality of two-dimensional sub-picture tracks comprises blending an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. ### Claim 15 (depends on 11) The apparatus of claim 11, wherein decoding the composition of the composition operation comprises decoding the composition from the group consisting of: a track overlay composition specifying an overlay operation and an order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; a track grid composition specifying a grid order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; and a track matrix composition specifying an order and a matrix used to overlay each of the plurality of two-dimensional sub-picture tracks on the canvas. ### Claim 16 An apparatus for encoding video data, the apparatus comprising a processor in communication with memory, the processor being configured to execute instructions stored in the memory that cause the processor to: encode video data in a hierarchical track structure, comprising: receiving the video data including a plurality of two-dimensional sub-picture tracks associated with a viewport, wherein the plurality of two-dimensional sub-picture tracks are at a first level of the hierarchical track structure; and encoding a composition operation for composing the plurality of two-dimensional sub-picture tracks for the viewport, wherein the composition operation is associated with a composite track at a second level in the hierarchical track structure that is above the first level of the plurality of two-dimensional sub-picture tracks, and comprises data indicative of: a composition to perform on the plurality of two-dimensional sub-picture tracks to compose the plurality of two-dimensional sub-picture tracks into a canvas associated with the viewport, wherein the canvas comprises a plurality of portions and each of the plurality of two-dimensional sub-picture tracks comprises video data for at least an associated portion of the plurality of portions; and a composition layout operation to adjust the composition if the canvas comprises a composition layout created by two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas; and provide the received video data and the encoded composition operation. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the composition layout comprises a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas, or both. ### Claim 18 (depends on 16) The apparatus of claim 16, wherein encoding the composition layout operation of the composition operation comprises encoding one or more of a background color, a background image, or a background video to use to fill a gap between the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. ### Claim 19 (depends on 16) The apparatus of claim 16, wherein encoding the composition layout operation of the composition operation comprises encoding blending data to use to blend an overlap of the two or more of the plurality of two-dimensional sub-picture tracks composed on the canvas. ### Claim 20 (depends on 16) The apparatus of claim 16, wherein encoding the composition of the composition operation comprises selecting the composition from the group consisting of: a track overlay composition specifying an overlay operation and an order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; a track grid composition specifying a grid order for overlaying each of the plurality of two-dimensional sub-picture tracks on the canvas; and a track matrix composition specifying an order and a matrix used to overlay each of the plurality of two-dimensional sub-picture tracks on the canvas.