# US 11,115,680 B2 · Apparatuses and methods for encoding and decoding a panoramic video signal > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,680 B2 - Document ID: US11115680B2 - Status: Granted - Kind: B2 - Application number: 16/664,466 - Filed: 2019-10-25 - Published: 2020-02-20 - Granted: 2021-09-07 - Earliest claimed: 2017-04-26 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Sergey Yurievich Ikonin - Web page: https://patentgrep.com/patent/US11115680B2 ## CPC - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N 19/103: Selection of coding mode or of prediction mode - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/43: Hardware specially adapted for motion estimation or compensation - H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109) - H04N 19/51: Motion estimation or motion compensation - H04N 19/513: Processing of motion vectors - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The disclosure relates to an apparatus for encoding a video signal, wherein the video signal is a two-dimensional projection of a panoramic video signal and comprises a plurality of successive frames, including a reference frame and a current frame, wherein each frame of the plurality of successive frames comprises a plurality of video coding blocks and wherein each video coding block comprises a plurality of pixels. ## Claims ### Claim 1 An encoding apparatus for encoding a video signal, the video signal being a two-dimensional (2D) projection of a panoramic video signal and comprising a plurality of successive frames including a reference frame and a current frame, each frame of the plurality of successive frames comprising a plurality of video coding blocks, each video coding block comprising a plurality of pixels, the encoding apparatus comprising computer hardware implementing a plurality of units, including: an inter prediction unit for generating a predicted video coding block for a current video coding block of the current frame based on a corresponding video coding block of the reference frame, wherein the inter prediction unit is configured to: determine a three-dimensional (3D) reference motion vector based on a first 3D position defined by a projection of a first pixel of the current video coding block of the current frame onto a viewing sphere and based on a second 3D position defined by a projection of a second pixel of the corresponding video coding block of the reference frame onto the viewing sphere, and predict at least one pixel of the current video coding block based on the 3D reference motion vector; and an encoding unit configured to encode the current video coding block of the current frame based on the predicted video coding block, wherein the inter prediction unit is configured to predict the at least one pixel of the current video coding block based on the 3D reference motion vector by: selecting a further pixel from the corresponding video coding block of the reference frame; generating a modified 3D reference motion vector based on the 3D reference motion vector, wherein the modified 3D reference motion vector and the 3D reference motion vector have a same length, but different orientations; adding the modified 3D reference motion vector to a third 3D position defined by a projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere to generate a fourth 3D position; and determining a 2D position of the at least one pixel of the current video coding block by de-projecting the fourth 3D position onto the current frame. ### Claim 2 (depends on 1) The encoding apparatus of claim 1, wherein the inter prediction unit is further configured to: determine the 3D reference motion vector as a vector connecting the first 3D position defined by the projection of the first pixel of the current video coding block of the current frame onto the viewing sphere with the second 3D position defined by the projection of the second pixel of the corresponding video coding block of the reference frame onto the viewing sphere. ### Claim 3 (depends on 1) The encoding apparatus of claim 1, wherein the inter prediction unit is configured to predict the at least one pixel of the current video coding block based on the 3D reference motion vector by: selecting a further pixel from the corresponding video coding block of the reference frame; generating a fourth 3D position by adding the 3D reference motion vector to a third 3D position defined by a projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere; and determining a 2D position of the at least one pixel of the current video coding block by de-projecting the fourth 3D position onto the current frame. ### Claim 4 (depends on 1) The encoding apparatus of claim 1, wherein the current frame defines an equatorial line corresponding to an equatorial plane of the viewing sphere, and the inter prediction unit is configured to generate the modified 3D reference motion vector based on the 3D reference motion vector by: determining a first great circle of the viewing sphere connecting the first 3D position defined by the projection of the first pixel of the current video coding block of the current frame onto the viewing sphere with the second 3D position defined by the projection of the second pixel of the corresponding video coding block of the reference frame onto the viewing sphere; determining a second great circle of the viewing sphere containing the third 3D position defined by the projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere, wherein the first great circle and the second great circle intersect in the equatorial plane of the viewing sphere; and determining the modified 3D motion vector as the vector connecting the third 3D position defined by the projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere and the fourth 3D position, wherein the fourth 3D position lies on the second great circle, and wherein the distance between the fourth 3D position and the third 3D position is equal to the distance between the first 3D position and the second 3D position. ### Claim 5 (depends on 1) The encoding apparatus of claim 1, wherein the inter prediction unit is further configured to: predict a first subset of the plurality of pixels of the current video coding block based on the 3D reference motion vector and a second subset of the plurality of pixels of the current video coding block by interpolating the predicted first subset of the plurality of pixels to generate the predicted video coding block for the current video coding block of the current frame based on the corresponding video coding block of the reference frame. ### Claim 6 (depends on 1) The encoding apparatus of claim 1, wherein the encoding unit is further configured to encode the current video coding block of the current frame in combination with the 3D reference motion vector and/or a 2D projection of the 3D reference motion vector. ### Claim 7 A method for encoding a video signal, the video signal being a two-dimensional (2D) projection of a panoramic video signal and comprising a plurality of successive frames including a reference frame and a current frame, each frame of the plurality of successive frames comprising a plurality of video coding blocks, each video coding block comprising a plurality of pixels, the encoding method comprising: determining a three-dimensional (3D) reference motion vector based on a first 3D position defined by a projection of a first pixel of a current video coding block of the current frame onto a viewing sphere and based on a second 3D position defined by a projection of a second pixel of a corresponding video coding block of the reference frame onto the viewing sphere; predicting at least one pixel of the current video coding block based on the 3D reference motion vector to generate a predicted video coding block for the current video coding block of the current frame based on the corresponding video coding block of the reference frame; and encoding the current video coding block of the current frame based on the predicted video coding block, wherein predicting the at least one pixel of the current video coding block based on the 3D reference motion vector comprises: selecting a further pixel from the corresponding video coding block of the reference frame; generating a modified 3D reference motion vector based on the 3D reference motion vector, wherein the modified 3D reference motion vector and the 3D reference motion vector have a same length, but different orientations; adding the modified 3D reference motion vector to a third 3D position defined by a projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere to generate a fourth 3D position; and determining a 2D position of the at least one pixel of the current video coding block by de-projecting the fourth 3D position onto the current frame. ### Claim 8 A decoding apparatus for decoding a video signal, the video signal being a two-dimensional (2D) projection of a panoramic video signal and comprising a plurality of successive frames including a reference frame and a current frame, each frame of the plurality of successive frames comprising a plurality of video coding blocks, each video coding block comprising a plurality of pixels, the decoding apparatus comprising computer hardware implementing a plurality of units, including: a decoding unit configured to decode the video signal to provide a residual video coding block associated with a current video coding block of the current frame; an inter prediction unit for generating a predicted video coding block for a current video coding block of the current frame based on a corresponding video coding block of the reference frame, wherein the inter prediction unit is configured to: obtain a three-dimensional (3D) reference motion vector based on a first 3D position defined by a projection of a first pixel of the current video coding block of the current frame onto a viewing sphere and a second 3D position defined by a projection of a second pixel of the corresponding video coding block of the reference frame onto the viewing sphere, and predict at least one pixel of the current video coding block based on the 3D reference motion vector; and a restoration unit configured to restore the current video coding block based on the residual video coding block and the predicted video coding block, wherein the inter prediction unit is configured to predict the at least one pixel of the current video coding block based on the 3D reference motion vector by: selecting a further pixel from the corresponding video coding block of the reference frame; generating a modified three-dimensional reference motion vector based on the 3D reference motion vector, wherein the modified 3D reference motion vector and the 3D reference motion vector have the same length, but different orientations; adding the modified 3D reference motion vector to a third 3D position defined by a projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere to generate a fourth three-dimensional position; and determining a 2D position of the at least one pixel of the current video coding block by de-projecting the fourth 3D position onto the current frame. ### Claim 9 (depends on 8) The decoding apparatus of claim 8, wherein the 3D reference motion vector is a vector connecting the first 3D position defined by the projection of the first pixel of the current video coding block of the current frame onto the viewing sphere with the second 3D position defined by the projection of the second pixel of the corresponding video coding block of the reference frame onto the viewing sphere. ### Claim 10 (depends on 8) The decoding apparatus of claim 8, wherein the inter prediction unit is configured to predict the at least one pixel of the current video coding block based on the 3D reference motion vector by: electing a further pixel from the corresponding video coding block of the reference frame; generating a fourth 3D position by adding the 3D reference motion vector to a third 3D position defined by a projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere; and determining a 2D position of the at least one pixel of the current video coding block by de-projecting the fourth 3D position onto the current frame. ### Claim 11 (depends on 8) The decoding apparatus of claim 8, wherein the current frame defines an equatorial line corresponding to an equatorial plane of the viewing sphere and the inter prediction unit is configured to generate the modified 3D reference motion vector based on the 3D reference motion vector by: determining a first great circle of the viewing sphere connecting the first 3D position defined by the projection of the first pixel of the current video coding block of the current frame onto the viewing sphere with the second 3D position defined by the projection of the second pixel of the corresponding video coding block of the reference frame onto the viewing sphere; determining a second great circle of the viewing sphere containing the third 3D position defined by the projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere, wherein the first great circle and the second great circle intersect in the equatorial plane of the viewing sphere; and determining the modified 3D motion vector as the vector connecting the third 3D position defined by the projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere and the fourth 3D position, wherein the fourth 3D position lies on the second great circle and wherein the distance between the fourth 3D position and the third 3D position is equal to the distance between the first 3D position and the second 3D position. ### Claim 12 (depends on 8) The decoding apparatus of any one of claim 8, wherein the inter prediction unit is configured to predict a first subset of the plurality of pixels of the current video coding block based on the 3D reference motion vector and a second subset of the plurality of pixels of the current video coding block by interpolating the predicted first subset of the plurality of pixels for generating the predicted video coding block for the current video coding block of the current frame based on the corresponding video coding block of the reference frame. ### Claim 13 (depends on 8) The decoding apparatus of any one of claim 8, wherein the inter-prediction unit is configured to obtain the 3D reference motion vector and/or a 2D projection of the 3D reference motion vector by extracting the 3D reference motion vector and/or a 2D projection of the 3D reference motion vector from the encoded video signal. ### Claim 14 A method for decoding a video signal, the video signal being a two-dimensional (2D) projection of a panoramic video signal and comprising a plurality of successive frames including a reference frame and a current frame, each frame of the plurality of successive frames comprising a plurality of video coding blocks, each video coding block comprising a plurality of pixels, the decoding method comprising: decoding the video signal for providing a residual video coding block associated with a current video coding block of the current frame; obtaining a three-dimensional (3D) reference motion vector based on a first 3D position defined by a projection of a first pixel of the current video coding block of the current frame onto a viewing sphere and a second 3D position defined by a projection of a second pixel of a corresponding video coding block of the reference frame onto the viewing sphere; predicting at least one pixel of the current video coding block based on the 3D reference motion vector to generate a predicted video coding block for the current video coding block of the current frame based on the corresponding video coding block of the reference frame; and restoring the current video coding block based on the residual video coding block and the predicted video coding block, wherein predicting the at least one pixel of the current video coding block based on the 3D reference motion vector comprises: selecting a further pixel from the corresponding video coding block of the reference frame; generating a modified three-dimensional reference motion vector based on the 3D reference motion vector, wherein the modified 3D reference motion vector and the 3D reference motion vector have the same length, but different orientations; adding the modified 3D reference motion vector to a third 3D position defined by a projection of the further pixel from the corresponding video coding block of the reference frame onto the viewing sphere to generate a fourth three-dimensional position; and determining a 2D position of the at least one pixel of the current video coding block by de-projecting the fourth 3D position onto the current frame. ### Claim 15 (depends on 7) A computer program stored on a non-transitory computer readable medium comprising program code for performing the method of claim 7 when executed by hardware on a computer. ### Claim 16 (depends on 14) A computer program stored on a non-transitory computer readable medium comprising program code for performing the method of claim 14 when executed by hardware on a computer.