# US 2019/0246141 A1 · Processing of Multi-Directional Images in Spatially-Ordered Video Coding Applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0246141 A1 - Document ID: US20190246141A1 - Status: Published application - Kind: A1 - Application number: 15/888,559 - Filed: 2018-02-05 - Published: 2019-08-08 - Earliest claimed: 2018-02-05 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Jae Hoon Kim; Dazhong Zhang; Hang Yuan; Jiefu Zhai; Ming Chen; Xiaosong Zhou; Chris Y. Chung; Hsi-Jung Wu - Web page: https://patentgrep.com/patent/US20190246141A1 ## CPC - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability - H04N 19/162: User input - H04N 19/167: Position within a video image, e.g. region of interest [ROI] - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/52: by predictive encoding - H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Image processing techniques may accelerate coding of viewport data contained within multi-view image data. According to such techniques, an encoder may shifting content of a multi-directional image data according to the viewport location data provided by a decoder. The encoder may code the shifted multi-directional image data by predictive coding, and transmit to the decoder, the coded multi-directional image data and data identifying an amount of the shift. Doing so may move the viewport location to positions in the image data that are coded earlier than the positions that the viewport location naturally occupies and, thereby, may accelerate coding. On decode, a decoder may compare its present viewport location with viewport location data provided by the encoder with coded video data. The decoder may decode the coded video data and extract a portion of the decoded video data corresponding to a present viewport location for display. ## Claims ### Claim 1 An image coding method, comprising: responsive to data from a decoder identifying a location of a viewport, shifting multi-directional image data to be coded according to the viewport location data, coding the shifted multi-directional image data by predictive coding, and transmitting to a decoder, the coded multi-directional image data and data identifying an amount of the shift. ### Claim 2 (depends on 1) The coding method of claim 1, further comprising, prior to the shifting, parsing the multi-directional image data into a plurality of tiles, wherein: the coding occurs on a tile-by-tile basis, and the shifting places a location of the viewport in a tile that is coded earlier than a tile in which the viewport would be coded without the shifting. ### Claim 3 (depends on 2) The coding method of claim 2, wherein the predictive coding of the multi-direcitonal image codes a tile of the image that contains the viewport at a higher level of quality than the coding of a tile of the image that does not contain the viewport. ### Claim 4 (depends on 1) The coding method of claim 1, wherein the shifting places a location of the viewport at an origin of the shifted multi-directional image, and the coding begins at the origin of the shifted multi-directional image. ### Claim 5 (depends on 1) The coding method of claim 1, wherein the shifting places a location of the viewport at an origin of the shifted multi-directional image offset by a padding amount, and the coding begins at the origin of the shifted multi-directional image. ### Claim 6 (depends on 1) The coding method of claim 1, further comprising shifting a reference frame that provides a prediction reference for the predictive coding of the multi-directional image data according to the viewport location. ### Claim 7 (depends on 1) The coding method of claim 1, wherein the viewport location data represents a display condition at a decoder of a previously-coded frame. ### Claim 8 (depends on 1) The coding method of claim 1, wherein the viewport location data contains x and y offset data. ### Claim 9 (depends on 1) The coding method of claim 1, wherein the viewport location data contains angular offset data. ### Claim 10 (depends on 1) The coding method of claim 1, wherein the multi-directional image is a cube map image. ### Claim 11 (depends on 1) The coding method of claim 1, wherein the multi-directional image is an omni-directional image. ### Claim 12 An image coder, comprising: an image processor, responsive to data from a decoder identifying a location of a viewport, to shift a multi-directional image according to the viewport location data, a video coder having an input coupled to an output of the image processor, and a transmitter having an input for coded video data from the video coder and for data identifying an amount of the shift applied by the image processor. ### Claim 13 (depends on 12) The coder of claim 12, wherein the video coder operates on a multi-directional image data that is parsed into a plurality of tiles, on a tile-by-tile basis, and the image processor places a location of the viewport in a tile that is coded earlier than a tile in which the viewport would be coded without the shifting. ### Claim 14 (depends on 12) The coder of claim 12, wherein the image processor places a location of the viewport at an origin of the shifted multi-directional image, and the video coder begins coding at the origin of the shifted multi-directional image. ### Claim 15 (depends on 12) The coder of claim 12, wherein the video comprises a reference picture buffer that stores reference frames for predictive video coding operations, and the image coder shifts the reference frames according to the viewport location data. ### Claim 16 (depends on 12) The coder of claim 12, wherein the image processor shifts a location of the viewport at an origin of the shifted multi-directional image offset by a padding amount, and the video coder begins coding at the origin of the shifted multi-directional image. ### Claim 17 (depends on 12) The coder of claim 12, wherein the viewport location data represents a display condition at a decoder of a previously-coded frame. ### Claim 18 (depends on 12) The coder of claim 12, wherein the viewport location data contains x and y offset data. ### Claim 19 (depends on 12) The coder of claim 12, wherein the viewport location data contains angular offset data. ### Claim 20 (depends on 12) The coder of claim 12, further comprising an image source to generate the multi-directional image as a cube map image. ### Claim 21 (depends on 12) The coder of claim 12, further comprising an image source to generate the multi-directional image as an omni-directional image. ### Claim 22 An image decoding method, comprising: decoding coded multi-directional image data received from an encoder, extracting a subset of data from the decoded multi-directional image by comparing viewport location data received from the encoder to viewport location data previously communicated to the encoder; and displaying the extracted subset of data. ### Claim 23 (depends on 22) The decoding method of claim 22, further comprising: storing the decoded multi-directional image in a reference picture buffer, and communicating to the encoder, a location of the extracted subset of data as a new viewport location. ### Claim 24 (depends on 22) The decoding method of claim 22, wherein the decoding occurs on a tile-by-tile basis, the tiles representing different spatial areas of the decoded multi-directional image. ### Claim 25 (depends on 22) The decoding method of claim 22, wherein the decoding begins at the origin of the multi-directional image; and the subset is extracted from a position of the decoded multi-directional image that includes the origin. ### Claim 26 A terminal, comprising: a receiver, having an input for coded multi-directional image data and viewport location data; a video decoder having an input for the coded multi-directional image data coupled to an output of the receiver; and an image processor to extract a subset of data from decoded multi-directional image data output by the video decoder by comparing viewport location data received from the encoder to viewport location data previously communicated to the encoder; and a display to display the extracted subset of data. ### Claim 27 (depends on 26) The terminal of claim 26, wherein the video decoder includes a reference picture buffer to store the decoded multi-directional image, and the terminal comprises a transceiver to communicate to an encoder, a location of the extracted subset of data as a new viewport location. ### Claim 28 (depends on 26) The terminal of claim 26, wherein the video decoder operates on a tile-by-tile basis, the tiles representing different spatial areas of the decoded multi-directional image. ### Claim 29 (depends on 26) The terminal of claim 26, wherein the video decoder begins decoding at an origin of the multi-directional image; and the image process extracts the subset from a position of the decoded multi-directional image that includes the origin. ### Claim 30 (depends on 26) The terminal of claim 26, wherein the terminal is a head mounted display.