# US 2019/0246084 A1 · Re-projecting flat projections of pictures of panoramic video for rendering by application > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0246084 A1 - Document ID: US20190246084A1 - Status: Published application - Kind: A1 - Application number: 16/269,230 - Filed: 2019-02-06 - Published: 2019-08-08 - Earliest claimed: 2016-11-29 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Matthew Wozniak; Bakkama Srinath Reddy; Vladislav Alexandrov; David H. Mebane; Dibyajyoti Roy; Shyam Sadhwani - Web page: https://patentgrep.com/patent/US20190246084A1 ## CPC - H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)} - G06T 3/073: Transforming surfaces of revolution to planar images, e.g. cylindrical surfaces to planar images - H04N 9/646: {for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters} - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - H04N 9/67: for matrixing (camera processing pipelines for matrixing of colour signals H04N23/85) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Innovations in reconstruction and rendering of panoramic video are described. For example, a view-dependent operation controller of a panoramic video playback system receives an indication of a view direction for an application and, based at least in part on the view direction, identifies a section of a picture of panoramic video in an input projection. The view-dependent operation controller limits operations of a color converter, video decoder, and/or streaming controller to the identified section. In this way, the panoramic video playback system can avoid performing operations to reconstruct sections of the picture of panoramic video that will not be viewed. As another example, a mapper of a panoramic video playback system re-projects at least some sample values in an input flat projection towards a center location for a view direction, producing an output flat projection, which an application can use to generate one or more screen projections. ## Claims ### Claim at position 1 -20. (canceled) ### Claim 21 A computer system comprising one or more processing units and memory, wherein the computer system implements a panoramic video playback system that includes: a view-dependent operation controller configured to: receive an indication of a view direction for an application; based at least in part on the view direction, identify a section of a picture of panoramic video in an input flat projection, the picture of panoramic video in the input flat projection being based on sample values of different camera views recorded at a given time in multiple directions around a central position, wherein the identified section includes a portion of the picture of panoramic video in the input flat projection; and define a bounding box, for the identified section, around the portion of the picture of panoramic video in the input flat projection, the bounding box including a buffer area outside the portion of the picture of panoramic video in the input flat projection; a streaming controller configured to request encoded data for one or more spatial regions, wherein the bounding box aligns with boundaries of the one or more spatial regions or fits within the one or more spatial regions, and wherein the one or more spatial regions are slices or tiles; an input buffer configured to store the encoded data; a video decoder configured to decode at least some of the encoded data, thereby producing sample values of the identified section, the sample values of the identified section including sample values of the portion of the picture of panoramic video in the input flat projection and sample values of the buffer area; and a mapper configured to map at least some of the sample values of the identified section in the input flat projection to an output flat projection, wherein the mapper is configured to re-project the at least some of the sample values of the identified section in the input flat projection towards a center location for the view direction. ### Claim 22 (depends on 21) The computer system of claim 21, wherein the view direction is a current view direction or a predicted view direction. ### Claim 23 (depends on 21) The computer system of claim 21, wherein the mapper is configured to: for locations in the output flat projection, determine corresponding locations in the input flat projection, wherein the locations in the output flat projection and the corresponding locations in the input flat projection are associated with locations in a view section of an intermediate, spherical projection, the view section of the intermediate, spherical projection being indicated by the view direction; assign sample values to the locations in the output flat projection based on the at least some of the sample values of the identified section in the input flat projection at the corresponding locations in the input flat projection, respectively; and provide the output flat projection to the application. ### Claim 24 (depends on 21) The computer system of claim 21, wherein the mapper is configured to, when the view direction changes rapidly, use at least some of the sample values of the buffer area in the re-projection; and ### Claim 25 (depends on 21) The computer system of claim 21, wherein the decoder is configured to, when the view direction changes gradually, use at least some of the sample values of the buffer area for reference in later decoding operations. ### Claim 26 (depends on 21) The computer system of claim 21, wherein the mapper is configured to re-project the at least some of the sample values of the identified section in the input flat projection so that the output flat projection is centered at the center location for the view direction. ### Claim 27 (depends on 21) The computer system of claim 21, wherein the mapper is configured to scale the identified section of the input flat projection to substantially fill the output flat projection. ### Claim 28 (depends on 21) The computer system of claim 21, wherein the panoramic video playback system further includes the application, and wherein the application includes: a second mapper configured to map at least some sample values from the output flat projection to a screen projection. ### Claim 29 In a computer system, a method comprising: receiving an indication of a view direction for an application; based at least in part on the view direction, identifying a section of a picture of panoramic video in an input flat projection, the picture of panoramic video in the input flat projection being based on sample values of different camera views recorded at a given time in multiple directions around a central position, wherein the identified section includes a portion of the picture of panoramic video in the input flat projection; defining a bounding box, for the identified section, around the portion of the picture of panoramic video in the input flat projection, the bounding box including a buffer area outside the portion of the picture of panoramic video in the input flat projection; receiving encoded data for one or more spatial regions, wherein the bounding box aligns with boundaries of the one or more spatial regions or fits within the one or more spatial regions, and wherein the one or more spatial regions are slices or tiles; decoding at least some of the encoded data, thereby producing sample values of the identified section, the sample values of the identified section including sample values of the portion of the picture of panoramic video in the input flat projection and sample values of the buffer area; and mapping at least some of the sample values of the identified section in the input flat projection to an output flat projection, including re-projecting the at least some of the sample values of the identified section in the input flat projection towards a center location for the view direction. ### Claim 30 (depends on 29) The method of claim 29, wherein the view direction is a current view direction or a predicted view direction. ### Claim 31 (depends on 29) The method of claim 29, wherein the mapping further includes, when the view direction changes rapidly, using at least some of the sample values of the buffer area in the re-projecting. ### Claim 32 (depends on 29) The method of claim 29, further comprising, when the view direction changes gradually, using at least some of the sample values of the buffer area for reference in later decoding operations. ### Claim 33 (depends on 29) The method of claim 29, wherein the mapping includes scaling the identified section of the input flat projection to substantially fill the output flat projection. ### Claim 34 (depends on 29) The method of claim 29, further comprising: with the application, mapping at least some sample values from the output flat projection to a screen projection. ### Claim 35 One or more computer-readable media storing computer-executable instructions for causing a computer system, when programmed thereby, to perform operations comprising: receiving an indication of a view direction for an application; based at least in part on the view direction, identifying a section of a picture of panoramic video in an input flat projection, the picture of panoramic video in the input flat projection being based on sample values of different camera views recorded at a given time in multiple directions around a central position, wherein the identified section includes a portion of the picture of panoramic video in the input flat projection; defining a bounding box, for the identified section, around the portion of the picture of panoramic video in the input flat projection, the bounding box including a buffer area outside the portion of the picture of panoramic video in the input flat projection; receiving encoded data for one or more spatial regions, wherein the bounding box aligns with boundaries of the one or more spatial regions or fits within the one or more spatial regions, and wherein the one or more spatial regions are slices or tiles; decoding at least some of the encoded data, thereby producing sample values of the identified section, the sample values of the identified section including sample values of the portion of the picture of panoramic video in the input flat projection and sample values of the buffer area; and mapping at least some of the sample values of the identified section in the input flat projection to an output flat projection, including re-projecting the at least some of the sample values of the identified section in the input flat projection towards a center location for the view direction. ### Claim 36 (depends on 35) The one or more computer-readable media of claim 35, wherein the mapping includes: for locations in the output flat projection, determining corresponding locations in the input flat projection, wherein the locations in the output flat projection and the corresponding locations in the input flat projection are associated with locations in a view section of an intermediate, spherical projection, the view section of the intermediate, spherical projection being indicated by the view direction; and assigning sample values to the locations in the output flat projection based on the at least some of the sample values of the identified section in the input flat projection at the corresponding locations in the input flat projection, respectively. ### Claim 37 (depends on 35) The one or more computer-readable media of claim 35, wherein the mapping further includes, when the view direction changes rapidly, using at least some of the sample values of the buffer area in the re-projecting. ### Claim 38 (depends on 35) The one or more computer-readable media of claim 35, wherein the operations further comprise, when the view direction changes gradually, using at least some of the sample values of the buffer area for reference in later decoding operations. ### Claim 39 (depends on 35) The one or more computer-readable media of claim 35, wherein the mapping includes scaling the identified section of the input flat projection to substantially fill the output flat projection and cropping portions of the bounding box outside the identified section. ### Claim 40 (depends on 35) The one or more computer-readable media of claim 35, wherein the operations further comprise: with the application, mapping at least some sample values from the output flat projection to a screen projection.