# US 11,451,710 B2 · Adaptive panoramic video streaming using composite pictures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,710 B2 - Document ID: US11451710B2 - Status: Granted - Kind: B2 - Application number: 16/935,476 - Filed: 2020-07-22 - Published: 2020-11-05 - Granted: 2022-09-20 - Earliest claimed: 2018-05-25 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Matthew Wozniak; Bakkama Srinath Reddy; Amr Martini - Web page: https://patentgrep.com/patent/US11451710B2 ## CPC - H04N 5/23238 - G06T 11/003 - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - H04N 19/184: the unit being bits, e.g. of the compressed video stream - H04N 19/33: in the spatial domain - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Innovations in stream configuration operations and playback operations for adaptive streaming of panoramic video are described. The innovations include features of adaptive streaming of panoramic video with composite pictures. For example, a stream configuration tool splits an input picture of panoramic video into multiple sections and creates multiple composite pictures. A composite picture includes one of the sections as well as a low-resolution version of the input picture. A playback tool reconstructs one or more composite pictures. Under normal operation, the playback tool can use the reconstructed section(s) of the composite picture(s) to render high-quality views of the panoramic video. If the view window dramatically changes, however, or if encoded data for a section is lost or corrupted, the playback tool can use the low-resolution version of the input picture to render lower-quality details for views of the panoramic video, without disruption of playback. ## Claims ### Claim 1 A computer system comprising one or more processing units and memory, wherein the computer system implements a panoramic video playback tool that includes: a view controller configured to: determine a view window for playback of panoramic video; from among multiple sections of the panoramic video that correspond to an input picture, identify one or more sections that contain at least part of the view window, wherein each of the multiple sections is part of a respective composite picture that also includes a low-resolution version of the input picture; and for the one or more identified sections, select one or more bitstreams among multiple bitstreams, each of the one or more bitstreams corresponding to a given one of the one or more identified sections; a streaming controller configured to request encoded data, in the one or more selected bitstreams for the one or more identified sections, respectively, for the input picture; one or more input buffers configured to store the encoded data; multiple video decoders configured to decode the encoded data to reconstruct the one or more identified sections for the input picture and reconstruct the low-resolution version of the input picture; a mapper configured to, based at least in part on the one or more reconstructed sections and the reconstructed low-resolution version of the input picture, create an output picture, wherein the mapper is further configured to determine one or more portions of the output picture cannot be created using the one or more reconstructed sections, and, for the one or more portions of the output picture that cannot be created using the one or more reconstructed sections, create that portion of the output picture using the reconstructed low-resolution version of the input picture; and one or more output buffers configured to store the output picture for output to a display device. ### Claim 2 (depends on 1) The computer system of claim 1, wherein the view controller is configured to, in multiple iterations, perform operations to determine the view window, identify the one or more sections that contain at least part of the view window, and, for the one or more identified sections, select the one or more bitstreams. ### Claim 3 (depends on 1) The computer system of claim 1, wherein at least part of the output picture is created using the one or more reconstructed sections, and wherein at least part of the output picture is created using the reconstructed low-resolution version of the input picture. ### Claim 4 (depends on 1) The computer system of claim 1, wherein the low-resolution version of the input picture and the one or more sections in the respective composite picture are located at pre-defined positions within the respective composite picture. ### Claim 5 (depends on 1) The computer system of claim 1, wherein a manifest file includes information that indicates where the low-resolution version of the input picture is positioned in the respective composite pictures. ### Claim 6 (depends on 1) The computer system of claim 1, wherein the input picture has a first spatial resolution, wherein the low-resolution version of the input picture has a second spatial resolution lower than the first spatial resolution, and wherein each of the multiple sections has a third spatial resolution lower than the first spatial resolution. ### Claim 7 (depends on 1) The computer system of claim 1, wherein the mapper is further configured to: when creating the output picture, project the one or more reconstructed sections from an intermediate projection to an output projection. ### Claim 8 (depends on 7) The computer system of claim 7, wherein the intermediate projection is a sinusoidal projection, and wherein the output projection is a screen projection. ### Claim 9 (depends on 1) The computer system of claim 1, wherein the mapper is further configured to: when creating the output picture, project the low-resolution version of the input picture from an input projection to an output projection. ### Claim 10 In a computer system that implements a panoramic video playback tool, a method comprising: determining a view window for playback of panoramic video; from among multiple sections of the panoramic video that correspond to an input picture, identifying one or more sections that contain at least part of the view window, wherein each of the multiple sections is part of a respective composite picture that also includes a low-resolution version of the input picture; for the one or more identified sections, selecting one or more bitstreams among multiple bitstreams, each of the one or more bitstreams corresponding to a given one of the one or more identified sections; requesting encoded data, in the one or more selected bitstreams for the one or more identified sections, respectively, for the input picture; storing the encoded data in one or more input buffers; decoding the encoded data to: reconstruct the one or more identified sections for the input picture; and reconstruct the low-resolution version of the input picture; creating an output picture based at least in part on the one or more reconstructed sections and the reconstructed low-resolution version of the input picture at least in part by determining one or more portions of the output picture cannot be created using the one or more reconstructed sections, and for the one or more portions of the output picture that cannot be created using the one or more reconstructed sections, creating that portion of the output picture using the reconstructed low-resolution version of the input picture; and storing the output picture, in one or more output buffers, for output to a display device. ### Claim 11 (depends on 10) The method of claim 10, wherein multiple iterations of the determining, identifying, and selecting operations are performed. ### Claim 12 (depends on 10) The method of claim 10, wherein at least part of the output picture is created using the one or more reconstructed sections, and wherein at least part of the output picture is created using the reconstructed low-resolution version of the input picture. ### Claim 13 (depends on 10) The method of claim 10, wherein the input picture has a first spatial resolution, wherein the low-resolution version of the input picture has a second spatial resolution lower than the first spatial resolution, and wherein each of the multiple sections has a third spatial resolution lower than the first spatial resolution. ### Claim 14 (depends on 10) The method of claim 10, wherein the creating the output picture further comprises: projecting the one or more reconstructed sections from an intermediate projection to an output projection. ### Claim 15 (depends on 14) The method of claim 14, wherein the intermediate projection is a sinusoidal projection, and wherein the output projection is a screen projection. ### Claim 16 (depends on 10) The method of claim 10, wherein the creating the output picture further comprises: projecting the low-resolution version of the input picture from an input projection to an output projection. ### Claim 17 (depends on 10) The method of claim 10, wherein the low-resolution version of the input picture is a low-resolution version of an entire extent of the input picture. ### Claim 18 One or more computer-readable media storing instructions which, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations comprising: determining a view window for playback of panoramic video; from among multiple sections of the panoramic video that correspond to an input picture, identifying one or more sections that contain at least part of the view window, wherein each of the multiple sections is part of a respective composite picture that also includes a low-resolution version of the input picture; for the one or more identified sections, selecting one or more bitstreams among multiple bitstreams, each of the one or more bitstreams corresponding to a given one of the one or more identified sections; requesting encoded data, in the one or more selected bitstreams for the one or more identified sections, respectively, for the input picture; storing the encoded data in one or more input buffers; decoding the encoded data to: reconstruct the one or more identified sections for the input picture; and reconstruct the low-resolution version of the input picture; creating an output picture based at least in part on the one or more reconstructed sections and the reconstructed low-resolution version of the input picture at least in part by determining one or more portions of the output picture cannot be created using the one or more reconstructed sections, and for the one or more portions of the output picture that cannot be created using the one or more reconstructed sections, creating that portion of the output picture using the reconstructed low-resolution version of the input picture; and storing the output picture, in one or more output buffers, for output to a display device.