# US 2019/0174150 A1 · Spatially Tiled Omnidirectional Video Streaming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0174150 A1
- Document ID: US20190174150A1
- Status: Published application
- Kind: A1
- Application number: 16/302,500
- Filed: 2017-05-24
- Published: 2019-06-06
- Earliest claimed: 2016-05-25
- Assignees: Koninklijke KPN N.V.
- Applicants: Koninklijke KPN N.V.
- Inventors: Lucia D'Acunto; Emmanuel Thomas; Mattijs Oskar Van Deventer; Ray Van Brandenburg; Arjen Veenhuizen
- Web page: https://patentgrep.com/patent/US20190174150A1

## CPC

- H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras}
- H04L 65/65: Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 21/23439: {for generating different versions}
- H04N 21/26258: {for generating a list of items to be played back in a given order, e.g. playlist, or scheduling item distribution according to such list (retrieval of multimedia data based on playlists G06F16/40)}
- H04N 21/4307: {Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen}
- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/6587: Control parameters, e.g. trick play commands, viewpoint selection
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A method is described for processing an omnidirectional video by a client device, said omnidirectional video being associated with a 3D coordinate system for determining a spatial position of video data of said omnidirectional video on a curved surface, preferably a sphere, said method comprising: receiving a manifest file, the manifest file comprising a plurality of tile stream identifiers for identifying a plurality of tile streams, the tile streams comprising video frames having image views, whereby the image views of video frames of different tile streams cover different regions of a 2D projection of the omnidirectional video, a region defining a tile; selecting on the basis of spatial relation information in the manifest file and on the basis of a viewpoint of a user of the client device a first tile streams associated with a first resolution and a first tile position and a second tile stream associated with a second resolution and a second tile position, the second resolution being lower than the first resolution, wherein the spatial relation information defines for each tile a tile position and wherein the viewpoint defines a viewing direction of the user, tile position and the viewpoint being defined on the basis of coordinates of the 3D coordinate system.

## Claims

### Claim 1

A method of processing an omnidirectional video by a client device, said omnidirectional video being associated with a 3D coordinate system for determining a spatial position of video data of said omnidirectional video on a curved surface, said method comprising:
receiving, by the client device, a manifest file, the manifest file comprising a plurality of tile stream identifiers for identifying a plurality of tile streams, the tile streams comprising video frames with 2D projected video data, the video frames having image views, whereby the image views of video frames of different tile streams cover different regions of a 2D projection of the omnidirectional video, a region defining a tile;
selecting, by the client device, on the basis of spatial relation information in the manifest file and on the basis of a viewpoint of a user of the client device a first tile stream associated with a first resolution and a first tile position and a second tile stream associated with a second resolution and a second tile position, the second resolution being lower than the first resolution, wherein the spatial relation information defines for each tile a tile position and wherein the viewpoint defines a viewing direction of the user, the tile position and the viewpoint being defined on the basis of coordinates of the 3D coordinate system.

### Claim 2 (depends on 1)

Method according to claim 1 further comprising:
requesting, by the client device, the selected one or more tile streams from one or more tile stream storage devices; and,
receiving the video data of the one or more selected tile streams and receiving mapping information, the mapping information providing the client device with information for enabling the client device to map the 2D projected video data of the tile streams as omnidirectional video data onto the curved surface;
processing the video data of said received tile streams on the basis of the spatial relation information and the mapping information.

### Claim 3 (depends on 2)

Method according to claim 2 wherein the mapping information comprises an indication of the type of 2D projection that is used for projecting the omnidirectional video data onto a planar surface.

### Claim 4 (depends on 1)

Method according to claim 1 wherein the viewpoint is associated with a field of view of the user, the selecting including:
selecting a tile stream of a high resolution for a tile position located in at least part of a central part of the field of view; and,
selecting a tile stream of a lower resolution than the high resolution for a tile position located in at least part of a peripheral part of the field of view.

### Claim 5 (depends on 2)

Method according to claim 2 wherein said processing the video data further comprises:
decoding the video data of said received tile streams into video frames representing tiles;
spatially arranging the video frames of the tiles associated with one time instance and stitching the video frames into one video frame; and,
mapping 2D projected video data of the video frame as omnidirectional video data on the curved surface on the basis of the mapping information.

### Claim 6 (depends on 2)

Method according to claim 2 wherein the requesting and receiving of said selected one or more tile streams is based on an HTTP adaptive streaming protocol.

### Claim 7 (depends on 1)

Method according to claim 1 wherein the manifest file further defines one or more representations of a tile stream.

### Claim 8 (depends on 1)

Method according to claim 1 wherein the manifest file comprises one or more spatial relation descriptors (SRD), a spatial relation descriptor comprising one or more SRD parameters for defining the tile position and dimensions of the tile on the basis of coordinates of the 3D coordinate system.

### Claim 9 (depends on 8)

Method according to claim 8 wherein the one or more spatial relation descriptors (SRD) further comprise one or more SRD parameters for signaling a type of 2D projection that is used for projecting the omnidirectional video data onto a planar surface.

### Claim 10

A client device for processing an omnidirectional video, said omnidirectional video being associated with a 3D coordinate system for determining a spatial position of video data of said omnidirectional video on a curved surface, the client device comprising:
a computer readable storage medium having at least part of a program embodied therewith, the computer readable storage medium comprising a manifest file; and,
a computer readable storage medium having computer readable program code embodied therewith, and a processor coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising:
receiving a manifest file, the manifest file comprising a plurality of tile stream identifiers for identifying a plurality of tile streams, the tile streams comprising video frames with 2D projected video data, the video frames having image views, whereby the image views of video frames of different tile streams cover different regions of a 2D projection of the omnidirectional video, a region defining a tile; and,
selecting on the basis of spatial relation information in the manifest file and on the basis of a viewpoint of a user of the client device a first tile stream associated with a first resolution and a first tile position and a second tile stream associated with a second resolution and a second tile position, the second resolution being lower than the first resolution, wherein the spatial relation information defines for each tile a tile position and wherein the viewpoint defines a viewing direction of the user, the tile position and the viewpoint being defined on the basis of coordinates of the 3D coordinate system.

### Claim 11 (depends on 10)

Client device according to claim 10, wherein the executable operations further comprise:
requesting the selected one or more tile streams from one or more tile stream storage devices; and,
receiving the video data of the one or more selected tile streams and receiving mapping information, the mapping information providing the client device with information for enabling the client device to map the 2D projected video data of the tile streams as omnidirectional video data onto the curved surface;
processing the video data of said received tile streams on the basis of the spatial relation information and the mapping information.

### Claim 12 (depends on 10)

Client device according to claim 10, wherein the manifest file comprises one or more spatial relation descriptors (SRD), a spatial relation descriptor comprising one or more SRD parameters for defining the tile position and dimensions of the tile on the basis of coordinates of the 3D coordinate system.

### Claim 13

Non-transitory computer-readable storage media for storing a manifest file for a client device, said client device being configured to process an omnidirectional video by on the basis of said manifest file, said omnidirectional video being associated with a 3D coordinate system for determining a spatial position of video data of said omnidirectional video on a curved surface, said manifest file comprising computer readable data, said data comprising:
a plurality of tile stream identifiers for identifying a plurality of tile streams, the tile streams comprising video frames with 2D projected video data, the video frames having image views, whereby the image views of video frames of different tile streams cover different regions of a 2D projection of the omnidirectional video, a region defining a tile;
spatial relation information, the spatial relation information defining for each tile a tile position, the tile position and the viewpoint being defined on the basis of coordinates of the 3D coordinate system; and,
mapping information for providing the client device with information for enabling the client device to map the 2D projected video data of the tile streams identified in the manifest file as omnidirectional video data onto the curved surface.

### Claim 14 (depends on 13)

Non-transitory computer-readable storage media according to claim 13, wherein the computer readable data further comprises:
one or more spatial relation descriptors (SRD), a spatial relation descriptor comprising one or more SRD parameters for defining the tile position and dimensions of the tile on the basis of coordinates of the 3D coordinate system; and/or, one or more SRD parameters for signaling a type of 2D projection that is used for projecting the omnidirectional video data onto a planar surface.

### Claim 15 (depends on 1)

Computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to claim 1.

### Claim 16 (depends on 3)

Method according to claim 3, wherein the mapping information further comprises one or more parameters that are needed for mapping the 2D projected video data onto the curved surface.

### Claim 17 (depends on 12)

Client device according to claim 12, wherein the one or more spatial relation descriptors further comprise one or more SRD parameters for signaling a type of 2D projection that is used for projecting the omnidirectional video data onto a planar surface.