System and method for predicting user viewpoint using location information of sound source in 360 VR contents

A method of predicting a location of a user's viewport using location information of a sound source in 360 virtual reality (VR) content, the method comprising:

• (a) by a content production unit, spatially dividing a panorama video into multiple segment tiles to generate an MPD for each tile and compression-coding the generated MPD and the segment tiles at resolutions for respective tiles to transmit the resulting in a form of media data to a HTTP server;
• (b) by the HTTP server, dividing the received media data into segments on a per-predefined time basis in accordance with a MPD request command together with a uniform resource locator (URL) included in the MPD received through a network and then transmitting the MPD including a video MPD and an audio MPD and the segment tiles to a client device with a reference bandwidth of the network;
• (c) by the client device, predicting the location of the user's viewport on the basis of the segment tiles and the video MPD and the audio MPD received through the network, determining a segment bitrate to be provided next adaptively on the basis of bandwidth condition, and making a request for a segment tile in the location of the user's viewport predicted at the determined segment bitrate to the HTTP server;
• (d) by the HTTP server, transmitting a segment to be provided next to the client device with an allocated bandwidth provided by the client device;
• and
• (e) by the client device, performing decoding on the segment tiles transmitted through the network to acquire video and audio, and rendering the acquired video and audio in a 360-degree space to be regenerated,
• wherein the audio MPD includes spatial representation description (SRD) and sound localization information (SLI),
• in which the SRD includes source_id (non-negative integer in a decimal representation providing an identifier for a source of content and implicitly defining a coordinate system), object_x (non-negative integer in decimal representation expressing a horizontal location of a top-left corner of associated media assets in the coordinate system), object_y (non-negative integer in decimal representation expressing a vertical location of a top-left corner of the associated media assets in the coordinate system), object_width (non-negative integer in decimal representation expressing a width of the associated media assets in the coordinate system), object_height (non-negative integer in decimal representation expressing a height of the associated media assets in the coordinate system), total_width (optional non-negative integer in decimal representation expressing a width of the extent of all media assets in the coordinate system), total_height (optional non-negative integer in decimal representation expressing a height of the extent of all media assets in the coordinate system), and spatial_set_id (optional non-negative integer in decimal representation providing an identifier for a group of media assets),
• wherein a sound localization information description (SLID) includes:
• a sound source localization identifier (SLI_id),
• a location (x, y, z axis value) of the sound source in a 360-degree space having sound_R (information on right phase of the sound source in 360 VR content), sound_L (information on left phase of the sound source in 360 VR content), sound_spatial_hori (horizontal angle information for a user to sense the sound source in 360 VR content), and sound_spatial_verti (vertical angle information for the user to sense, and
• a panning model,
• wherein the step (c) is configured to
• determine the segment bitrate to be provided next on the basis of a weight of a tile for at least one item of a region of interest (ROI), sound source localization information (SLI), a current viewport, and a pole, a reference bitrate, a rank for each bandwidth condition, and tiles located in the ROI, the SLI, the current viewport, and the pole, and
• transmit the determined bitrate to the HTTP server,
• wherein the weights in the step (c) are stored in such a manner as to be matched to respective items of the ROI, the SLI, the current viewport, and the pole that are predetermined for each bandwidth condition, and
• wherein the step (c) is provided to
• determine that the bandwidth condition is very good when a rank r derived from the sum of weights of respective items is one,
• set each of the weight wS of the tile corresponding to the SLI, the weight wR of the tile corresponding to the ROI, the weight wV of the tile corresponding to the current viewport, and the weight wP of the tile corresponding to the pole as a maximum weight set wX, and
• derive a bitrate Ri of an i-th tile from the total sum of products of the maximum weight max wX, the i-th tile ti, and the reference bitrate Rf.