# US 11,122,314 B2 · Bitrate optimizations for immersive multimedia streaming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,122,314 B2
- Document ID: US11122314B2
- Status: Granted
- Kind: B2
- Application number: 16/613,961
- Filed: 2017-12-12
- Published: 2021-04-15
- Granted: 2021-09-14
- Earliest claimed: 2017-12-12
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Neil Birkbeck; Balineedu Adsumilli; Damien Kelly
- Web page: https://patentgrep.com/patent/US11122314B2

## CPC

- H04N 21/2662: Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
- H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/23439: {for generating different versions}
- H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
- H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)}
- H04N 21/816: {involving special video data, e.g 3D video}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Signals of an immersive multimedia item are jointly considered for optimizing the quality of experience for the immersive multimedia item. During encoding, portions of available bitrate are allocated to the signals (e.g., a video signal and an audio signal) according to the overall contribution of those signals to the immersive experience for the immersive multimedia item. For example, in the spatial dimension, multimedia signals are processed to determine spatial regions of the immersive multimedia item to render using greater bitrate allocations, such as based on locations of audio content of interest, video content of interest, or both. In another example, in the temporal dimension, multimedia signals are processed in time intervals to adjust allocations of bitrate between the signals based on the relative importance of such signals during those time intervals. Other techniques for bitrate optimizations for immersive multimedia streaming are also described herein.

## Claims

### Claim 1

A method for adjusting allocations of bitrate between a video signal and an audio signal of an immersive multimedia item, the method comprising:
producing a video saliency map representing changes in the video signal over a period of time within a spatial region of the immersive multimedia item;
producing an audio saliency map representing changes in the audio signal over the period of time within the spatial region, wherein at least one of the changes in the video signal or the changes in the audio signal reflect content of interest;
combining at least a portion of the video saliency map and at least a portion of the audio saliency map to produce a fused saliency map, wherein the combining includes adjusting relative weights of the audio saliency map and of the video saliency map; and
enhancing the content of interest by adjusting the allocations of the bitrate between the video signal and the audio signal according to the fused saliency map.

### Claim 2 (depends on 1)

The method of claim 1, wherein producing the video saliency map representing the changes in the video signal over the period of time within the spatial region of the immersive multimedia item comprises:
rendering multiple viewports of the immersive multimedia item, wherein at least some viewports of the multiple viewports correspond to the spatial region;
determining saliencies of at least some of the multiple viewports, wherein the saliencies reflect changes in video content within the at least some of the multiple viewports; and
aggregating the saliencies within a common parameterization domain.

### Claim 3 (depends on 2)

The method of claim 2, wherein rendering multiple viewports of the immersive multimedia item comprises:
selecting the multiple viewports based on statistical data indicating frequently viewed spatial regions of the immersive multimedia item over the period of time.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
parameterizing the audio signal in a direction of a spherical representation of the immersive multimedia item to identify changes in loudness of audio content of the immersive multimedia item, the direction corresponding to the spatial region.

### Claim 5 (depends on 4)

The method of claim 4, wherein producing the audio saliency map representing the changes in the audio signal over the period of time within the spatial region comprises:
calculating a weighted sum of convolved contributions of power of the parameterized audio signal per audio class and frequency; and
determining the changes in the audio signal over the period of time based on the weighted sum.

### Claim 6 (depends on 5)

The method of claim 5, wherein the weighted sum is calculated subsequent to determining that the audio content for which the changes in loudness are identified correspond to a relevant audio class.

### Claim 7 (depends on 1)

The method of claim 1, wherein combining the at least a portion of the video saliency map and the at least a portion of the audio saliency map to produce the fused saliency map comprises:
determining a first value by applying first parameters against the video saliency map;
determining a second value by applying second parameters against the audio saliency map;
determining a third value by applying third parameters against a product of the video saliency map and the audio saliency map; and
combining the first value, the second value, and the third value.

### Claim 8 (depends on 7)

The method of claim 7, wherein each of the first parameters, the second parameters, and the third parameters includes a linear weight parameter and a non-linear gain parameter.

### Claim 9 (depends on 1)

The method of claim 1, wherein enhancing the content of interest by adjusting the allocations of the bitrate between the video signal and the audio signal according to the fused saliency map comprises:
determining whether salient regions of the fused saliency map reflect a greater presence of audio content or of video content;
responsive to determining that the salient regions reflect a greater presence of audio content, adjusting the allocations of the bitrate by increasing a bitrate allocation for the audio signal and decreasing a bitrate allocation for the video signal over the period of time; and
responsive to determining that the salient regions reflect a greater presence of video content, adjusting the allocations of the bitrate by increasing a bitrate allocation for the video signal and decreasing a bitrate allocation for the audio signal over the period of time.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
aggregating data indicative of frequently viewed spatial regions of the immersive multimedia item to produce a probability map indicative of a probable view direction; and
readjusting the allocations of the bitrate between the video signal and the audio signal according to the probability map.

### Claim 11

An apparatus for adjusting allocations of bitrate between a video signal and an audio signal of an immersive multimedia item, the apparatus comprising:
a processor
configured to:
produce a video saliency map representative of changes in the video signal over a period of time within a spatial region of the immersive multimedia item;
produce an audio saliency map representative of changes in the audio signal over the period of time within the spatial region, wherein at least one of the changes in the video signal or the changes in the audio signal reflect content of interest;
combine at least a portion of the video saliency map and at least a portion of the audio saliency map to produce a fused saliency map, wherein to combine comprises to adjust relative weights of the audio saliency map and of the video saliency map; and
enhance the content of interest by adjusting the allocations of the bitrate between the video signal and the audio signal according to the fused saliency map.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein to produce the video saliency map representative of the changes in the video signal over the period of time within the spatial region of the immersive multimedia item comprises to:
render multiple viewports of the immersive multimedia item, wherein at least some viewports of the multiple viewports correspond to the spatial region;
determine saliencies of at least some of the multiple viewports, wherein the saliencies reflect changes in video content within the at least some of the multiple viewports; and
aggregate the saliencies within a common parameterization domain.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein to render multiple viewports of the immersive multimedia item comprises to:
select the multiple viewports based on statistical data indicative of frequently viewed spatial regions of the immersive multimedia item over the period of time.

### Claim 14 (depends on 11)

The apparatus of claim 11, wherein the processor is configured to:
parameterize the audio signal in a direction of a spherical representation of the immersive multimedia item to identify changes in loudness of audio content of the immersive multimedia item, the direction corresponding to the spatial region.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein to produce the audio saliency map representative of the changes in the audio signal over the period of time within the spatial region comprises to:
calculate a weighted sum of convolved contributions of power of the parameterized audio signal per audio class and frequency; and
determine the changes in the audio signal over the period of time based on the weighted sum.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the weighted sum is calculated subsequent to a determination that the audio content for which the changes in loudness are identified correspond to a relevant audio class.

### Claim 17 (depends on 11)

The apparatus of claim 11, wherein to combine the at least a portion of the video saliency map and the at least a portion of the audio saliency map to produce the fused saliency map comprises to:
determine a first value by applying first parameters against the audio saliency map;
determine a second value by applying second parameters against the video saliency map;
determine a third value by applying third parameters against a product of the video saliency map and the audio saliency map; and
combine the first value, the second value, and the third value.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein each of the first parameters, the second parameters, and the third parameters includes a linear weight parameter and a non-linear gain parameter.

### Claim 19 (depends on 11)

The apparatus of claim 11, wherein to enhance the content of interest by adjusting the allocations of the bitrate between the video signal and the audio signal according to the fused saliency map comprises to:
determine whether salient regions of the fused saliency map reflect a greater presence of audio content or of video content;
responsive to a determination that the salient regions reflect a greater presence of audio content, adjust the allocations of the bitrate by increasing a bitrate allocation for the audio signal and decreasing a bitrate allocation for the video signal over the period of time; and
responsive to a determination that the salient regions reflect a greater presence of video content, adjust the allocations of the bitrate by increasing a bitrate allocation for the video signal and decreasing a bitrate allocation for the audio signal over the period of time.

### Claim 20 (depends on 11)

The apparatus of claim 11, wherein the processor is configured to:
aggregate data indicative of frequently viewed spatial regions of the immersive multimedia item to produce a probability map indicative of a probable view direction; and
readjust the allocations of the bitrate between the video signal and the audio signal according to the probability map.