# US 2021/0275915 A1 · Methods, apparatus and systems for optimizing communication between sender(s) and receiver(s) in computer-mediated reality applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0275915 A1
- Document ID: US20210275915A1
- Status: Published application
- Kind: A1
- Application number: 17/208,991
- Filed: 2021-03-22
- Published: 2021-09-09
- Earliest claimed: 2017-06-15
- Assignees: Dolby International AB; Dolby Laboratories Licensing Corporation
- Applicants: Dolby International AB; Dolby Laboratories Licensing Corporation
- Inventors: Christof Fersch; Nicolas R. Tsingos
- Web page: https://patentgrep.com/patent/US20210275915A1

## CPC

- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- H04S 7/302: {Electronic adaptation of stereophonic sound system to listener position or orientation (H04S7/301 takes precedence)}
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
- A63F 13/5255: according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
- A63F 13/54: involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
- G06F 3/012: {Head tracking input arrangements}
- G09G 5/36: characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)}
- H04S 7/303: {Tracking of listener position or orientation}
- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/212: using sensors worn by the player, e.g. for measuring heart beat or leg activity
- G09G 2354/00: Aspects of interface with display user
- H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S 2420/11: Application of ambisonics in stereophonic audio systems
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- H04S: STEREOPHONIC SYSTEMS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The present invention is directed to systems, methods and apparatus for processing media content for reproduction by a first apparatus. The method includes obtaining pose information indicative of a position and/or orientation of a user. The pose information is transmitted to a second apparatus that provides the media content. The media content is rendered based on the pose information to obtain rendered media content. The rendered media content is transmitted to the first apparatus for reproduction. The present invention may include a first apparatus for reproducing media content and a second apparatus storing the media content. The first apparatus is configured to obtain pose information indicative and transmit the pose information to the second apparatus; and the second apparatus is adapted to: render the media content based on the pose information to obtain rendered media content; and transmit the rendered media content to the first apparatus for reproduction.

## Claims

### Claim 1

A system comprising a first apparatus for reproducing media content and a second apparatus storing the media content,
wherein the first apparatus is adapted to:
obtain pose information indicative of a position and/or orientation of a user; and
transmit the pose information to the second apparatus; and
the second apparatus is adapted to:
render the media content based on the pose information to obtain rendered media content; and
transmit the rendered media content to the first apparatus for reproduction.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the media content comprises audio content and the rendered media content comprises rendered audio content; and/or
the media content comprises video content and the rendered media content comprises rendered video content.

### Claim 3 (depends on 1)

The system according to claim 1, wherein the media content comprises audio content and the rendered media content comprises rendered audio content; and
the first apparatus is further adapted to generate an audible representation of the rendered audio content.

### Claim 4 (depends on 2)

The system according to claim 2, wherein the audio content is one of First Order Ambisonics, FOA, -based, Higher Order Ambisonics, HOA, -based, object-based, or channel based audio content, or a combination of two or more of FOA-based, HOA-based, object-based, or channel based audio content.

### Claim 5 (depends on 2)

The system according to claim 2, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content.

### Claim 6 (depends on 1)

The system according to claim 1, wherein the second apparatus is further adapted to:
obtain predicted pose information based on the pose information and previous pose information; and
render the media content based on the predicted pose information to obtain the rendered media content.

### Claim 7 (depends on 6)

The system according to claim 6, wherein the second apparatus is further adapted to:
transmit the predicted pose information to the first apparatus together with the rendered media content.

### Claim 8 (depends on 7)

The system according to claim 7, wherein the first apparatus is further adapted to:
compare the predicted pose information to actual pose information; and
update the rendered media content based on a result of the comparison.

### Claim 9 (depends on 8)

The system according to claim 8, wherein the predicted pose information is predicted for an estimate of a timing at which the rendered media content is expected to be processed by the first apparatus for reproduction; and
the actual pose information is pose information obtained at a timing at which the rendered media content is actually processed by the first apparatus for reproduction.

### Claim 10 (depends on 1)

The system according to claim 1, wherein the rendered media content is transmitted to the first apparatus in uncompressed form.

### Claim 11 (depends on 1)

The system according to claim 1, wherein the second apparatus is further adapted to encode the rendered media content before transmission to the first apparatus; and
the first apparatus is further adapted to decode the encoded rendered media content after reception at the first apparatus.

### Claim 12 (depends on 9)

The system according to claim 9, wherein the estimate of the timing at which the rendered audio content is expected to be processed by the first apparatus for reproduction includes an estimation of a time that is necessary for encoding and decoding the rendered audio content and/or an estimate of a time that is necessary for transmitting the rendered media content to the first apparatus.

### Claim 13 (depends on 6)

The system according to claim 6, wherein the predicted pose information is obtained further based on an estimate of a time that is necessary for encoding and decoding the rendered media content and/or an estimate of a time that is necessary for transmitting the rendered media content to the first apparatus.

### Claim 14 (depends on 1)

The system according to claim 1, wherein the first apparatus is further adapted to:
compare the pose information that has been used for rendering the media content to current pose information; and
update the rendered media content based on a result of the comparison.

### Claim 15 (depends on 1)

The system according to claim 1, wherein the second apparatus is further adapted to:
determine gradient information indicative of how the rendered media content changes in response to changes of the pose information; and
transmit the gradient information to the first apparatus together with the rendered media content; and
the first apparatus is further adapted to:
compare the pose information that has been used for rendering the media content to current pose information; and
update the rendered media content based on the gradient information and a result of the comparison.

### Claim 16 (depends on 1)

The system according to claim 1, wherein the media content comprises audio content and the rendered media content comprises rendered audio content;
the first apparatus is further adapted to transmit environmental information indicative of acoustic characteristics of an environment in which the first apparatus is located to the second apparatus; and
the rendering the media content is further based on the environmental information.

### Claim 17 (depends on 1)

The system according to claim 1, wherein the media content comprises audio content and the rendered media content comprises rendered audio content;
the first apparatus is further adapted to transmit morphologic information indicative of a morphology of the user or part of the user to the second apparatus; and
the rendering the media content is further based on the morphologic information.

### Claim 18

A second apparatus for providing media content for reproduction by a first apparatus, the second apparatus adapted to:
receive pose information indicative of a position and/or orientation of a user of the first apparatus;
render the media content based on the pose information to obtain rendered media content; and
transmit the rendered media content to the first apparatus for reproduction.

### Claim 19 (depends on 18)

The second apparatus according to claim 18, wherein the media content comprises audio content and the rendered media content comprises rendered audio content; and/or
the media content comprises video content and the rendered media content comprises rendered video content.

### Claim 20 (depends on 19)

The second apparatus according to claim 19, wherein the audio content is one of First Order Ambisonics, FOA, -based, Higher Order Ambisonics, HOA, -based, object-based, or channel based audio content, or a combination of two or more of FOA-based, HOA-based, object-based, or channel based audio content.

### Claim 21 (depends on 19)

The second apparatus according to claim 19, wherein the rendered audio content is one of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content, or a combination of two or more of binaural audio content, FOA audio content, HOA audio content, or channel-based audio content.

### Claim 22 (depends on 18)

The second apparatus according to claim 18, further adapted to:
obtain predicted pose information based on the pose information and previous pose information; and
render the media content based on the predicted pose information to obtain the rendered media content.

### Claim 23 (depends on 22)

The second apparatus according to claim 22, further adapted to:
transmit the predicted pose information to the first apparatus together with the rendered media content.

### Claim 24 (depends on 22)

The second apparatus according to claim 22, wherein the predicted pose information is predicted for an estimate of a timing at which the rendered media content is expected to be processed by the first apparatus for reproduction.

### Claim 25 (depends on 18)

The second apparatus according to claim 18, wherein the rendered media content is transmitted to the first apparatus in uncompressed form.

### Claim 26 (depends on 18)

The second apparatus according to claim 18, further adapted to encode the rendered media content before transmission to the first apparatus.

### Claim 27 (depends on 24)

The second apparatus according to claim 24, wherein the estimate of the timing at which the rendered audio content is expected to be processed by the first apparatus for reproduction includes an estimation of a time that is necessary for encoding and decoding the rendered audio content and/or an estimate of a time that is necessary for transmitting the rendered media content to the first apparatus.

### Claim 28 (depends on 22)

The second apparatus according to claim 22, wherein the predicted pose information is obtained further based on an estimate of a time that is necessary for encoding and decoding the rendered media content and/or an estimate of a time that is necessary for transmitting the rendered media content to the first apparatus.

### Claim 29 (depends on 18)

The second apparatus according to claim 18, further adapted to:
determine gradient information indicative of how the rendered media content changes in response to changes of the pose information; and
transmit the gradient information to the first apparatus together with the rendered media content.

### Claim 30 (depends on 18)

The second apparatus according to claim 18, wherein the media content comprises audio content and the rendered media content comprises rendered audio content;
the second apparatus is further adapted to receive environmental information indicative of acoustic characteristics of an environment in which the first apparatus is located from the first apparatus; and
the rendering the media content is further based on the environmental information.

### Claim 31 (depends on 18)

The second apparatus according to claim 18, wherein the media content comprises audio content and the rendered media content comprises rendered audio content;
the second apparatus is further adapted to receive morphologic information indicative of a morphology of the user or part of the user from the first apparatus; and
the rendering the media content is further based on the morphologic information.

### Claim 32

A first apparatus for reproducing media content provided by a second apparatus, the first apparatus adapted to:
obtain pose information indicative of a position and/or orientation of a user of the first apparatus;
transmit the pose information to the second apparatus;
receive rendered media content from the second apparatus, wherein the rendered media content has been obtained by rendering the media content based on the pose information; and
reproduce the rendered media content.

### Claim at position 33

33-47. (canceled)
