# US 2024/0201494 A1 · Methods and systems for adding real-world sounds to virtual reality scenes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0201494 A1
- Document ID: US20240201494A1
- Status: Published application
- Kind: A1
- Application number: 18/589,388
- Filed: 2024-02-27
- Published: 2024-06-20
- Earliest claimed: 2022-05-27
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment LLC; Sony Interactive Entertainment Inc.
- Inventors: Victoria Dorn; Celeste Bean; Tatianna Manzon-Gutzman; Sarah Karp
- Web page: https://patentgrep.com/patent/US20240201494A1

## CPC

- G02B 27/017: {Head mounted}
- A63F 13/424: involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
- A63F 13/54: involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
- A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- H04S 7/304: {For headphones}
- H04S 7/40: {Visual indication of stereophonic sound image}
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

Methods and systems are provided for integrating media cues into virtual reality scenes presented on a head mounted display (HMD) is disclosed. The method includes presenting a virtual reality scene on a display of an HMD. The method further includes receiving sensor data from one or more sensors in a real-world space in which the HMD is located. Then, identifying an object location of an object in the real-world space that produces a sound. The method includes generating a media cue in the virtual reality scene presented in the HMD. The media cue is presented in a virtual location that is correlated to the object location of the object in the real-world space.

## Claims

### Claim 1

A method for integrating media cues into virtual reality scenes presented on a display, comprising:
presenting a virtual reality scene on the display;
outputting audio related to the virtual reality space via headphones;
receiving sensor data from one or more sensors in a real-world space in which the display is located;
identifying an object location of an object in the real-world space that produces a sound using the sensor data; and
generating a media cue in the virtual reality scene presented in the display, the media cue is presented in a virtual location that is correlated to the object location of the object in the real-world space, the media cue is rendered as audio data associated with the sound that is captured from the real-world space and is mixed with audio related to the virtual reality space when output via said headphones.

### Claim 2 (depends on 1)

The method of claim 1, wherein the virtual location of the media cue is presented relative to a point of view (POV) of a user viewing the display.

### Claim 3 (depends on 1)

The method of claim 1, wherein the media cue is additionally represented as image data for conveying the sound, the image data provides an indicator of direction of the sound relative to a point of view (POV) of a user viewing the display.

### Claim 4 (depends on 1)

The method of claim 1, wherein interactivity and audio related to the virtual reality scene is active when the media cue is presented.

### Claim 5 (depends on 1)

The method of claim 1, wherein the media cue is additionally represented as image data that conveys an identity of the object in a graphical form.

### Claim 6 (depends on 1)

The method of claim 1, wherein the audio related to the virtual reality space native audio from the virtual reality scene.

### Claim 7 (depends on 1)

The method of claim 1, wherein the virtual location of the media cue is dynamically changed based on changes or movements of the object in the real-world space.

### Claim 8 (depends on 1)

The method of claim 1, wherein the virtual location of the media cue moves to represent movements of the object in the real-world space, and updating a correlation of the object location of the object in the real-world space relative to the a point of view of a user viewing the display.

### Claim 9 (depends on 1)

The method of claim 1, wherein the audio data is a computer-generated sound that represents the sound that is captured from the real-world space.
