# US 11,451,689 B2 · System and method for matching audio content to virtual reality visual content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,689 B2 - Document ID: US11451689B2 - Status: Granted - Kind: B2 - Application number: 15/948,417 - Filed: 2018-04-09 - Published: 2018-10-11 - Granted: 2022-09-20 - Earliest claimed: 2017-04-09 - Assignees: InSoundz Ltd. - Applicants: InSoundz Ltd. - Inventors: Tomer Goshen; Emil Winebrand - Web page: https://patentgrep.com/patent/US11451689B2 ## CPC - H04N 5/06: Generation of synchronising signals - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/2368: Multiplexing of audio and video streams - H04N 21/6587: Control parameters, e.g. trick play commands, viewpoint selection - H04N 21/80: Generation or processing of content or additional data by content creator independently of the distribution process; Content per se {(arrangements for generating broadcast information H04H60/02)} - H04N 21/8106: {involving special audio data, e.g. different tracks for different languages} - H04N 21/8126: {involving additional data, e.g. news, sports, stocks, weather forecasts} - H04N 21/8146: {involving graphical data, e.g. 3D object, 2D graphics} - H04N 21/816: {involving special video data, e.g 3D video} - H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} - H04N 21/8547: involving timestamps for synchronizing content - H04R 27/00: Public address systems (circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; {monitoring or testing arrangements for public address systems H04R29/007}; amplifiers H03F) - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - H04S: STEREOPHONIC SYSTEMS ## Abstract A system and method for matching audio content to virtual reality visual content. The method includes analyzing received visual content and received metadata to determine an optimal audio source associated with the received visual content; configuring the optimal audio source to capture audio content; synthesizing the captured audio content with the received visual content; and providing the synthesized captured audio content and received visual content to a virtual reality (VR) device. ## Claims ### Claim 1 A method for matching audio content to virtual reality visual content, comprising: analyzing received visual content and metadata to determine an optimal audio source associated with the received visual content, wherein analyzing the received visual content and metadata further comprises determining a field of view of the visual content, wherein the metadata is associated with the visual content and includes at least one parameter indicating an area of capture of the visual content, wherein the field of view is determined based on the at least one parameter, wherein the optimal audio source is closest to the field of view among a plurality of available audio sources, wherein each audio source of the plurality of available audio sources is located in proximity to the area of captured of the visual content and is configured to capture sound beams associated with the visual content such that the optimal audio source provides the clearest sound associated with the visual content among the plurality of available audio sources; configuring the optimal audio source to capture audio content; synthesizing the audio content with the received visual content; and providing the synthesized audio content and received visual content to a virtual reality (VR) device. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining a new field of view of the visual content; and updating the determined optimal audio source based on the determined new field of view. ### Claim 3 (depends on 2) The method of claim 2, wherein the new field of view is determined based on the received metadata. ### Claim 4 (depends on 1) The method of claim 1, wherein the received metadata includes at least one of: location pointers, time pointers, perspective indicators, a position of a VR device relative to a starting position or a predetermined baseline, a speed at which the position of the VR device changes, eye tracking parameters, gyroscope measurements, inertial measurement unit measurements, and accelerometer measurements. ### Claim 5 (depends on 1) The method of claim 1, wherein synthesizing the audio content further comprises: matching the received visual content to the audio content with minimal lag or buffering. ### Claim 6 (depends on 1) The method of claim 1, wherein the audio content and received visual content are received over a live stream. ### Claim 7 (depends on 1) The method of claim 1, wherein the audio content and received visual content are previously recorded and stored on and retrieved from a storage. ### Claim 8 A system for matching audio content to virtual reality visual content, comprising: a processing circuitry; and a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to: analyze received visual content and metadata to determine an optimal audio source associated with the received visual content, wherein analyzing the received visual content and metadata further includes determining a field of view of the visual content, wherein the metadata is associated with the visual content and includes at least one parameter indicating an area of capture of the visual content, wherein the field of view is determined based on the at least one parameter, wherein the optimal audio source is closest to the field of view among a plurality of available audio sources, wherein each audio source of the plurality of available audio sources is located in proximity to the area of captured of the visual content and is configured to capture sound beams associated with the visual content such that the optimal audio source provides the clearest sound associated with the visual content among the plurality of available audio sources; configure the optimal audio source to capture audio content; synthesize the audio content with the received visual content; and provide the synthesized audio content and received visual content to a virtual reality (VR) device. ### Claim 9 (depends on 8) The system of claim 8, the system further configured to: determine a new field of view of the visual content; and update the determined optimal audio source based on the determined new field of view. ### Claim 10 (depends on 9) The system of claim 9, wherein the new field of view is determined based on the received metadata. ### Claim 11 (depends on 8) The system of claim 8, wherein the received metadata includes at least one of: location pointers; time pointers; perspective indicators; a position of a VR device relative to a starting position or a predetermined baseline; a speed at which the position of the VR device changes; eye tracking parameters; gyroscope measurements, inertial measurement unit measurements, and accelerometer measurements. ### Claim 12 (depends on 8) The system of claim 8, wherein synthesizing the audio content further comprises: matching the received visual content to the audio content with minimal lag or buffering. ### Claim 13 (depends on 8) The system of claim 8, wherein the audio content and received visual content are received over a live stream. ### Claim 14 (depends on 8) The system of claim 8, wherein the audio content and received visual content are previously recorded and stored on and retrieved from a storage. ### Claim 15 A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to perform a process, the process comprising: analyzing received visual content and metadata to determine an optimal audio source associated with the received visual content, wherein analyzing the received visual content and metadata further comprises determining a field of view of the visual content, wherein the metadata is associated with the visual content and includes at least one parameter indicating an area of capture of the visual content, wherein the field of view is determined based on the at least one parameter, wherein the optimal audio source is closest to the field of view among a plurality of available audio sources, wherein each audio source of the plurality of available audio sources is located in proximity to the area of captured of the visual content and is configured to capture sound beams associated with the visual content such that the optimal audio source provides the clearest sound associated with the visual content among the plurality of available audio sources; configuring the optimal audio source to capture audio content; synthesizing the audio content with the received visual content; and providing the synthesized audio content and received visual content to a virtual reality (VR) device.