# US 12,190,916 B2 · Integration of audio into a multi-view interactive digital media representation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,190,916 B2 - Document ID: US12190916B2 - Status: Granted - Kind: B2 - Application number: 18/458,084 - Filed: 2023-08-29 - Published: 2023-12-14 - Granted: 2025-01-07 - Earliest claimed: 2015-09-22 - Assignees: Fyusion, Inc. - Applicants: Fyusion, Inc. - Inventors: Stefan Johannes Josef Holzer; Radu Bogdan Rusu; Vladimir Roumenov Glavtchev; Alexander Jay Bruen Trevor - Web page: https://patentgrep.com/patent/US12190916B2 ## CPC - G11B 27/32: on separate auxiliary tracks of the same or an auxiliary record carrier - G06F 16/686: {using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings} - G06F 16/687: using geographical or spatial information, e.g. location - G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel - G11B 27/322: {used signal is digitally coded} - H04N 13/349: Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking (for viewing without the aid of special glasses using fly-eye lenses H04N13/307) - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract According to particular embodiments, one process includes retrieving a multi-view interactive digital media representation that includes numerous images fused together into content and context models. The process next includes retrieving and processing audio data to be integrated into the multi-view interactive digital media representation. A first segment of audio data may be associated with a first position in the multi-view interactive digital media representation. In other examples, a first segment of audio data may be associated with a visual position or the location of a camera in the multi-view interactive digital media representation. The audio data may be played in coordination with the multi-view interactive digital media representation based on a user's navigation through the multi-view interactive digital media representation, where the first segment is played when the first position or first visual position is reached. ## Claims ### Claim 1 A method comprising: retrieving a Multi-view Interactive Digital Media Representation (MVIDMR) of a car, the MVIDMR being navigable by a user in one or more dimensions by selecting a viewpoint from a plurality of different viewpoints from which to view the car, each viewpoint corresponding to a different frame in a plurality of frames; attaching audio data to designated frames in the MVIDMR, the audio data being captured separately from the frames of the MVIDMR, the designated frames corresponding to designated viewpoints of the MVIDMR such that designated portions of the audio are played when the designated frames in the MVIDMR are reached during navigation; and causing, responsive to the user navigating to a first position in the MVIDMR, a first segment of the audio data to be played such that as the user navigates closer to a designated position in the designated frames, volume of corresponding portions of audio increases, wherein navigating the MVIDMR in one direction plays the audio data forward, navigating the MVIDMR in the opposite direction plays the audio data backwards and speed of playing the audio data corresponds to navigation speed. ### Claim 2 (depends on 1) The method of claim 1, wherein the audio data is processed to segment the audio data into the first segment and a second segment. ### Claim 3 (depends on 2) The method of claim 2, wherein the second segment is associated with a second position in the multi-view interactive digital media representation, wherein the second position in the MVIDMR triggers playback of the second segment. ### Claim 4 (depends on 2) The method of claim 2, wherein the MVIDMR is a surround view. ### Claim 5 (depends on 2) The method of claim 2, wherein the MVIDMR is a multi-view image. ### Claim 6 (depends on 2) The method of claim 2, wherein the MVIDMR is a three dimensional model. ### Claim 7 (depends on 2) The method of claim 2, wherein the audio data includes an audio recording that was obtained concurrently with a recording of images included in the multi-view interactive digital media representation. ### Claim 8 (depends on 2) The method of claim 2, wherein the audio data is an audio recording that was obtained separately from images included in the multi-view interactive digital media representation. ### Claim 9 (depends on 2) The method of claim 2, wherein the audio data is a pre-recorded audio file. ### Claim 10 (depends on 2) The method of claim 2, wherein the first segment and second segment are extracted from the audio data using independent component analysis. ### Claim 11 (depends on 2) The method of claim 2, wherein the first segment includes a first set of sound sources and the second segment includes a second set of sound sources. ### Claim 12 (depends on 11) The method of claim 11, wherein the first set of sound sources includes voices and the second set of sound sources includes background noises. ### Claim 13 (depends on 11) The method of claim 11, wherein the first position is a video object position associated with a captured object in a surround view. ### Claim 14 (depends on 11) The method of claim 11, wherein the first position is a video capture position with the position of a capture device. ### Claim 15 (depends on 2) The method of claim 2, wherein the first segment includes sounds occurring near the first position in the multi-view interactive digital media representation, and wherein the second segment includes sounds occurring near a second position in the MVIDMR. ### Claim 16 (depends on 2) The method of claim 2, wherein the user's navigation through the MVIDMR includes selecting automatic playback, wherein automatic playback triggers play of a designated navigation through the MVIDMR, wherein the designated navigation includes a set sequence of views and the audio data. ### Claim 17 (depends on 2) The method of claim 2, wherein the user's navigation through the MVIDMR includes navigating to the first position in the MVIDMR and navigating to a second position in the MVIDMR.