# US 12,009,013 B2 · System and method for AI/XI based automatic song finding method for videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,009,013 B2
- Document ID: US12009013B2
- Status: Granted
- Kind: B2
- Application number: 18/078,329
- Filed: 2022-12-09
- Published: 2023-06-15
- Granted: 2024-06-11
- Earliest claimed: 2021-12-09
- Assignees: Bellevue Investments GmbH & Co. Kgaa
- Applicants: BELLEVUE INVESTMENTS GmbH & Co. KGaA
- Inventors: Dieter Rein; Jurgen Jaron
- Web page: https://patentgrep.com/patent/US12009013B2

## CPC

- G10H 1/0025: {Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece (automatically producing a series of tones G10H1/26)}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G06F 16/435: Filtering based on additional data, e.g. user or group profiles
- G06F 16/48: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G10H 1/0008: {Associated control or indicating means}
- G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)}
- G10H 2240/075: Musical metadata derived from musical analysis or for use in electrophonic musical instruments
- G10H 2240/085: Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
- G10H 2240/131: Library retrieval, i.e. searching a database or selecting a specific musical piece, segment, pattern, rule or parameter set
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

According to a first embodiment, one method presented herein involves methods of finding the best fitting song from a large audio database for a selected video production. The content of these songs utilized in the instant invention has been tagged by emotion tags describing the energy, the emotion of these songs over time—meaning that each song can contain a plurality of, even overlapping, emotions.

## Claims

### Claim 1

A method of automatically matching one or more songs to a video, said video having a length associated therewith, comprising the steps of: (a) submitting said video to a video analysis tool, said video analysis tool automatically determining locations of scenes, segments, and objects in said video and assigning a plurality of time-varying emotion tags to said video; (b) in a computer, the processor performing the steps of: (i) determining an emotion level for each of said video scenes; (ii) calculating an energy level for each of said video scenes; (iii) determining one or more emotions associated with each of said video scenes; (iv) scaling each of said emotion values by a length of an associated tag; (v) determining a single value representative of all of said emotion values; (vi) searching an audio database containing a plurality of songs for one or more matching songs, each of said songs in said audio database having one or more emotion tags associated therewith, said emotion tags describing a time-varying energy and emotion of said song; (vii) select at least one song from said audio database based on said search; and (viii) adapting a length of said at least one song to match said video length, thereby automatically matching one or more songs to said video.

### Claim 2 (depends on 1)

The method according to claim 1, wherein step (vi) comprises the steps of: (vi1) determining one or more energy levels for said video; (vi2) determining one or more emotion values for said video; (vi3) determining a main emotion of said video; (vi4) determining a genre of said video; (vi5) filtering said plurality of songs in said database by said main emotion and genre; and (vi6) further filtering said filtered plurality of songs by a length of said video.

### Claim 3

A method of matching at least one song to a video work selected by a user, wherein is provided an audio database containing a plurality of songs therein, each of said songs having at least a genre, an emotion, an energy level, and a length associated therewith, comprising the steps of: (a) submitting said video work to a video analysis tool, said video analysis tool automatically determining locations of scenes, segments, and objects in said video work and assigning a plurality of video tags to said video work, said plurality of tags being associated with said determined locations of scenes, segments, and objects of said video work; (b) in a computer, the processor performing the steps of: (i) using said video tags to determine cuts and transitions in said video work; (ii) using at least said cuts, transitions and said video tags to generate at least one video scene from said video work; (iii) determining an energy level for each of said at least one video scenes, thereby obtaining at least one energy level associated with said video work; (iv) determining a video emotion value associated with each of said video tags; (v) using said video tags to scale each of said video emotion values, thereby determining a main emotion of said video work; (vi) using each of said at least one energy levels associated with said video work to determine a dominant energy level of said video work; (vii) using said main emotion of said video work to determine a video genre; (viii) determining a length of said video work; (ix) using said main emotion of said video work, said length of said video work, said genre of said video work, and said dominant energy level of said video work to conduct a first search of said audio database for candidate songs; (1) if at least one candidate song is returned from said first search of said audio database, (ix1) presenting said at least one candidate songs to the user, and (ix2) allowing the user to select at least one of said at least one candidate songs, thereby matching at least one song to said video work; (x) if no candidate songs are returned from said first search of said audio database, using said main emotion of said video work, said length of said video work, and said genre of said video work to conduct a second search of said audio database for candidate songs; (xi) if at least one candidate song is returned from said second search of said audio database, (xi1) presenting said at least one candidate songs to the user, and (xi2) allowing the user to select at least one of said at least one candidate songs, thereby matching at least one song to said video work; (xii) if no candidate songs are returned from first search and from said second search of said audio database, using said main emotion of said video work and said length of said video work to conduct a third search of said audio database for candidate songs; (xiii) if at least one candidate song is returned from said third search of said audio database, (xiii1) presenting said at least one candidate songs to the user, and (xiii2) allowing the user to select at least one of said at least one candidate songs, thereby matching at least one song to said video work.

### Claim 4

A method of matching at least one song to a video work selected by a user, wherein is provided an audio database containing a plurality of songs therein, each of said songs having at least a genre, an emotion, an energy level, and a length associated therewith, comprising the steps of: (a) submitting said video work to a video analysis tool, said video analysis tool automatically determining locations of scenes, segments, and objects in said video work and assigning a plurality of video tags to said video work, said plurality of tags being associated with said determined locations of scenes, segments, and objects of said video work; (b) in a computer, the processor performing the steps of: (i) using said video tags to determine cuts and transitions in said video work; (ii) using at least said cuts, transitions and said video tags to generate at least one video scene from said video work; (iii) determining an energy level for each of said at least one video scenes, thereby obtaining at least one energy level associated with said video work; (iv) determining a video emotion value associated with each of said video tags; (v) using said video tags to scale each of said video emotion values, thereby determining a main emotion of said video work; (vi) using each of said at least one energy levels associated with said video work to determine a dominant energy level of said video work; (vii) using said main emotion value and said video emotion values to search the audio database for a candidate song; (viii) adapting said candidate song to said video length; and (ix) applying said adapted candidate song to said video work, thereby matching said candidate song to said video work.