US 12,119,026 B2Granted

Multimedia music creation using visual input

AssigneeArtiphon, Inc.

InventorMichael V. Butera

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system for creating music using visual input. The system detects events and metrics (e.g., objects, gestures, etc.) in user input (e.g., video, audio, music data, touch, motion, etc.) and generates music and visual effects that are synchronized with the detected events and correspond to the detected metrics. To generate the music, the system selects parts from a library of stored music data and assigns each part to the detected events and metrics (e.g., using heuristics to match musical attributes to visual attributes in the user input). To generate the visual effects, the system applies rules (e.g., that map musical attributes to visual attributes) to translate the generated music data to visual effects. Because the visual effects are generated using music data that is generated using the detected events/metrics, both the generated music and the visual effects are synchronized with—and correspond to—the user input.

Claims20

Dependent claims

A computer-implemented method for creating music using visual input, the method comprising:

storing a library of stored music data, the stored music data comprising a plurality of parts, each part comprising musical notes;
receiving, from a user, user input data that includes visual input;
detecting, in the user input data, events and metrics;
and
generating audio that is synchronized with the detected events and corresponds to the detected metrics by:
selecting, from the library of stored music data, parts of the stored music data;
generating music data by assigning each of the selected parts of the stored music data to the detected events and metrics;
and
translating the events and metrics detected in the user input data to audio by synthesizing the musical notes of the assigned parts of the generated music data;
generating visual effects that are synchronized with the events detected in the user input data and correspond to the metrics detected in the user input data by:
storing rules for translating the stored music data to visual effects;
and
using the stored rules to translate the assigned parts to visual effects;
and
generating video that includes the visual input, the generated audio, and the generated visual effects.

12 more elements

A system for creating music using visual input, comprising:

non-transitory computer readable storage media that stores a library of stored music data, the stored music data comprising a plurality of parts, each part comprising musical notes;
an event/metric detection unit that:
receives user input data, from a user, that includes visual input;
and
detects events and metrics in the user input data;
a music data translation unit that translates the events and metrics detected in the user input data to generated music data by:
selecting parts of the stored music data;
and
assigning each of the selected parts to the detected events and metrics;
an audio engine that generates audio that is synchronized with the detected events and corresponds to the detected metrics by synthesizing the musical notes of the assigned parts of the generated music data;
and
a video engine that:
generates visual effects that are synchronized with the events detected in the user input data and correspond to the metrics detected in the user input data by applying rules to translate the generated music data to visual effects;
and
generates video includes the generated audio, the visual input, and the generated visual effects.

12 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/722,202Filed 15 Apr 2022 · us provisional application
Priority: 15 Apr 2021
Published: US 2022/0335974 A120 Oct 2022
Granted: 15 Oct 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification17 codes

G11B 27/031
Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers: electronic editing of digitised analogue…
G11B 27/10
Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel: indexing; Addressing; Timing or synchronising;…
G06V 10/82
Using pattern recognition or machine learning: using neural networks
G06V 20/41
In video content: higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian…