# US 2022/0044663 A1 · Music context system audio track structure and method of real-time synchronization of musical content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044663 A1
- Document ID: US20220044663A1
- Status: Published application
- Kind: A1
- Application number: 17/466,275
- Filed: 2021-09-03
- Published: 2022-02-10
- Earliest claimed: 2015-06-22
- Assignees: Mashtraxx Limited
- Applicants: Mashtraxx Limited
- Inventors: Joseph Michael William LYSKE
- Web page: https://patentgrep.com/patent/US20220044663A1

## CPC

- G10H 1/0025: {Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece (automatically producing a series of tones G10H1/26)}
- A63B 69/00: Training appliances or apparatus for special sports (training of parachutists B64D23/00)
- G06F 16/24575: {using context}
- G10H 1/368: {displaying animated or moving pictures synchronized with the music or audio part}
- G10H 1/40: Rhythm
- G11B 20/10527: {Audio or video recording; Data buffering arrangements (G11B20/12 - G11B20/18 take precedence)}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/036: Insert-editing
- G11B 27/038: Cross-faders therefor
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- H04H 60/06: Arrangements for scheduling broadcast services or broadcast-related services
- H04H 60/33: Arrangements for monitoring the users' behaviour or opinions
- H04H 60/65: for using the result on users' side
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- A63B 71/0686: {Timers, rhythm indicators or pacing apparatus using electric or electronic means}
- G06F 16/61: Indexing; Data structures therefor; Storage structures
- G10H 1/057: by envelope-forming circuits
- G10H 1/361: {Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems}
- G10H 1/46: Volume control
- G10H 7/008: {Means for controlling the transition from one tone waveform to another}
- G10H 2210/031: Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
- G10H 2210/051: for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings
- G10H 2210/061: for extraction of musical phrases, isolation of musically relevant segments, e.g. musical thumbnail generation, or for temporal structure analysis of a musical piece, e.g. determination of the movement sequence of a musical work
- G10H 2210/076: for extraction of timing, tempo; Beat detection
- G10H 2210/091: for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
- G10H 2210/105: Composing aid, e.g. for supporting creation, edition or modification of a piece of music
- G10H 2210/125: Medley, i.e. linking parts of different musical pieces in one single piece, e.g. sound collage, DJ mix
- G10H 2210/131: Morphing, i.e. transformation of a musical piece into a new different one, e.g. remix
- G10H 2210/371: Rhythm syncopation, i.e. timing offset of rhythmic stresses or accents, e.g. note extended from weak to strong beat or started before strong beat
- G10H 2210/375: Tempo or beat alterations; Music timing control
- G10H 2210/391: Automatic tempo adjustment, correction or control
- G10H 2220/126: for graphical editing of individual notes, parts or phrases represented as variable length segments on a 2D or 3D representation, e.g. graphical edition of musical collage, remix files or pianoroll representations of MIDI-like files
- G10H 2230/015: PDA [personal digital assistant] or palmtop computing devices used for musical purposes, e.g. portable music players, tablet computers, e-readers or smart phones in which mobile telephony functions need not be used
- G10H 2240/075: Musical metadata derived from musical analysis or for use in electrophonic musical instruments
- G10H 2240/085: Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
- G10H 2240/121: Musical libraries, i.e. musical databases indexed by musical parameters, wavetables, indexing schemes using musical parameters, musical rule bases or knowledge bases, e.g. for automatic composing methods
- G10H 2240/131: Library retrieval, i.e. searching a database or selecting a specific musical piece, segment, pattern, rule or parameter set
- G10H 2240/325: Synchronizing two or more audio tracks or files according to musical features or musical timings
- G10H 2250/035: Crossfade, i.e. time domain amplitude envelope control of the transition between musical sounds or melodies, obtained for musical purposes, e.g. for ADSR tone generation, articulations, medley, remix
- G11B 27/005: {Reproducing at a different information rate from the information rate of recording (for television signals H04N5/783)}
- G11B 27/105: {of operating discs}
- G11B 2020/1062: {Data buffering arrangements, e.g. recording or playback buffers}
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- A63B: APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT (apparatus for passive exercising, massage A61H)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system is described that permits identified musical phrases or themes to be synchronized and linked into changing real-world events. The achieved synchronization includes a seamless musical transition—achieved using a timing offset, such as relative advancement of an significant musical “onset,” that is inserted to align with a pre-existing but identified music signature, beat or timebase—between potentially disparate pre-identified musical phrases having different emotive themes defined by their respective time signatures, intensities, keys, musical rhythms and/or musical phrasing. The system operates to augment an overall sensory experience of a user in the real world by dynamically changing, re-ordering or repeating and then playing audio themes within the context of what is occurring in the surrounding physical environment, e.g. during different phases of a cardio workout in a step class the music rate and intensity increase during sprint periods and decrease during recovery periods.

## Claims

### Claim 1

A method of determining onsets and their position in a digital signal, the method comprising:
separating transient parts of the digital signal from a relatively stationary part of the digital signal to produce a time domain representation;
generating an amplitude envelope of the transient part;
setting a power threshold in the amplitude envelope and detecting localised peaks in the amplitude envelope that exceed the power threshold; and
with respect to each localised peak, working backwards in time from that localized peak to find a sampling point at which a characteristic of the digital signal before that point in time is maximally different from that same characteristic of the digital signal after that point in time; and
selecting, as the position of the onset for the localized peak, the sampling point that is maximally different.

### Claim 2 (depends on 1)

The method of claim 1, wherein the step of identifying the onset corresponds to a largest measured power ratio between adjacent predetermined periods.

### Claim 3 (depends on 1)

The method of claim 1, wherein the characteristics includes a standard deviation of the first derivative of the transient signal before and after each sampling point.

### Claim 4 (depends on 1)

The method of claim 1, wherein the transient parts occur at relatively high frequencies in the audio signal.

### Claim 5 (depends on 1)

The method of claim 1, wherein the amplitude envelope is produced by applying a convolution function.

### Claim 6 (depends on 1)

The method of claim 1, wherein said relatively stationary parts are frequencies that appear across consecutive sampling windows and transient parts are frequencies that evolve in contiguous sampling windows, wherein each sampling window is between about ten milliseconds and about fifty milliseconds.

### Claim 7 (depends on 1)

A computer program product that, when executed by a processor,
causes the processor to execute procedure to determine the presence of an onset in a section of an audio signal, the procedure executing the method of claim 1.

### Claim 8

A digital signal processor “DSP” for separating a transient part of a digital representation of a signal from a relatively stationary part of the digital representation of the signal and to find a boundary position of the transient part, the DSP arranged to:
produce a time domain representation of the digital signal;
generate an amplitude envelope of the transient part;
set a power threshold and detect localised peaks in the amplitude envelope that exceed the power threshold; and
with respect to each localised peak, working backwards in time from that localized peak to find a sampling point at which a characteristic of the digital signal before that point in time is maximally different from that same characteristic of the digital signal after that point in time; and
select, as the boundary position of the onset for the localized peak, the sampling point that is maximally different.

### Claim 9 (depends on 8)

An audio system comprising the DSP of claim 8:

### Claim 10

A system comprising a digital signal processor arranged, for a digital audio file having transient parts and relatively stationary parts, to:
separate the transient parts from the relatively stationary parts to produce a time domain representation, wherein said relatively stationary parts are frequencies that appear across consecutive sampling windows and transient parts are frequencies that demonstrate significant changes in energy in contiguous sampling windows where each sampling window has a duration of between about 10 milliseconds and about 50 milliseconds;
generate an amplitude envelope of the transient parts;
set a power threshold and detect a localized peak in the amplitude envelope that exceed the power threshold; and
from the localized peak, to iterate backwards in time on a sample-by-sample basis over the duration of the sampling window to find a sample point at which a measurable characteristic of the digital audio signal, measured during a first period having a first duration extending backwards in time before the sampling point, is maximally different from the measurable characteristic of the digital audio signal measured during a second period also having the first duration, the second period extending forwards in time from the sampling point; and
select, as the onset, the sample that is maximally different.

### Claim 11 (depends on 10)

The system of claim 10, further comprising:
a database containing a multiplicity of digital files each partitioned into at least one section characterized by a contextual theme, each of said at least one sections having an onset representative of a significant temporal event; and
a processor arranged to use said identified onset to transition between a first contextual theme to a second contextual theme, the first contextual theme and the second contextual theme associated with at least one of said digital files.

### Claim 12 (depends on 11)

The audio system of claim 11, wherein the first contextual theme is the same as the second contextual theme.

### Claim 13 (depends on 11)

The system of claim 11, wherein the digital files are stored in a database and the processor is arranged remotely to access the database over a network connection, the network connection supporting one of download and streaming of digital files to the processor of a subscriber device.
