# US 2018/0322855 A1 · Auditory augmentation system and method of composing a media product

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322855 A1
- Document ID: US20180322855A1
- Status: Published application
- Kind: A1
- Application number: 16/035,234
- Filed: 2018-07-13
- Published: 2018-11-08
- Earliest claimed: 2015-06-22
- Assignees: Time Machine Capital Limited
- Applicants: Time Machine Capital Limited
- Inventors: Joseph Michael William LYSKE
- Web page: https://patentgrep.com/patent/US20180322855A1

## CPC

- G10H 1/0025: {Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece (automatically producing a series of tones G10H1/26)}
- A63B 69/00: Training appliances or apparatus for special sports (training of parachutists B64D23/00)
- G06F 16/24575: {using context}
- G10H 1/368: {displaying animated or moving pictures synchronized with the music or audio part}
- G10H 1/40: Rhythm
- G11B 20/10527: {Audio or video recording; Data buffering arrangements (G11B20/12 - G11B20/18 take precedence)}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/036: Insert-editing
- G11B 27/038: Cross-faders therefor
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- H04H 60/06: Arrangements for scheduling broadcast services or broadcast-related services
- H04H 60/33: Arrangements for monitoring the users' behaviour or opinions
- H04H 60/65: for using the result on users' side
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- A63B 71/0686: {Timers, rhythm indicators or pacing apparatus using electric or electronic means}
- G06F 16/61: Indexing; Data structures therefor; Storage structures
- G10H 1/057: by envelope-forming circuits
- G10H 1/361: {Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems}
- G10H 1/46: Volume control
- G10H 7/008: {Means for controlling the transition from one tone waveform to another}
- G10H 2210/031: Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
- G10H 2210/051: for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings
- G10H 2210/061: for extraction of musical phrases, isolation of musically relevant segments, e.g. musical thumbnail generation, or for temporal structure analysis of a musical piece, e.g. determination of the movement sequence of a musical work
- G10H 2210/076: for extraction of timing, tempo; Beat detection
- G10H 2210/091: for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
- G10H 2210/105: Composing aid, e.g. for supporting creation, edition or modification of a piece of music
- G10H 2210/125: Medley, i.e. linking parts of different musical pieces in one single piece, e.g. sound collage, DJ mix
- G10H 2210/131: Morphing, i.e. transformation of a musical piece into a new different one, e.g. remix
- G10H 2210/371: Rhythm syncopation, i.e. timing offset of rhythmic stresses or accents, e.g. note extended from weak to strong beat or started before strong beat
- G10H 2210/375: Tempo or beat alterations; Music timing control
- G10H 2210/391: Automatic tempo adjustment, correction or control
- G10H 2220/126: for graphical editing of individual notes, parts or phrases represented as variable length segments on a 2D or 3D representation, e.g. graphical edition of musical collage, remix files or pianoroll representations of MIDI-like files
- G10H 2230/015: PDA [personal digital assistant] or palmtop computing devices used for musical purposes, e.g. portable music players, tablet computers, e-readers or smart phones in which mobile telephony functions need not be used
- G10H 2240/075: Musical metadata derived from musical analysis or for use in electrophonic musical instruments
- G10H 2240/085: Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
- G10H 2240/121: Musical libraries, i.e. musical databases indexed by musical parameters, wavetables, indexing schemes using musical parameters, musical rule bases or knowledge bases, e.g. for automatic composing methods
- G10H 2240/131: Library retrieval, i.e. searching a database or selecting a specific musical piece, segment, pattern, rule or parameter set
- G10H 2240/325: Synchronizing two or more audio tracks or files according to musical features or musical timings
- G10H 2250/035: Crossfade, i.e. time domain amplitude envelope control of the transition between musical sounds or melodies, obtained for musical purposes, e.g. for ADSR tone generation, articulations, medley, remix
- G11B 27/005: {Reproducing at a different information rate from the information rate of recording (for television signals H04N5/783)}
- G11B 27/105: {of operating discs}
- G11B 2020/1062: {Data buffering arrangements, e.g. recording or playback buffers}
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- A63B: APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT (apparatus for passive exercising, massage A61H)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An auditory augmentation system includes a database with a multiplicity of audio sections and associated metadata for digital audio files. Each audio section is mapped to a contextual theme, each contextual theme being mapped to an audio section having an entry point and an exit point. The entry and exit points support seamless splice or fade transitions between different audio sections. A processing system couples to the database along with an input; the input is in the form of temporally-varying events data that defines a temporal input. The processing system resolves the temporal input into one or more of a plurality of categorized contextual themes, correlates the categorized contextual themes with metadata associated with selected audio sections relevant to the one or more categorized contextual themes, and splices or fades together selected audio sections, and generates, as an output, a media product in which transitions between audio sections are seamless.

## Claims

### Claim 1

An auditory augmentation system comprising:
a database containing a multiplicity of audio sections and associated metadata for digital audio files, each audio section mapped to at least one contextual theme defined with respect to identified musical timings, each contextual theme itself mapped to an audio section having at least one entry point and at least one exit point, said entry and exit points supporting seamless splice or fade transitions between different audio sections and wherein each of said at least one entry point and said at least one exit point relates to an anacrusis event for the section as expressed in terms of beats and fractions;
a processing system coupled to the database and responsive to said metadata; and
an input coupled to the processing system, the input in the form of temporally-varying events data that defines a temporal input;
wherein the processing system is arranged:
to resolve the temporal input into one or more of a plurality of categorized contextual themes;
to correlate said categorized contextual themes with metadata associated with selected audio sections relevant to the one or more of the plurality of categorized contextual themes, and thereafter
to splice or fade together selected audio sections to reflect said events as the temporal input varies, thereby to generate, as an output, a media product in which transitions between audio sections are seamless and the audio sections are aligned with the temporally-varying events.

### Claim 2 (depends on 1)

The auditory augmentation system according to claim 1, wherein the processing system selects the audio track substantially in real-time to accompany changing physical or local environmental events.

### Claim 3 (depends on 1)

The auditory augmentation system according to claim 1, wherein the system is arranged to augment an overall sensory experience of a user in the real world by dynamically changing, re-ordering or repeating and then playing audio sections within the context of said categorized themes associated with the changing nature of events presented as the temporal input to the processing system.

### Claim 4 (depends on 1)

The auditory augmentation system according to claim 1, wherein the processing system is a distributed set of processing systems.

### Claim 5 (depends on 1)

The auditory augmentation system according to claim 1, wherein said entry and exit points support seamless rhythmic transition in audio output between audio sections and wherein the spice or fade preserves integrity of musical timing between audio sections.

### Claim 6 (depends on 1)

The auditory augmentation system according to claim 1, wherein said musical timings take the form of one of:
a beginning and an end in the case of a sustained feature for the contextual theme, wherein the sustained features is one of a form function and a hit point over time; and
a specific point in time for a hit point.

### Claim 7 (depends on 6)

The auditory augmentation system according to claim 6, wherein the form function is selected from one of the group consisting of:
a static section;
a drop-out;
a fade in intensity;
a build in intensity;
a relative intensity of the audio section; and
an underscore.

### Claim 8 (depends on 6)

The auditory augmentation system according to claim 6, wherein the hit point over time is selected from one of the group consisting of:
a sound effect;
speech;
riff; and
instrument type.

### Claim 9 (depends on 1)

The auditory augmentation system according to claim 1, wherein the temporal input takes the form of an intensity arising from at least one of:
a relative change in light levels;
a relative change in sound levels;
changing rates in a monitorable function; and
a relative change of audio characteristics between a current audio section and a new audio section that is to be cut or spliced into the current audio section.

### Claim 10 (depends on 1)

The auditory augmentation system according to claim 1, wherein the temporal input takes the form of a scaling of intensities arising from at least one of:
a relative change in light levels;
a relative change in sound levels;
changing rates in a monitorable function; and
a relative change of audio characteristics between a current audio section and a new audio section that is to be cut or spliced into the current audio section.

### Claim 11 (depends on 1)

The auditory augmentation system according to claim 1, wherein said musical timings take the form of one of:
(i) a beginning and an end in the case of a sustained feature for the contextual theme, wherein the sustained features is one of a form function and a hit point over time; and
(ii) a specific point in time for a hit point

### Claim 12

An auditory augmentation system comprising:
a database containing a multiplicity of audio sections for digital audio files, each audio section mapped to a contextual theme defined with respect to identifying musical timing for the audio section, each theme itself mapped to at least one entry point and at least one exit point, said entry and exit points supporting a seamless fade transition between different audio sections and wherein each of said at least one entry point and said at least one exit point relates to an anacrusis event for the section as expressed in terms of beats and fractions;
a processing system coupled to the database and responsive to said metadata; and
an input coupled to the processing system, the input identifying a changing nature of events presented as a temporal input to the processing system, each event categorized with a theme;
wherein the processing system is arranged to correlate the categorized themes of said changing events with said contextual themes for audio sections and is further arranged automatically to select and splice together audio sections to reflect said events as the temporal input varies.

### Claim 13

A method of aligning audio sections in one or more digital audio tracks with temporally-varying events data to compose a media product, the method comprising:
receiving, as temporal input, a stream of data containing temporally-changing events data;
accessing a database containing a multiplicity of audio sections and associated metadata for digital audio files, each audio section mapped to at least one contextual theme defined with respect to identified musical timings, each contextual theme itself mapped to an audio section having at least one entry point and at least one exit point, said entry and exit points supporting seamless splice or fade transitions between different audio sections and wherein each of said at least one entry point and said at least one exit point relates to an anacrusis event for the section as expressed in terms of beats and fractions and wherein said musical timings take the form of one of: (i) a beginning and an end in the case of a sustained feature for the contextual theme, wherein the sustained features is one of a form function and a hit point over time; and (ii) a specific point in time for a hit point;
resolving the temporal input into one of a plurality of relevant categorized contextual themes;
correlating said categorized contextual themes with metadata associated with selected audio sections relevant to the one or more of the plurality of categorized contextual themes; and
splicing or fading together selected audio sections from one or more selected digital audio files of the database to reflect said events as the temporal input varies, thereby generating, as an output, a media product in which transitions between audio sections are seamless and the audio sections are aligned with the temporally-changing events.

### Claim 14 (depends on 13)

The method of claim 13, wherein the audio track is selected to accompany changing physical or local environmental events.

### Claim 15 (depends on 13)

The method of claim 13, wherein the method augments an overall sensory experience of a user in the real world by dynamically changing, re-ordering or repeating and then playing audio sections within the context of said categorized themes associated with the changing nature of events presented as the temporal input to the processing system.

### Claim 16

A method of aligning audio sections in one or more digital audio tracks with temporally-varying events data to compose a media product, the method comprising:
receiving, as temporal input, a stream of data containing temporally-changing events data;
accessing a database containing a multiplicity of audio sections and associated metadata for digital audio files, each audio section mapped to at least one contextual theme defined with respect to identified musical timings, each contextual theme itself mapped to an audio section having at least one entry point and at least one exit point, said entry and exit points supporting seamless splice or fade transitions between different audio sections and wherein each of said at least one entry point and said at least one exit point relates to an anacrusis event for the section as expressed in terms of beats and fractions and wherein said musical timings take the form of one of: (i) a beginning and an end in the case of a sustained feature for the contextual theme, wherein the sustained features is one of a form function and a hit point over time; and (ii) a specific point in time for a hit point;
selecting at least one relevant contextual theme from a plurality of contextual themes;
correlating said at least one selected relevant contextual theme with metadata associated with selected audio sections relevant to the one or more of the plurality of categorized contextual themes; and
splicing or fading together selected audio sections from one or more selected digital audio files of the database to reflect said events as the temporal input varies, thereby generating, as an output, a media product in which transitions between audio sections are seamless and the audio sections are aligned with the temporally-changing events.

### Claim 17 (depends on 16)

The method of claim 16, wherein the audio track is selected to accompany changing physical or local environmental events.

### Claim 18 (depends on 16)

The method of claim 16, wherein the method augments an overall sensory experience of a user in the real world by dynamically changing, re-ordering or repeating and then playing audio sections within the context of said categorized themes associated with the changing nature of events presented as the temporal input to the processing system.