# US 10,504,498 B2 · Real-time jamming assistance for groups of musicians > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,498 B2 - Document ID: US10504498B2 - Status: Granted - Kind: B2 - Application number: 15/820,636 - Filed: 2017-11-22 - Published: 2019-05-23 - Granted: 2019-12-10 - Earliest claimed: 2017-11-22 - Assignees: Yousician Oy - Applicants: Yousician Oy - Inventors: Matti Ryynänen; Anssi Petteri Klapuri - Web page: https://patentgrep.com/patent/US10504498B2 ## CPC - G10H 1/383: {Chord detection and/or recognition, e.g. for correction, or automatic bass generation} - G10H 1/0025: {Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece (automatically producing a series of tones G10H1/26)} - G10H 1/368: {displaying animated or moving pictures synchronized with the music or audio part} - G10H 1/40: Rhythm - G10H 2210/061: for extraction of musical phrases, isolation of musically relevant segments, e.g. musical thumbnail generation, or for temporal structure analysis of a musical piece, e.g. determination of the movement sequence of a musical work - G10H 2210/066: for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental - G10H 2210/076: for extraction of timing, tempo; Beat detection - G10H 2210/105: Composing aid, e.g. for supporting creation, edition or modification of a piece of music - G10H 2210/125: Medley, i.e. linking parts of different musical pieces in one single piece, e.g. sound collage, DJ mix - G10H 2210/141: Riff, i.e. improvisation, e.g. repeated motif or phrase, automatically added to a piece, e.g. in real time - G10H 2210/571: Chords; Chord sequences - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE ## Abstract Real-time jamming is automatically assisted for musicians. A real-time audio signal is received of played music that is played by at least one person. Beat is tracked of the played music from the real-time audio signal and accordingly a time of a next beat is predicted. At least one of chords; notes; and drum sounds is recognized from the real-time audio signal and repetitions in the played music are accordingly detected. A next development is predicted in the played music, based on the detected repetitions, including at least one of chords; notes; and drum sounds that will be played next, and respective timing based on the predicted time of the next beat. A real-time output is produced based on the predicted next development in the played music. ## Claims ### Claim 1 A method comprising: automatically receiving a real-time audio signal of played music that is played by at least one person; automatically tracking beat of the played music from the real-time audio signal and accordingly automatically predicting a time of a next beat; automatically recognising from the real-time audio signal at least one of chords; notes; and drum sounds and accordingly automatically detecting repetitions in the played music; automatically predicting a next development in the played music, based on the detected repetitions, comprising at least one of chords; notes; and drum sounds that will be played next, and respective timing based on the predicted time of the next beat; and automatically producing a real-time output based on the predicted next development in the played music wherein: the predicting of the at least one of chords; notes; and drum sounds is performed by detecting self-similarity in the played music; and the user is allowed to select a desired musical style and the predicting of the next development is performed accordingly. ### Claim 2 (depends on 1) The method of claim 1, wherein a pre-defined library of musical patterns is used to assist in the predicting of the next development in the played music. ### Claim 3 (depends on 2) The method of claim 2, wherein the library contains any one or more musical patterns selected from a group consisting of: popular chord progressions; musical rules about note progressions; and popular drum sound patterns. ### Claim 4 A method comprising: automatically receiving a real-time audio signal of played music that is played by at least one person; automatically tracking beat of the played music from the real-time audio signal and accordingly automatically predicting a time of a next beat; automatically recognising from the real-time audio signal at least one of chords; notes; and drum sounds and accordingly automatically detecting repetitions in the played music; automatically predicting a next development in the played music, based on the detected repetitions, comprising at least one of chords; notes; and drum sounds that will be played next, and respective timing based on the predicted time of the next beat; and automatically producing a real-time output based on the predicted next development in the played music; wherein the predicting of the at least one of the chords; notes; and drum sounds is performed by detecting self-similarity in the played music; the method further comprising: computing a self-similarity matrix; updating the matrix by comparing the frame against all the previously buffered frames when a new frame is formed from the real-time audio signal. ### Claim 5 A method comprising: automatically receiving a real-time audio signal of played music that is played by at least one person; automatically tracking beat of the played music from the real-time audio signal and accordingly automatically predicting a time of a next beat; automatically recognising from the real-time audio signal at least one of chords; notes; and drum sounds and accordingly automatically detecting repetitions in the played music; automatically predicting a next development in the played music, based on the detected repetitions, comprising at least one of chords; notes; and drum sounds that will be played next, and respective timing based on the predicted time of the next beat; and automatically producing a real-time output based on the predicted next development in the played music; wherein: the predicting of the at least one of chords; notes; and drum sounds in performed by detecting self-similarity in the played music; wherein hashing is used to enable using longer periods of the received audio signal; and locality sensitive hashing (LSH) is used to detect a sequence of past frames of the received audio signal that matches the latest sequence of frames. ### Claim 6 (depends on 5) The method of claim 5, wherein the tracking of the beat of the played music from the real-time audio signal adapts to fluctuation of the tempo of the played music. ### Claim 7 (depends on 5) The method of claim 5, wherein: the self-similarity is calculated using analysing of the received real-time audio signal so as to extract an internal representation for the played music; and the internal representation comprises: a sequence of feature vectors that represent the musical contents of a short segments of received audio signal; or a sequence of high-level descriptors of the received audio signal, wherein the high-level descriptors comprise any one or more of chords; notes; and drum sound notes. ### Claim 8 (depends on 5) The method of claim 5, wherein the producing of the real-time output comprises displaying any one or more of: musical notation; chords; drum notes; given fret indication; an instrument key indication; and a drum specific indication. ### Claim 9 (depends on 5) The method of claim 5, wherein the producing of the real-time output comprises displaying a timeline with indication of events placed on the timeline such that the timeline comprises several rows on the screen. ### Claim 10 (depends on 5) The method of claim 5, wherein the producing of the real-time output comprises visualising repeating sequences. ### Claim 11 (depends on 5) The method of claim 5, wherein the user is allowed to select one or more recorded songs and the recorded songs are processed as if previously received in the real-time audio signal. ### Claim 12 (depends on 5) The method of claim 5, wherein the producing of the real-time output comprises performing one or more instruments along with the played music. ### Claim 13 (depends on 1) A non-transitory memory medium comprising computer executable program code which when executed by at least one processor causes an apparatus at least to perform the method of claim 1. ### Claim 14 A method comprising: automatically receiving a real-time audio signal of played music that is played by at least one person; automatically tracking beat of the played music from the real-time audio signal and accordingly automatically predicting a time of a next beat; automatically recognising from the real-time audio signal at least one of chords; notes; and drum sounds and accordingly automatically detecting repetitions in the played music; automatically predicting a next development in the played music, based on the detected repetitions, comprising at least one of chords; notes; and drum sounds that will be played next, and respective timing based on the predicted time of the next beat; and automatically producing a real-time output based on the predicted next development in the played music wherein: the predicting of the at least one of chords; notes; and drum sounds is performed by detecting self-similarity in the played music; and a musical key of the played music is shown to the user. ### Claim 15 (depends on 14) A non-transitory memory medium comprising computer executable program code which when executed by at least one processor causes an apparatus at least to perform the method of claim 14. ### Claim 16 An apparatus comprising a processor and computer program code configured to cause the apparatus to automatically perform, on executing by the processor of the computer program code: receiving a real-time audio signal of played music that is played by at least one person; tracking beat of the played music from the real-time audio signal and accordingly predicting a time of next beat; recognising from the real-time audio signal at least one of chords; notes; and drum sounds and accordingly detecting repetitions in the played music; predicting a next development in the played music, based on the detected repetitions, comprising at least one of chords; notes; and drum sounds that will be played next, and respective timing based on the predicted time of the next beat; and producing a real-time output based on the predicted next development in the played music; wherein the processor and computer program code are configured to cause the apparatus to perform, on executing by the processor; the predicting of the at least one of: chords; notes; and drum sounds by detecting self-similarity in the played music; and allowing the user to select a desired musical style and performing accordingly the predicting of the next development. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the processor and computer program code are configured to cause the apparatus to perform, on executing by the processor, the tracking of the beat of the played music from the real-time audio signal adapting to fluctuation of the tempo of the played music. ### Claim 18 An apparatus comprising a processor and computer program code configured to cause the apparatus to automatically perform, on executing by the processor of the computer program code: receiving a real-time audio signal of played music that is played by at least one person; tracking beat of the played music from the real-time audio signal and accordingly predicting a time of next beat; recognising from the real-time audio signal at least one of chords; notes; and drum sounds and accordingly detecting repetitions in the played music; predicting a next development in the played music, based on the detected repetitions, comprising at least one of chords; notes; and drum sounds that will be played next, and respective timing based on the predicted time of the next beat; and producing a real-time output based on the predicted next development in the played music; wherein the processor and computer program code are further configured to cause the apparatus to: perform the predicting of the at least one of chords; notes; and drum sounds by detecting self-similarity in the played music; and show a musical key of the played music to the user.