# US 8,772,621 B2 · System and method for capture and rendering of performance on synthetic string instrument

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,772,621 B2
- Document ID: US8772621B2
- Status: Granted
- Kind: B2
- Application number: 13/292,773
- Filed: 2011-11-09
- Published: 2012-07-12
- Granted: 2014-07-08
- Earliest claimed: 2010-11-09
- Assignees: Smule, Inc.
- Applicants: Ge Wang; Jeannie Yang; Jieun Oh; Tom Lieber
- Inventors: Ge Wang; Jeannie Yang; Jieun Oh; Tom Lieber
- Web page: https://patentgrep.com/patent/US8772621B2

## CPC

- G10H 1/0016: {Means for indicating which keys, frets or strings are to be actuated, e.g. using lights or leds}
- G06F 3/041: Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G10H 1/0008: {Associated control or indicating means}
- G10H 1/0033: {Recording/reproducing or transmission of music for electrophonic musical instruments}
- G10H 2220/096: using a touch screen
- G10H 2220/106: using icons, e.g. selecting, moving or linking icons, on-screen symbols, screen regions or segments representing musical elements or parameters
- G10H 2230/015: PDA [personal digital assistant] or palmtop computing devices used for musical purposes, e.g. portable music players, tablet computers, e-readers or smart phones in which mobile telephony functions need not be used
- G10H 2230/075: Spint stringed, i.e. mimicking stringed instrument features, electrophonic aspects of acoustic stringed musical instruments without keyboard; MIDI-like control therefor
- G10H 2240/305: Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Synthetic multi-string musical instruments have been developed for capturing and rendering musical performances on handheld or other portable devices in which a multi-touch sensitive display provides one of the input vectors for an expressive performance by a user or musician. Visual cues may be provided on the multi-touch sensitive display to guide the user in a performance based on a musical score. Alternatively, or in addition, uncued freestyle modes of operation may be provided. In either case, it is not the musical score that drives digital synthesis and audible rendering of the synthetic multi-string musical instrument. Rather, it is the stream of user gestures captured at least in part using the multi-touch sensitive display that drives the digital synthesis and audible rendering.

## Claims

### Claim 1

A method comprising:
using a first portable computing device as a synthetic string instrument;
presenting on a multi-touch sensitive display of the portable computing device, and in correspondence with a musical score, temporally synchronized visual cues relative to respective strings of the synthetic string instrument;
capturing user gestures indicative of length of respective strings of the synthetic string instrument from data sampled in correspondence with respective finger contacts with the multi-touch sensitive display along visual depictions of the respective strings;
capturing user gestures indicative of excitation of at least one of the strings;
encoding a gesture stream for a performance of the user by parameterizing at least a subset of the string length and string excitation indicative user gestures; and
audibly rendering the performance on the portable computing device using the encoded gesture stream as an input to a digital synthesis of the synthetic string instrument executing on the first portable computing device, wherein the captured gesture stream, and not the musical score itself, drives the digital synthesis.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the string excitation indicative gestures are distinct from the string length indicative gestures.

### Claim 3 (depends on 1)

The method of claim 1,
wherein the digital synthesis includes a model of acoustic response for one of a violin, a viola, a cello and a double bass; and
wherein the captured string excitation indicative gestures include a bow traversal gesture indicated by contact of the user's thumb or finger with a rotating visual on the multi-touch sensitive display.

### Claim 4 (depends on 3)

The method of claim 3,
wherein radial excursion of the user's thumb or finger contact with the rotating visual is indicative of bow speed.

### Claim 5 (depends on 3)

The method of claim 3,
wherein the captured string excitation indicative gestures further include pluck- or strum-type gestures indicated by contact of the user's thumb or finger with respective visual indications of the strings.

### Claim 6 (depends on 1)

The method of claim 1,
wherein the digital synthesis is of one of a guitar, banjo, ukulele, lute, mandolin or sitar; and
wherein the captured string excitation indicative gestures include pluck- or strum-type gestures indicated by contact of the user's thumb or finger with respective visual indications of the strings.

### Claim 7 (depends on 6)

The method of claim 6,
wherein the digital synthesis includes wavetable or frequency modulation synthesis in correspondence with the parameterization of at least the string length indicative user gestures.

### Claim 8 (depends on 6)

The method of claim 6,
wherein the digital synthesis includes physical model of acoustic response for the guitar, banjo, ukulele, lute, mandolin or sitar.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
capturing a damping or muting indicative gesture expressed by the user; and
in the digital synthesis, attenuating in correspondence with a parameterization of the damping or muting indicative gesture.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
responsive to a user selection on the multi-touch sensitive display, retrieving an encoding of the musical score via the communications interface.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
grading the user's performance based at least in part on correspondence of captured finger contact gestures with notes or positions and timings encoded in the musical score.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
grading the user's performance based at least in part on correspondence of captured finger contact gestures with vibrato, trilling or other spatio-temporal effects encoded in the musical score.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
presenting on the multi-touch sensitive display a lesson plan of exercises, wherein the captured gesture stream corresponds to performance by the user of a particular one of the exercises; and
advancing the user to a next exercise of the lesson plan based on a grading of the user's performance of the particular exercise.

### Claim 14 (depends on 1)

The method of claim 1, wherein presenting on a multi-touch sensitive display of the portable computing device, and in correspondence with a musical score, temporally synchronized visual cues relative to respective strings of the synthetic string instrument further includes temporally synchronized haptic feedback cues that include the indication to contact at least one of the respective strings at a particular length and indication to excite the at least one of the respective strings.

### Claim 15

A method comprising:
using a first portable computing device as a synthetic string instrument;
capturing user gestures relative to respective strings of the synthetic string instrument from data sampled in correspondence with respective finger contacts with a multi-touch sensitive display of the portable computing device;
capturing user gestures indicative of bow traversal of at least one of the strings;
encoding a gesture stream for a performance of the user by parameterizing at least a subset of events captured from the finger contacts and bow traversal;
audibly rendering the performance on the portable computing device using the encoded gesture stream as an input to a digital synthesis of the synthetic string instrument executing on the first portable computing device; and
presenting on the multi-touch sensitive display, and in correspondence with a musical score, temporally synchronized visual cues to guide the user's gestures relative to the respective strings of the synthetic string instrument,
wherein the gesture stream captured from the user's performance, and not the musical score itself, drives the digital synthesis.

### Claim 16 (depends on 15)

The method of claim 15,
wherein the user gestures indicative of bow traversal are captured in correspondence with thumb or finger contact with the multi-touch sensitive display.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
dynamically correcting captured finger contact gestures in accord with notes or positions encoded in the musical score.

### Claim 18 (depends on 15)

The method of claim 15, further comprising:
grading the user's performance based at least in part on correspondence of captured finger contact gestures with notes or positions and timings encoded in the musical score.

### Claim 19 (depends on 15)

The method of claim 15, further comprising:
grading the user's performance based at least in part on correspondence of captured finger contact gestures with vibrato, trilling or other spatio-temporal effects encoded in the musical score.

### Claim 20 (depends on 15)

The method of claim 15, further comprising:
manipulating a second portable device relative to the first, wherein the user gestures indicative of bow traversal include lateral movement and inclination of the second portable device captured in correspondence with the manipulation.

### Claim 21 (depends on 20)

The method of claim 20,
wherein the second portable computing device includes either or both of multi-axis accelerometer and a gyroscopic sensor for capture of orientation and motion dynamics of the second portable computing device, and
wherein the second portable computing device is configured to compute and wirelessly communicate to the first portable computing device the bow traversal gestures based on the captured orientation and motion dynamics.

### Claim 22 (depends on 20)

The method of claim 20, further comprising:
capturing at a multi-touch sensitive display of the second portable computing device further user gestures indicative of one or more of bow pressure and bow-on-string.

### Claim 23 (depends on 20)

The method of claim 20, further comprising:
grading the user's performance based at least in part on correspondence of one or more parameterizations of the captured bow traversal gestures with bowing techniques encoded in the musical score,
wherein the one or more parameterizations are selected from the set of speed of travel, position of a sounding point, and method of attack.

### Claim 24 (depends on 15)

The method of claim 15, further comprising
responsive to a user selection on the multi-touch sensitive display, retrieving an encoding of the musical score via the communications interface.

### Claim 25 (depends on 15)

The method of claim 15, further comprising:
presenting on the multi-touch sensitive display a lesson plan of exercises, wherein the captured gesture stream corresponds to performance by the user of a particular one of the exercises; and
advancing the user to a next exercise of the lesson plan based on a grading of the user's performance of the particular exercise.

### Claim 26 (depends on 15)

The method of claim 15,
wherein the first portable computing device includes a communications interface,
the method further comprising, transmitting the encoded gesture stream via the communications interface for rendering of the performance on a remote device.

### Claim 27 (depends on 26)

The method of claim 26,
wherein the remote device includes a server from which the rendered performance is subsequently supplied as one or more audio encodings thereof.

### Claim 28 (depends on 26)

The method of claim 26, further comprising:
audibly rendering a second performance on the portable computing device using a second gesture stream encoding received via the communications interface directly or indirectly from a second remote device, the second performance rendering using the received second gesture stream encoding as an input to the digital synthesis of the musical instrument.

### Claim 29 (depends on 26)

The method of claim 26, further comprising:
geocoding the transmitted gesture stream; and
displaying a geographic origin for, and in correspondence with audible rendering of, a third performance encoded as a third gesture stream received via the communications interface directly or indirectly from a third remote device.

### Claim 30 (depends on 15)

The method of claim 15, wherein the digital synthesis includes a model of acoustic response for one of a violin, a viola, a cello and a double bass.

### Claim 31 (depends on 20)

The method of claim 20, wherein the first and second portable computing devices are each selected from the group of:
a compute pad, a laptop computer, notebook computer or netbook;
a personal digital assistant or book reader; and
a mobile phone or media player.

### Claim 32 (depends on 15)

A computer program product encoded in one or more media, the computer program product including instructions executable on a processor of the portable computing device to cause the portable computing device to perform the method of claim 15.

### Claim 33 (depends on 32)

The computer program product of claim 32, wherein the one or more media are readable by the portable computing device or readable incident to a computer program product conveying transmission to the portable computing device.

### Claim 34

An apparatus comprising:
a portable computing device having a multi-touch display interface; and
machine readable code executable on the portable computing device to implement the synthetic musical instrument, the machine readable code including instructions executable to capture both: (i) user gestures relative to respective strings of the synthetic string instrument from data sampled in correspondence with respective finger contacts with a multi-touch sensitive display of the portable computing device and (ii) user gestures indicative of bow traversal of at least one of the strings and to encoded a gesture stream for a performance of the user by parameterizing at least a subset of events captured from the finger contacts and bow traversal; and
the machine readable code further executable to audibly render the performance on the portable computing device using the encoded gesture stream as an input to a digital synthesis of the synthetic string instrument executing on the portable computing device;
wherein the apparatus is configured to wirelessly communicate with a second portable computing device proximate thereto, the second portable computing device including either or both of multi-axis accelerometer and a gyroscopic sensor for capture of orientation and motion dynamics of the second portable computing device, and
machine readable code executable on the second portable computing device to compute and wirelessly communicate to the first portable computing device the bow traversal gestures based on the captured orientation and motion dynamics.

### Claim 35 (depends on 34)

The apparatus of claim 34,
embodied as one or more of a compute pad, a handheld mobile device, a mobile phone, a laptop or notebook computer, a personal digital assistant, a smart phone, a media player, a netbook, and a book reader.

### Claim 36

A computer program product encoded in media and including instructions executable to implement a synthetic musical instrument on a portable computing device having a multi-touch display interface, the computer program product encoding and comprising:
instructions executable to capture both: (i) user gestures relative to respective strings of the synthetic string instrument from data sampled in correspondence with respective finger contacts with a multi-touch sensitive display of the portable computing device and (ii) user gestures indicative of bow traversal of at least one of the strings and to encode a gesture stream for a performance of the user by parameterizing at least a subset of events captured from the finger contacts and bow traversal,
further instructions executable to audibly render the performance on the portable computing device using the encoded gesture stream as an input to a digital synthesis of the synthetic string instrument executing on the portable computing device; and
further instructions executable to present on the multi-touch sensitive display, and in correspondence with a musical score, temporally synchronized visual cues to guide the user's gestures relative to the respective strings of the synthetic string instrument,
wherein the gesture stream captured from the user's performance, and not the musical score itself, drives the digital synthesis.