System and method for capture and rendering of performance on synthetic string instrument

A method comprising:

• using a first portable computing device as a synthetic string instrument;
• capturing user gestures relative to respective strings of the synthetic string instrument from data sampled in correspondence with respective finger contacts with a multi-touch sensitive display of the portable computing device;
• capturing user gestures indicative of bow traversal of at least one of the strings;
• encoding a gesture stream for a performance of the user by parameterizing at least a subset of events captured from the finger contacts and bow traversal;
• audibly rendering the performance on the portable computing device using the encoded gesture stream as an input to a digital synthesis of the synthetic string instrument executing on the first portable computing device;
• and
• presenting on the multi-touch sensitive display, and in correspondence with a musical score, temporally synchronized visual cues to guide the user's gestures relative to the respective strings of the synthetic string instrument,
• wherein the gesture stream captured from the user's performance, and not the musical score itself, drives the digital synthesis.

An apparatus comprising:

• a portable computing device having a multi-touch display interface;
• and
• machine readable code executable on the portable computing device to implement the synthetic musical instrument, the machine readable code including instructions executable to capture both:
• (i) user gestures relative to respective strings of the synthetic string instrument from data sampled in correspondence with respective finger contacts with a multi-touch sensitive display of the portable computing device and (ii) user gestures indicative of bow traversal of at least one of the strings and to encoded a gesture stream for a performance of the user by parameterizing at least a subset of events captured from the finger contacts and bow traversal;
• and
• the machine readable code further executable to audibly render the performance on the portable computing device using the encoded gesture stream as an input to a digital synthesis of the synthetic string instrument executing on the portable computing device;
• wherein the apparatus is configured to wirelessly communicate with a second portable computing device proximate thereto, the second portable computing device including either or both of multi-axis accelerometer and a gyroscopic sensor for capture of orientation and motion dynamics of the second portable computing device, and
• machine readable code executable on the second portable computing device to compute and wirelessly communicate to the first portable computing device the bow traversal gestures based on the captured orientation and motion dynamics.

A computer program product encoded in media and including instructions executable to implement a synthetic musical instrument on a portable computing device having a multi-touch display interface, the computer program product encoding and comprising:

• instructions executable to capture both:
• (i) user gestures relative to respective strings of the synthetic string instrument from data sampled in correspondence with respective finger contacts with a multi-touch sensitive display of the portable computing device and (ii) user gestures indicative of bow traversal of at least one of the strings and to encode a gesture stream for a performance of the user by parameterizing at least a subset of events captured from the finger contacts and bow traversal,
• further instructions executable to audibly render the performance on the portable computing device using the encoded gesture stream as an input to a digital synthesis of the synthetic string instrument executing on the portable computing device;
• and
• further instructions executable to present on the multi-touch sensitive display, and in correspondence with a musical score, temporally synchronized visual cues to guide the user's gestures relative to the respective strings of the synthetic string instrument,
• wherein the gesture stream captured from the user's performance, and not the musical score itself, drives the digital synthesis.