Virtual tutorials for musical instruments with finger tracking in augmented reality

A system for presenting a tutorial using an eyewear device, the eyewear device comprising a processor, a memory, a camera, and a display, the system comprising:

• programming in the memory, wherein execution of the programming by the processor configures the eyewear device to perform functions, including functions to:
• capture frames of video data with the camera;
• register a marker location associated with a musical instrument in a physical environment, wherein the musical instrument comprises a plurality of physical actuator locations;
• estimate a local position of the eyewear device relative to the marker location based on the frames of video data;
• retrieve from the memory a song file associated with the musical instrument, wherein the song file comprises correct fingertip positions, a tempo, a sequence of notes and note values, and a series of virtual tutorial objects;
• present on the display, based on the local position, the series of virtual tutorial objects relative to the marker location in accordance with the song file;
• present on the display a pictogram resembling the musical instrument, wherein the pictogram comprises a plurality of simulated actuator locations, each correlated with a respective one of the plurality of physical actuator locations;
• detect a hand shape in a first frame of the frames of video data, wherein the first frame includes first depth information for a first plurality of pixels;
• calculate a set of expected fingertip coordinates based on the hand shape;
• for each note in the sequence of notes in the song file, calculate a sum of the geodesic distances between the set of expected fingertip coordinates and the correct fingertip positions;
• and
• in response to determining that the sum is greater than a threshold accuracy value present a failure indicator on the display near the plurality of simulated actuator locations in accordance with the song file.

A non-transitory computer-readable medium storing program code that, when executed, is operative to cause a processor coupled to an eyewear device to perform the steps of:

• capturing frames of video data with a camera coupled to the eyewear device, the eyewear device further comprising a memory and a display;
• registering a marker location associated with a musical instrument in a physical environment, wherein the musical instrument comprises a plurality of physical actuator locations;
• estimating a local position of the eyewear device relative to the marker location based on the frames of video data;
• retrieving from the memory a song file associated with the musical instrument, wherein the song file comprises correct fingertip positions, a tempo, a sequence of notes and note values, and a series of virtual tutorial objects;
• presenting on the display, based on the local position, the series of virtual tutorial objects relative to the marker location in accordance with the song file;
• presenting on the display a pictogram resembling the musical instrument, wherein the pictogram comprises a plurality of simulated actuator locations, each correlated with a respective one of the plurality of physical actuator locations;
• detecting a hand shape in a first frame of the frames of video data, wherein the first frame includes first depth information for a first plurality of pixels;
• calculating a set of expected fingertip coordinates based on the hand shape;
• for each note in the sequence of notes in the song file, calculating a sum of the geodesic distances between the set of expected fingertip coordinates and the correct fingertip positions;
• and
• in response to determining that the sum is greater than a threshold accuracy value presenting a failure indicator on the display near the plurality of simulated actuator locations in accordance with the song file.