Apparatus and method for playing musical instrument using augmented reality technique in mobile terminal

An apparatus for identifying and providing a virtual musical instrument for play using an augmented reality technique in a mobile terminal, the apparatus comprising:

• a display unit;
• an image recognizer that recognizes a visual representation of a musical instrument;
• and
• a sound source processor that outputs the recognized visual representation of the musical instrument to the display unit for using the visual representation for virtual play, and provides audible output of notes matching touched positions of the image of the musical instrument output on the display unit, wherein the visual representation recognized by the image recognizer is a hand drawing of an instrument.

An apparatus for identifying and providing a virtual musical instrument for play using an augmented reality technique in a mobile terminal, the apparatus comprising:

• a display unit;
• an image recognizer that recognizes a visual representation of a musical instrument;
• and
• a sound source processor that outputs the recognized visual representation of the musical instrument to the display unit for using the visual representation for virtual play, and provides audible output of notes matching touched positions of the image of the musical instrument output on the display unit, wherein the visual representation is made with a stylus or finger and recognized by the image recognizer.

A method for providing a musical instrument for virtual play using an augmented reality technique, the method comprising:

• capturing, by using a camera, a visual representation of the musical instrument;
• processing the visual representation, by using an image recognition technique, to recognize the musical instrument and detect whether the recognized musical instrument is one of a plurality of virtual musical instruments that the mobile terminal is capable of providing;
• outputting, by a processor, the visual representation of the musical instrument for display on a display unit and for a virtual play of the recognized musical instrument when it is detected that the recognized musical instrument is one of the plurality of virtual musical instruments that the mobile terminal is capable of providing;
• and
• matching user manipulation of the musical instrument while being virtually played to the visual representation of the musical instrument displayed by the display unit.

An apparatus for providing a musical play using an augmented reality technique in a mobile terminal, the apparatus comprising:

• a camera for capturing an image of a musical instrument;
• a controller for:
• receiving the image from the camera,
• processing the image by using an image recognition technique for recognizing the musical instrument,
• detecting whether the recognized musical instrument is one of a plurality of musical instruments that the mobile terminal is capable of providing,
• when the musical instrument is one of the plurality of musical instruments the mobile terminal is capable of providing, generating a sound source based on the received image;
• recognizing through additional images of the musical instrument captured by the camera a position of a hand motion, and
• audibly outputting a sound based on the sound source and the position of the hand motion;
• wherein the sound source includes a mapping of locations in the image received from the camera to different sounds.

An method for providing a musical play comprising:

• capturing, by a camera, an image of a musical instrument;
• processing, by a controller, the image via an image recognition technique to identify the musical instrument;
• detecting whether the identified musical instrument is a musical instruments that the mobile terminal is capable of providing;
• when the identified musical instrument is a musical instruments that the mobile terminal is capable of providing, generating a sound source for the identified musical instrument based on the image;
• capturing additional images of the musical instrument by the camera;
• processing the additional images to identify a position of a hand motion that is recorded in the additional images;
• and
• generating a sound based on the position of the hand motion and the sound source;
• wherein the sound source includes a mapping of locations in the image captured by the camera to different sounds.

A method for identifying and providing a virtual musical instrument for play, the method comprising:

• receiving, by a touchscreen of an electronic device, an input that draws a visual representation of a musical instrument on the touchscreen;
• and
• displaying the visual representation of the musical instrument on the touchscreen for using the visual representation of the musical instrument for virtual play;
• receiving, by the touchscreen, input at a position in the visual representation of the musical instrument;
• and
• audibly outputting a musical note corresponding to the position in the visual representation of the musical instrument where the input is received.