Augmented reality experiences with object manipulation

An augmented reality method for use with an eyewear device having a position detection system, a display system, an image capture system, and an inertial measurement unit, the method comprising:

• operating the position detection system to construct a map of a physical environment and determine a location and pose of the eyewear device in the map using images of the physical environment captured using the image capture system and input from the inertial measurement unit in the eyewear device;
• controlling a music player;
• operating a virtual object rendering utility to render, on the display system, a visual graphic representation of at least one of:
• a plurality of control inputs associated with the music player;
• or
• a menu of a plurality of hand gestures and a plurality of control aspects of the music player corresponding to each of the hand gestures;
• detecting at least one of a hand gesture or movement in the physical environment using images captured from the image capture system, wherein the detected at least one hand gesture or movement corresponds to one or more of the plurality of control inputs or the plurality of hand gestures;
• associating the detected at least one hand gesture or movement with a control setting of the music player stored in a memory;
• changing the control setting of the music player responsive to the detected at least one hand gesture or movement;
• providing audio output that reflects the changed control setting of the music player responsive to the detected at least one hand gesture or movement;
• displaying a visual graphic representation of a movable virtual object in the physical environment using the display system of the eyewear device;
• detecting at least one of a second hand gesture or movement in the physical environment using images captured from the image capture system and input from the inertial measurement unit;
• and
• changing at least one of a position or an orientation of the movable virtual object based on the detected at least one second hand gesture or movement.

A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:

• operating a position detection system to construct a map of a physical environment and determine a location and pose of an eyewear device in the map using images of the physical environment captured using an image capture system in the eyewear device and input from an inertial measurement unit in the eyewear device;
• controlling a music player;
• operating a virtual object rendering utility in the eyewear device to render, on a display system of the eyewear device, a visual graphic representation of at least one of:
• a plurality of control inputs associated with the music player;
• or
• a menu of a plurality of hand gestures and a plurality of control aspects of the music player corresponding to each of the hand gestures;
• detecting at least one of a hand gesture or movement in the physical environment using images captured from the image capture system, wherein the detected at least one hand gesture or movement corresponds to one or more of the plurality of control inputs or the plurality of hand gestures;
• associating the detected at least one hand gesture or movement with a control setting of the music player stored in a memory;
• changing the control setting of the music player responsive to the detected at least one hand gesture or movement;
• providing audio output that reflects the changed control setting of the music player responsive to the detected at least one hand gesture or movement;
• displaying a visual graphic representation of a movable virtual object in the physical environment using the display system of the eyewear device;
• detecting at least one of a second hand gesture or movement in the physical environment using images captured from the image capture system and input from the inertial measurement unit;
• and
• operating the virtual object rendering utility to change a position or an orientation of the movable virtual object based on the detected at least one second hand gesture or movement.