Bimanual gestures for controlling virtual and graphical elements

A bimanual control system, comprising:

• an eyewear device comprising a processor, a memory, a camera system, an image processing system, a localization system, and a display;
• programming in the memory, wherein execution of the programming by the processor configures the eyewear device to perform functions, including functions to:
• present on the display one or more graphical elements at a first position relative to a display plane;
• capture frames of video data with the camera system;
• detect a first hand in the captured frames of video data with the image processing system;
• define, with the image processing system, a first input plane associated with a surface of the detected first hand, wherein the first input plane is defined relative to the display plane;
• determine, with the localization system, a current eyewear location relative to the defined first input plane;
• calculate a correlation between the defined first input plane and the display plane, in accordance with the current eyewear location;
• detect, with the image processing system, a series of bimanual hand shapes comprising a series of current fingertip locations associated with a fingertip of a second hand, relative to the defined first input plane;
• present on the display a first movable element at a current first element position, in accordance with both a current fingertip location and the calculated correlation;
• determine, with the image processing system, whether the detected series of bimanual hand shapes matches a predefined hand gesture selected from a plurality of predefined hand gestures;
• and
• execute a selecting action in accordance with both the matching predefined hand gesture and the current first element position.

A non-transitory computer-readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:

• presenting on a display one or more graphical elements at a first position relative to a display plane, wherein the display is coupled to an eyewear device comprising a processor, a memory, a camera system, a localization system, and an image processing system;
• capturing frames of video data with the camera system;
• detecting a first hand in the captured frames of video data with the image processing system;
• defining, with the image processing system, a first input plane associated with a surface of the detected first hand, wherein the first input plane is defined relative to the display plane;
• determining, with the localization system, a current eyewear location relative to the defined first input plane;
• calculating a correlation between the defined first input plane and the display plane, in accordance with the current eyewear location;
• detecting, with the image processing system, a series of bimanual hand shapes comprising a series of current fingertip locations associated with a fingertip of a second hand, relative to the defined first input plane;
• presenting on the display a first movable element at a current first element position, in accordance with both a current fingertip location and the calculated correlation;
• determining, with the image processing system, whether the detected series of bimanual hand shapes matches a predefined hand gesture selected from a plurality of predefined hand gestures;
• and
• executing a selecting action in accordance with both the matching predefined hand gesture and the current first element position.