# US 11,307,666 B2 · Systems and methods of direct pointing detection for interaction with a digital device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,307,666 B2 - Document ID: US11307666B2 - Status: Granted - Kind: B2 - Application number: 16/371,565 - Filed: 2019-04-01 - Published: 2019-10-24 - Granted: 2022-04-19 - Earliest claimed: 2012-03-13 - Assignees: Eyesight Mobile Technologies Ltd. - Applicants: Eyesight Mobile Technologies Ltd. - Inventors: Itay Katz; Amnon Shenfeld - Web page: https://patentgrep.com/patent/US11307666B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system and method for recognizing an aimed point on a plane is provided. Images captured by one or more image sensor are processed for obtaining data indicative of location of at least one pointing element in the viewing space and data indicative of at least one predefined user's body part in the viewing space; using the obtained data, an aimed point on the plane is identified. In case it is determined that a predefined condition is met a predefined command and/or message is executed. ## Claims ### Claim 1 A method of recognizing an aimed point or area in a space, the method comprising: obtaining, using an image sensor, at least one image of the space; processing the at least one image by at least one processor operatively connected to the image sensor and obtaining data indicative of more than one location of at least one pointing element in the space; obtaining data indicative of a location of at least one predefined body part of a user in the space; determining by the at least one processor, a location of the aimed point or area in the space using a combination of: the data indicative of the more than one location of the at least one pointing element, the data indicative of the location of the at least one predefined body part, and a plurality of selected position data components associated with a motion path of the at least one pointing element during a pointing gesture, wherein the at least one processor determines the location of the aimed point or area by: determining a respective viewing ray for each of the plurality of selected position data components, the respective viewing ray extending from the location of the at least one predefined body part through the location of the at least one pointing element, and intersecting a plane on which at least one graphical element is displayed, thus yielding a respective candidate plane, wherein the location of the at least one pointing element corresponds to the given position data component; determining an overlapping area between the respective candidate planes, and determining the aimed point or area using data indicative of the overlapping area; determining a change in a motion vector of the at least one pointing element during the pointing gesture, wherein the change in the motion vector relates to a change in a direction of the motion vector in at least one of a vertical axis or a horizontal axis; determining a user selection of an icon or a graphical element, based on the change in the motion vector and the determined location of the aimed point or area; and executing a predefined command or message associated with the selected icon or the graphical element. ### Claim 2 (depends on 1) The method of claim 1, wherein the space comprises a three-dimensional viewing space, and the data indicative of the location of the at least one pointing element includes x, y, and z coordinates. ### Claim 3 (depends on 1) The method of claim 1, further comprising detecting a change in a speed of the at least one pointing element along a motion path of the at least one pointing element during a pointing gesture. ### Claim 4 (depends on 1) The method of claim 1, further comprising detecting a deceleration of the at least one pointing element along a motion path of the at least one pointing element during a pointing gesture. ### Claim 5 (depends on 1) The method of claim 1, further comprising: displaying at least one graphical element in the space; and identifying, by the at least one processor, a given graphical element from the at least one displayed graphical element, using data indicative of the location of the aimed point or area in the space. ### Claim 6 (depends on 1) The method of claim 1, further comprising: determining the location of the at least one pointing element using location features extracted from a motion path of the at least one pointing element during the pointing gesture. ### Claim 7 (depends on 1) The method of claim 1, wherein: the of selected position data components comply with a predefined criterion. ### Claim 8 (depends on 1) The method of claim 1, wherein the image sensor is a proximity sensor. ### Claim 9 (depends on 1) The method of claim 1, further comprising determining a clicking gesture. ### Claim 10 A non-transitory computer readable medium storing instructions which, when executed, cause at least one processor to perform operations for recognizing an aimed point or area in a space, the operations comprising: obtaining, using an image sensor, at least one image of the space; processing the at least one image by at least one processor operatively connected to the image sensor and obtaining data indicative of more than one location of at least one pointing element in the space; obtaining data indicative of a location of at least one predefined body part of a user in the space; determining by the at least one processor, a location of the aimed point or area in the space using a combination of: the data indicative of the more than one location of the at least one pointing element, the data indicative of the location of the at least one predefined body part, and a plurality of selected position data components associated with a motion path of the at least one pointing element during a pointing gesture, wherein the at least one processor determines the location of the aimed point or area by: determining a respective viewing ray for each of the plurality of selected position data components, the respective viewing ray extending from the location of the at least one predefined body part through the location of the at least one pointing element, and intersecting a plane on which at least one graphical element is displayed, thus yielding a respective candidate plane, wherein the location of the at least one pointing element corresponds to the given position data component; determining an overlapping area between the respective candidate planes, and determining the aimed point or area using data indicative of the overlapping area; determining a change in a motion vector of the at least one pointing element during the pointing gesture, wherein the change in the motion vector relates to a change in a direction of the motion vector in at least one of a vertical axis or a horizontal axis; determining a user selection of an icon or a graphical element, based on the change in the motion vector and the determined location of the aimed point or area; and executing a predefined command or message associated with the selected icon or the graphical element. ### Claim 11 (depends on 10) The computer readable medium of claim 10, wherein the space comprises a three-dimensional viewing space, and the data indicative of the location of the at least one pointing element includes x, y, and z coordinates. ### Claim 12 (depends on 10) The computer readable medium of claim 10, the operations further comprising: displaying at least one graphical element in the space; and identifying, by the at least one processor, a given graphical element from the at least one displayed graphical element, using data indicative of the location of the aimed point or area in the space. ### Claim 13 (depends on 10) The computer readable medium of claim 10, the operations further comprising: determining the location of the at least one pointing element using location features extracted from a motion path of the at least one pointing element during the pointing gesture. ### Claim 14 (depends on 10) The computer readable medium of claim 10, wherein: the selected position data components. ### Claim 15 (depends on 10) The computer readable medium of claim 10, wherein the image sensor is a proximity sensor. ### Claim 16 (depends on 10) The computer readable medium of claim 10, the operations further comprising determining a clicking gesture. ### Claim 17 A system for recognizing an aimed point or area in a space, the system comprising at least one processor configured to: obtain, using an image sensor, at least one image of the space; process the at least one image by at least one processor operatively connected to the image sensor and obtaining data indicative of more than one location of at least one pointing element in the space; obtain data indicative of a location of at least one predefined body part of a user in the space; determine by the at least one processor, a location of the aimed point or area in the space using a combination of: the data indicative of the more than one location of the at least one pointing element, the data indicative of the location of the at least one predefined body part, and a plurality of selected position data components associated with a motion path of the at least one pointing element during a pointing gesture, wherein the at least one processor determines the location of the aimed point or area by: determining a respective viewing ray for each of the plurality of selected position data components, the respective viewing ray extending from the location of the at least one predefined body part through the location of the at least one pointing element, and intersecting a plane on which at least one graphical element is displayed, thus yielding a respective candidate plane, wherein the location of the at least one pointing element corresponds to the given position data component; determining an overlapping area between the respective candidate planes, and determining the aimed point or area using data indicative of the overlapping area; determine a change in a motion vector of the at least one pointing element during the pointing gesture, wherein the change in the motion vector relates to a change in a direction of the motion vector in at least one of a vertical axis or a horizontal axis; determining a user selection of an icon or a graphical element, based on the change in the motion vector and the determined location of the aimed point or area; and execute a predefined command or message associated with the selected icon or the graphical element. ### Claim 18 (depends on 17) The system of claim 17, wherein the space comprises a three-dimensional viewing space, and the data indicative of the location of the at least one pointing element includes x, y, and z coordinates. ### Claim 19 (depends on 17) The system of claim 17, the at least one processor being further configured to: determine the location of the at least one pointing element using location features extracted from the motion path of the at least one pointing element during the pointing gesture, wherein the features with respect to the motion path of the at least one pointing element further include a plurality of selected position data components of the at least one pointing element during the pointing gesture.