# US 8,823,642 B2 · Methods and systems for controlling devices using gestures and related 3D sensor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,823,642 B2 - Document ID: US8823642B2 - Status: Granted - Kind: B2 - Application number: 13/478,378 - Filed: 2012-05-23 - Published: 2013-01-10 - Granted: 2014-09-02 - Earliest claimed: 2011-07-04 - Assignees: 3Divi - Applicants: Andrey Valik; Pavel Zaitsev; Dmitry Morozov - Inventors: Andrey Valik; Pavel Zaitsev; Dmitry Morozov - Web page: https://patentgrep.com/patent/US8823642B2 ## CPC - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Provided are computer-implemented methods and systems for controlling devices using gestures and a 3D sensor that enables implementing the above. In one embodiment, the method proposed herein may be based on defining at least one sensor area within the space surrounding a user of a controlled device; associating this sensor area with at least one user gesture; associating the combination of the user gesture and sensor area with an actionable command; identifying the direction of the line of sight of the user and a focal point of the line of sight of the user; and, if the line of sight of the user is directed a sensor area, issuing an actionable command corresponding to the combination of the sensor area and the gesture that the user makes while looking at this sensor area. ## Claims ### Claim 1 A method for controlling a device using gestures, the method comprising: capturing a three-dimensional (3D) image associated with a user of the device; identifying whether a line of site of the user is directed towards a predetermined area in a space surrounding the user, the predetermined area being a sensor area, the line of sight being an imaginary straight line between a position of eyes of the user and an object that the user views, wherein the line of sight is a beam crossing a point associated with a hand of the user; identifying a gesture made by the user substantially contemporaneously with the current line of sight; comparing the gesture to one or more predetermined gestures associated with the sensor area; and based on the comparison, selectively providing to the device a command corresponding to the combination of the sensor area and the gesture, wherein the sensor area is substantially flat and is disposed within a same plane as a display communicatively coupled to the device, wherein the sensor area includes a sensor area located outside the display and a sensor area adjacent with a boundary of the display. ### Claim 2 (depends on 1) The method of claim 1, wherein the direction of the line of sight is determined based on one or more of the following: position of the eyes of the user, an orientation of a head of the user, a position of a hand of a user, an angle of inclination of the head, and a rotation of the head. ### Claim 3 (depends on 1) The method of claim 1, wherein the direction of the line of sight is calculated based on a position of the pupils and/or a contour of the irises of the eyes. ### Claim 4 (depends on 1) The method of claim 1, wherein determination of the line of sight is based on a predetermined manual or automatic calibration of a 3D sensor being used to capture the 3D image. ### Claim 5 (depends on 1) The method of claim 1, wherein a center of one of the eyes of the user is taken as the position of eyes. ### Claim 6 (depends on 1) The method of claim 1, wherein a point on an interval connecting centers of eyes of the user is taken as the position of eyes. ### Claim 7 (depends on 1) The method of claim 1, wherein the sensor area further includes a sensor area located within the display. ### Claim 8 (depends on 7) The method of claim 7, wherein a pointer is displayed on the display and moved on the display in accordance with motions of a hand or a pointing device. ### Claim 9 (depends on 7) The method of claim 7, wherein a pointer is displayed on the display and moved in accordance with motions of a focal point of the vision line. ### Claim 10 (depends on 1) The method of claim 1, further comprising determining a focal point of the vision line and tracking the focal point. ### Claim 11 (depends on 10) The method of claim 10, wherein a movement of the focal point from outside to inside the display, a graphic object is displayed on the display. ### Claim 12 (depends on 11) The method of claim 11, wherein a horizontal movement of a hand directed to the display displaying a number of graphical objects is associated with a scrolling command. ### Claim 13 (depends on 1) The method of claim 1, wherein the gesture is recognized based on an analysis of fingers of the user. ### Claim 14 (depends on 13) The method of claim 13, wherein bending of fingers previously unbent and directed toward the sensor area connected to an object on a display is associated with a command to select an object. ### Claim 15 (depends on 13) The method of claim 13, wherein opening a palm, previously clenched in a fist or a pinch, towards an object on a display, is associated with a command to maximize the object. ### Claim 16 (depends on 1) The method of claim 1, wherein a position and a movement of a pointing device held by the user is analyzed with any elongated object being used as the pointing device. ### Claim 17 (depends on 16) The method of claim 16, wherein single or multiple touches by a finger of the pointing device held by the rest of the fingers of the same hand are recognized. ### Claim 18 (depends on 1) The method of claim 1, wherein a face is recognized and the user is identified based on the 3D image. ### Claim 19 A system for controlling a device using gestures, the system comprising: an infrared (IR) projector to generate modulated light; an IR camera to capture a three-dimensional (3D) image associated with a user of the device; and a computing unit communicatively coupled to the IR projector and the IR camera, the computing unit being configured to analyze the 3D image and, based on the analysis, to: identify whether a line of site of the user is directed towards a predetermined area in a space surrounding the user, the predetermined area being a sensor area, the line of sight being an imaginary straight line between a position of eyes of the user and an object that the user views, wherein the line of sight is a beam crossing a point associated with a hand of the user, identify a gesture made by the user substantially contemporaneously with the current line of sight, compare the gesture to one or more predetermined gestures associated with the sensor area; and based on the comparison, selectively provide to the device a command corresponding to the combination of the sensor area and the gesture, wherein the sensor area is substantially flat and is disposed within a same plane as a display communicatively coupled to the device, wherein the sensor area includes a sensor area located outside the display and a sensor area adjacent with a boundary of the display. ### Claim 20 (depends on 19) The system of claim 19, wherein in order to analyze the 3D image, the computing unit is configured to: filter the 3D image; receive a map of folded phases; transform the map of folded phases into a map unfolded phases; calculate a map of depths; and calculate 3D coordinates of parts of a body of the user. ### Claim 21 (depends on 19) The system of claim 19, further comprising a color video camera. ### Claim 22 (depends on 21) The system of claim 21, wherein the IR projector, color video camera, and IR camera are installed in a common housing. ### Claim 23 (depends on 21) The system of claim 21, wherein the color video camera and/or the IR camera are equipped with liquid lenses. ### Claim 24 (depends on 19) The system of claim 19, wherein the computing unit is equipped with an interface to transmit a video signal to a display.