# US 11,029,761 B2 · Context based gesture control > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,029,761 B2 - Document ID: US11029761B2 - Status: Granted - Kind: B2 - Application number: 16/052,667 - Filed: 2018-08-02 - Published: 2020-02-06 - Granted: 2021-06-08 - Earliest claimed: 2018-08-02 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Eric V. Kline; Sarbajit K. Rakshit - Web page: https://patentgrep.com/patent/US11029761B2 ## CPC - G06F 16/784: {the detected or recognised objects being people} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06V 40/113: {Recognition of static hand signs} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06F 16/22: Indexing; Data structures therefor; Storage structures - G06V 40/172: {Classification, e.g. identification} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods, systems and computer program products for operating smart devices using context based gesture control are provided. Aspects include observing, via a first device, a first user making a first gesture and a second gesture and identifying, by a processor, an intended device based at least in part on the first gesture. Aspects also include identifying, by the processor, an intended action for the intended device based at least in part on the second gesture. Aspects further include instructing, by the processor, the intended device to perform the intended action. ## Claims ### Claim 1 A method for operating smart devices using context based gesture control, comprising: observing, via a first device, a first gesture and a second gesture performed by a first user, wherein the first gesture is made with a first hand of the first user and the second gesture is made with a second hand of the first user; identifying, by a processor, a first intended device based at least in part on the first gesture, wherein the first gesture includes the first user pointing at the first intended device; identifying, by the processor, a first intended action to be performed by the first intended device based at least in part on the second gesture; observing, via the first device, a third gesture and a fourth gesture performed by a second user, wherein the third gesture is made with a first hand of the second user and the third gesture is made with a second hand of the second user; identifying, by the processor, a second intended device based at least in part on the third gesture, wherein the third gesture includes the second user pointing at the second intended device; identifying, by the processor, a second intended action to be performed by the second intended device based at least in part on the fourth gesture; obtaining, by the processor, a hierarchy of users; instructing, by the processor, the first intended device to perform the first intended action based on a determination that the first user has a higher priority than the second user in the hierarchy of users; and instructing, by the processor, the second intended device to perform the second intended action based on a determination that the second user has a higher priority than the first user in the hierarchy of users. ### Claim 2 (depends on 1) The method of claim 1, wherein identifying the first intended device and the first intended action is performed by obtaining a user action database that stores relationships between gestures and first intended devices and actions for a plurality of users. ### Claim 3 (depends on 1) The method of claim 1, further comprising identifying the first user by the processor and wherein the identification of the first intended device and the first intended action are further based at least in part on the identity of the first user. ### Claim 4 (depends on 1) The method of claim 1, further composing obtaining, via the first device, one or more environmental conditions and wherein the identification of the first intended device and the first intended action are further based at least in part on one or more environmental conditions. ### Claim 5 (depends on 1) The method of claim 1, wherein the identification of the first intended device and the first intended action are further based at least in part on an operation state of one or more smart devices in communication with the first device. ### Claim 6 (depends on 1) The method of claim 1, wherein the first device includes a camera, a transceiver, and the processor and wherein the first intended device is in communication with the first device and includes a controller configured to perform the first intended action. ### Claim 7 A system for operating smart devices using context based gesture control, comprising: a memory having computer readable instructions; and a processor for executing the computer readable instructions, the computer readable instructions including instructions for: observing, via a first device, a first user making a first gesture and a second gesture; identifying, by a processor, a first intended device based at least in part on the first gesture, wherein the first gesture includes the first user pointing at the first intended device and wherein the first gesture is made with a first hand of the first user and the second gesture is made with a second hand of the first user; identifying, by the processor, a first intended action for the first intended device based at least in part on the second gesture; observing, via the first device, a third gesture and a fourth gesture performed by a second user, wherein the third gesture is made with a first hand of the second user and the third gesture is made with a second hand of the second user; identifying, by the processor, a second intended device based at least in part on the third gesture, wherein the third gesture includes the second user pointing at the second intended device; identifying, by the processor, a second intended action to be performed by the second intended device based at least in part on the fourth gesture; obtaining, by the processor, a hierarchy of users; instructing, by the processor, the first intended device to perform the first intended action based on a determination that the first user has a higher priority than the second user in the hierarchy of users; and instructing, by the processor, the second intended device to perform the second intended action based on a determination that the second user has a higher priority than the first user in the hierarchy of users. ### Claim 8 (depends on 7) The system of claim 7, wherein identifying the first intended device and the first intended action is performed by obtaining a user action database that stores relationships between gestures and first intended devices and actions for a plurality of users. ### Claim 9 (depends on 7) The system of claim 7, wherein the computer readable instructions further include instructions for: identifying the first user by the processor and wherein the identification of the first intended device and the first intended action are further based at least in part on the identity of the first user. ### Claim 10 (depends on 7) The system of claim 7, wherein the computer readable instructions further include instructions for: obtaining, via the first device, one or more environmental conditions and wherein the identification of the first intended device and the first intended action are further based at least in part on one or more environmental conditions. ### Claim 11 (depends on 7) The system of claim 7, wherein the identification of the first intended device and the first intended action are further based at least in part on an operation state of one or more smart devices in communication with the first device. ### Claim 12 (depends on 7) The system of claim 7, wherein the first device includes a camera, a transceiver, and the processor and wherein the first intended device is in communication with the first device and includes a controller configured to perform the first intended action. ### Claim 13 A computer program product comprising a computer readable storage medium having program instructions embodied therewith the program instructions executable by a computer processor to cause the computer processor to perform a method, comprising: observing, via a first device, a first user making a first gesture and a second gesture; identifying, by a processor, a first intended device based at least in part on the first gesture, wherein the first gesture includes the first user pointing at the first intended device and wherein the first gesture is made with a first hand of the first user and the second gesture is made with a second hand of the first user; identifying, by the processor, a first intended action for the first intended device based at least in part on the second gesture; observing, via the first device, a third gesture and a fourth gesture performed by a second user, wherein the third gesture is made with a first hand of the second user and the third gesture is made with a second hand of the second user; identifying, by the processor, a second intended device based at least in part on the third gesture, wherein the third gesture includes the second user pointing at the second intended device; identifying, by the processor, a second intended action to be performed by the second intended device based at least in part on the fourth gesture; obtaining, by the processor, a hierarchy of users; instructing, by the processor, the first intended device to perform the first intended action based on a determination that the first user has a higher priority than the second user in the hierarchy of users; and instructing, by the processor, the second intended device to perform the second intended action based on a determination that the second user has a higher priority than the first user in the hierarchy of users. ### Claim 14 (depends on 13) The computer program product of claim 13, wherein the method further comprises identifying the first user by the processor and wherein the identification of the first intended device and the first intended action are further based at least in part on the identity of the first user.