# US 2013/0156296 A1 · Three Dimensional Gesture Recognition in Vehicles > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0156296 A1 - Document ID: US20130156296A1 - Status: Published application - Kind: A1 - Application number: 13/762,332 - Filed: 2013-02-07 - Published: 2013-06-20 - Earliest claimed: 2010-05-20 - Assignees: Edge3 Technologies LLC - Inventors: Tarek El Dokor - Web page: https://patentgrep.com/patent/US20130156296A1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - B62D 1/00: Steering controls, i.e. means for initiating a change of direction of the vehicle - G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/107: {Static hand or arm} - G06V 40/113: {Recognition of static hand signs} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06T 2207/20201: Motion blur correction - G06T 2207/30196: Human being; Person - G06T 2207/30232: Surveillance - G06T 2207/30268: Vehicle interior - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method and system for performing gesture recognition of a vehicle occupant employing a time of flight (TOF) sensor and a computing system in a vehicle. An embodiment of the method of the invention includes the steps of receiving one or more raw frames from the TOF sensor, performing clustering to locate one or more body part clusters of the vehicle occupant, calculating the location of the tip of the hand of the vehicle occupant, determining whether the hand has performed a dynamic or a static gesture, retrieving a command corresponding to one of the determined static or dynamic gestures, and executing the command. ## Claims ### Claim 1 A method for performing gesture recognition in a vehicle, comprising the steps of: receiving one or more raw frames from a time of flight (TOF) camera of a vehicle occupant; performing clustering to determine one or more objects of interest of the vehicle occupant; locating one or more arm clusters of the vehicle occupant; locating a palm cluster of the vehicle occupant; and calculating a location of a tip of a hand of the vehicle occupant. ### Claim 2 (depends on 1) The method of claim 1, further comprising the step of cleaning a depth data of each of the one or more raw frames. ### Claim 3 (depends on 1) The method of claim 1, further comprising the steps of: defining one or more predetermined search spaces in accordance with the location of the tip of the hand; defining one or more partition spaces within the one or more search spaces; and locating the hand within the partition and search spaces. ### Claim 4 (depends on 3) The method of claim 3, further comprising the steps of: observing whether the hand has performed a dynamic or a static gesture; if the hand has performed a dynamic gesture, matching the observed gesture with a set of known masks to determine an intended static gesture; and if the hand has performed a dynamic gesture, extracting features from one or more clips of the observed gesture and match with one or more predefined motion streams to determine an intended dynamic gesture. ### Claim 5 (depends on 4) The method of claim 4, further comprising the steps of: retrieving a command corresponding to one of the determined static and dynamic gesture; and executing the command. ### Claim 6 (depends on 5) The method of claim 5, wherein the command is executed in accordance with a computing system and display within the vehicle. ### Claim 7 (depends on 4) The method of claim 4, wherein the known masks are determined for a particular individual vehicle occupant. ### Claim 8 (depends on 4) The method of claim 4, wherein the predefined motion streams are determined for a particular individual vehicle occupant. ### Claim 9 (depends on 1) The method of claim 1, further comprising the step of selecting the vehicle occupant from among a plurality of individuals in the vehicle. ### Claim 10 (depends on 1) The system of claim 1, wherein a processor is adapted to determine the vehicle occupant from among two or more individuals in the vehicle. ### Claim 11 A method for performing gesture recognition of a vehicle occupant employing a time of flight (TOF) sensor and a computing system in a vehicle, comprising the steps of: receiving one or more raw frames from the TOF sensor; performing clustering to locate one or more body part clusters of the vehicle occupant; calculating a location of a tip of a hand of the vehicle occupant; determining whether the hand has performed a dynamic or a static gesture; retrieving a command corresponding to one of the determined static and dynamic gesture; and executing the command. ### Claim 12 A system for performing gesture recognition in a vehicle, comprising: a time of flight (TOF) camera for providing one or more raw frames of a vehicle occupant; and a processor for performing clustering to determine one or more objects of interest of the vehicle occupant, locating one or more arm clusters of the vehicle occupant as one of the objects of interest, and calculating a location of a tip of a hand of the vehicle occupant. ### Claim 13 (depends on 12) The system of claim 12, wherein the processor is further adapted to clean a depth data of each of the one or more raw frames. ### Claim 14 (depends on 12) The system of claim 12, wherein the processor is further adapted to define one or more predetermined search spaces in accordance with the location of the tip of the hand, define one or more partition spaces within the one or more search spaces, and locate the hand within the partition and search spaces. ### Claim 15 (depends on 14) The system of claim 14, wherein the processor is further adapted to observe whether the hand has performed a dynamic or a static gesture, if the hand has performed a dynamic gesture, match the observed gesture with a set of known masks to determine an intended static gesture, and if the hand has performed a dynamic gesture, extract features from one or more clips of the observed gesture and match with one or more predefined motion streams to determine an intended dynamic gesture. ### Claim 16 (depends on 15) The system of claim 15, wherein the processor is further adapted to retrieve a command corresponding to one of the determined static and dynamic gesture, and execute the command to perform one or more vehicle actions. ### Claim 17 (depends on 16) The system of claim 16, further comprising a computing system for implementing the command. ### Claim 18 (depends on 16) The system of claim 16, wherein the computing system further comprises a display for displaying a response to the command to the vehicle occupant. ### Claim 19 (depends on 15) The system of claim 15, wherein the processor is adapted to determine the known masks for a particular individual vehicle occupant. ### Claim 20 (depends on 15) The system of claim 15, wherein the processor is adapted to determine the predefined motion streams for a particular individual vehicle occupant.