Real-time gesture recognition method and apparatus

An apparatus for real-time recognition of a gesture communicated by a subject, the apparatus comprising:

• one or more multi-threaded processors;
• and
• a non-transitory memory with instructions stored thereon, the instructions upon execution by the one or more multi-threaded processors, causing the one or more multi-threaded processors to:
• receive, by a first thread of the one or more multi-threaded processors, a first set of image frames associated with the gesture, the first set of image frames captured during a first time interval;
• perform, by the first thread, pose estimation on each frame of the first set of image frames including eliminating background information from each frame to obtain one or more areas of interest;
• store information representative of the one or more areas of interest in a shared memory accessible to the one or more multi-threaded processors;
• and
• perform, by a second thread of the one or more multi-threaded processors, a gesture recognition operation on a second set of image frames associated with the gesture, the second set of image frames captured during a second time interval that is different from the first time interval,
• wherein the instructions upon execution by the one or more multi-threaded processors cause the one or more multi-threaded processors, as part of performing the gesture recognition operation, to:
• use a first processor of the one or more multi-threaded processors that implements a first three-dimensional convolutional neural network (3D CNN) to perform an optical flow operation on the information representative of the one or more areas of interest that is accessed from the shared memory, wherein the optical flow operation is enabled to recognize a motion associated with the gesture;
• use a second processor of the one or more multi-threaded processors that implements a second 3D CNN to perform spatial and color processing operations on the information representative of the one or more areas of interest that is accessed from the shared memory;
• fuse results of the optical flow operation and results of the spatial and color processing operations to produce an identification of the gesture;
• and
• use a recurrent neural network (RNN) to determine that the identification corresponds to a singular gesture across at least the first and second sets of image frames,
• wherein the first thread and the second thread execute concurrently on the one or more multi-threaded processors, wherein the first thread continuously receives and processes new image frames from the first set of image frames while the second thread performs the gesture recognition operation on the second set of image frames, and wherein the pose estimation and the gesture recognition operation are performed in parallel on different sets of image frames captured at different time intervals, and wherein the first time interval and the second time interval are non-overlapping time intervals.