System and method for virtual character animation using motion capture

A computer-based method for motion capture of an actor to estimate a hand pose prediction of the actor for rendering a three-dimensional (3D) virtual environment, comprising:

• receiving raw motion capture data from at least one wearable sensor and at least one optical marker-based sensor;
• normalizing the received raw motion capture data via one or more sensor pre-processors;
• receiving the normalized motion capture data from the sensor pre-processors via a convolutional neural network;
• generating a learning model via the convolutional neural network to detect one or more patterns over a predetermined time window, the detected patterns being encoded by a matrix;
• receiving a measurement of bone length from the actor and the matrix of the convolutional neural network via a hand pose dense neural network;
• determining an intermediate hand pose prediction at a selected point in time via the hand pose dense neural network based on the received measurement of bone length and the matrix of the convolutional neural network;
• learn a mapping of the received measurement of bone lengths and the determined intermediate hand pose prediction via a hand scale dense neural network based on the outputs from the convolutional neural network;
• and
• rescaling and offsetting the determined intermediate hand pose prediction based on the learned mapping via a post processor to generate the hand pose prediction.

A non-transitory computer-readable storage medium for motion capture of an actor to estimate a hand pose prediction of the actor for rendering a three-dimensional (3D) virtual environment, wherein the non-transitory computer-readable storage medium includes instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:

• receiving raw motion capture data from at least one wearable sensor and at least one optical marker-based sensor;
• normalizing the received raw motion capture data via one or more sensor pre-processors;
• receiving the normalized motion capture data from the sensor pre-processors via a convolutional neural network;
• generating a learning model via the convolutional neural network to detect one or more patterns over a predetermined time window, the detected patterns being encoded by a matrix;
• receiving a measurement of bone length from the actor and the matrix of the convolutional neural network via a hand pose dense neural network;
• determining an intermediate hand pose prediction at a selected point in time via the hand pose dense neural network based on the received measurement of bone length and the matrix of the convolutional neural network;
• learn a mapping of the received measurement of bone lengths and the determined intermediate hand pose prediction via a hand scale dense neural network based on the outputs from the convolutional neural network;
• and
• rescaling and offsetting the determined intermediate hand pose prediction based on the learned mapping via a post processor to generate the hand pose prediction.