Systems and methods for an environment-aware predictive modeling framework for human-robot symbiotic walking

A system, comprising:

• a prosthetic or orthotic joint configured to receive one or more control signals and operate in response to the one or more control signals;
• a camera that captures image data indicative of a surrounding environment around the prosthetic or orthotic joint;
• and
• a computing device in operative communication with the prosthetic or orthotic joint and the camera, the computing device including a processor in communication with a memory, the memory including instructions, which, when executed, cause the processor to:
• receive, at the processor, the image data from the camera;
• extract, by a depth estimation network formulated at the processor, a set of depth features indicative of perceived spatial depth information of a surrounding environment from the image data;
• and
• generate, by a control output module formulated at the processor, a control signal to be applied to the prosthetic or orthotic joint based on the set of depth features
• wherein the memory further includes instructions, which, when executed, cause the processor to:
• uniformly sample, at the control output module, an ensemble of latent observations from one or more observed behavior demonstrations of the prosthetic or orthotic joint that incorporate human kinematic properties and environmental features within the latent space, a trajectory of the prosthetic or orthotic joint being collectively described by a plurality of basis functions;
• iteratively propagate, at the control output module, the ensemble forward by one step using a state transition function as the prosthetic or orthotic joint operates;
• iteratively update, at the control output module, one or more measurements of the ensemble using the set of depth features and an orientation of the prosthetic or orthotic joint;
• iteratively project, at the control output module, a mean and variance of one or more latent components of the ensemble into a trajectory space through the plurality of basis functions and based on the one or more measurements of the ensemble;
• and
• update, at the control output module, the control signal based on a difference between a new observation at a first time t and an expected observation at a second time t−1, the expected observation being indicative of one or more measurements of the ensemble taken at time t−1 and the new observation being indicative of one or more measurements of the ensemble taken at time t.