Lockstep processor recovery for vehicle applications

A computer implemented method comprising:

• providing communication agents in a plurality of processing nodes in a computing environment, each processing nodes including a processing unit and associated memory storing instructions;
• creating a communications group comprising a first plurality of processing nodes comprising worker processing nodes and a second plurality of processing nodes comprising redundant processing nodes;
• instructing each of the worker processing nodes to execute a plurality of same tasks in parallel;
• receiving, at intervals specified by a master processing node, intermediate processing results of the plurality of same tasks performed by the worker processing nodes from each of the communication agents associated with a worker processing node;
• detecting an application fault in one of the worker processing nodes by checking one or more intermediate results of each of the plurality of same tasks performed by the worker processing nodes;
• creating a new worker processing node from one of the redundant processing nodes to replace the one of the worker nodes with the fault using at least one of the communication agents;
• creating a new communications group comprising a portion of the first plurality of processing nodes and the new worker processing node;
• writing checkpoint backup data to each of the worker processing nodes in the new communications group;
• and
• causing the worker processing nodes to execute the plurality of same tasks in parallel based on an application state reflected in the checkpoint backup data.

A distributed computing system in a motor vehicle, comprising:

• a plurality of distributed nodes each comprising processing circuitry and a memory associated with the processing circuitry, each of the distributed nodes including a message passing interface (MPI) agent;
• and
• a network bus connecting each of the plurality of distributed nodes, at least one of the plurality of distributed nodes configured as a manager node, the manager node having memory including instructions, the instructions causing the manager node to:
• create a communications group comprising a first plurality of processing nodes comprising worker processing nodes and a second plurality of processing nodes comprising redundant processing nodes;
• instruct each of the worker processing nodes to execute a plurality of same tasks for an application in lockstep with each other worker processing nodes;
• receive, at intervals specified by a master processing node, intermediate processing results of the plurality of same tasks performed by the worker processing nodes from each of the message passing interface agents associated with a worker processing node;
• detect a fault in one of the worker processing nodes based on the results of the plurality of same tasks performed by the one of the worker processing nodes by checking the one or more intermediate processing results of each of the plurality of same tasks performed by the worker processing nodes;
• remove the one of the worker processing nodes exhibiting the fault from the communications group;
• create a new worker processing node from one of the redundant processing nodes to replace the one of the worker processing nodes exhibiting the fault using the message passing interface agent;
• create a new communications group comprising the first plurality of processing nodes remaining in the communications group and the new worker processing node;
• write checkpoint backup data to each of the worker processing nodes in the new communications group;
• and
• cause each of the worker processing nodes to execute the plurality of same tasks in parallel starting at an application state reflected in the checkpoint backup data.