Integrating Tactile Nonvirtual Controls in a Virtual Reality (VR) Training Simulator

A training simulator software component for use on a computer in association with a system for integrating nonvirtual tactile controls in a virtual reality (VR) training simulator for training a trainee operator in the operation of equipment, said equipment having a physical control cab with a plurality of operator-actuated controls for use in operation thereof positioned in relation to an operator position, said system comprising:

• a computer for hosting and execution of the training simulator software component comprising:
• a processor and memory;
• a human display interface connected thereto for the display to a trainee operator of a three-dimensional VR operating environment within which the trainee operator can operate a virtual version of the equipment;
• and
• a control bus for the capture of control inputs and video signal stream data by the computer for use in the rendering of the VR operating environment;
• a training control cab for use by the trainee operator comprising:
• a plurality of nonvirtual controls corresponding to and emulating a plurality of the operator-actuated controls of the physical control cab of the equipment, said nonvirtual controls being positioned in relation to a trainee position corresponding approximately to the location of the operator position in the physical control cab and from which the trainee operator can interact with said nonvirtual controls, each nonvirtual control being equipped with sensors to permit capture of operating values therefrom upon manual actuation of same and being operatively connected to the control bus of the computer, whereby the operating values from the nonvirtual controls are captured via the control bus for use by the training simulator software component;
• and
• a video camera connected to the computer and positioned to capture a view of the trainee operator in the trainee position and the positioning and orientation of limbs of the trainee operator in relation to the nonvirtual controls, and generate a video signal data stream corresponding thereto which is captured via the control bus for use by the training simulator software component;
• said software capable of facilitating the method of integrating nonvirtual controls in a VR training simulator when executed by the computer by, in conjunction with hardware components of the computer:
• on initiation of the method and throughout the time of training operation by the trainee operator, rendering and updating the view of the VR operating environment to the human display interface, said view seen from the perspective of the trainee operator of the equipment in the trainee position in the physical control cab and including:
• placement, behavior and operation of the virtual version of the equipment in relation to an operating area;
• a virtual control cab of the equipment including rendered versions of at least each operator-actuated control, being rendered cab controls, showing current status of the rendered cab controls based on the corresponding nonvirtual controls in the training control cab;
• and
• simulated view of portions of the limbs of the trainee operator in relation to their position in the training control cab of the equipment and in interaction with each of the rendered cab controls, based upon the limb positioning of the trainee operator in relation to the nonvirtual controls in the training control cab;
• when the trainee operator manually actuates any of the nonvirtual controls in the training control cab:
• translating operating values corresponding to said manual actuation received via the control bus into updated control parameters to be applied to the behaviour of the virtual version of the equipment within the VR operating environment;
• and
• applying any updated control parameters to the rendering and behaviour of the virtual version of the equipment and the corresponding rendered cab controls within the virtual control cab;
• on an ongoing basis during operation of the virtual version of the equipment by the trainee operator:
• applying an image recognition function to the video signal data stream to yield the necessary limb rendering data corresponding to the positioning and orientation of the portions of the limbs of the trainee operator in relation to the nonvirtual controls within the training control cab, to permit the rendering of virtual versions of the portions of the limbs of the trainee operator within the VR operating environment;
• and
• upon any movement of the limbs of the trainee operator resulting in modified limb rendering data, applying any updated limb rendering data to the rendering and behavior of the simulated versions of the portions of the limbs of the trainee operator within the VR operating environment;
• wherein during operation of the virtual version of the equipment within the VR operating environment, the trainee operator will be able to see a rendered version of at least portions of their limbs interacting with controls in the virtual control cab while physically feeling the actuation of the nonvirtual controls within the training control cab.

An equipment operation simulation system integrating nonvirtual controls in a virtual reality (VR) training environment for training the trainee operator in the operation of equipment, said equipment having a physical control cab with a plurality of operator-actuated controls for use in operation thereof positioned in relation to an operator position, said system comprising:

• a computer comprising:
• a processor and memory;
• a human display interface connected to the computer for display to a trainee operator of a three-dimensional VR operating environment;
• a training simulator software component programmed to render and display the VR operating environment in accordance with the remainder of the method;
• a control bus for capture of control inputs and video signal stream data by the computer for use in the rendering of the VR operating environment;
• and
• a training control cab, comprising:
• a plurality of nonvirtual controls corresponding to and emulating a plurality of the operator-actuated controls of the physical control cab of the equipment, said nonvirtual controls being positioned in relation to a trainee position corresponding approximately to the location of the operator position in the physical control cab and from which the trainee operator can interact with said nonvirtual controls, each nonvirtual control being equipped with sensors to permit capture of operating values therefrom upon manual actuation of same and being operatively connected to the control bus of the computer, whereby the operating values from the nonvirtual controls are captured via the control bus for use by the training simulator software component;
• a video camera connected to the computer and positioned to capture a view of the trainee operator in the trainee position and the positioning and orientation of limbs of the trainee operator in relation to the nonvirtual controls, and generate a video signal data stream corresponding thereto which is captured via the control bus for use by the training simulator software component;
• where in operation of the system the training simulator software component and computer will:
• on initiation of the method and throughout the time of training operation by the trainee operator, render and update the view of the VR operating environment to the human display interface within which the trainee operator can simulate the operation of a virtual version of the equipment, said view seen from the perspective of the trainee operator of the equipment in the trainee position in the physical control cab and including:
• placement, behavior and operation of the virtual version of the equipment in relation to an operating area;
• a virtual control cab of the equipment including rendered versions of at least each operator-actuated control, being rendered cab controls, showing current status of the rendered cab controls based on the corresponding nonvirtual controls in the training control cab;
• and
• simulated view of portions of the limbs of the trainee operator in relation to their position in the training control cab of the equipment and in interaction with each of the rendered cab controls, based upon the limb positioning of the trainee operator in relation to the nonvirtual controls in the training control cab;
• when the trainee operator manually actuates any of the nonvirtual controls in the training control cab:
• translate operating values corresponding to said manual actuation received via the control bus into updated control parameters to be applied to the behaviour of the virtual version of the equipment within the VR operating environment;
• and
• apply any updated control parameters to the rendering and behaviour of the virtual version of the equipment and the corresponding rendered cab controls within the virtual control cab;
• on an ongoing basis during operation of the virtual version of the equipment by the trainee operator:
• apply an image recognition function to the video signal data stream to yield the necessary limb rendering data corresponding to the positioning and orientation of the limbs of the trainee operator in relation to the nonvirtual controls within the training control cab, to permit the rendering of virtual versions of the portions of the limbs of the trainee operator within the VR operating environment;
• and
• upon any movement of the limbs of the trainee operator resulting in modified limb rendering data, apply any updated limb rendering data to the rendering and behavior of the simulated versions of the portions of the limbs of the trainee operator within the VR operating environment;
• wherein during operation of the virtual version of the equipment within the VR operating environment, the trainee operator will be able to see a rendered version of at least portions of their limbs interacting with controls in the virtual control cab while physically feeling the actuation of the nonvirtual controls within the training control cab.