Interactive mixed masking system, method and computer program product for a simulator

A tangible, non-transitory computer readable media having program instructions stored thereon which when executed cause at least one processor to:

• register a location of a non-simulated chromakey mask, a user touch input device in a frame image of an ambient environment, and a virtual portal location within a virtual reality (VR) scene including virtual pixels of virtual user-interactive simulator (VUIS) controls;
• detect and isolate within the non-simulated chromakey mask, in a current frame image of the ambient environment, first pixels having a non-chromakey color overlapping a portion of the non-simulated chromakey mask and second pixels matching a chromakey color of the non-simulated chromakey mask;
• preserve the first pixels in the current frame image to form a camera pass-through image;
• and
• form a composite mixed reality scene with the VR scene and the first pixels preserved in the current frame image to form the camera pass-through image registered to the VR scene, wherein virtual pixels of the VUIS controls corresponding to the first pixels are omitted from the VR scene and remaining virtual pixels of the VUIS controls overlap and link to the user touch input device in the ambient environment.

A method comprising:

• capturing, by at least one camera, a frame image of an ambient environment;
• registering, by at least one processor, a location of a non-simulated chromakey mask, a user touch input device in the frame image of the ambient environment, and a virtual portal location within a virtual reality (VR) scene including virtual pixels of virtual user-interactive simulator (VUIS) controls;
• detecting and isolating, by the at least one processor, within the non-simulated chromakey mask in a current frame image of the ambient environment, first pixels having a non-chromakey color overlapping a portion of the non-simulated chromakey mask and second pixels matching a chromakey color of the non-simulated chromakey mask;
• preserving, by the at least one processor, the first pixels in the current frame image to form a camera pass-through image;
• and
• forming, by the at least one processor, a composite mixed reality scene with the VR scene and the first pixels preserved in the current frame image to form the camera pass-through image registered to the VR scene, wherein virtual pixels of the VUIS controls corresponding to the first pixels are omitted from the VR scene and remaining virtual pixels of the VUIS controls overlap and link to the user touch input device in the ambient environment.