# US 12,112,440 B2 · Mixed-reality visor for in-situ vehicular operations training > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,440 B2 - Document ID: US12112440B2 - Status: Granted - Kind: B2 - Application number: 17/919,304 - Filed: 2021-12-17 - Published: 2023-10-19 - Granted: 2024-10-08 - Earliest claimed: 2020-12-20 - Applicants: Wael Zohni - Inventors: Wael Zohni - Web page: https://patentgrep.com/patent/US12112440B2 ## CPC - G02B 27/017: {Head mounted} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G01S 17/89: for mapping or imaging - G02B 27/0172: {characterised by optical features} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 15/00: Three-dimensional [3D] image rendering - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A Mixed-Reality visor (MR-visor) system and method utilizing environmental sensor feedback for replicating restricted external visibility during operation of manned vehicles, such as marine or aircraft. Adaptive hardware and software enable the user to reliably limit, modify and/or block views outside window(s) areas of the vehicle while maintaining visibility of the cabin interior and instrument control panel(s) without need for complex mechanical hardware alignment and setup. In the case of aircraft pilot training, the MR-visor can be worn by a pilot to replicate Instrument Meteorological Conditions (IMC) and other challenging scenarios. ## Claims ### Claim 1 A system for modifying a view perceived by a user who is substantially contained within an enclosure, the system comprising: a view-blocking wearable user visor-headset having a display surface and see-through camera; a distinguishing system configured to detect regions corresponding to an exterior region of the enclosure from an interior region contained within the enclosure and output a region signal, the distinguishing system is configured to generate a three-dimensional point cloud model of a substantial portion of the interior region of the enclosure; computer memory storage and data transfer capability of the three-dimensional point cloud model; a vision system configured to overlay imagery graphics upon the display surface of the view-blocking wearable user visor-headset based on the region signal; first position and motion detection sensors affixable to a portion of the enclosure; second position and motion detection sensors integrated with the view-blocking wearable user visor-headset; and hardware configured for data communication between the first position and motion detection sensors, the second position and motion detection sensors, and at least one processing unit that is configured to generate and utilize differential data between the first and the second position and motion detection sensors to reduce vision mapping compute requirements. ### Claim 2 (depends on 1) The system of claim 1, wherein the first and the second position and motion sensors include at least one of an inertial measuring unit, a gyroscopic sensor, and an accelerometer. ### Claim 3 (depends on 1) The system of claim 1, further comprising at least one electromagnetic energy emitter and at least one electromagnetic energy receiver operably coupled to a processing system. ### Claim 4 A system for modifying a user's view of a region of interest from within an enclosure in which the user is present, the system comprising: a user wearable device having a transparent display screen substantially covering a user's field of view; a first processing unit configured to detect regions of interest and output a region signal; second processing units configured to detect and transmit object three-dimensional orientation provided by first position and motion sensors affixed in place and second position and motion sensors integrated with the user wearable device; a vision system configured to receive the region signal and display an image upon at least one selected region of interest; at least one electromagnetic energy emitter and at least one electromagnetic energy receiver operably coupled to the vision system, the at least one electromagnetic energy emitter and the at least one electromagnetic energy receiver utilize infra-red electromagnetic energy; and a computer vision enabled three-dimensional point-cloud rendering subsystem configured to identify targeted regions of interest, the computer vision enabled three-dimensional point-cloud rendering subsystem comprising: at least one camera with a focal length positioned to provide binocular three-dimensional views of a surrounding environment; and a computer-based system integrating at least one of a three-dimensional point cloud model of internal features of the enclosure in which the user is present, an inertial measurement unit (IMU), and light-detection-and-ranging (LIDAR) depth sensing filter for determining limits of the enclosure and providing environmental references for point cloud model overlay, sizing, and position. ### Claim 5 (depends on 4) The system of claim 4, wherein the transparent display screen is a Micro-Electro-Mechanical-Systems (MEMS) display utilizing holographic or diffractive extraction of light from a waveguide where at least two (2) micro-displays beam an image through a lens toward a surface of the transparent display screen. ### Claim 6 (depends on 5) The system of claim 5, wherein headset onboard inward-facing visible light emitters direct light energy towards eyes of the user. ### Claim 7 (depends on 5) The system of claim 5, wherein at least one electromagnetic energy emitter is located within an interior of the enclosure in which the user is present. ### Claim 8 (depends on 4) The system of claim 4, wherein the enclosure is an aircraft cockpit with interior regions including an instrument panel, and windowpane regions of interest providing view of exterior regions to the aircraft. ### Claim 9 (depends on 4) The system of claim 4, wherein the at least one camera is an optical see-through camera that alters between near- and far-vision by utilizing at least one of mechanical cycling of multiple sets of physical lenses and high-speed auto-focusing lenses. ### Claim 10 (depends on 4) The system of claim 4, further comprising a programmable Global Positioning System (GPS) tracking system that provides at least one of location, orientation, speed, and acceleration data to an image-generating Central-Processing-Unit (CPU). ### Claim 11 (depends on 4) The system of claim 4, further comprising inward-facing sensors or cameras utilized for at least one of gaze-tracking, eye-monitoring, and face-tracking of the user. ### Claim 12 (depends on 4) The system of claim 4, further comprising supplemental physiological measurement devices having at least one of a heart rate sensor, electroencephalogram (EEG) sensors, and Galvanic Skin Response (GSR) sensors. ### Claim 13 (depends on 4) The system of claim 4, further comprising an embedded surround sound personal audio system component to the user wearable device. ### Claim 14 (depends on 5) The system of claim 5, wherein at least one electromagnetic energy emitter is located within an exterior of the enclosure in which the user is present. ### Claim 15 (depends on 3) The system of claim 3, wherein the at least one electromagnetic energy emitter and the at least one electromagnetic energy receiver utilize infra-red electromagnetic energy. ### Claim 16 A system for modifying a view perceived by a user who is substantially contained within an enclosure, the system comprising: a view-blocking wearable user visor-headset having a display surface and see-through camera; a distinguishing system configured to detect regions corresponding to an exterior region of the enclosure from an interior region contained within the enclosure and output a region signal; a vision system configured to overlay imagery graphics upon the display surface of the view-blocking wearable user visor-headset based on the region signal; and a computer vision enabled three-dimensional point-cloud rendering subsystem configured to identify targeted regions of interest, the computer vision enabled three-dimensional point-cloud rendering subsystem comprising: at least one camera with a focal length positioned to provide binocular three-dimensional views of a surrounding environment; and a computer-based system integrating at least one of a three-dimensional point cloud model of internal features of the enclosure in which the user is present, an inertial measurement unit (IMU), and light-detection-and-ranging (LIDAR) depth sensing filter for determining limits of the enclosure and providing environmental references for point cloud model overlay, sizing, and position. ### Claim 17 (depends on 16) The system of claim 16, wherein the see-through camera is configured to alter between near- and far-vision by utilizing at least one of mechanical cycling of multiple sets of physical lenses and high-speed auto-focusing lenses. ### Claim 18 (depends on 1) The system of claim 1, wherein the display surface is a Micro-Electro-Mechanical-Systems (MEMS) display utilizing holographic or diffractive extraction of light from a waveguide where at least two (2) micro-displays beam an image through a lens toward a surface of the display surface. ### Claim 19 (depends on 18) The system of claim 18, wherein inward-facing visible light emitters onboard the view-blocking wearable user visor-headset direct light energy towards eyes of the user. ### Claim 20 (depends on 3) The system of claim 3, wherein at least one electromagnetic energy emitter is located within an interior of the enclosure in which the user is present. ### Claim 21 (depends on 3) The system of claim 3, wherein at least one electromagnetic energy emitter is located within an exterior of the enclosure in which the user is present. ### Claim 22 (depends on 1) The system of claim 1, wherein the enclosure is an aircraft cockpit with interior regions including an instrument panel, and windowpane regions of interest providing view of exterior regions to the aircraft. ### Claim 23 (depends on 1) The system of claim 1, further comprising a programmable Global Positioning System (GPS) tracking system that provides at least one of location, orientation, speed, and acceleration data to an image-generating Central-Processing-Unit (CPU). ### Claim 24 (depends on 1) The system of claim 1, further comprising inward-facing sensors or cameras utilized for at least one of gaze-tracking, eye-monitoring, and face-tracking of the user. ### Claim 25 (depends on 1) The system of claim 1, further comprising supplemental physiological measurement devices having at least one of a heart rate sensor, electroencephalogram (EEG) sensors, and Galvanic Skin Response (GSR) sensors. ### Claim 26 (depends on 1) The system of claim 1, further comprising an embedded surround sound personal audio system component to the user wearable device. ### Claim 27 (depends on 16) The system of claim 16, further comprising at least one electromagnetic energy emitter and at least one electromagnetic energy receiver operably coupled to a processing system. ### Claim 28 (depends on 27) The system of claim 27, wherein the at least one electromagnetic energy emitter and the at least one electromagnetic energy receiver utilize infra-red electromagnetic energy. ### Claim 29 (depends on 27) The system of claim 27, wherein the at least one electromagnetic energy emitter is located within the interior of the enclosure in which the user is present. ### Claim 30 (depends on 27) The system of claim 27, wherein at least one electromagnetic energy emitter is located within an exterior of the enclosure in which the user is present. ### Claim 31 (depends on 1) The system of claim 1, wherein the see-through camera is configured to alter between near- and far-vision by utilizing at least one of mechanical cycling of multiple sets of physical lenses and high-speed auto-focusing lenses. ### Claim 32 (depends on 16) The system of claim 16, wherein the display surface is a Micro-Electro-Mechanical-Systems (MEMS) display utilizing holographic or diffractive extraction of light from a waveguide where at least two micro-displays beam an image through a lens toward a surface of the display surface. ### Claim 33 (depends on 32) The system of claim 32, wherein the view-blocking wearable user visor-headset comprises inward-facing visible light emitters onboard directing light energy towards eyes of the user. ### Claim 34 (depends on 16) The system of claim 16, wherein the enclosure is an aircraft cockpit having interior regions including an instrument panel and windowpane regions of interest providing view of exterior regions to the aircraft. ### Claim 35 (depends on 16) The system of claim 16, further comprising a programmable Global Positioning System (GPS) tracking system configured to provide at least one of location, orientation, speed, and acceleration data to an image-generating Central-Processing-Unit (CPU). ### Claim 36 (depends on 16) The system of claim 16, wherein the view-blocking wearable user visor-headset comprises inward-facing sensors or cameras configured for at least one of gaze-tracking, eye-monitoring, and face-tracking of the user. ### Claim 37 (depends on 16) The system of claim 16, further comprising a supplemental physiological measurement device having at least one of a heart rate sensor, electroencephalogram (EEG) sensors, and Galvanic Skin Response (GSR) sensors. ### Claim 38 (depends on 16) The system of claim 16, further comprising an embedded surround sound personal audio system component to the view-blocking wearable user visor-headset. ### Claim 39 (depends on 1) The system of claim 1 wherein the see-through camera comprises a liquid lens configured for dynamic autofocus and zoom, a system configured to adjust a lens curvature and focal length of the liquid lens via at least one of an electric field and electro-wetting mechanism. ### Claim 40 (depends on 16) The system of claim 16 wherein the see-through camera comprises a liquid lens configured for dynamic autofocus and zoom, a system configured to adjust a lens curvature and focal length of the liquid lens via at least one of an electric field and electro-wetting mechanism. ### Claim 41 (depends on 1) The system of claim 1 wherein the distinguishing system is configured to employ machine learning algorithms to detect regions corresponding to the exterior region of the enclosure from the interior region over time. ### Claim 42 (depends on 16) The system of claim 16 wherein the distinguishing system is configured to employ machine learning algorithms to detect regions corresponding to the exterior region of the enclosure from the interior region over time. ### Claim 43 (depends on 1) The system of claim 1 wherein the hardware configured for data communication is configured for use with a plurality communication protocols comprising Bluetooth, Wi-Fi, and NFC. ### Claim 44 (depends on 1) The system of claim 1 further comprising alternative peripherals configured to interface with the vision system. ### Claim 45 (depends on 4) The system of claim 4 further comprising alternative peripherals configured to interface with the vision system. ### Claim 46 (depends on 16) The system of claim 16 further comprising alternative peripherals configured to interface with the vision system′. ### Claim 47 (depends on 1) The system of claim 1 wherein the vision system is configured to automatically adapt the imagery graphics overlay based on changes in the surrounding environment. ### Claim 48 (depends on 4) The system of claim 4 wherein the vision system is configured to automatically adapt the imagery graphics overlay based on changes in the surrounding environment. ### Claim 49 (depends on 16) The system of claim 16 wherein the vision system is configured to automatically adapt the imagery graphics overlay based on changes in the surrounding environment. ### Claim 50 (depends on 1) The system of claim 1 wherein the three-dimensional point cloud model is configured to be exported in a standardized format. ### Claim 51 (depends on 4) The system of claim 4 wherein the three-dimensional point cloud model is configured to be exported in a standardized format. ### Claim 52 (depends on 16) The system of claim 16 wherein the three-dimensional point cloud model is configured to be exported in a standardized format. ### Claim 53 (depends on 1) The system of claim 1 further comprising a gesture recognition module configured to allow the user to manually define and modify regions corresponding to the exterior region or the interior region through hand or object gestures, the distinguishing system configured to detect the hand or object gesture and update the region signal. ### Claim 54 (depends on 4) The system of claim 4 further comprising a gesture recognition module configured to allow the user to manually define and modify regions corresponding to the exterior region or the interior region through hand or object gestures, the distinguishing system configured to detect the hand or object gesture and update the region signal. ### Claim 55 (depends on 16) The system of claim 16 further comprising a gesture recognition module configured to allow the user to manually define and modify regions corresponding to the exterior region or the interior region through hand or object gestures, the distinguishing system configured to detect the hand or object gesture and update the region signal. ### Claim 56 (depends on 1) The system of claim 1 wherein the distinguishing system is configured to employ artificial intelligence and machine learning algorithms. ### Claim 57 (depends on 4) The system of claim 4 wherein the distinguishing system is configured to employ artificial intelligence and machine learning algorithms. ### Claim 58 (depends on 16) The system of claim 16 wherein the distinguishing system is configured to employ artificial intelligence and machine learning algorithms.