# US 11,305,887 B2 · Method and system for detecting and remedying situation awareness failures in operators of remotely operated vehicles > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,305,887 B2 - Document ID: US11305887B2 - Status: Granted - Kind: B2 - Application number: 16/569,854 - Filed: 2019-09-13 - Published: 2021-03-25 - Granted: 2022-04-19 - Earliest claimed: 2019-09-13 - Assignees: The Boeing Company - Applicants: The Boeing Company - Inventors: Nathan C. Perry; Jesse Lee Kresevic - Web page: https://patentgrep.com/patent/US11305887B2 ## CPC - B64D 45/00: Aircraft indicators or protectors not otherwise provided for - G06V 40/18: Eye characteristics, e.g. of the iris - B64D 47/02: Arrangements or adaptations of signal or lighting devices - B64U 20/87: Mounting of imaging devices, e.g. mounting of gimbals - G05D 1/0011: {associated with a remote control arrangement} - G05D 1/0022: {characterised by the communication link (data switching networks in general H04L12/00)} - G05D 1/0061: {for transition from automatic pilot to manual pilot and vice versa} - G06V 40/19: Sensors therefor - G08G 5/22: located on the ground - G08G 5/26: Transmission of traffic-related information between aircraft and ground stations - G08G 5/34: for flight plan modification - G08G 5/55: for a single aircraft - B64U 2101/30: for imaging, photography or videography - B64U 2201/20: Remote controls - G08G 5/57: for unmanned aircraft - G08G 5/58: for emergency situations, e.g. hijacking or bird strikes - B64D: EQUIPMENT FOR FITTING IN OR TO AIRCRAFT; FLIGHT SUITS; PARACHUTES; ARRANGEMENT OR MOUNTING OF POWER PLANTS OR PROPULSION TRANSMISSIONS IN AIRCRAFT - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) ## Abstract A system and method that detect situation awareness failures in operators of remotely operated vehicles and consequently alert the operators and/or activate a fail-safe mode in the vehicles to reduce potential for an accident. Real-time values of indicators displayed on a control station are periodically stored in a database and compared to inspected values of each indicator stored when the operator last viewed each indicator to determine if there is a difference representing an operator's level of awareness. An eye-tracking system is used to monitor the operator's point-of-gaze and awareness of each indicator. If there is a difference beyond an acceptable level of deviation between the real-time value and the inspected value, the system alerts the operator and/or commands the remotely operated vehicle to commence a fail-safe operation. ## Claims ### Claim 1 A method for detecting and remedying a situation awareness failure of an operator of a remotely operated vehicle, the method comprising: storing a location of an indicator contained within a display of a control station for remotely operating the remotely operated vehicle; monitoring a point-of-gaze of the operator of the control station; storing an inspected value of the indicator representing an inspected operating state of the remotely operated vehicle at a time when the operator last inspected the indicator; periodically storing a real-time value of the indicator, the real-time value representing a current operating state of the remotely operated vehicle; performing a comparative analysis comparing the real-time value of the indicator with the inspected value of the indicator; and assessing a situation awareness of the operator based on the comparative analysis of the real-time value and the inspected value of the indicator. ### Claim 2 (depends on 1) The method of claim 1, further comprising alerting the operator and starting a timer if the situation awareness of the operator is assessed to be insufficient. ### Claim 3 (depends on 1) The method of claim 1, further comprising sending a command to activate a fail-safe control mode in the remotely operated vehicle if the situation awareness of the operator is assessed to be insufficient. ### Claim 4 (depends on 2) The method of claim 2, further comprising sending a command to activate a fail-safe control mode in the remotely operated vehicle if the operator fails to remedy an insufficient situation awareness before the timer expires. ### Claim 5 (depends on 1) The method of claim 1, wherein the step of assessing the situation awareness of the operator comprises determining if a difference between the real-time value of the indicator and the inspected value of the indicator exceeds a threshold for an acceptable level of deviation between the real-time value and the inspected value of the indicator. ### Claim 6 (depends on 5) The method of claim 5, further comprising alerting the operator and starting a timer if the difference between the real-time value of the indicator and the inspected value of the indicator exceeds the threshold for the acceptable level of deviation between the real-time value and the inspected value of the indicator. ### Claim 7 (depends on 5) The method of claim 5, further comprising sending a command to activate a fail-safe control mode in the remotely operated vehicle if the difference between the real-time value of the indicator and the inspected value of the indicator exceeds the threshold for the acceptable level of deviation between the real-time value and the inspected value of the indicator. ### Claim 8 (depends on 6) The method of claim 6, further comprising sending a command to activate a fail-safe control mode in the remotely operated vehicle if the operator fails to remedy an insufficient situation awareness before the timer expires. ### Claim 9 (depends on 1) The method of claim 1, wherein the real-time value of the indicator is stored at a frequency between 30 Hz and 400 Hz. ### Claim 10 (depends on 1) The method of claim 1, wherein inspected value of the indicator is stored at a frequency between 30 Hz and 400 Hz. ### Claim 11 A system for detecting and remedying a situation awareness failure of an operator of a remotely operated vehicle, the system comprising: an eye tracker device for monitoring a point-of-gaze of the operator; one or more databases for storing (i) a location of an indicator contained within a display of a control station for remotely operating the remotely operated vehicle, (ii) an inspected value of the indicator when the point-of-gaze of the operator intersects the location of the indicator, the inspected value representing an inspected operating state of the remotely operated vehicle at a time when the operator last inspected the indicator, and (iii) a real-time value of the indicator, the real-time value representing a current operating state of the remotely operated vehicle; and a computing device programmed to receive the inspected value and the real-time value of the indicator from the one or more databases, and to assess a situation awareness of the operator based on a comparative analysis of the real-time value and the inspected value of the indicator. ### Claim 12 (depends on 11) The system of claim 11, wherein the computing device is programmed to alert the operator and start a timer if the situation awareness of the operator is assessed to be insufficient. ### Claim 13 (depends on 11) The system of claim 11, wherein the computing device is programmed to generate a command to activate a fail-safe control mode in the remotely operated vehicle if the situation awareness of the operator is assessed to be insufficient. ### Claim 14 (depends on 12) The system of claim 12, wherein the computing device is programmed to generate a command to activate a fail-safe control mode in the remotely operated vehicle if the operator fails to remedy an insufficient situation awareness before the timer expires. ### Claim 15 (depends on 11) The system of claim 11, wherein the computing device is programmed to determine if a difference between the real-time value of the indicator and the inspected value of the indicator exceeds a threshold for an acceptable level of deviation between the real-time value and the inspected value of the indicator. ### Claim 16 (depends on 15) The system of claim 15, wherein the computing device is programmed to alert the operator and start a timer if the difference between the real-time value of the indicator and the inspected value of the indicator exceeds the threshold for the acceptable level of deviation between the real-time value and the inspected value of the indicator. ### Claim 17 (depends on 15) The system of claim 15, wherein the computing device is programmed to generate a command to activate a fail-safe control mode in the remotely operated vehicle if the difference between the real-time value of the indicator and the inspected value of the indicator exceeds the threshold for the acceptable level of deviation between the real-time value and the inspected value of the indicator. ### Claim 18 (depends on 16) The system of claim 16, wherein the computing device is programmed to generate a command to activate a fail-safe control mode in the remotely operated vehicle if the operator fails to remedy an insufficient situation awareness before the timer expires. ### Claim 19 A ground control system for detecting and remedying a situation awareness failure of a remotely piloted aircraft, the ground control system comprising: a ground control station comprising a display; an eye tracker device for monitoring a point-of-gaze of a pilot; one or more databases for storing (i) a location of a plurality of indicators contained within the display for controlling the remotely piloted aircraft, (ii) inspected values of each of the indicators when the point-of-gaze of the remote pilot intersects the location of each of the indicators, the inspected values representing an inspected operating state of the remotely piloted aircraft at a time when a remote pilot last inspected each of the indicators, and (iii) real-time values of each of the indicators, the real-time values representing a current operating state of the remotely piloted aircraft; and a computing device programmed to receive the inspected value and the real-time value of each of the indicators from the one or more databases, and to assess a situation awareness of the pilot based on a comparative analysis of the real-time value and the inspected value of each of the indicators. ### Claim 20 (depends on 19) The ground control system of claim 19, wherein the computing device is programmed to alert the pilot and start a timer if the situation awareness of the pilot is assessed to be insufficient, and to generate a command to activate a fail-safe control mode in the remotely piloted aircraft if the pilot fails to remedy an insufficient situation awareness before the timer expires.