# US 11,993,394 B2 · Flight safety demonstration and infotainment through mixed reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,993,394 B2
- Document ID: US11993394B2
- Status: Granted
- Kind: B2
- Application number: 17/559,043
- Filed: 2021-12-22
- Published: 2023-05-11
- Granted: 2024-05-28
- Earliest claimed: 2021-11-10
- Assignees: Rockwell Collins, Inc.
- Applicants: Rockwell Collins, Inc.
- Inventors: Poornima Kathiresan; Aditya Kumar; Santhosh Ramamurthy
- Web page: https://patentgrep.com/patent/US11993394B2

## CPC

- B64D 45/00: Aircraft indicators or protectors not otherwise provided for
- B64D 11/0015: {Arrangements for entertainment or communications, e.g. radio, television}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/147: using display panels
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/73: using feature-based methods
- G06T 11/00: Two-dimensional [2D] image generation
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- B64D 2045/007: {Indicators or signs in the cabin, e.g. exit signs or seat numbering}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30244: Camera pose
- G06T 2207/30268: Vehicle interior
- G06V 40/161: {Detection; Localisation; Normalisation}
- G09G 2380/12: Avionics applications
- B64D: EQUIPMENT FOR FITTING IN OR TO AIRCRAFT; FLIGHT SUITS; PARACHUTES; ARRANGEMENT OR MOUNTING OF POWER PLANTS OR PROPULSION TRANSMISSIONS IN AIRCRAFT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

The present disclosure is generally directed to the use of mixed reality to provide safety information as well as infotainment during flight. Demonstrations of the safety information for the aircraft are augmented in a 3-D way with a virtual representation of the aircraft. The virtual representation provides the safety instructions to the passengers. The mixed reality concept provides the safety information with the help of video-see through mixed reality by way of a head-mounted display worn by the passengers. The safety instructions are provided with a complete working view of locations and operations for the safety jackets, emergency exits, and oxygen passenger service units.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving a first video signal and pose information; wherein the first video signal is received from a camera of a head-mounted display within an aircraft environment, the first video signal corresponding to images captured of the aircraft environment; wherein the pose information is received from a head tracker of the head-mounted display; wherein the first video signal and the pose information are received by way of an avionics network;
detecting a feature vector within the first video signal by digital image processing the first video signal using a feature detector algorithm, the feature vector including at least a first feature, the first feature corresponding to an aircraft safety device, the aircraft safety device including one of a life preserver stowage, an oxygen mask passenger service unit, a seatbelt, or an emergency exit door;
estimating a pose of the aircraft safety device within the first video signal based on the feature vector together with the pose information from the head tracker;
generating a second video signal of the aircraft environment, the second video signal including the first video signal and one or more virtual objects composited with the first video signal, the one or more virtual objects corresponding to a safety instruction associated with the aircraft safety device, the one or more virtual objects positioned and oriented within the second video signal from a perspective of the head-mounted display based on the pose of the aircraft safety device; and
causing a display of the head-mounted display to display the second video signal.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the first feature includes one or more edges associated with the aircraft safety device.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the feature detector algorithm comprises a speeded up robust features (SURF) method.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the pose of the aircraft safety device within the first video signal is estimated by calculating a transformation matrix based on the pose information from the head tracker together with the first feature associated with the aircraft safety device included in the feature vector.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the aircraft safety device comprises the life preserver stowage; wherein the safety instruction displayed by the head-mounted display includes a life preserver procedure; wherein the life preserver procedure includes displaying a virtual representation of operating the life preserver stowage.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the aircraft safety device comprises the oxygen passenger service unit, wherein the safety instruction displayed by the head-mounted display includes an oxygen mask procedure; wherein the oxygen mask procedure includes displaying a virtual representation of operating the oxygen mask passenger service unit.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the aircraft safety device comprises the seatbelt; wherein the feature vector includes a second feature, the second feature including a fasten seatbelt sign; wherein the safety instruction displayed by the head-mounted display includes a fasten seatbelt procedure, wherein the fasten seatbelt procedure includes displaying a light on the fasten seatbelt sign and a virtual representation of operating the seatbelt.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the aircraft safety device comprises the emergency exit door; wherein the safety instruction displayed by the head-mounted display includes an emergency exit door procedure, wherein the emergency exit door procedure includes displaying a virtual representation of operating the emergency exit door.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising receiving a passenger language from the avionics network, wherein the second video signal includes an auditory sound output by the head-mounted display, wherein the auditory sound is selected from memory based on the passenger language.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, further comprising receiving an infotainment signal from the avionics network and causing the display of the head-mounted display to display imagery based on the infotainment signal.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising recognizing a gesture of a hand within the first video signal, wherein the gesture comprises pointing to the aircraft safety device, wherein the second video signal of the aircraft environment is generated in response to recognizing the gesture.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein the gesture of the hand within the first video signal is recognized by applying a hand tracking and segmentation algorithm to the first video signal.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the second video signal of the aircraft environment is generated in response to detecting the feature vector within the first video signal.

### Claim 14

An aircraft system comprising:
a head-mounted display within an aircraft environment, the head-mounted display including;
a frame;
at least one camera fixed to the frame and configured to generate a first video signal corresponding to images captured of the aircraft environment;
a head tracker fixed to the frame and configured to generate pose information indicative of a position and orientation of the at least one camera;
a display fixed to the frame and configured to generate an image;
an avionics network; and
a processor communicatively coupled to the head-mounted display by the avionics network, the processor configured to execute program instructions maintained on a memory causing the processor to:
receive the first video signal and the pose information from the head-mounted display by way of the avionics network;
detect a feature vector within the first video signal by digital image processing the first video signal using a feature detector algorithm, the feature vector including at least a first feature corresponding to an aircraft safety device including one of a life preserver stowage, an oxygen mask passenger service unit, a seatbelt, or an emergency exit door;
estimate a pose of the aircraft safety device within the first video signal based on the feature vector together with the pose information from the head tracker;
generate a second video signal by compositing the first video signal with one or more virtual objects corresponding to a safety instruction associated with the aircraft safety device, the one or more virtual objects positioned and oriented within the second video signal from a perspective of the head-mounted display based on the pose of the aircraft safety device; and
cause the display of the head-mounted display to display the second video signal.

### Claim 15 (depends on 14)

The aircraft system of claim 14, further comprising a line replaceable unit, the line replaceable unit including the processor.