# US 2023/0066765 A1 · Controlling devices in operations areas based on environmental states detected using machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0066765 A1
- Document ID: US20230066765A1
- Status: Published application
- Kind: A1
- Application number: 17/458,076
- Filed: 2021-08-26
- Published: 2023-03-02
- Earliest claimed: 2021-08-26
- Assignees: JBT AeroTech Corporation
- Applicants: JBT AeroTech Corporation
- Inventors: Grant Thomas; Stephen C. Tatton
- Web page: https://patentgrep.com/patent/US20230066765A1

## CPC

- G06T 7/0008: {checking presence/absence}
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- B64F 5/60: Testing or inspecting aircraft components or systems
- G06N 20/00: Machine learning
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/50: Context or environment of the image
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G08B 3/10: using electric transmission; using electromagnetic transmission
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B64F: GROUND OR AIRCRAFT-CARRIER-DECK INSTALLATIONS SPECIALLY ADAPTED FOR USE IN CONNECTION WITH AIRCRAFT; DESIGNING, MANUFACTURING, ASSEMBLING, CLEANING, MAINTAINING OR REPAIRING AIRCRAFT, NOT OTHERWISE PROVIDED FOR; HANDLING, TRANSPORTING, TESTING OR INSPECTING AIRCRAFT COMPONENTS, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

In some embodiments, a system is provided that includes an edge computing device and at least one camera configured to obtain image data depicting at least a portion of an operations area. The edge computing device includes a non-transitory computer-readable medium that has a model data store and computer-executable instructions stored thereon. The instructions cause the edge computing device to perform actions including receiving at least one image from the at least one camera; processing the at least one image using at least one machine learning model stored in the model data store to determine at least one environmental state within the operations area; and controlling a device based on the determined at least one environmental state. The machine learning model is trained by a model management computing system that obtains training data via low-bandwidth connections to edge computing devices.

## Claims

### Claim 1

A system, comprising:
at least one camera configured to obtain image data depicting at least a portion of an operations area; and
an edge computing device having at least one processor and a non-transitory computer-readable medium;
wherein the edge computing device is communicatively coupled to the at least one camera;
wherein the non-transitory computer-readable medium has a model data store and computer-executable instructions stored thereon; and
wherein the instructions, in response to execution by the at least one processor, cause the edge computing device to perform actions comprising:
receiving, by the edge computing device from a model management computing system, at least one machine learning model;
storing, by the edge computing device, the at least one machine learning model in the model data store;
receiving, by the edge computing device, at least one image from the at least one camera;
processing, by the edge computing device, the at least one image using the at least one machine learning model to determine at least one environmental state within the operations area; and
controlling, by the edge computing device, a device based on the determined at least one environmental state.

### Claim 2 (depends on 1)

The system of claim 1, wherein determining the at least one environmental state includes determining whether one or more of a subject apparatus and one or more servicing apparatuses are present in the operations area.

### Claim 3 (depends on 2)

The system of claim 2, wherein determining the at least one environmental state includes determining a state of at least one of the subject apparatus and the one or more servicing apparatuses.

### Claim 4 (depends on 3)

The system of claim 3, wherein the subject apparatus is an aircraft.

### Claim 5 (depends on 4)

The system of claim 4, wherein the one or more servicing apparatuses includes an auto level wheel of a passenger boarding bridge.

### Claim 6 (depends on 5)

The system of claim 5, wherein determining the state of at least one of the subject apparatus and the one or more servicing apparatuses includes:
determining whether a door of the aircraft is open; and
determining whether the auto level wheel is extended.

### Claim 7 (depends on 6)

The system of claim 6, wherein controlling the device based on the determined at least one environmental state includes, in response to determining that the aircraft is present, the door is open, and the auto level wheel is not engaged, controlling the device to present an alert.

### Claim 8 (depends on 7)

The system of claim 7, wherein the device includes a loudspeaker, and wherein controlling the device to present the alert includes controlling the device to present the alert via the loudspeaker.

### Claim 9 (depends on 7)

The system of claim 7, wherein the device includes an Aircraft Communications Addressing and Reporting System (ACARS) transmitter, and wherein controlling the device to present the alert includes controlling the device to transmit the alert via the ACARS transmitter.

### Claim 10

A method of collecting training data and training one or more machine learning models for use on edge computing devices, the method comprising:
receiving, by a model management computing system from one or more edge computing devices via low-bandwidth connections, combined data streams including one or more images;
processing, by the model management computing system, the combined data streams to obtain the one or more images;
for each image of the one or more images:
receiving, by the model management computing system, one or more annotations of one or more objects within the image; and
adding, by the model management computing system, the one or more annotations and the image to a training data set;
training, by the model management computing system, one or more machine learning models using the training data set to generate new annotations of new images; and
transmitting, by the model management computing system, the trained one or more machine learning models to one or more edge computing devices.

### Claim 11 (depends on 10)

The method of claim 10, wherein the one or more annotations indicate at least one of a presence of a piece of equipment and a status of a piece of equipment.

### Claim 12 (depends on 11)

The method of claim 11, wherein the piece of equipment is an aircraft, an aircraft door, an auto level wheel, or a de-icing machine.

### Claim 13 (depends on 10)

The method of claim 10, wherein processing the combined data streams to obtain the one or more images includes, for a given combined data stream:
extracting a set of low-bandwidth video streams in a bandwidth-capped codec from the given combined data stream;
transcoding the low-bandwidth video streams from the bandwidth-capped codec to a variable-bandwidth codec to create a set of recovered video streams; and
extracting the one or more images from the set of recovered video streams.

### Claim 14 (depends on 13)

The method of claim 13, wherein processing the combined data streams to obtain the one or more images further includes:
segmenting at least one predetermined portion of at least one of the one or more images to obtain at least one segmented image; and
providing the at least one segmented image for inclusion in the one or more images.

### Claim 15 (depends on 14)

The method of claim 14, wherein receiving one or more annotations of one or more objects within the image includes, for each predetermined portion and each segmented image associated with the predetermined portion, receiving one or more segment annotations of one or more objects within the segmented image.

### Claim 16 (depends on 15)

The method of claim 15, further comprising adding the one or more segment annotations and the segmented image associated with the predetermined portion to a segment training data set associated with the predetermined portion.

### Claim 17 (depends on 16)

The method of claim 16, further comprising training a segment machine learning model for each predetermined portion using the segment training data set associated with each predetermined portion.

### Claim 18 (depends on 10)

The method of claim 10, further comprising:
receiving, by an edge computing device, a high-bandwidth video stream at a first bandwidth in a variable-bandwidth codec from a camera;
downsampling, by the edge computing device, the high-bandwidth video stream to a second bandwidth;
transcoding, by the edge computing device, the downsampled high-bandwidth video stream to a bandwidth-capped codec to create a low-bandwidth video stream; and
transmitting, by the edge computing device, the low-bandwidth video stream to the model management computing system.

### Claim 19 (depends on 18)

The method of claim 18, wherein transmitting the low-bandwidth video stream to the model management computing system includes:
combining, by the edge computing device, the low-bandwidth video stream with at least one other low-bandwidth video stream to create a combined data stream; and
transmitting, by the edge computing device, the combined data stream to the model management computing system via a low-bandwidth connection.

### Claim 20 (depends on 19)

The method of claim 19, wherein transmitting the low-bandwidth video stream to the model management computing system further includes:
receiving, by the edge computing device, an auxiliary data stream from an equipment device; and
adding, by the edge computing device, the auxiliary data stream to the combined data stream.
