# US 2026/0073322 A1 · Method and apparatus for processing personnel workload state based on multimodal data, and device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0073322 A1
- Document ID: US20260073322A1
- Status: Published application
- Kind: A1
- Application number: 19/390,557
- Filed: 2025-11-15
- Published: 2026-03-12
- Earliest claimed: 2023-12-28
- Assignees: Kingfar International Inc.
- Applicants: Kingfar International Inc.
- Inventors: Qichao ZHAO; Ran YANG; Qingju WANG
- Web page: https://patentgrep.com/patent/US20260073322A1

## CPC

- G06Q 10/063114: {Status monitoring or status determination for a person or group}
- A61B 5/18: for vehicle drivers {or machine operators}
- G06V 10/806: {of extracted features}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Provided are a method, and an apparatus for processing a personnel workload state based on multimodal data, and a device. The method includes: acquiring state information of a first user; determining workload information of the first user according to the acquired state information, the state information includes at least two of: physiological information, eye movement information, electroencephalogram information, brain function imaging information, motion capture information, spatiotemporal acquisition information, behavior acquisition information, and facial expression and state information; and feeding back the workload information to a first management account, and formulating a training scheme matching the workload information. According to the embodiments of the present disclosure, the workload of the user can be recognized based on the multimodal data of the user.

## Claims

### Claim 1

A method for processing a workload state based on multimodal data, comprising:
acquiring state information of a first user;
determining workload information of the first user based on the acquired state information, wherein the state information comprises at least two of: physiological information, eye movement information, electroencephalogram information, brain function imaging information, motion capture information, spatiotemporal acquisition information, behavior acquisition information, and facial expression and state information; and
feeding back the workload information to a first management account, and formulating a training scheme matching the workload information.

### Claim 2 (depends on 1)

The method according to claim 1, wherein when the state information of the first user comprises a first electroencephalogram signal and a first near-infrared brain function imaging signal of the first user, said determining workload information of the first user based on the acquired state information comprises:
respectively extracting a first feature of the first electroencephalogram signal and a second feature of the first near-infrared brain function imaging signal; and
determining the workload information of the first user based on the first feature and the second feature.

### Claim 3 (depends on 2)

The method according to claim 2, wherein said determining the workload information of the first user based on the first feature and the second feature comprises:
fusing the first feature and the second feature to obtain a first fusion feature; and
inputting the first fusion feature into a preset workload recognition model, and taking workload information output by the workload recognition model as the workload information of the first user.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the first feature comprises a first temporal feature and a first image feature, and the second feature comprises a second temporal feature and a second image feature; and
said fusing the first feature and the second feature to obtain a first fusion feature comprises:
fusing the first temporal feature and the second temporal feature to obtain a third temporal feature;
fusing the first image feature and the second image feature to obtain a third image feature; and
fusing the third temporal feature and the third image feature to obtain the first fusion feature.

### Claim 5 (depends on 4)

The method according to claim 4, wherein said fusing the first temporal feature and the second temporal feature to obtain a third temporal feature comprises:
converting the first temporal feature into a fourth image feature;
converting the second temporal feature into a fifth image feature; and
fusing the fourth image feature and the fifth image feature to obtain a sixth image feature, and taking the sixth image feature as the third temporal feature.

### Claim 6 (depends on 4)

The method according to claim 4, wherein the first temporal feature comprises: a time-domain feature, a frequency-domain feature, a time-frequency-domain feature, a nonlinear feature, and/or a brain function connection feature, and/or
wherein the first image feature comprises: a two-dimensional (2D) map and/or a three-dimensional (3D) dense connection network, and/or
wherein the second temporal feature comprises: concentrations of oxygenated and deoxygenated hemoglobin, a β-index of a single channel, and/or a Pearson correlation coefficient between channels, and/or
wherein the second image feature comprises: a 3D channel activation map, a correlation matrix formed between channels, and/or a brain network connection map.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the workload information comprises: a workload level and/or a workload type.

### Claim 8 (depends on 2)

The method according to claim 2, wherein the first feature comprises a first temporal feature and a first image feature, the second feature comprises a second temporal feature and a second image feature, and the workload information comprises a workload level and a workload type;
said determining the workload information of the first user based on the first feature and the second feature comprises:
fusing the first temporal feature and the second temporal feature to obtain a third temporal feature, and recognizing the workload level of the first user based on the third temporal feature; and
fusing the first image feature and the second image feature to obtain a third image feature, and recognizing the workload type of the first user based on the third image feature.

### Claim 9 (depends on 8)

The method according to claim 8, wherein said recognizing the workload level of the first user based on the third temporal feature comprises:
inputting the third temporal feature into a load level recognition model, and taking a workload level output by the load level recognition model as the workload level of the first user.

### Claim 10 (depends on 8)

The method according to claim 8, wherein said recognizing the workload type of the first user based on the third image feature comprises:
inputting the third image feature into a load type recognition model, and taking a workload type output by the load type recognition model as the workload type of the first user.

### Claim 11 (depends on 8)

The method according to claim 8, wherein the first temporal feature comprises: a time-domain feature, a frequency-domain feature, a time-frequency-domain feature, a nonlinear feature, and/or a brain function connection feature, and/or
wherein the first image feature comprises: a brain topographic map and/or a power spectrum topographic map, and/or
wherein the second temporal feature comprises: concentrations of oxygenated and deoxygenated hemoglobin, a β-index of a single channel, and/or a Pearson correlation coefficient between channels, and/or
wherein the second image feature comprises: a 3D channel activation map, a complex brain network, a default brain network, a correlation heat map, and/or a brain network connection map.

### Claim 12 (depends on 7)

The method according to claim 7, wherein the workload type comprises: auditory, visual, attention, executive, and/or planning.

### Claim 13 (depends on 4)

The method according to claim 4, wherein extracting the second image feature of the first near-infrared brain function imaging signal comprises:
extracting brain network maps with different functions from the first near-infrared brain function imaging signal as the second image feature.

### Claim 14 (depends on 7)

The method according to claim 7, further comprising:
performing a warning processing on user workload based on the workload information.

### Claim 15 (depends on 14)

The method according to claim 14, wherein said performing a warning processing on user workload based on the workload information comprises:
when the workload level in the workload information is not lower than a preset level, and the workload type in the workload information comprises a first type, alarming for the workload type of the first type.

### Claim 16 (depends on 14)

The method according to claim 14, wherein said performing a warning processing on user workload based on the workload information comprises:
when the workload level in the workload information is not lower than a preset level and the workload type in the workload information comprises a second type, determining a proportion of the second type among a plurality of workload information obtained within a preset first duration; and
when the proportion is not lower than a preset proportion threshold, alarming for the workload type of the second type.

### Claim 17 (depends on 1)

The method according to claim 1, wherein said formulating a training scheme matching the workload information comprises:
obtaining a training scheme corresponding to the workload information as the training scheme of the first user.

### Claim 18

An electronic device, comprising:
a processor, and a memory, wherein one or more computer programs are stored in the memory, the one or more computer programs comprise instructions, and the electronic device, when executed by the processor, is configured to:
acquire state information of a first user;
determine workload information of the first user based on the acquired state information, wherein the state information comprises at least two of: physiological information, eye movement information, electroencephalogram information, brain function imaging information, motion capture information, spatiotemporal acquisition information, behavior acquisition information, and facial expression and state information; and
feed back the workload information to a first management account, and formulate a training scheme matching the workload information.

### Claim 19

A non-transitory computer-readable storage medium on which a computer program is stored, wherein the computer program, when running on a computer, enable the computer to implement:
acquiring state information of a first user;
determining workload information of the first user based on the acquired state information, wherein the state information comprises at least two of: physiological information, eye movement information, electroencephalogram information, brain function imaging information, motion capture information, spatiotemporal acquisition information, behavior acquisition information, and facial expression and state information; and
feeding back the workload information to a first management account, and formulating a training scheme matching the workload information.
