# US 2026/0019658 A1 · Multimodal Multimedia Processing For Wearable Device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0019658 A1
- Document ID: US20260019658A1
- Status: Published application
- Kind: A1
- Application number: 18/772,870
- Filed: 2024-07-15
- Published: 2026-01-15
- Earliest claimed: 2024-07-15
- Assignees: Zepp, Inc.
- Applicants: Zepp, Inc.
- Inventors: Wang Huang; Kongqiao Wang
- Web page: https://patentgrep.com/patent/US20260019658A1

## CPC

- G16H 30/40: for processing medical images, e.g. editing
- H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/8547: involving timestamps for synchronizing content
- G16H 20/30: relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of multimodal multimedia processing for at least one wearable device comprising an image sensor and at least one processor. The method includes in response to the image sensor being turned on, obtaining a first measurement of at least one of a physiological parameter of an individual or an environmental parameter captured by the at least one wearable device in a vicinity of the individual; determining whether a triggering event has occurred based on the first measurement, the triggering event associated with generating tagging information based on the first measurement using a machine learning model adapted to run on the at least one processor; and in response to determining that the triggering event has occurred, editing a selected clip from a multimedia stream currently being captured by the image sensor to include the tagging information generated based on the first measurement at a corresponding timestamp.

## Claims

### Claim 1

A method of multimodal multimedia processing for at least one wearable device comprising an image sensor and at least one processor, the method comprising:
in response to the image sensor of the at least one wearable device being turned on, obtaining, by the at least one wearable device, a first measurement of at least one of a physiological parameter of an individual or an environmental parameter captured by the at least one wearable device in a vicinity of the individual;
determining, by the at least one processor of the at least one wearable device, whether a triggering event has occurred based on the first measurement, the triggering event associated with generating tagging information based on the first measurement for the image sensor of the at least one wearable device using a machine learning model adapted to run on the at least one processor of the at least one wearable device; and
in response to determining that the triggering event has occurred, editing, by the at least one processor of the at least one wearable device, a selected clip from a multimedia stream currently being captured by the image sensor of the at least one wearable device to include the tagging information generated based on the first measurement at a corresponding timestamp.

### Claim 2 (depends on 1)

The method of claim 1, wherein the at least one wearable device comprises the image sensor and a second sensor, wherein the second sensor performs multimodal cooperation with the image sensor.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining, by the at least one processor of the at least one wearable device, whether the triggering event has occurred based on the first measurement further comprises:
obtaining, by the second sensor, a second measurement of at least one of another physiological parameter of the individual or another environmental parameter captured by the second sensor in the vicinity of the individual; and
determining, by the at least one processor of the at least one wearable device, whether the triggering event has occurred based on the first measurement and the second measurement.

### Claim 4 (depends on 1)

The method of claim 1, wherein the physiological parameter comprises at least one of: heart rate, heart rate variability, blood pressure, blood glucose level, body temperature, or respiration rate.

### Claim 5 (depends on 1)

The method of claim 1, wherein the environmental parameter comprises at least one of: altitude, GPS location, ambient temperature, ambient humidity, ambient noise index, or an environmental pollution index.

### Claim 6 (depends on 1)

The method of claim 1, wherein the tagging information comprises the first measurement, information extracted from the multimedia stream, and the corresponding timestamp.

### Claim 7 (depends on 1)

The method of claim 1, wherein the selected clip is to be analyzed with other tagged clips to determine a personalized multimedia lifelog entry.

### Claim 8 (depends on 7)

The method of claim 7, wherein the machine learning model adapted to run on the at least one processor of the at least one wearable device comprises a first large language model customized for the individual and adapted to run on the at least one processor of the at least one wearable device, the method further comprising:
sending the selected clip to a server, wherein the selected clip including the tagging information is analyzed to update the personalized multimedia lifelog entry using a second large language model and an expert knowledge base interacting with the second large language model.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
generating, by the at least one processor of the at least one wearable device, an instruction to direct the image sensor of the at least one wearable device to switch to perform a task based on the first measurement, wherein the task is generated using at least one of the first large language model or the second large language model.

### Claim 10 (depends on 9)

The method of claim 9, wherein the task to be performed by the image sensor of the at least one wearable device comprises at least one of: updating a frame rate of the multimedia stream currently being captured, or taking a high-resolution still photo.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
generating, by the at least one processor of the at least one wearable device, an instruction to provide a recommendation to the individual based on the first measurement and the personalized multimedia lifelog entry for the individual using at least one of the first large language model or the second large language model.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
detecting, by the at least one processor of the at least one wearable device, at least one object from the selected clip based on the tagging information; and
determining a task associated with the at least one object using the machine learning model.

### Claim 13 (depends on 12)

The method of claim 12, wherein a parameter derived from the first measurement is used to determine a type of the at least one object in the task.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
transmitting, by the at least one wearable device to a recipient monitoring the triggering event for the individual, the selected clip edited to include the tagging information with an alert.

### Claim 15

A wearable device for multimodal multimedia processing, comprising:
an image sensor;
a non-transitory memory; and
at least one processor configured to execute instructions stored in the non-transitory memory to:
in response to the image sensor of the wearable device being turned on, obtain a first measurement of at least one of a physiological parameter of an individual or an environmental parameter captured by the wearable device in a vicinity of the individual;
determine whether a triggering event has occurred based on the first measurement, the triggering event associated with generating tagging information based on the first measurement for the image sensor using a machine learning model adapted to run on the at least one processor; and
in response to determining that the triggering event has occurred, edit a selected clip from a multimedia stream currently being captured by the image sensor to include the tagging information generated based on the first measurement at a corresponding timestamp.

### Claim 16 (depends on 15)

The wearable device of claim 15, further comprising a second sensor, wherein the second sensor performs multimodal cooperation with the image sensor, and the instructions to determine whether a triggering event has occurred based on the first measurement comprise instructions to:
obtain, by the second sensor, a second measurement of at least one of another physiological parameter of the individual or another environmental parameter captured by the second sensor in the vicinity of the individual; and
determine, by the at least one processor, whether the triggering event has occurred based on the first measurement and the second measurement.

### Claim 17 (depends on 15)

The wearable device of claim 15, wherein the physiological parameter comprises at least one of: heart rate, heart rate variability, blood pressure, blood glucose level, body temperature, or respiration rate, and the environmental parameter comprises at least one of: altitude, GPS location, ambient temperature, ambient humidity, ambient noise index, or an environmental pollution index, and the tagging information comprises the first measurement, information extracted from the multimedia stream, and the corresponding timestamp.

### Claim 18 (depends on 15)

The wearable device of claim 15, wherein the selected clip is to be analyzed with other tagged clips to determine a personalized multimedia lifelog entry, and the machine learning model adapted to run on the at least one processor comprises a first large language model customized for the individual and adapted to run on the at least one processor, and the instructions stored in the non-transitory memory further comprise instructions to:
send the selected clip to a server, wherein the selected clip including the tagging information is analyzed to update the personalized multimedia lifelog entry using a second large language model and an expert knowledge base interacting with the second large language model.

### Claim 19 (depends on 18)

The wearable device of claim 18, wherein the instructions stored in the non-transitory memory further comprise instructions to:
generate an instruction to direct the image sensor to switch to perform a task based on the first measurement, wherein the task is generated using at least one of the first large language model or the second large language model; or
generate an instruction to provide a recommendation to the individual based on the first measurement and the personalized multimedia lifelog entry for the individual using at least one of the first large language model or the second large language model.

### Claim 20 (depends on 1)

A non-transitory computer-readable storage medium configured to store computer programs for multimodal multimedia processing using at least one wearable device, the computer programs comprising instructions executable by at least one processor to perform the method of claim 1.
