# US 2023/0274549 A1 · Systems and Methods for Video Event Segmentation Derived from Simultaneously Recorded Sensor Data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274549 A1
- Document ID: US20230274549A1
- Status: Published application
- Kind: A1
- Application number: 17/683,142
- Filed: 2022-02-28
- Published: 2023-08-31
- Earliest claimed: 2022-02-28
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Company, Ltd.
- Inventors: Kathleen Sofia Hajash; Thomas Brenner; Brian Harms; Catherine S. Kim; Imran Mohammed
- Web page: https://patentgrep.com/patent/US20230274549A1

## CPC

- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/44: {Event detection}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one embodiment, a method includes, by an electronic device, accessing activity data containing one or more non-image-based sensor signals from a first wearable device, where the activity data corresponds to an activity a user performs during a first timeframe, accessing from a first camera device, one or more cameras of the first camera device, where the video data corresponds to the first activity of the first user during the first timeframe, segmenting the activity data based on one or more features of the one or more non-image-based sensor signals to identify one or more segments of activity data corresponding to a second timeframe, classifying the one or more segments of the video data based on the one or more identified events associated with the first activity during the second timeframe, classifying the segments of the video data based on the one or more events during the second timeframe.

## Claims

### Claim 1

A method comprising, by an electronic device:
accessing, from a first wearable device on a first user, activity data comprising one or more non-image-based sensor signals from one or more sensors of the first wearable device, wherein activity data correspond to a first activity of the first user during a first timeframe;
accessing, from a first camera device, video data from one or more cameras of the first camera device, wherein the video data corresponds to the first activity of the first user during the first timeframe;
segmenting the activity data based on one or more features of the one or more non-image-based sensor signals to identify one or more segments of the activity data, wherein the one or more segments of activity data correspond to one or more second timeframes within the first timeframe, respectively;
classifying each of the one or more segments of activity data based on the one or more non-image-based signals to identify one or more events associated with the first activity of the first user during the second timeframe corresponding to the respective segment of activity data; and
classifying one or more segments of the video data based on the identified one or more events, wherein the one or more segments of video data correspond to the one or more segments of activity data, respectively, during the second timeframe corresponding to the respective segment of activity data.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
aligning the activity data and the video data based on one or more of timestamps, audio elements, visual elements, or sensor signal elements.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
determining the first activity from a plurality of activities based on the one or more non-image-based sensor signals.

### Claim 4 (depends on 3)

The method of claim 3, wherein the first activity is further determined based on one or more of a global positioning system (GPS) location, a date, a time, a temperature, or a previous activity.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first activity is selected from a plurality of hierarchically organized activities, wherein each of the plurality of activities is associated with a respective subclass of one or more actions or one or more events, and wherein the identified one or more events associated with the first activity of the first user are selected from a subclass of one or more actions associated with the first activity.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
receiving a user input specifying the first activity from a plurality of activities.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
providing instructions for presenting a user interface comprising the one or more segments of the video data, wherein the user interface comprises one or more activatable elements for filtering the one or more segments of the video data based on the one or more events associated with the respective segments of the video data.

### Claim 8

An electronic device comprising:
one or more displays;
one or more sensors
one or more non-transitory computer-readable storage media including instructions; and
one or more processors coupled to the storage media, the one or more processors configured to execute the instructions to:
access, from a first wearable device on a first user, activity data comprising one or more non-image-based sensor signals from one or more sensors of the first wearable device, wherein activity data correspond to a first activity of the first user during a first timeframe;
access, from a first camera device, video data from one or more cameras of the first camera device, wherein the video data corresponds to the first activity of the first user during the first timeframe;
segment the activity data based on one or more features of the one or more non-image-based sensor signals to identify one or more segments of the activity data, wherein the one or more segments of activity data correspond to one or more second timeframes within the first timeframe, respectively;
classify each of the one or more segments of activity data based on the one or more non-image-based sensor signals to identify one or more events associated with the first activity of the first user during the second timeframe corresponding to the respective segment of activity data; and
classify one or more segments of the video data based on the identified one or more events, wherein the one or more segments of video data correspond to the one or more segments of activity data, respectively, during the second timeframe corresponding to the respective segment of activity data.

### Claim 9 (depends on 8)

The electronic device of claim 8, wherein the processors are further configured to execute instructions to:
align the activity data and the video data based on one or more of timestamps, audio elements, visual elements, or sensor signal elements.

### Claim 10 (depends on 8)

The electronic device of claim 8, wherein the processors are further configured to execute instructions to:
determine the first activity from a plurality of activities based on the one or more non-image-based sensor signals.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein the first activity is further determined based on one or more of a global positioning system (GPS) location, a date, a time, a temperature, or a previous activity.

### Claim 12 (depends on 8)

The electronic device of claim 8, wherein the first activity is selected from a plurality of hierarchically organized activities, wherein each of the plurality of activities is associated with a respective subclass of one or more actions or one or more events, and wherein the identified one or more events associated with the first activity of the first user are selected from a subclass of one or more actions associated with the first activity.

### Claim 13 (depends on 8)

The electronic device of claim 8, wherein the processors are further configured to execute instructions to:
receive a user input specifying the first activity from a plurality of activities.

### Claim 14 (depends on 8)

The electronic device of claim 8, wherein the processors are further configured to execute instructions to:
provide instructions for presenting a user interface comprising the one or more segments of the video data, wherein the user interface comprises one or more activatable elements for filtering the one or more segments of the video data based on the one or more events associated with the respective segments of the video data.

### Claim 15

A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of an electronic device, cause the one or more processors to:
access, from a first wearable device on a first user, activity data comprising one or more signals from one or more sensors of the first wearable device, wherein activity data correspond to a first activity of the first user during a first timeframe;
access, from a first camera device, video data from one or more cameras of the first camera device, wherein the video data corresponds to the first activity of the first user during the first timeframe;
segment the activity data based on one or more features of the one or more non-image-based sensor signals to identify one or more segments of the activity data, wherein the one or more segments of activity data correspond to one or more second timeframes within the first timeframe, respectively;
classify each of the one or more segments of activity data based on the one or more non-image-based sensor signals to identify one or more events associated with the first activity of the first user during the second timeframe corresponding to the respective segment of activity data; and
classify one or more segments of the video data based on the identified one or more events, wherein the one or more segments of video data correspond to the one or more segments of activity data, respectively, during the second timeframe corresponding to the respective segment of activity data.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the instructions further comprises instructions to:
align the one or more non-image-based sensor signals and the video data based at least on a time stamp associated with a particular segment of activity data, audio signal, visual detection of an event, or sensor signal analysis.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the instructions further comprises instructions to:
determine the first activity from a plurality of activities based on the one or more non-image-based sensor signals.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the first activity is further determined based on one or more of a global positioning system (GPS) location, a date, a time, a temperature, or a previous activity.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the first activity is selected from a plurality of hierarchically organized activities, wherein each of the plurality of activities is associated with a respective subclass of one or more actions or one or more events, and wherein the identified one or more events associated with the first activity of the first user are selected from a subclass of one or more actions associated with the first activity.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the instructions further comprises instructions to:
receive a user input specifying the first activity from a plurality of activities.
