# US 2023/0274536 A1 · Augmentation method of visual training data for behavior detection machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274536 A1
- Document ID: US20230274536A1
- Status: Published application
- Kind: A1
- Application number: 17/681,786
- Filed: 2022-02-27
- Published: 2023-08-31
- Earliest claimed: 2022-02-27
- Assignees: Viisights Solutions Ltd.
- Applicants: Viisights Solutions Ltd.
- Inventors: Simon POLAK; Shiri GORDON; Menashe ROTHSCHILD; Asaf BIRENZVIEG
- Web page: https://patentgrep.com/patent/US20230274536A1

## CPC

- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/254: involving subtraction of images
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Disclosed herein are methods and systems for training machine learning (ML) models to classify activity of objects, comprising selecting a set of frames depicting one or more objects from one or more video sequences each comprising a plurality of consecutive frames, associating the object(s) with each pixel included in a bounding box of the object(s) identified in each frame of the set, computing a motion mask for each frame of the set indicating whether each pixel associated with the object(s) in the frame is changed or unchanged compared to a corresponding pixel in a preceding frame, augmenting an image of the object(s) in each frame of a subset of frames of the set to depict only the changed pixels associated with the object(s) by cutting out the unchanged pixels, and training one or more ML models, using the set of frames, to classify one or more activities of the object(s).

## Claims

### Claim 1

A computer implemented method of training machine learning (ML) models to classify activity of objects, comprising:
using at least one processor for:
selecting a set of frames depicting at least one object from at least one video sequence comprising a plurality of consecutive frames;
associating the at least one object with each of a plurality of pixels included in a bounding box of the at least one object identified in each of the frames of the set;
computing a motion mask for each frame of the set indicating whether each pixel associated with the at least one object in the respective frame is changed or unchanged compared to a corresponding pixel in a preceding frame of the set;
augmenting an image of the at least one object in each frame of a subset of frames selected from the set of frames to depict only the changed pixels associated with the at least one object by cutting out the unchanged pixels associated with the at least one object; and
training at least one ML model, using the set of frames, to classify at least one activity of the at least one object.

### Claim 2 (depends on 1)

The computer implemented method of claim 1, wherein the at least one activity is a member of a group consisting of: an action conducted by the at least one object, an activity the at least one object is engaged in, an interaction of the at least one object with at least one another object, an event in which the at least one object participates, and a behavior of the at least one object.

### Claim 3 (depends on 1)

The computer implemented method of claim 1, wherein the bounding box of the at least one object is identified using at least one visual analysis algorithm.

### Claim 4 (depends on 1)

The computer implemented method of claim 1, wherein the bounding box of the at least one object is defined manually by at least one annotator.

### Claim 5 (depends on 1)

The computer implemented method of claim 1, wherein each of the changed pixels is a member of a group comprising: a moved pixel, a displaced pixel and a transformed pixel.

### Claim 6 (depends on 1)

The computer implemented method of claim 1, wherein the changed and unchanged pixels in each frame compared to its preceding frame are identified using at least one motion detection algorithm.

### Claim 7 (depends on 1)

The computer implemented method of claim 1, wherein the subset of frames is selected from the set by computing a random value for each frame of the set using at least one random number module, and including the respective frame in the subset in case the random value computed for the respective frame does not exceed a predefined probability threshold.

### Claim 8 (depends on 1)

The computer implemented method of claim 1, wherein cutting out the unchanged pixels comprises blackening each unchanged pixel associated with the at least one object.

### Claim 9 (depends on 1)

The computer implemented method of claim 1, further comprising augmenting independently a plurality of subsets of images for a plurality of objects, each of the plurality of subsets comprises images of a respective set of frames selected from the plurality of consecutive frames for a respective one of the plurality of objects.

### Claim 10

A system for training machine learning (ML) models to classify activity of objects, comprising:
at least one processor adapted to execute a code, the code comprising:
code instructions to select a set of frames depicting at least one object from at least one video sequence comprising a plurality of consecutive frames;
code instructions to associate the at least one object with each of a plurality of pixels included in a bounding box of the at least one object identified in each of the frames of the set;
code instructions to compute a motion mask for each frame of the set indicating whether each pixel associated with the at least one object in the respective frame is changed or unchanged compared to a corresponding pixel in a preceding frame of the set;
code instructions to augment an image of the at least one object in each frame of a subset of frames selected from the set of frames to depict only the changed pixels associated with the at least one object by cutting out the unchanged pixels associated with the at least one object; and
code instructions to train at least one ML model, using the set of frames, to classify at least one activity of the at least one object.

### Claim 11 (depends on 10)

The system of claim 10, wherein the at least one activity is a member of a group consisting of: an action conducted by the at least one object, an activity the at least one object is engaged in, an interaction of the at least one object with at least one another object, an event in which the at least one object participates, and a behavior of the at least one object.

### Claim 12 (depends on 10)

The system of claim 10, wherein the bounding box of the at least one object is created using at least one visual analysis algorithm.

### Claim 13 (depends on 10)

The system of claim 10, wherein the bounding box of the at least one object is defined manually by at least one annotator.

### Claim 14 (depends on 10)

The system of claim 10, wherein each of the changed pixels is a member of a group comprising: a moved pixel, a displaced pixel and a transformed pixel.

### Claim 15 (depends on 10)

The system of claim 10, wherein the changed and unchanged pixels in each frame compared to its preceding frame are identified using at least one motion detection algorithm.

### Claim 16 (depends on 10)

The system of claim 10, wherein the subset of frames is selected from the set by computing a random value for each frame of the set using at least one random number module, and including the respective frame in the subset in case the random value computed for the respective frame does not exceed a predefined probability threshold.

### Claim 17 (depends on 10)

The system of claim 10, wherein cutting out the unchanged pixels comprises blackening each unchanged pixel associated with the at least one object.

### Claim 18 (depends on 10)

The system of claim 10, wherein the code further comprises code instructions to augment independently a plurality of subsets of frames for a plurality of objects, each of the plurality of subsets comprises images of a respective set of frames selected from the plurality of consecutive frames for a respective one of the plurality of objects.

### Claim 19

A computer implemented method of using a trained machine learning (ML) model to classify activity of objects, comprising:
using at least one processor for:
receiving at least one frame depicting at least one object;
applying at least one trained ML model to the at least one frame to classify activity of the at least one object, the at least one ML model is trained using a plurality of consecutive frames of a video sequence depicting the at least one object, wherein the at least one object is augmented in a subset of frames of the plurality of consecutive frames by cutting out from each of the frames of the subset pixels associated with the at least one object which are unchanged compared to corresponding pixels in a preceding frame;
receiving a classification score for at least one activity label estimated by the at least one ML model for an activity of the at least one object; and
outputting the at least one activity label in case the classification score exceeds a certain classification confidence threshold.

### Claim 20

A system for using a trained machine learning (ML) model to classify activity of objects, comprising:
at least one processor adapted to execute a code, the code comprising:
code instructions to receive at least one frame depicting at least one object;
code instructions to apply at least one trained ML model to the at least one frame to classify activity of the at least one object, the at least one ML model is trained using a plurality of consecutive frames of a video sequence depicting the at least one object, wherein an image of the at least one object is augmented in a subset of frames of the plurality of consecutive frames by cutting out from each of the frames of the subset pixels associated with the at least one object which are unchanged compared to corresponding pixels in a preceding frame;
code instructions to receive a classification score for at least one activity label estimated by the at least one ML model for an activity of the at least one object; and
code instructions to output the at least one activity label in case the classification score exceeds a certain classification confidence threshold.
