# US 2023/0054918 A1 · Motion-based human video detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0054918 A1
- Document ID: US20230054918A1
- Status: Published application
- Kind: A1
- Application number: 17/980,715
- Filed: 2022-11-04
- Published: 2023-02-23
- Earliest claimed: 2019-10-22
- Assignees: ObjectVideo Labs, LLC
- Applicants: ObjectVideo Labs, LLC
- Inventors: Sima Taheri; Gang Qian; Sung Chun Lee; Sravanthi Bondugula; Allison Beach
- Web page: https://patentgrep.com/patent/US20230054918A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06T 7/254: involving subtraction of images
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher}
- G06V 10/82: using neural networks
- G06V 10/98: Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06N 3/045: Combinations of networks
- G06N 5/046: Forward inferencing; Production systems
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20224: Image subtraction
- G06T 2207/30196: Human being; Person
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and apparatus for motion-based human video detection are disclosed. A method includes generating a representation of a difference between two frames of a video; providing, to an object detector, a particular frame of the two frames and the representation of the difference between two frames of the video; receiving an indication that the object detector detected an object in the particular frame; determining that detection of the object in the particular frame was a false positive detection; determining an amount of motion energy where the object was detected in the particular frame; and training the object detector based on penalization of the false positive detection in accordance with the amount of motion energy where the object was detected in the particular frame.

## Claims

### Claim 1

(canceled)

### Claim 2

A computer-implemented method comprising:
determining, for two or more images captured by a camera, a first false positive detection of a first object and a corresponding first amount of motion energy for an area in the two or more images in which the first object was detected; and
assigning, for training an object detector, a penalty to the first false positive detection that is higher than a second penalty assigned to a second false positive detection with a higher motion energy than the first amount of motion energy.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the first false positive detection of the first object comprises:
determining whether a comparison of the first object and a ground truth satisfies a threshold criteria; and
in response to determining that the comparison does not satisfy the threshold criteria, determining the first false positive detection.

### Claim 4 (depends on 3)

The method of claim 3, wherein the ground truth comprises data indicating that at least one of i) the first object is not present in the two or more images, or ii) the first object is not a human.

### Claim 5 (depends on 2)

The method of claim 2, wherein determining the first false positive detection of the first object comprises:
generating a single-channel grayscale image using a difference between sequential images of the two or more images captured by the camera;
generating a four-channel image by combining the single-channel grayscale image and one of the two or more images captured by a camera; and
providing the four-channel image to the object detector to determine the first false positive detection.

### Claim 6 (depends on 5)

The method of claim 5, wherein the two or more images captured by the camera comprise three-channel RGB images.

### Claim 7 (depends on 5)

The method of claim 5, wherein the corresponding first amount of motion energy is determined using an average pixel value of pixels within the area in the single-channel grayscale image in which the first object was detected.

### Claim 8 (depends on 2)

The method of claim 2, comprising training the object detector by:
obtaining an image classifier trained on three-channel images;
generating the object detector by retraining the image classifier using four-channel images included in training data; and
training the object detector using the penalty.

### Claim 9 (depends on 2)

The method of claim 2, wherein the penalty assigned to the first false positive detection is inversely proportional to the corresponding first amount of motion energy for the area in the two or more images in which the first object was detected.

### Claim 10 (depends on 2)

The method of claim 2, comprising:
generating a representation of a difference between another two or more images captured by the camera;
providing, to the object detector, a particular image of the other two or more images and the representation of the difference between the other two or more images captured by the camera;
receiving an indication that the object detector detected an object in the particular image; and
in response to receiving the indication that the object detector detected an object in the particular image, performing one or more actions.

### Claim 11

A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
determining, for two or more images captured by a camera, a first false positive detection of a first object and a corresponding first amount of motion energy for an area in the two or more images in which the first object was detected; and
assigning, for training an object detector, a penalty to the first false positive detection that is higher than a second penalty assigned to a second false positive detection with a higher motion energy than the first amount of motion energy.

### Claim 12 (depends on 11)

The system of claim 11, wherein determining the first false positive detection of the first object comprises:
determining whether a comparison of the first object and a ground truth satisfies a threshold criteria; and
in response to determining that the comparison does not satisfy the threshold criteria, determining the first false positive detection.

### Claim 13 (depends on 12)

The system of claim 12, wherein the ground truth comprises data indicating that at least one of i) the first object is not present in the two or more images, or ii) the first object is not a human.

### Claim 14 (depends on 11)

The system of claim 11, wherein determining the first false positive detection of the first object comprises:
generating a single-channel grayscale image using a difference between sequential images of the two or more images captured by the camera;
generating a four-channel image by combining the single-channel grayscale image and one of the two or more images captured by a camera; and
providing the four-channel image to the object detector to determine the first false positive detection.

### Claim 15 (depends on 14)

The system of claim 14, wherein the two or more images captured by the camera comprise three-channel RGB images.

### Claim 16 (depends on 14)

The system of claim 14, wherein the corresponding first amount of motion energy is determined using an average pixel value of pixels within the area in the single-channel grayscale image in which the first object was detected.

### Claim 17 (depends on 11)

The system of claim 11, the operations comprising training the object detector by:
obtaining an image classifier trained on three-channel images;
generating the object detector by retraining the image classifier using four-channel images included in training data; and
training the object detector using the penalty.

### Claim 18 (depends on 11)

The system of claim 11, wherein the penalty assigned to the first false positive detection is inversely proportional to the corresponding first amount of motion energy for the area in the two or more images in which the first object was detected.

### Claim 19 (depends on 11)

The system of claim 11, the operations comprising:
generating a representation of a difference between another two or more images captured by the camera;
providing, to the object detector, a particular image of the other two or more images and the representation of the difference between the other two or more images captured by the camera;
receiving an indication that the object detector detected an object in the particular image; and
in response to receiving the indication that the object detector detected an object in the particular image, performing one or more actions.

### Claim 20

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
determining, for two or more images captured by a camera, a first false positive detection of a first object and a corresponding first amount of motion energy for an area in the two or more images in which the first object was detected; and
assigning, for training an object detector, a penalty to the first false positive detection that is higher than a second penalty assigned to a second false positive detection with a higher motion energy than the first amount of motion energy.

### Claim 21 (depends on 20)

The computer storage media of claim 20, wherein determining the first false positive detection of the first object comprises:
determining whether a comparison of the first object and a ground truth satisfies a threshold criteria; and
in response to determining that the comparison does not satisfy the threshold criteria, determining the first false positive detection.