# US 2021/0279527 A1 · Detecting content in a real-time video stream recorded by a detection unit

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279527 A1
- Document ID: US20210279527A1
- Status: Published application
- Kind: A1
- Application number: 16/809,543
- Filed: 2020-03-04
- Published: 2021-09-09
- Earliest claimed: 2020-03-04
- Assignees: Matroid, Inc.
- Applicants: Matroid, Inc.
- Inventors: REZA ZADEH; RYAN WONG; JOHN GODDARD; JIAHANG LI; STEVEN CHEN; XIAOYUN YANG
- Web page: https://patentgrep.com/patent/US20210279527A1

## CPC

- G06F 16/7837: {using objects detected or recognised in the video content}
- G06F 16/26: Visual data mining; Browsing structured data
- G06F 18/2178: {based on feedback of a supervisor}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/945: {User interactive design; Environments; Toolboxes}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06N 3/045: Combinations of networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Described herein are systems and methods that search videos and other media content to identify items, objects, faces, or other entities within the media content. Detectors identify objects within media content by, for instance, detecting a predetermined set of visual features corresponding to the objects. Detectors configured to identify an object can be trained using a machine learned model (e.g., a convolutional neural network) as applied to a set of example media content items that include the object. The systems comprise an integrated detection unit configured to record media content, identify preferred content, and communicate the identifications of preferred content for storage in a computationally efficient manner.

## Claims

### Claim 1

A detection unit comprising:
a camera configured to record live media content, the camera storing a detector configured to identify preferred content in frames of the recorded live media content; and
a non-transitory computer-readable storage medium storing executable computer instructions that, when executed by a hardware processor, are configured to cause the hardware processor to perform steps comprising:
applying the detector to the live media content recorded by the camera to identify the preferred content within the frames of the live media content;
responsive to identifying the preferred content in the frames of the live media content, communicating an alert to a user, the alert comprising timestamps describing when each frame of the live media content including the preferred content was recorded by the camera;
receiving a request from the user to define a first subset of the frames of the live media content including the preferred content occurring at specific timestamps of interest and to define a second subset of the frames of the live media content not including the preferred content;
responsive to receiving the request from the user, providing the first subset of the frames to a cloud server; and
storing the second subset of frames at the detection unit.

### Claim 2 (depends on 1)

The detection unit of claim 1, wherein the user uploads the detector for storage on the detection unit and selects preferred content for the detector to identify in frames of the recorded live media content.

### Claim 3 (depends on 1)

The detection unit of claim 1, wherein the detector is a neural network generated by the user to output a confidence score indicating a likelihood that the preferred content is present within a frame of media content, the detector trained based on a first set of positive images of the preferred content and a second set of negative images that exclude the preferred content.

### Claim 4 (depends on 1)

The detection unit of claim 1, wherein the user uploads a plurality of detectors for storage on the detection unit, each detector configured to output a confidence score indicating a likelihood that a unique preferred content is present within a frame of media content and trained based on a first set of positive images of the unique preferred content and a second set of negative images that excludes the unique preferred content.

### Claim 5 (depends on 4)

The detection unit of claim 4, wherein the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
for each detector of the plurality;
inputting live media content recorded by the camera to the detector to identify unique preferred content in the live media content;
responsive to identifying unique preferred content appearing in a frame the live media content, identifying a timestamp at which the frame was recorded by the camera;
generating an aggregate alert comprising timestamps at describing when each identified instance of unique preferred content was recorded by the camera; and
communicating the aggregate alert to the user.

### Claim 6 (depends on 1)

The detection unit of claim 1, wherein the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
receiving confirmation from the user designating no preferred content of interest was identified in the first subset of frames; and
responsive to receiving the confirmation, designating all frames of the live media content in the second subset of frames; and
storing the second subset of frames on the detection unit.

### Claim 7 (depends on 1)

The detection unit of claim 1, wherein the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
determining that the live media content includes no frames in which preferred content is present;
designating all frames of the live media content in the second subset of frames; and
storing the second subset of frames at the detection unit.

### Claim 8

A non-transitory computer-readable storage medium storing executable computer instructions that, when executed by a hardware processor, are configured to cause the hardware processor to perform steps comprising:
applying the detector to the live media content recorded by the camera to identify the preferred content within the frames of the live media content;
responsive to identifying the preferred content in the frames of the live media content, communicating an alert to a user, the alert comprising timestamps describing when each frame of the live media content including the preferred content was recorded by the camera;
receiving a request from the user to define a first subset of the frames of the live media content including the preferred content occurring at specific timestamps of interest and to define a second subset of the frames of the live media content not including the preferred content;
responsive to receiving the request from the user, providing the first subset of the frames to a cloud server; and
storing the second subset of frames at the detection unit.

### Claim 9 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein the user uploads the detector for storage on the detection unit and selects preferred content for the detector to identify in frames of the recorded live media content.

### Claim 10 (depends on 8)

The non-transitory computer readable storage medium of claim 8, wherein the detector is a neural network generated by the user to output a confidence score indicating a likelihood that the preferred content is present within a frame of media content, the detector trained based on a first set of positive images of the preferred content and a second set of negative images that excludes the preferred content.

### Claim 11 (depends on 8)

The non-transitory computer readable storage medium claim 8, wherein the user uploads a plurality of detectors for storage on the detection unit, each detector configured to output a confidence score indicating a likelihood that a unique preferred content is present within a frame of media content and trained based on a first set of positive images of the unique preferred content and a second set of negative images that exclude the unique preferred content.

### Claim 12 (depends on 11)

The non-transitory computer readable storage medium claim 11, wherein the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
for each detector of the plurality;
inputting live media content recorded by the camera to the detector to identify unique preferred content in the live media content;
responsive to identifying unique preferred content appearing in a frame the live media content, identifying a timestamp at which the frame was recorded by the camera;
generating an aggregate alert comprising timestamps at describing when each identified instance of unique preferred content was recorded by the camera; and
communicating the aggregate alert to the user.

### Claim 13 (depends on 8)

The non-transitory computer readable storage medium claim 8, wherein the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
receiving confirmation from the user designating no preferred content of interest was identified in the first subset of frames; and
responsive to receiving the confirmation, designating all frames of the live media content in the second subset of frames; and
storing the second subset of frames on the detection unit.

### Claim 14 (depends on 8)

The non-transitory computer readable storage medium claim 8, wherein the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
determining that the live media content includes no frames in which preferred content is present;
designating all frames of the live media content in the second subset of frames; and
storing the second subset of frames at the detection unit.

### Claim 15

A computer-implemented method comprising:
applying the detector to the live media content recorded by the camera to identify the preferred content within the frames of the live media content;
responsive to identifying the preferred content in the frames of the live media content, communicating an alert to a user, the alert comprising timestamps describing when each frame of the live media content including the preferred content was recorded by the camera;
receiving a request from the user to define a first subset of the frames of the live media content including the preferred content occurring at specific timestamps of interest and to define a second subset of the frames of the live media content not including the preferred content;
responsive to receiving the request from the user, providing the first subset of the frames to a cloud server; and
storing the second subset of frames at the detection unit.

### Claim 16 (depends on 15)

The method of claim 15, wherein the user uploads the detector for storage on the detection unit and selects preferred content for the detector to identify in frames of the recorded live media content.

### Claim 17 (depends on 15)

The method of claim 15, wherein the detector is a neural network generated by the user to output a confidence score indicating a likelihood that the preferred content is present within a frame of media content, the detector trained based on a first set of positive images of the preferred content and a second set of negative images that excludes the preferred content.

### Claim 18 (depends on 15)

The method of claim 15, wherein the user uploads a plurality of detectors for storage on the detection unit, each detector configured to output a confidence score indicating a likelihood that a unique preferred content is present within a frame of media content and trained based on a first set of positive images of the unique preferred content and a second set of negative images that excludes the unique preferred content.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
for each detector of the plurality;
inputting live media content recorded by the camera to the detector to identify unique preferred content in the live media content;
responsive to identifying unique preferred content appearing in a frame the live media content, identifying a timestamp at which the frame was recorded by the camera;
generating an aggregate alert comprising timestamps at describing when each identified instance of unique preferred content was recorded by the camera; and
communicating the alert to the user.

### Claim 20 (depends on 15)

The method of claim 15, further comprising:
receiving confirmation from the user designating no preferred content of interest was identified in the first subset of frames; and
responsive to receiving the confirmation, designating all frames of the live media content in the second subset of frames; and
storing the second subset of frames on the detection unit.

### Claim 21 (depends on 15)

The method of claim 15, further comprising:
determining that the live media content includes no frames in which preferred content is present;
designating all frames of the live media content in the second subset of frames; and
storing the second subset of frames at the detection unit.
