# US 2021/0279470 A1 · Detecting content in a real-time video stream using machine-learning classifiers

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279470 A1
- Document ID: US20210279470A1
- Status: Published application
- Kind: A1
- Application number: 16/809,542
- Filed: 2020-03-04
- Published: 2021-09-09
- Earliest claimed: 2020-03-04
- Assignees: Matroid, Inc.
- Applicants: Matroid, Inc.
- Inventors: REZA ZADEH; RYAN WONG; JOHN GODDARD; JIAHANG LI; STEVEN CHEN; XIAOYUN YANG
- Web page: https://patentgrep.com/patent/US20210279470A1

## CPC

- G06V 10/82: using neural networks
- G06F 11/327: {Alarm or error message display}
- G06F 18/24: Classification techniques
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/20: in augmented reality scenes
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Described herein are systems and methods that search videos and other media content to identify items, objects, faces, or other entities within the media content. Detectors identify objects within media content by, for instance, detecting a predetermined set of visual features corresponding to the objects. Detectors configured to identify an object can be trained using a machine learned model (e.g., a convolutional neural network) as applied to a set of example media content items that include the object. The systems and methods describe techniques that search videos and media content to determine the presence of unknown objects, generate novel detectors trained to identify the unknown objects, and apply the novel detectors to historical media content to identify previous appearances of the unknown objects.

## Claims

### Claim 1

A computer-implemented method, the method comprising:
receiving live frames of media content recorded by a media source;
for each live frame of media content, identifying a presence of one or more unknown objects in the live frame using a classifier that is unable to identify a type of each unknown object;
for each present unknown object:
responsive to identifying the unknown object, accessing media content recorded by the media source;
generating a novel detector based on the visual features of the unknown object identified in the live frames of media content, the detector configured to output a confidence score indicating a likelihood that the unknown object is present within a frame of media content; and
applying the novel detector to the accessed media content to identify previous appearances of the unknown object in frames of the accessed media content; and
generating, for presentation to a user, a user interface identifying one or more frames of the accessed media content in which at least one of the unknown objects is present and a location of each unknown object present in each identified frame.

### Claim 2 (depends on 1)

The method of claim 1, wherein the live frames of media content are assigned a source label identifying the media source and the history of media content is assigned the same source label.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
responsive to identifying the unknown object, generating an alert, wherein the alert describes that the unknown object is present in the live frames of media content and identifies the media source and a timestamp at which the unknown object was identified; and
presenting the alert to the user via the user interface.

### Claim 4 (depends on 1)

The method of claim 1, wherein identifying the presence of one or more unknown objects in the frames of media content comprises:
receiving, from the user, a request to search the live frames of media content for preferred content, the preferred content representing visual features of interest to the user;
applying one or more detectors to the live frames of media content to search for the preferred content in the live frames of media content, each detector of the one or more detectors configured to identify preferred content in the live frames of media content; and
identifying, by the one or more detectors, a presence of one or more unknown objects in the frames of media content, wherein each unknown object represents a type of content the one or more detectors is unable to identify.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
for each frame of the live frames of media content in which an unknown object is present,
determining a location of identified preferred content in the frame; and
determining a location of the unknown object in the frame.

### Claim 6 (depends on 4)

The method of claim 4, wherein each detector of the one or more detectors is configured to output a confidence score indicating a likelihood that a type of content is present within a frame of the live frames of media content, the method further comprising:
classifying objects assigned greater than a threshold score as preferred content present in the frame; and
classifying objects assigned confidence scores less than the threshold score as unknown objects present in the frame.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the novel detector based on visual features of the unknown object comprises:
for each identified unknown object,
inputting the live frame in which the unknown object is present to a machine learned model to extract one or more visual features of the unknown object from the live frame of media content; and
training the novel detector to classify the unknown object based on the extracted visual features, the classification based on the confidence score.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating a detector based on visual features of the unknown object comprises:
for each identified unknown object,
generating, for presentation to the user, a user interface identifying the unknown object present in the live frame of media content;
receiving, by the user interface, a selection of one or more visual features representing the unknown object and a plurality of negative examples;
training the novel detector to classify the unknown object using a confidence score based on the selection of visual features and the plurality of negative examples.

### Claim 9 (depends on 1)

The method of claim 1, wherein the user interface presented to the user comprises:
a progress bar interface element segmented into frames of the live media content and the history of media content, wherein each frame in which an unknown object is present is a marked with an alert interface element; and
a display interface element for presenting a frame with highlighted markings of locations of unknown objects present in the frame, wherein the frame is presented at the display interface element in response to a selection of the frame from the progress bar interface element.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
determining a set of analytics based on a number of appearances of the unknown object in the live frames and the history of media content; and
updating the user interface with an analytics interface element listing the determined set of analytics.

### Claim 11

A non-transitory computer-readable storage medium storing executable computer instructions that, when executed by a hardware processor, are configured to cause the hardware processor to perform steps comprising:
receiving live frames of media content recorded by a media source;
for each live frame of media content, identifying a presence of one or more unknown objects in the live frame using a classifier that is unable to identify a type of each unknown object;
for each identified unknown object:
responsive to identifying the unknown object, accessing media content recorded by the media source;
generating a novel detector based on the visual features of the unknown object identified in the live frames of media content, the detector configured to output a confidence score indicating a likelihood that the unknown object is present within a frame of media content; and
applying the novel detector to the accessed media content to identify previous appearances of the unknown object in frames of the accessed media content; and
generating, for presentation to a user, a user interface identifying one or more frames of the accessed media content in which at least one of the unknown objects is present and a location of each unknown object present in each identified frame.

### Claim 12 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
responsive to identifying the unknown object, generating an alert, wherein the alert describes that the unknown object is present in the live frames of media content and identifies the media source and a timestamp at which the unknown object was identified; and
presenting the alert to the user via the user interface.

### Claim 13 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the computer instructions for identifying one or more unknown objects in the frames of media content are further configured to cause the hardware processor to perform the steps comprising:
receiving, from the user, a request to search the live frames of media content for preferred content, the preferred content representing visual features of interest to the user;
applying one or more detectors to the live frames of media content to search for the preferred content in the live frames of media content, each detector of the one or more detectors configured to identify preferred content in the live frames of media content; and
identifying, by the one or more detectors, a presence of one or more unknown objects in the frames of media content, wherein each unknown object represents a type of content the one or more detectors is unable to identify.

### Claim 14 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
for each frame of the live frames of media content in which an unknown object is present,
determining a location of identified preferred content in the frame; and
determining a location of the unknown object in the frame.

### Claim 15 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein each detector of the one or more detectors is configured to output a confidence score indicating a likelihood that a type of content is present within a frame of the live frames of media content, the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
classifying objects assigned greater than a threshold scores as preferred content present in the frame; and
classifying objects assigned confidence scores less than the threshold score as unknown objects present in the frame.

### Claim 16 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the computer instructions for generating a detector based on visual features of the unknown object are further configured to cause the hardware processor to perform the steps comprising:
for each identified unknown object,
inputting the live frame in which the unknown object is present to a machine learned model to extract one or more visual features of the unknown object from the live frame of media content; and
training the novel detector to classify the unknown object based on the extracted visual features, the classification based on the confidence score.

### Claim 17 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the computer instructions for generating a detector based on visual features of the unknown object are further configured to cause the hardware processor to perform the steps comprising:
for each identified unknown object,
generating, for presentation to the user, a user interface identifying the unknown object present in the live frame of media content;
receiving, by the user interface, a selection of one or more visual features representing the unknown object and a plurality of negative examples;
training the novel detector to classify the unknown object using a confidence score based on the selection of visual features and the plurality of negative examples.

### Claim 18 (depends on 11)

The non-transitory computer readable storage medium of claim 11, wherein the user interface presented to the user comprises:
a progress bar interface element segmented into frames of the live media content and the history of media content, wherein each frame in which an unknown object is present is a marked with an alert interface element; and
a display interface element for presenting a frame with highlighted markings of locations of unknown objects present in the frame, wherein the frame is presented at the display interface element in response to a selection of the frame from the progress bar interface element.

### Claim 19 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the computer instructions are further configured to cause the hardware processor to perform the steps comprising:
determining a set of analytics based on a number of appearances of the unknown object in the live frames and the history of media content; and
updating the user interface with an analytics interface element listing the determined set of analytics.

### Claim 20

A system comprising:
a computer processor; and
a non-transitory computer-readable storage medium storing executable computer instructions that, when executed by a hardware processor, are configured to cause the hardware processor to perform steps comprising:
receiving live frames of media content recorded by a media source;
for each live frame of media content, identifying a presence of one or more unknown objects in the live frame using a classifier that is unable to identify a type of each unknown object;
for each identified unknown object:
responsive to identifying the unknown object, accessing media content recorded by the media source;
generating a novel detector based on the visual features of the unknown object identified in the live frames of media content, the detector configured to output a confidence score indicating a likelihood that the unknown object is present within a frame of media content; and
applying the novel detector to the accessed media content to identify previous appearances of the unknown object in frames of the accessed media content; and
generating, for presentation to a user, a user interface identifying one or more frames of the accessed media content in which at least one of the unknown objects is present and a location of each unknown object present in each identified frame.