US 11,961,300 B2Granted

Dynamic media content categorization method

AssigneeEcole Polytechnique Federale De Lausanne (Epfl)

InventorsGabriel Autès, Sami Arpa, Sabine Süsstrunk

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method of classifying a media includes receiving a media file and extracting therefrom first and second data streams including first and second media content, respectively, the media content being associated with the media item. First and second feature vectors describing the first and second media content, respectively, are generated. At least a first single feature vector representing the first sequence of first feature vectors and the second sequence of second feature vectors is generated, or at least a first single feature vector representing at least the first sequence of first feature vectors is generated. A second feature vector representing at least the second sequence of second feature vectors is generated. A probability vector from the first single feature vector, or from the first and second single feature vectors is generated. A user profile suitability class is assigned to the media item based on the probability vector.

Claims24

Dependent claims

A computer implemented method of classifying a media item into one or more user profile suitability classes in a classification system comprising a set of pre-trained artificial neural networks, the method comprising:

receiving a media file;
extracting a first data stream and a second data stream from the media file, the first media stream comprising first media content of a first media content form, the second data stream comprising second media content of a second, different media content form, the first and second data streams being comprised in the media item, the first content form being one of the following content forms:
audio, image and text, while the second media content form being any one of the remaining two content forms;
generating a first sequence of first feature vectors describing the first media content, the first sequence of first feature vectors being generated by a first set of convolution neural networks;
generating a second sequence of second feature vectors describing the second media content, the second sequence of second feature vectors being generated by a second set of convolution neural networks or by a set of word embedding matrices;
generating at least a first single feature vector representing the first sequence of first feature vectors and the second sequence of second feature vectors, or generating at least a first single feature vector representing at least the first sequence of first feature vectors, and generating a second single feature vector representing at least the second sequence of second feature vectors;
generating a probability vector at least from the first single feature vector in the case where the first single feature vector represents the first sequence of first feature vectors and the second sequence of second feature vectors, or at least from the first and second single feature vectors by first concatenating the first and second single feature vectors in the case where the first single feature vector represents at least the first sequence of first feature vectors, and the second single feature vector represents at least the second sequence of second feature vectors, the probability vector being generated by a feedforward artificial neural network;
and
assigning a user profile suitability class to the media item based on the probability vector.

6 more elements

A classification system for classifying a media item into one or more user profile suitability classes, the system comprising a set of pre-trained artificial neural networks and being configured to perform operations comprising:

receive a media file;
extract a first data stream and a second data stream from the media file, the first media stream comprising first media content of a first media content form, the second data stream comprising second media content of a second, different media content form, the first and second data streams being comprised in the media item, the first content form being one of the following content forms:
audio, image and text, while the second media content form being any one of the remaining two content forms;
generate by a first set of convolution neural networks a first sequence of first feature vectors describing the first media content;
generate by a second set of convolution neural networks or by a set of word embedding matrices a second sequence of second feature vectors describing the second media content;
generate at least a first single feature vector representing the first sequence of first feature vectors and the second sequence of second feature vectors, or generating at least a first single feature vector representing at least the first sequence of first feature vectors, and generating a second single feature vector representing at least the second sequence of second feature vectors;
generate by a feedforward artificial neural network a probability vector at least from the first single feature vector in the case where the first single feature vector represents the first sequence of first feature vectors and the second sequence of second feature vectors, or at least from the first and second single feature vectors by first concatenating the first and second single feature vectors in the case where the first single feature vector represents at least the first sequence of first feature vectors, and the second single feature vector represents at least the second sequence of second feature vectors;
and
assign a user profile suitability class to the media item based on the probability vector.

6 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/607,393Filed 29 Apr 2019
Published: US 2022/0207864 A130 Jun 2022
Granted: 16 Apr 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification8 codes

G06V 20/47
Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames: detecting features for…
G06F 18/254
Fusion techniques: of classification results, e.g. of results related to same input data
G06V 10/454
With interaction between the filter responses, e.g. cortical complex cells: integrating the filters into a hierarchical structure, e.g.…
G06V 10/765
Using classification, e.g. of video objects: using rules for classification or partitioning the feature space