US 12,608,940 B1Granted

Detecting relevant portions of video files based on classifications of images

AssigneeAmazon Technologies, Inc.

InventorsShachar Flora Ben Dayan, Michael Rotman, Ido Yerushalmy

Life of this patent

Estimating the term…

Loading patent…

Abstract

Relevant portions of broadcasts of live events, such as replays, may be marked or framed by patterns of one or more images. A model is trained with pairs of images and text descriptors to generate image embeddings and text embeddings representative of the images and text descriptors, respectively. Where a pattern of images includes one or more images, a user may specify one or more text descriptors representative of images of the pattern and identify portions of a media file with respect to such images. The model generates text embeddings of the text descriptors, as well as image embeddings of image frames of a video file broadcasted to viewers, including a relevant portion marked or framed by the pattern. Where an image embedding is sufficiently similar to one of the text embeddings, the pattern is detected, and the relevant portion is identified based on the pattern.

Claims20

Dependent claims

A system comprising:

at least one computer server having at least one computer processor,
wherein the at least one computer server is programmed with one or more sets of instructions that, when executed, cause the system to execute a method comprising:
training a model to generate image embeddings and text embeddings based at least in part on a first set of images and a first set of text descriptors, wherein each of the first set of text descriptors describes one of the first set of images, and wherein the model is trained to maximize cosine similarities of image embeddings generated for images and text embeddings generated for text descriptors of the images;
identifying at least one multimedia file transmitted to a plurality of personal devices, wherein the at least one multimedia file comprises a second set of images and a second set of audio signals transmitted to the plurality of personal devices in a broadcast of an organized event;
receiving a plurality of classification terms from a user;
providing the plurality of classification terms as inputs to the model;
receiving a plurality of text embeddings as outputs from the model, wherein each of the plurality of text embeddings is generated by a text encoder for one of the plurality of classification terms;
providing the second set of images as inputs to the model;
receiving a plurality of image embeddings as outputs from the model, wherein each of the plurality of image embeddings is generated by an image encoder for one of the second set of images;
calculating probabilities that each of the second set of images corresponds to one of the plurality of classification terms based at least in part on the plurality of text embeddings and the plurality of image embeddings;
determining that a first probability that a first image of the second set of images displayed at a first time corresponds to a first classification term of the plurality of classification terms exceeds a predetermined threshold;
determining that a second probability that a second image of the second set of images displayed at a second time corresponds to a second classification term of the plurality of classification terms exceeds the predetermined threshold;
and
identifying at least one portion of the at least one multimedia file transmitted to the plurality of personal devices between the second time and the first time;
and
after the organized event has concluded,
transmitting the at least one portion of the at least one multimedia file to a first personal device,
wherein the first personal device is not one of the plurality of personal devices.

15 more elements

A computer-implemented method comprising:

transmitting at least a first video file to a plurality of personal devices in a broadcast of a first organized event over one or more networks;
receiving information regarding at least a first text descriptor from a user;
providing at least the first text descriptor to a model as a first input, wherein the model comprises a text encoder and an image encoder, and wherein the model is trained to generate image embeddings representative of images and text embeddings representative of sets of text in a latent space;
receiving at least a first text embedding from the model as a first output in response to the first input, wherein the first text embedding represents the first text descriptor;
providing a first plurality of images to the model as second inputs, wherein each of the first plurality of images is included in the first video file;
receiving a first plurality of image embeddings from the model as second outputs in response to the second inputs, wherein each of the first plurality of image embeddings is in the latent space with at least the first text embedding;
determining probabilities that each of the first plurality of images is consistent with at least the first text descriptor based at least in part on the first output and the second outputs;
selecting a first portion of the first video file based at least in part on a first image of the first plurality of images and a first probability that the first image is consistent with the first text descriptor;
and
after the first organized event has concluded,
transmitting at least the first portion of the first video file to a first personal device over the one or more networks,
wherein the first personal device is not one of the plurality of personal devices.

9 more elements

A computer-implemented method comprising:

receiving information regarding a pattern of images from a user, wherein the information regarding the pattern of images identifies:
a plurality of text descriptors;
and
a duration;
providing at least a first text descriptor and a second text descriptor to a model as first inputs, wherein the plurality of text descriptors includes at least the first text descriptor and the second text descriptor, wherein the model comprises a text encoder trained to at least generate text embeddings from text descriptors and an image encoder trained to at least generate image embeddings from images, and wherein the model is further trained to determine similarities between text embeddings and image embeddings;
receiving at least a first text embedding and a second text embedding from the model as first outputs in response to the first inputs, wherein the first text embedding represents the first text descriptor, and wherein the second text embedding represents the second text descriptor;
providing a plurality of images to the model as second inputs, wherein each of the plurality of images is included in a video file transmitted in a broadcast of an organized event;
receiving a plurality of image embeddings from the model as second outputs in response to the second inputs, wherein each of the plurality of image embeddings is in a latent space with at least the first text embedding and the second text embedding;
determining probabilities that each of the plurality of images is consistent with at least the first text descriptor and the second text descriptor based at least in part on the first outputs and the second outputs;
determining that a first probability that a first image of the plurality of images is consistent with the first text descriptor exceeds a predetermined threshold, wherein the first image was transmitted at a first time during the broadcast of the organized event;
determining that a second probability that a second image of the plurality of images is consistent with a second text descriptor of the plurality of text descriptors exceeds the predetermined threshold, wherein the second image was transmitted at a second time during the broadcast of the organized event, and wherein a difference between the second time and the first time is not greater than the duration;
selecting a portion of the video file based at least in part on the first image, the first probability, the second image and the second probability, wherein the portion of the video file was transmitted during the broadcast of the organized event between the first time and the second time.

9 more elements

A computer-implemented method comprising:

receiving information regarding at least a first text descriptor from a user;
providing at least the first text descriptor to a model as a first input, wherein the model comprises a text encoder and an image encoder, and wherein the model is trained to generate image embeddings representative of images that are similar to text embeddings representative of sets of text in a latent space;
receiving at least a first text embedding from the model as a first output in response to the first input, wherein the first text embedding represents the first text descriptor;
providing a first plurality of images to the model as second inputs, wherein each of the first plurality of images is included in a first video file transmitted in a broadcast of a first organized event;
receiving a first plurality of image embeddings from the model as second outputs in response to the second inputs, wherein each of the first plurality of image embeddings is in the latent space with at least the first text embedding;
determining probabilities that each of the first plurality of images is consistent with at least the first text descriptor based at least in part on the first output and the second outputs;
and
selecting a first portion of the first video file based at least in part on a first image of the first plurality of images and a first probability that the first image is consistent with the first text descriptor, wherein the first portion of the first video file is transmitted following a second portion of the first video file and prior to a third portion of the first video file, and
wherein the first portion of the first video file depicts at least one activity that occurred prior to the transmission of the first portion of the first video file.

6 more elements

A computer-implemented method comprising:

providing a first plurality of images to a model as first inputs, wherein each of the first plurality of images is included in a first video file transmitted during a broadcast of an organized event, wherein the model comprises a text encoder and an image encoder, and wherein the model is trained to maximize cosine similarities of image embeddings generated for images in a latent space and text embeddings generated for text descriptors of the images in the latent space;
receiving a first plurality of image embeddings from the model as first outputs in response to the first inputs;
providing a first plurality of text descriptors to the model as second inputs;
receiving a first plurality of text embeddings from the model as second outputs in response to the second inputs, wherein each of the first plurality of text embeddings represents one of the first plurality of text descriptors;
calculating probabilities that each of the first plurality of images is consistent with one of the first plurality of text descriptors based at least in part on the first outputs and the second outputs;
determining that a first probability that a first image of the first plurality of images transmitted at a first time is consistent with a first descriptor of the first plurality of text descriptors exceeds a predetermined threshold;
determining that a second probability that a second image of the first plurality of images transmitted at a second time is consistent with a second descriptor of the first plurality of text descriptors exceeds the predetermined threshold;
and
identifying a second plurality of images, wherein each of the second plurality of images is included in the first video file, and wherein each of the second plurality of images was transmitted between the first time and the second time;
and
after the organized event has concluded,
transmitting a second video file including the second plurality of images to at least one personal device.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/322,015Filed 23 May 2023
Granted: 21 Apr 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification17 codes

G06V 20/47
Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames: detecting features for…
G06V 10/774
Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or…:…
G06V 10/82
Using pattern recognition or machine learning: using neural networks
G06V 20/41
In video content: higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian…