Back to results
US 11,768,871 B2Granted
Systems and methods for contextualizing computer vision generated tags using natural language processing
AssigneeEntefy Inc.
InventorsKonstantinos Rapantzikos, Alston Ghafourifar
Life of this patent
Estimating the term…
Abstract
This disclosure relates to systems, methods, and computer readable media for performing filtering of computer vision generated tags in a media file for the individual user in a multi-format, multi-protocol communication system. One or more media files may be received at a user client. The one or more media files may be automatically analyzed using computer vision models, and computer vision generated tags may be generated in response to analyzing the media file. The tags may then be filtered using Natural Language Processing (NLP) models, and information obtained during NLP tag filtering may be used to train and/or fine-tune one or more of the computer vision models and the NLP models.
Claims20
A non-transitory computer readable medium comprising computer readable instructions, which, upon execution by one or more processing units, cause the one or more processing units to:
- receive one or more media files comprising one or more objects;
- automatically analyze the one or more media files using a plurality of computer vision models responsive to receiving the one or more media files, wherein automatically analyzing the one or more media files comprises:
- determining content associated with the one or more media files;
- and
- running a subset of the plurality of computer vision models based on the content;
- generate computer vision tags for the media file responsive to automatically analyzing the one or more media files, wherein each of the computer vision tags comprises a determined confidence value;
- determine a prioritization of the generated computer vision tags based on one or more Natural Language Processing models;
- filter the generated computer vision tags based on the prioritization;
- identify a set of the generated computer vision tags having highest confidence values of the determined confidence values of the computer vision tags;
- identify one of the generated computer vision tags having one of the determined confidence values at or below a predetermined threshold;
- determine that the one of the generated computer vision tags is an irrelevant tag based on reprocessing the one of the generated computer vision tags using the plurality of computer vision models and the one or more Natural Language Processing models;
- prioritize the filtered generated computer vision tags based on the determined confidence values of the generated computer vision tags, wherein prioritizing the filtered generated computer vision tags comprises ranking the filtered generated computer vision tags by increasing the prioritization of the set of the filtered generated computer vision tags and decreasing the prioritization of the irrelevant tag;
- and
- associate the prioritized filtered generated computer vision tags with the one or more media files.
11 more elements
A system, comprising:
- a memory;
- and
- one or more processing units, communicatively coupled to the memory, wherein the memory stores instructions, when executed, cause the one or more processing units to:
- receive a media file comprising an image or a video, wherein the media file includes one or more objects;
- automatically analyze the media file using computer vision models responsive to receiving the media file, wherein automatically analyzing the media file comprises:
- determining content associated with the media file;
- utilizing a subset of the plurality of computer vision models with the content;
- generate computer vision tags for the media file responsive to automatically analyzing the media file, wherein each of the computer vision tags comprises a determined confidence value;
- determine a prioritization of the computer vision tags based on one or more Natural Language Processing models;
- filter the computer vision tags based on the prioritization, wherein the instructions to filter comprise using an inter-level tag filtering to re-rank determined confidence values of the computer vision tags;
- identify a set of the computer vision tags having highest confidence values of the determined confidence values of the computer vision tags;
- identify one of the computer vision tags having one of the determined confidence values at or below a predetermined threshold;
- determine that the one of the computer vision tags is an irrelevant tag based on reprocessing the one of the computer vision tags using the plurality of computer vision models and the one or more Natural Language Processing models;
- select the computer vision tags based on the determined confidence values of the computer vision tags, wherein selecting the computer vision tags comprises ranking the computer vision tags by increasing the prioritization of the set of the computer vision tags and decreasing the prioritization of the irrelevant tag;
- and
- associate the selected computer vision tags with the media file.
13 more elements
A computer-implemented method, comprising:
- receiving a media file comprising one or more objects;
- automatically analyzing the media file using a plurality of computer vision models responsive to receiving the media file, wherein automatically analyzing the media file comprises:
- determining content associated with the media file;
- and
- running a subset of the plurality of computer vision models using the content;
- generating computer vision tags for the media file responsive to automatically analyzing the media file, wherein each tag comprises a determined confidence value;
- determining a prioritization of the computer vision tags based on one or more Natural Language Processing models;
- filtering the computer vision tags based on the prioritization and via an inter-level tag filtering;
- identify a set of the computer vision tags having highest confidence values of the determined confidence values of the computer vision tags;
- identify one of the computer vision tags having one of the determined confidence values at or below a predetermined threshold;
- determine that the one of the computer vision tags is an irrelevant tag based on reprocessing the one of the computer vision tags using the plurality of computer vision models and the one or more Natural Language Processing models;
- prioritizing the computer vision tags based on the determined confidence values of the computer vision tags, wherein the prioritizing comprises ranking the computer vision tags by increasing the prioritization of the set of the computer vision tags and decreasing the prioritization of the irrelevant tag;
- and
- associating the prioritized computer vision tags with the media file.
11 more elements
Dependencies are inferred from the claim text.