Multi-modal machine learning architectures integrating language models and computer vision systems
AssigneeSurgetech, LLC
InventorsMichael Love, Blake Love, Tiago Soromenho
Life of this patent
Estimating the term…
Abstract
Improved multi-modal machine learning networks integrate computer vision systems with language models. In certain embodiments, a computer vision system analyzes at least one image to generate a computer vision output. The language model generates an output based, at least in part, on a consideration of the computer vision output. The outputs of the language model can be generated by jointly considering textual information learned by the language model and visual content extracted by the computer vision system, thereby significantly improving the accuracy, breadth, and comprehensiveness of the outputs.
Claims20
A method implemented via execution of computing instructions by one or more processors and stored on one or more non-transitory computer-readable storage devices, the method comprising:
- providing a multi-modal application that includes:
- a computer vision system configured to execute one or more computer vision tasks and generate computer vision outputs;
- and
- a client interface that facilitates interactions between an end-user and a language model that generates language model outputs for communicating with the end-user;
- executing, using the computer vision system, at least one computer vision task on an image to generate a computer vision output;
- comparing the computer vision output generated by the computer vision system with one or more trigger events;
- in response to detecting a match between the computer vision output and the one or more trigger events, executing, by the language model, at least one natural language processing (NLP) task to generate a preemptive output;
- and
- outputting the preemptive output via the client interface.
A system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable storage devices storing computing instructions configured to be executed on the one or more processors and cause the one or more processors to execute functions comprising:
- providing a multi-modal application that includes:
- a computer vision system configured to execute one or more computer vision tasks and generate computer vision outputs;
- and
- a client interface that facilitates interactions between an end-user and a language model that generates language model outputs for communicating with the end-user;
- executing, using the computer vision system, at least one computer vision task on an image to generate a computer vision output;
- comparing the computer vision output generated by the computer vision system with one or more trigger events;
- in response to detecting a match between the computer vision output and the one or more trigger events, executing, by the language model, at least one natural language processing (NLP) task to generate a preemptive output;
- and
- outputting the preemptive output via the client interface.
A system comprising:
- one or more processor means;
- and
- one or more non-transitory storage means storing computing instructions configured to be executed on the one or more processor means and cause the one or more processor means to execute functions comprising:
- providing a multi-modal application means that includes:
- a computer vision means configured to execute one or more computer vision tasks and generate computer vision outputs;
- and
- an interface means that facilitates interactions between an end-user and a language model means that generates language model outputs for communicating with the end-user;
- executing, using the computer vision means, at least one computer vision task on an image to generate a computer vision output;
- comparing the computer vision output generated by the computer vision means with one or more trigger events;
- in response to detecting a match between the computer vision output and the one or more trigger events, executing, by the language model means, at least one natural language processing (NLP) task to generate a preemptive output;
- and
- outputting the preemptive output via the interface means.
Dependencies are inferred from the claim text.