Iterative input refinement for multi-modal language models
AssigneeNVIDIA Corporation
InventorsRajath Bellipady Shetty, Niral Lalit Pathak, Ratin Kumar
Life of this patent
Estimating the term…
Loading patent…
AssigneeNVIDIA Corporation
InventorsRajath Bellipady Shetty, Niral Lalit Pathak, Ratin Kumar
Estimating the term…
In various examples, a multi-modal language model such as a vision language model (VLM) may be iteratively prompted to identify and/or refine a region of interest (ROI) to evaluate for a designated detection task. For example, an initial prompt may broadly focus the VLM on identifying one or more ROIs within an image. After identifying an initial ROI, the VLM may be prompted to refine the ROI, for example, by prompting the VLM to evaluate the initial ROI and identify an ROI with more specific content than the initial prompt did, or by evaluating successive frames generated over time until a measure of confidence that an identified ROI contains the designated content meets a designate threshold, upon which, the multi-modal language model may be prompted to perform the detection task on the identified ROI.
One or more processors comprising processing circuitry to:
A method comprising:
A system comprising one or more processors to control, within a simulation that is rendered using one or more light transport simulation algorithms, one or more operations of a simulated ego-machine based at least on one or more outputs of one or more multi-modal language models, the one or more outputs generated based at least on applying one or more frames of simulated sensor data to the multi-modal language model and using the multi-modal language model to evaluate whether one or more conditions associated with a designated task are present in one or more regions of interest of the one or more frames of simulated sensor data, the one or more regions of interest identified in one or more initial outputs of the one or more multi-modal language models.
Dependencies are inferred from the claim text.