# US 2026/0100042 A1 · Iterative input refinement for multi-modal language models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0100042 A1
- Document ID: US20260100042A1
- Status: Published application
- Kind: A1
- Application number: 18/905,192
- Filed: 2024-10-03
- Published: 2026-04-09
- Earliest claimed: 2024-10-03
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Rajath Bellipady Shetty; Niral Lalit Pathak; Ratin Kumar
- Web page: https://patentgrep.com/patent/US20260100042A1

## CPC

- G06V 20/50: Context or environment of the image
- B60W 60/001: {Planning or execution of driving tasks}
- G05B 13/0265: {the criterion being a learning criterion}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 15/00: Three-dimensional [3D] image rendering
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/803: {of input or preprocessed data}
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- H04L 9/3213: {using tickets or tokens, e.g. Kerberos (network architectures or network communication protocols for entities authentication using tickets in a packet data network H04L63/0807)}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

In various examples, a multi-modal language model such as a vision language model (VLM) may be iteratively prompted to identify and/or refine a region of interest (ROI) to evaluate for a designated detection task. For example, an initial prompt may broadly focus the VLM on identifying one or more ROIs within an image. After identifying an initial ROI, the VLM may be prompted to refine the ROI, for example, by prompting the VLM to evaluate the initial ROI and identify an ROI with more specific content than the initial prompt did, or by evaluating successive frames generated over time until a measure of confidence that an identified ROI contains the designated content meets a designate threshold, upon which, the multi-modal language model may be prompted to perform the detection task on the identified ROI.

## Claims

### Claim 1

One or more processors comprising processing circuitry to:
prompt a multi-modal language model associated with an ego-machine to generate one or more initial responses associated with a designated task and indicating one or more identified regions of interest in one or more frames of sensor data applied to the multi-modal language model;
prompt the multi-modal language model to generate one or more subsequent responses indicating an evaluation of whether one or more conditions associated with the designated task are present in any of the one or more identified regions of interest; and
control one or more operations of the ego-machine based at least on the one or more subsequent responses.

### Claim 2 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to increase an input resolution of at least one frame of the one or more frames of sensor data applied to the multi-modal language model based at least on the multi-modal language model generating the one or more initial responses indicating the one or more identified regions of interest.

### Claim 3 (depends on 1)

The one or more processors of claim 1, wherein one or more identified regions of interest identified by the multi-modal language model have a resolution that corresponds to a maximum input resolution supported by the multi-modal language model.

### Claim 4 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to iteratively prompt the multi-modal language model to refine the one or more identified regions of interest within the one or more frames of sensor data.

### Claim 5 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to iteratively prompt the multi-modal language model to generate the one or more initial responses indicating the one or more identified regions of interest based at least on applying successive frames of the sensor data representing successive time slices to the multi-modal language model.

### Claim 6 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to prompt the multi-modal language model to evaluate whether the one or more conditions are present in the one or more identified regions of interest based at least on the one or more initial responses generated by the multi-modal language model indicating at least a threshold measure of confidence in the one or more identified regions of interest.

### Claim 7 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to apply, based at least on the one or more initial responses generated by the multi-modal language model indicating less than a threshold measure of confidence in the one or more identified regions of interest, a representation of a subsequent frame of the sensor data representing a subsequent time slice to the multi-modal language model.

### Claim 8 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to increase a rate of prompting the multi-modal language model based at least on the multi-modal language model indicating the one or more identified regions of interest.

### Claim 9 (depends on 1)

The one or more processors of claim 1, wherein the processing circuitry is further to prompt the multi-modal language model to generate the one or more initial responses based at least on periodically applying successive frames of the sensor data representing successive time slices to the multi-modal language model until the multi-modal language model identifies the one or more identified regions of interest.

### Claim 10 (depends on 1)

The one or more processors of claim 1, wherein the designated task comprises at least one computer vision task, and one or more subsequent responses by the multi-modal language model indicate one or more results of the at least one computer vision task, the at least one computer vision task comprising at least one of:
driver drowsiness detection,
driver distraction detection,
driver out-of-position detection,
driver or occupant identification,
seatbelt usage detection,
occupant presence detection,
occupant classification,
child presence detection,
gesture recognition,
sign recognition,
context-aware question answering,
recognition of one or more objects left behind, or
suspicious activity monitoring.

### Claim 11 (depends on 1)

The one or more processors of claim 1, wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system implementing one or more vision language models (VLMs);
a system implementing one or more multi-modal language models;
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system for performing one or more generative AI operations;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

### Claim 12

A method comprising:
based at least on a multi-modal language model generating one or more initial responses associated with a designated task and indicating one or more identified regions of interest in one or more frames of sensor data of an ego-machine, prompting the multi-modal language model to generate one or more subsequent responses evaluating whether one or more conditions associated with the designated task are present in the one or more identified regions of interest; and
controlling one or more operations of the ego-machine based at least on the one or more subsequent responses.

### Claim 13 (depends on 12)

The method of claim 12, further comprising increasing an input resolution of at least one frame of the one or more frames of sensor data applied to the multi-modal language model based at least on the multi-modal language model generating the one or more initial responses indicating the one or more identified regions of interest.

### Claim 14 (depends on 12)

The method of claim 12, wherein at least one frame of the one or more frames of sensor data depicting the one or more identified regions of interest identified by the multi-modal language model have a resolution that corresponds to a maximum input resolution supported by the multi-modal language model.

### Claim 15 (depends on 12)

The method of claim 12, further comprising iteratively prompting the multi-modal language model to refine the one or more identified regions of interest within the one or more frames of sensor data.

### Claim 16 (depends on 12)

The method of claim 12, further comprising iteratively prompting the multi-modal language model to generate the one or more initial responses indicating the one or more identified regions of interest based at least on applying successive frames of the sensor data representing successive time slices to the multi-modal language model.

### Claim 17 (depends on 12)

The method of claim 12, wherein the method is performed by at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system implementing one or more vision language models (VLMs);
a system implementing one or more multi-modal language models;
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system for performing one or more generative AI operations;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

### Claim 18

A system comprising one or more processors to control, within a simulation that is rendered using one or more light transport simulation algorithms, one or more operations of a simulated ego-machine based at least on one or more outputs of one or more multi-modal language models, the one or more outputs generated based at least on applying one or more frames of simulated sensor data to the multi-modal language model and using the multi-modal language model to evaluate whether one or more conditions associated with a designated task are present in one or more regions of interest of the one or more frames of simulated sensor data, the one or more regions of interest identified in one or more initial outputs of the one or more multi-modal language models.

### Claim 19 (depends on 18)

The system of claim 18, wherein the simulation is generated, at least in part, using a three-dimensional (3D) content collaboration platform for 3D assets.

### Claim 20 (depends on 18)

The system of claim 18, wherein the 3D content collaboration platform for 3D assets uses OpenUSD.
