Loading patent…
AssigneeNVIDIA Corporation
InventorsRajath Bellipady Shetty, Niral Lalit Pathak, Ratin Kumar
Estimating the term…
In various examples, multiple sensors of an ego-machine may be used to generate corresponding streams of sensor data, and a multi-modal language model may be used to select a stream from among multiple streams to evaluate for a given detection task. Taking image data generated using multiple exterior cameras as an example, a vision language model (VLM) may be prompted to evaluate an image from each camera (e.g., resized to a designated resolution supported by the VLM) and identify a relevant camera for a designated task. The VLM may be prompted at a designated frame rate, and once the VLM identifies a camera or corresponding video stream, the VLM may be prompted with a (e.g., subsequent, higher resolution) frame of image data from the identified camera to perform one or more subsequent tasks (e.g., a designated detection task, identify an ROI within the higher resolution frame of image data, etc.).
One or more processors comprising processing circuitry to:
A method comprising:
A system comprising one or more processors to control, within a simulation of an environment that is rendered using one or more light transport simulation algorithms, one or more operations of a simulated ego-machine in the simulated environment based at least on one or more outputs of one or more multi-modal language models, the one or more outputs generated based at least on the multi-modal language model evaluating whether one or more conditions associated with a designated task are present in one or more frames of simulated sensor data generated using a selected simulated sensor, the selected simulated sensor being selected from a plurality of simulated sensors based at least on the multi-modal language model evaluating a plurality of frames of simulated sensor data generated using the plurality of simulated sensors of the simulated ego-machine.
Dependencies are inferred from the claim text.