Loading patent…
AssigneeNVIDIA Corporation
InventorsRajath Bellipady Shetty, Niral Lalit Pathak, Ratin Kumar
Estimating the term…
In various examples, a multi-modal language model may be split up and hosted by multiple devices. For example, a modality (e.g., vision, audio) encoder and/or projector of the multi-modal language model (e.g., a vision language model) may be hosted on one device (e.g., an in-vehicle SoC) that encodes raw sensor data into corresponding tokens and streams the tokens to a second device (e.g., an external graphic processing unit (GPU) or artificial intelligence (AI) accelerator) that hosts an inference server and a language model (LM) of the multi-modal language model. The LM may return a response indicating the result(s) of the requested detection task, and the response may be used to take some responsive action (e.g., control one or more operations of an ego-machine).
One or more processors comprising processing circuitry to:
A method comprising:
A system comprising one or more processors to control, within a simulation that is rendered using one or more light transport simulation algorithms, one or more operations of a simulated ego-machine based at least on one or more outputs of one or more multi-modal language models, the one or more outputs generated based at least on a tokenized representation of simulated sensor data streamed from a first hardware platform used to generate the tokenized representation to a second hardware platform used to execute one or more large language models (LLMs) of the one or more multi-modal language models.
Dependencies are inferred from the claim text.