# US 2025/0375889 A1 · Techniques for vision-based robot control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0375889 A1
- Document ID: US20250375889A1
- Status: Published application
- Kind: A1
- Application number: 19/072,884
- Filed: 2025-03-06
- Published: 2025-12-11
- Earliest claimed: 2024-06-06
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Xiangyun MENG; Xuning YANG; Fabio TOZETO RAMOS; Dieter FOX
- Web page: https://patentgrep.com/patent/US20250375889A1

## CPC

- B25J 9/1697: {Vision controlled systems}
- B25J 5/007: {mounted on wheels}
- B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques for controlling a robot include receiving sensor data and one or more goal specifications, processing the sensor data, a robot size, and the one or more goal specifications using one or more trained encoders to generate a plurality context tokens, processing the plurality of context tokens using one or more trained decoders to generate a robot plan, and controlling a robot based on the robot plan.

## Claims

### Claim 1

A computer-implemented method for controlling a robot, the method comprising:
receiving sensor data and one or more goal specifications;
processing the sensor data, a robot size, and the one or more goal specifications using one or more trained encoders to generate a plurality context tokens;
processing the plurality of context tokens using one or more trained decoders to generate a robot plan; and
controlling a robot based on the robot plan.

### Claim 2 (depends on 1)

The method of claim 1, wherein the sensor data includes at least one of a plurality of red-green-blue images with depth (RGB-D) inputs, a plurality of light detection and ranging (LiDAR) inputs, or robot state data.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more goal specifications include at least one of a reference image, a look-at pose, and a target object mask.

### Claim 4 (depends on 1)

The method of claim 1, wherein processing the sensor data, the robot size, and the one or more goal specifications using the one or more trained encoders comprises:
processing a plurality of LiDAR inputs using a first trained encoder included in the one or more trained encoders to generate a plurality of LiDAR tokens;
processing a plurality of RGB-D inputs using a second trained encoder included in the one or more trained encoders to generate a plurality of RGB-D tokens;
processing a reference image using a third trained encoder included in the one or more trained encoders to generate a plurality of reference image tokens;
processing the plurality of reference image tokens and the plurality of RGB-D tokens using a fourth trained encoder included in the one or more trained encoders to generate a plurality of vision tokens; and
processing the plurality of vision tokens, the plurality of LiDAR tokens, the robot size, and a look-at pose using a fifth trained encoder included in the one or more trained encoders to generate the plurality of context tokens.

### Claim 5 (depends on 4)

The method of claim 4, wherein processing the plurality of LiDAR inputs comprises:
generating, based on the plurality of LiDAR inputs, one or more directional bins; and
generating, based on the one or more directional bins and using the first trained encoder, the plurality of LiDAR tokens.

### Claim 6 (depends on 4)

The method of claim 4, wherein processing the plurality of RGB-D inputs comprises generating, based on the plurality of RGB-inputs, a fixed size feature map using a frozen Masked Autoencoder.

### Claim 7 (depends on 4)

The method of claim 4, wherein processing the reference image comprises generating, based on the reference image and using a frozen Masked Autoencoder, a fixed size feature map.

### Claim 8 (depends on 4)

The method of claim 4, wherein processing the plurality of vision tokens, the plurality of LiDAR tokens, the robot size, and the look-at pose comprises:
generating, based on the robot size and using a first neural network, a plurality of robot size tokens;
generating, based on the look-at pose and using a second neural network, a plurality of look-at pose tokens; and
generating, based on the plurality of robot size tokens and the plurality of look-at pose tokens, the plurality of context tokens.

### Claim 9 (depends on 1)

The method of claim 1, wherein processing the plurality of context tokens comprises determining, based on the plurality of context tokens and using a first trained decoder included in the one or more trained decoders, a camera tilt.

### Claim 10 (depends on 1)

The method of claim 1, wherein processing the plurality of context tokens comprises:
generating, based on the plurality of context tokens, a plurality of cross-attention features; and
generating, based on the plurality of cross-attention features and using a first trained decoder included in the one or more trained decoders, a trajectory for the robot.

### Claim 11

One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving sensor data and one or more goal specifications;
processing the sensor data, a robot size, and the one or more goal specifications using one or more trained encoders to generate a plurality context tokens;
processing the plurality of context tokens using one or more trained decoders to generate a robot plan; and
controlling a robot based on the robot plan.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein processing the sensor data, the robot size, and the one or more goal specifications using the one or more trained encoders comprises:
generating, based on a plurality of LiDAR inputs, one or more directional bins;
generating, based on the one or more directional bins and using a first trained encoder included in the one or more trained encoders, a plurality of LiDAR tokens;
processing a plurality of RGB-D inputs using a second trained encoder included in the one or more trained encoders to generate a plurality of RGB-D tokens;
processing a reference image using a third trained encoder included in the one or more trained encoders to generate a plurality of reference image tokens;
processing the plurality of reference image tokens and the plurality of RGB-D tokens using a fourth trained encoder included in the one or more trained encoders to generate a plurality of vision tokens; and
processing the plurality of vision tokens, the plurality of LiDAR tokens, the robot size, and a look-at pose using a fifth trained encoder included in the one or more trained encoders to generate the plurality of context tokens.

### Claim 13 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein processing the plurality of vision tokens, the plurality of LiDAR tokens, the robot size, and the look-at pose comprises:
generating, based on the robot size and using a first neural network, a plurality of robot size tokens;
generating, based on the look-at pose and using a second neural network, a plurality of look-at pose tokens; and
generating, based on the plurality of robot size tokens and the plurality of look-at pose tokens, the plurality of context tokens.

### Claim 14 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein a first trained decoder included in the one or more trained decoders comprises at least one of a transformer decoder or a regression model.

### Claim 15 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein processing the plurality of context tokens comprises determining, based on the plurality of context tokens and using a first trained decoder included in the one or more trained decoders, a camera tilt.

### Claim 16 (depends on 15)

The one or more non-transitory computer-readable media of claim 15, wherein the first trained decoder comprises a multi-token classification technique with a residual model.

### Claim 17 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein processing the plurality of context tokens comprises:
generating, based on the plurality of context tokens, a plurality of cross-attention features; and
generating, based on the plurality of cross-attention features and using a first trained decoder included in the one or more trained decoders, a trajectory for the robot.

### Claim 18 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein the trajectory includes one or more waypoints in egocentric polar coordinates.

### Claim 19 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein a first trained encoder included in the one or more trained encoder receives as input camera extrinsics derived from odometry of the robot.

### Claim 20

A system comprising:
one or more memories storing instructions, and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
receive sensor data and one or more goal specifications,
process the sensor data, a robot size, and the one or more goal specifications using one or more trained encoders to generate a plurality context tokens,
process the plurality of context tokens using one or more trained decoders to generate a robot plan, and
control a robot based on the robot plan.
