# US 2026/0061605 A1 · Configuring a visual language model with spatial understanding for robotics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0061605 A1
- Document ID: US20260061605A1
- Status: Published application
- Kind: A1
- Application number: 19/305,349
- Filed: 2025-08-20
- Published: 2026-03-05
- Earliest claimed: 2024-08-30
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corp.
- Inventors: Chan Hee Song; Valts Blukis; Jonathan Tremblay; Stephen Tyree; Stanley Thomas Birchfield
- Web page: https://patentgrep.com/patent/US20260061605A1

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1697: {Vision controlled systems}
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})

## Abstract

Systems utilizing a vision-language model configured with a training dataset that includes images labeled with spatial question-and-answer pairs, the question-and-answer pairs encoding object-to-object relationships and object-to-space relationships depicted in the images, and at least one data processor configured to operate the vision-language model to carry out a robotic task.

## Claims

### Claim 1

A system comprising:
a vision-language model configured with a training dataset comprising images labeled with spatial question-and-answer pairs;
the question-and-answer pairs encoding object-to-object relationships and object-to-space relationships depicted in the images; and
at least one data processor configured to operate the vision-language model to carry out a robotic task.

### Claim 2 (depends on 1)

The system of claim 1, wherein the images comprise multiple reference frames.

### Claim 3 (depends on 1)

The system of claim 1, wherein the training dataset further comprises reference frame codes.

### Claim 4 (depends on 1)

The system of claim 1, wherein the question-and-answer pairs encode from among six spatial relationships.

### Claim 5 (depends on 4)

The system of claim 4, wherein the spatial relationships comprise “left”, “right”, “in front”, “behind”, “above”, and “below”.

### Claim 6 (depends on 1)

The system of claim 1, further comprising logic to generate content of the question-and-answer pairs from three dimensional (3D) bounding box annotations to the images.

### Claim 7 (depends on 6)

The system of claim 6, wherein the 3D bounding box annotation comprise an orientation setting.

### Claim 8 (depends on 7)

The system of claim 7, wherein the orientation setting comprises a rotation matrix.

### Claim 9 (depends on 6)

The system of claim 6, wherein questions of the question-and-answer pairs encode object-to-object relationships for objects depicted in the images.

### Claim 10 (depends on 6)

The system of claim 6, wherein questions of the question-and-answer pairs encode object-to-space relationships for objects depicted in the images.

### Claim 11 (depends on 6)

The system of claim 6, further comprising logic to project the 3D bounding box onto a plane.

### Claim 12 (depends on 11)

The system of claim 11, further comprising logic to mark a grid structure imposed on the plane with areas occupied by objects corresponding to the projected 3D bounding box.

### Claim 13 (depends on 1)

The system of claim 1, wherein answers of the question-and-answer pairs encode true responses to questions of the question-and-answer pairs for tuning hyperparameters of the vision-language model during training.

### Claim 14 (depends on 13)

The system of claim 13, wherein some of the answers are numeric.

### Claim 15 (depends on 14)

The system of claim 14, wherein numeric answers are configured to ground the vision-language model in a reference frame.

### Claim 16 (depends on 1)

The system of claim 1, wherein at least some of the question-and-answer pairs are configured in an ego-centric reference frame.

### Claim 17 (depends on 1)

The system of claim 1, wherein at least some of the question-and-answer pairs are configured in an object-centric reference frame.

### Claim 18 (depends on 1)

The system of claim 1, wherein at least some of the question-and-answer pairs are configured in an allo-centric reference frame.

### Claim 19 (depends on 1)

The system of claim 1, wherein the object-to-object relationships and object-to-space relationships are encoded as (Ii, ai, ti, si, ri, li), where Ii is an image, ai is an anchor object, ti is a target object or a target free-space point, si is size measure based on an object to be manipulated in the robotic system, ri∈{left, right, above, below, front, behind} is a spatial relationship, and li∈{allo-centric, object-centric, ego-centric} is a reference frame label.

### Claim 20 (depends on 1)

The system of claim 1, further comprising logic to generate a map of a robotic task environment, the map generated from the annotated 3D bounding box and randomly sampled points in empty areas that are a set distance from an object of the robotic task.

### Claim 21 (depends on 20)

The system of claim 20, the set distance based on a size of another object that is to be manipulated and placed in relation to the object in the robotic task.

### Claim 22

A system comprising:
a robot;
a vision-language model configured with a training dataset comprising images labeled with spatial question-and-answer pairs encoding object-to-object relationships and object-to-space relationships represented in the images; and
logic to operate the vision-language model to generate pick-and-place steps for the robot based on a natural language prompt.

### Claim 23 (depends on 22)

The system of claim 22, the vision-language model further configured to decompose the natural language prompt into a series of the pick-and-place steps for an object depicted in a scene.

### Claim 24 (depends on 23)

The system of claim 23, wherein each pick-and-place step is parameterized by a two-dimensional coordinate.

### Claim 25 (depends on 23)

The system of claim 23, further comprising a segmentation model configured to segment objects depicted in the scene.

### Claim 26 (depends on 22)

The system of claim 22, the vision-language model further configured to identify objects in its environment that are reasonable for the robot to manipulate.
