# US 2026/0099304 A1 · Techniques for closed-loop code generation for robot control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0099304 A1
- Document ID: US20260099304A1
- Status: Published application
- Kind: A1
- Application number: 19/249,907
- Filed: 2025-06-25
- Published: 2026-04-09
- Earliest claimed: 2024-10-03
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Valts BLUKIS; Alexander ZOOK; Stanley BIRCHFIELD; Balakumar SUNDARALINGAM; Jonathan TREMBLAY
- Web page: https://patentgrep.com/patent/US20260099304A1

## CPC

- B25J 9/1658: {characterised by programming language}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems}
- B25J 9/1697: {Vision controlled systems}
- G06F 8/35: model driven
- G06F 11/0736: {in functional embedded systems, i.e. in a data processing system designed as a combination of hardware and software dedicated to performing a certain function (testing or monitoring of automated control systems G05B23/02)}
- G06F 11/0793: {Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40)}
- G06F 11/3636: {by tracing the execution of the program}
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G06V 20/50: Context or environment of the image
- G06V 20/64: Three-dimensional [3D] objects
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

One embodiment of a method for processing data includes receiving an image; segmenting, using a first trained machine learning model, the image to generate a segmentation mask; generating one or more descriptions of one or more objects using a second machine learning model and based on the segmentation mask; generating program code using a third trained machine learning model and based on the one or more descriptions, the image, and a task; and causing a robot to move based on the program code.

## Claims

### Claim 1

A computer-implemented method for robot control, the method comprising:
receiving a first image;
segmenting, using a first trained machine learning model, the first image to generate a first segmentation mask;
generating one or more descriptions of one or more objects using a second trained machine learning model and based on the first segmentation mask;
generating first program code using a third trained machine learning model and based on the one or more descriptions, the first image, and a task; and
causing a robot to move based on the first program code.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising generating a scene description that includes the one or more descriptions, wherein the first program code is generated based on the scene description, the first image, and the task.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
receiving 3D information; and
determining a spatial representation of each object included in the one or more objects based on the first image, the 3D information, and the first segmentation mask, wherein the scene description further includes the spatial representation of each object.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
simulating execution of the first program code; and
in response to one or more errors, generating second program code using the third trained machine learning model and based on the one or more errors.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the first program code includes one or more calls to one or more functions associated with one or more skills that the robot is able to perform.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the first program code includes one or more calls to one or more functions of an application programming interface (API) that is queried with natural language statements.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a second image and 3D information;
segmenting, using the first trained machine learning model and based on the first segmentation mask, the second image to generate a second segmentation mask;
determining an updated spatial representation of each object included in the one or more objects based on the second image, the 3D information, and the second segmentation mask; and
generating second program code using the third trained machine learning model and based on the updated spatial representation of each object, the second image, and the task; and
causing a robot to move based on the second program code.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein causing the robot to move comprises:
performing one or more motion planning operations to determine one or more joint angle updates; and
transmitting the one or more joint angle updates to a controller of the robot.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein each description included in the one or more descriptions includes at least one of a shape, a color, a size, a branding, an orientation, or a text.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the first trained machine learning model comprises a segmentation model, wherein the second trained machine learning model comprises a vision-language model, and wherein the third trained machine learning model comprises a multimodal model.

### Claim 11

One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of:
receiving a first image;
segmenting, using a first trained machine learning model, the first image to generate a first segmentation mask;
generating one or more descriptions of one or more objects using a second trained machine learning model and based on the first segmentation mask;
generating first program code using a third trained machine learning model and based on the one or more descriptions, the first image, and a task; and
causing a robot to move based on the first program code.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of generating a scene description that includes the one or more descriptions, wherein the first program code is generated based on the scene description, the first image, and the task.

### Claim 13 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the steps of:
receiving 3D information; and
determining a spatial representation of each object included in the one or more objects based on the first image, the 3D information, and the first segmentation mask, wherein the scene description further includes the spatial representation of each object.

### Claim 14 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the steps of:
simulating execution of the first program code; and
in response to one or more errors, generating second program code using the third trained machine learning model and based on the one or more errors.

### Claim 15 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the first program code includes one or more calls to one or more functions associated with one or more skills that the robot is able to perform.

### Claim 16 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the steps of:
receiving a second image and 3D information;
segmenting, using the first trained machine learning model and based on the first segmentation mask, the second image to generate a second segmentation mask;
determining an updated spatial representation of each object included in the one or more objects based on the second image, the 3D information, and the second segmentation mask; and
generating second program code using the third trained machine learning model and based on the updated spatial representation of each object, the second image, and the task; and
causing the robot to move based on the second program code.

### Claim 17 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to perform the step of receiving natural language text describing the task.

### Claim 18 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein generating the one or more descriptions comprises:
asking the second trained machine learning model to identify manipulable objects in the first image; and
asking the second trained machine learning model to describe each object included in the one or more objects.

### Claim 19 (depends on 11)

The one or more non-transitory computer-readable media of claim 11, wherein the first trained machine learning model comprises a segmentation model, wherein the second trained machine learning model comprises a vision-language model, and wherein the third trained machine learning model comprises a multimodal model.

### Claim 20

A system, comprising:
a memory storing instructions; and
one or more processors, that when executing the instructions, are configured to perform the steps of:
receiving an image,
segmenting, using a first trained machine learning model, the image to generate a segmentation mask,
generating one or more descriptions of one or more objects using a second trained machine learning model and based on the segmentation mask,
generating program code using a third trained machine learning model and based on the one or more descriptions, the image, and a task, and
causing a robot to move based on the program code.
