# US 2026/0061622 A1 · Method and system for generating robotic instructions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0061622 A1
- Document ID: US20260061622A1
- Status: Published application
- Kind: A1
- Application number: 18/821,834
- Filed: 2024-08-30
- Published: 2026-03-05
- Earliest claimed: 2024-08-30
- Applicants: Accenture Global Solutions Limited
- Inventors: Kumar ABHINAV; Alpana DUBEY; Shubhashis SENGUPTA
- Web page: https://patentgrep.com/patent/US20260061622A1

## CPC

- B25J 9/1697: {Vision controlled systems}
- G06V 10/82: using neural networks
- G06F 40/151: Transformation
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/803: {of input or preprocessed data}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 20/50: Context or environment of the image
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-implemented method to generate robotic instructions is disclosed. The method may include receiving video data demonstrating one or more tasks and text data related to the one or more tasks. Further, the method may include encoding the video data and the text data, wherein the encoding is generated using at least one cross-attentional transformer. The method also includes receiving image data providing environmental data for at least one robotic task. Furthermore, the method may include encoding vision data corresponding to the image data. Consequently, the method may include generating robotic instructions based upon the video data, the text data and the vision data that was encoded.

## Claims

### Claim 1

A method for generating robotic instructions from contextual data comprising:
receiving, at one or more processors, video data demonstrating one or more tasks;
receiving, at the one or more processors, text data related to the one or more tasks;
encoding, at the one or more processors, the video data and the text data, wherein the encoding is generated using at least one cross-attentional transformer;
receiving, at the one or more processors, image data providing environmental data for at least one robotic task;
encoding, at the one or more processors, vision data corresponding to the image data; and
generating, at the one or more processors, based upon the video data and the text data that was encoded and the vision data that was encoded, robotic instructions.

### Claim 2 (depends on 1)

The method as recited in claim 1, further comprising:
splitting each frame of the video data into a predetermined number of patches;
flattening each of the predetermined number of patches into a vector;
projecting each vector into a linear projection; and
executing a multi-head attention module based upon the linear projections of each of the predetermined number of patches.

### Claim 3 (depends on 2)

The method as recited in claim 2, further comprising determining a plurality of portions of an input sequence that includes a plurality of temporal dependencies.

### Claim 4 (depends on 1)

The method as recited in claim 1, wherein the encoding of the text data includes text embeddings from input text using a bi-direction encoder representations from transformers.

### Claim 5 (depends on 1)

The method as recited in claim 1, wherein the encoding of the video data and the text data creates a dynamicity of an object.

### Claim 6 (depends on 5)

The method as recited in claim 5, further comprising:
contextualizing the text data using a cross-attention transfer block attending to the video data and outputting features; and
contextualizing the features using a transformer block with self-attention.

### Claim 7 (depends on 2)

The method as recited in claim 2, wherein the patches are non-overlapping and a same size.

### Claim 8 (depends on 2)

The method as recited in claim 2, further comprising generating an object dynamic motion block based on an object bounding box representation that generates a fused object representation.

### Claim 9

A system for generating robotic instructions from contextual data, the system comprising:
at least one memory storing instructions; and
at least one processor communicatively coupled with the at least one memory and configured to execute the instructions to perform operations comprising:
receiving video data demonstrating one or more tasks;
receiving text data related to the one or more tasks;
encoding the video data and the text data, wherein the encoding is generated using at least one cross-attentional transformer;
receiving image data providing environmental data for at least one robotic task;
encoding vision data corresponding to the image data; and
generating, based upon the video data and the text data that was encoded and the vision data that was encoded, robotic instructions.

### Claim 10 (depends on 9)

The system as recited in claim 9, wherein the operations further comprise:
splitting each frame of the video data into a predetermined number of patches;
flattening each of the predetermined number of patches into a vector;
projecting each vector into a linear projection; and
executing a multi-head attention module based upon the linear projections of each of the predetermined number of patches.

### Claim 11 (depends on 10)

The system as recited in claim 10, wherein the operations further comprise determining a plurality of portions of an input sequence that includes a plurality of temporal dependencies.

### Claim 12 (depends on 11)

The system as recited in claim 11, wherein the text data includes text embeddings from input text using a bi-direction encoder representations from transformers.

### Claim 13 (depends on 12)

The system as recited in claim 12, wherein the encoding of the video data and the text data creates a dynamicity of an object.

### Claim 14 (depends on 13)

The system as recited in claim 13, wherein the operations further comprise:
contextualizing the text data using a cross-attention transfer block attending to the video data and outputting features; and
contextualizing the features using a transformer block with self-attention.

### Claim 15 (depends on 10)

The system as recited in claim 10, wherein the patches are non-overlapping and a same size.

### Claim 16 (depends on 10)

The system as recited in claim 10, wherein the operations further comprise generating an object dynamic motion block based on an object bounding box representation that generates a fused object representation.

### Claim 17

A non-transitory computer-readable media (CRM) storing instructions thereon, which, when executed by at least one processor of a computing device, cause the computing device to generate robotic instructions from contextual data by performing operations comprising:
receiving video data demonstrating one or more tasks;
receiving text data related to the one or more tasks;
encoding the video data and the text data, wherein the encoding is generated using at least one cross-attentional transformer;
receiving image data providing environmental data for at least one robotic task;
encoding vision data corresponding to the image data; and
generating, based upon the video data and the text data that was encoded and the vision data that was encoded, robotic instructions.

### Claim 18 (depends on 17)

The non-transitory CRM as recited in claim 17, wherein the operations further comprise:
splitting each frame of the video data into a predetermined number of patches;
flattening each of the predetermined number of patches into a vector;
projecting each vector into a linear projection; and
executing a multi-head attention module based upon the linear projections of each of the predetermined number of patches.

### Claim 19 (depends on 18)

The non-transitory CRM as recited in claim 18, wherein the operations further comprise:
determining a plurality of portions of an input sequence that includes a plurality of temporal dependencies;
contextualizing the text data using a cross-attention transfer block attending to the video data and outputting features; and
contextualizing the features using a transformer block with self-attention,
wherein the text data includes text embeddings from input text using a bi-direction encoder representations from transformers, and
wherein the encoding of the video data and the text data creates a dynamicity of an object.

### Claim 20 (depends on 18)

The non-transitory CRM as recited in claim 18, wherein the operations further comprise generating an object dynamic motion block based on an object bounding box representation that generates a fused object representation, and wherein the patches are non-overlapping and a same size.
