# US 2024/0202453 A1 · Generalizable instruction following with pre-trained language and grounding models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202453 A1
- Document ID: US20240202453A1
- Status: Published application
- Kind: A1
- Application number: 18/181,103
- Filed: 2023-03-09
- Published: 2024-06-20
- Earliest claimed: 2022-12-16
- Assignees: Honda Motor Co., Ltd.
- Applicants: Honda Motor Co., Ltd.
- Inventors: Gaoussou Youssouf KEBE; Kwonjoon LEE; Teruhisa MISU
- Web page: https://patentgrep.com/patent/US20240202453A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 40/205: Parsing
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for generalizable instruction following are provided. In one embodiment, a method includes generating a plurality of abstract tuples based on a set of instructions. The method includes classifying each abstract tuple of the plurality of abstract tuples as a navigation abstract tuple, an executable interaction abstract tuple, or a complex interaction abstract tuple based on the action. The method includes discarding the abstract tuples classified as navigation abstract tuples. The method includes forming set of executable interactions including the executable interaction abstract tuple. The method includes detecting an object location for the at least one object. The method includes decoding the complex interaction abstract tuples into executable interaction abstract tuples. The method includes adding the decoded executable interaction abstract tuples to the set of executable interactions. The method includes generating an executable plan for an agent based on the set of executable interactions and the object location.

## Claims

### Claim 1

A system for generalizable instruction following, comprising:
a processor; and
a memory storing instructions that when executed by the processor cause the processor to:
generate a plurality of abstract tuples based on a set of instructions, wherein an abstract tuple of the plurality of abstract tuples includes at least one action and at least one object;
classify each abstract tuple of the plurality of abstract tuples as a navigation abstract tuple, an executable interaction abstract tuple, or a complex interaction abstract tuple based on the at least one action, wherein the executable interaction abstract tuple includes a single executable interaction and the complex interaction abstract tuple includes multiple executable interactions;
discard the abstract tuples classified as navigation abstract tuples;
form a set of executable interactions including the executable interaction abstract tuple;
detect an object location for the at least one object based on a knowledge base;
decode the complex interaction abstract tuples into executable interaction abstract tuples;
add the decoded executable interaction abstract tuples to the set of executable interactions; and
generate an executable plan for an agent based on the set of executable interactions and the object location.

### Claim 2 (depends on 1)

The system of claim 1, wherein the instructions when executed by the processor further cause the processor to detect the object location includes generating candidate locations based on proposing semantically meaningful candidates for masked tokens.

### Claim 3 (depends on 2)

The system of claim 2, wherein the masked tokens represent identified objects in a semantic map based on a depth prediction model generated from image data or depth data captured by the agent.

### Claim 4 (depends on 3)

The system of claim 3, wherein the semantic map is based on the depth prediction model and grounded language-image pre-training, wherein the grounded language-image pre-training converts object bounding boxes to natural language phrases.

### Claim 5 (depends on 1)

The system of claim 1, wherein the instructions when executed by the processor further cause the processor to:
receive a task from a biological entity; and
identify the set of instructions based on the task.

### Claim 6 (depends on 1)

The system of claim 1, wherein the executable plan is executed by the agent based on a semantic map.

### Claim 7 (depends on 1)

The system of claim 1, wherein the abstract tuples are based on semantic parsing.

### Claim 8

A method for generalizable instruction following, comprising:
generating a plurality of abstract tuples based on a set of instructions, wherein an abstract tuple of the plurality of abstract tuples includes at least one action and at least one object;
classifying each abstract tuple of the plurality of abstract tuples as a navigation abstract tuple, an executable interaction abstract tuple, or a complex interaction abstract tuple based on the at least one action, wherein the executable interaction abstract tuple includes a single executable interaction and the complex interaction abstract tuple includes multiple executable interactions;
discarding the abstract tuples classified as navigation abstract tuples;
forming a set of executable interactions including the executable interaction abstract tuple;
detecting an object location for the at least one object based on a knowledge base;
decoding the complex interaction abstract tuples into executable interaction abstract tuples;
adding the decoded executable interaction abstract tuples to the set of executable interactions; and
generating an executable plan for an agent based on the set of executable interactions and the object location.

### Claim 9 (depends on 8)

The method of claim 8, wherein detecting the object location includes generating candidate locations based on proposing semantically meaningful candidates for masked tokens.

### Claim 10 (depends on 9)

The method of claim 9, wherein the masked tokens represent identified objects in a semantic map based on a depth prediction model generated from image data or depth data captured by the agent.

### Claim 11 (depends on 10)

The method of claim 10, wherein the semantic map is based on the depth prediction model and grounded language-image pre-training, wherein the grounded language-image pre-training converts object bounding boxes to natural language phrases.

### Claim 12 (depends on 8)

The method of claim 8, further comprising:
receiving a task from a biological entity; and
identifying the set of instructions based on the task.

### Claim 13 (depends on 8)

The method of claim 8, wherein the executable plan is executed by the agent based on a semantic map.

### Claim 14 (depends on 8)

The method of claim 8, wherein the abstract tuples are based on semantic parsing.

### Claim 15

A non-transitory computer readable storage medium storing instructions that when executed by a computer, which includes a processor perform a method, the method comprising:
generating a plurality of abstract tuples based on a set of set of instructions, wherein an abstract tuple of the plurality of abstract tuples includes at least one action and at least one object;
classifying each abstract tuple of the plurality of abstract tuples as a navigation abstract tuple or an interaction abstract tuple based on the at least one action;
discarding the abstract tuples classified as navigation abstract tuples;
detecting an object location for the at least one object based on a knowledge base;
classifying each interaction abstract tuple as an executable interaction abstract tuple or a complex interaction abstract tuple, wherein the executable interaction abstract tuple includes a single executable interaction and the complex interaction abstract tuple includes multiple executable interactions;
forming a set of executable interactions including the executable interaction abstract tuple;
decoding the complex interaction abstract tuples into executable interaction abstract tuples;
adding the decoded executable interaction abstract tuples to the set of executable interactions; and
generating an executable plan for an agent based on the set of executable interactions and the object location.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein detecting the object location includes generating candidate locations based on proposing semantically meaningful candidates for masked tokens.

### Claim 17 (depends on 16)

The non-transitory computer readable storage medium of claim 16, wherein the masked tokens represent identified objects in a semantic map based on a depth prediction model generated from image data or depth data captured by the agent.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the semantic map is based on the depth prediction model and grounded language-image pre-training, wherein the grounded language-image pre-training converts object bounding boxes to natural language phrases.

### Claim 19 (depends on 15)

The non-transitory computer readable storage medium of claim 15, further comprising:
receiving a task from a biological entity; and
identifying the set of instructions based on the task.

### Claim 20 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the executable plan is executed by the agent based on a semantic map.
