# US 2023/0398696 A1 · Motion guidance and natural language commands based robotic systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0398696 A1
- Document ID: US20230398696A1
- Status: Published application
- Kind: A1
- Application number: 17/840,070
- Filed: 2022-06-14
- Published: 2023-12-14
- Earliest claimed: 2022-06-14
- Assignees: Toyota Research Institute, Inc.; Toyota Jidosha Kabushiki Kaisha
- Applicants: Toyota Research Institute, Inc.
- Inventors: Thomas Kollar
- Web page: https://patentgrep.com/patent/US20230398696A1

## CPC

- B25J 13/003: {by means of an audio-responsive input (audible safety signals B25J19/061)}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1658: {characterised by programming language}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 11/008: {Manipulators for service tasks}
- G06F 40/205: Parsing
- G05B 2219/40393: Learn natural high level command, associate its template with a plan, sequence
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A robotic system is contemplated. The robotic system comprises a robot comprising a camera, a microphone, memory, and a controller that is configured to receive a natural language command for performing an action within a real world environment, parse the natural language command, categorize the action as being associated with guidance for performing the action, receive the guidance for performing the action, the guidance including a motion applied to at least one portion of the robot within the real world environment for performing the action, and store, in the memory, the natural language command in correlation with the motion that is applied to the at least one portion of the robot.

## Claims

### Claim 1

A method implemented by a computing device of a robot, the method comprising:
receiving a natural language command for performing an action within a real world environment;
parsing, by the computing device, the natural language command;
categorizing the action as being associated with guidance for performing the action;
receiving the guidance for performing the action, the guidance including a motion applied to at least one portion of the robot within the real world environment for performing the action; and
storing the natural language command in correlation with the motion in memory of the computing device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising entering a training mode upon categorizing the action as being associated with the guidance for performing the action.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving an additional natural language command for performing the action within the real world environment;
parsing, by the computing device, the additional natural language command; and
categorizing the action in the additional natural language command as a learned subroutine.

### Claim 4 (depends on 3)

The method of claim 3, wherein the learned subroutine is at least one of a navigation operation or a cleaning operation.

### Claim 5 (depends on 3)

The method of claim 3, further comprising automatically performing the action within the real world environment independent of the guidance.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
receiving an additional natural language command for performing an additional action within the real world environment; and
categorizing the additional action as an engineered subroutine based on preprogrammed capabilities specific to the robot; and.

### Claim 7 (depends on 6)

The method of claim 6, further comprising automatically performing the additional action within the real world environment responsive to categorizing the additional action as the engineered subroutine.

### Claim 8 (depends on 7)

The method of claim 7, wherein the engineered subroutine is at least one of a grasping operation, a placing operation, a wiping operation, or a tool change operation.

### Claim 9 (depends on 1)

The method of claim 1, further comprising receiving an additional natural language command for performing a plurality of actions according to a sequence, the sequence including a first action that is an engineered subroutine task and a second action that is a learned subroutine.

### Claim 10 (depends on 9)

The method of claim 9, further comprising automatically performing each of the plurality of actions according to the sequence.

### Claim 11 (depends on 6)

The method of claim 6, wherein the engineered subroutine includes one or more of identifying dimensions of objects within the real world environment and moving one or more of these objects within a predefined range of the robot.

### Claim 12

A robotic system comprising:
a robot comprising:
a camera;
a microphone;
memory;
a controller that is configured to:
receive a natural language command for performing an action within a real world environment,
parse the natural language command,
categorize the action as being associated with guidance for performing the action,
receive the guidance for performing the action, the guidance including a motion applied to at least one portion of the robot within the real world environment for performing the action, and
store the natural language command in correlation with the motion in the memory.

### Claim 13 (depends on 12)

The robotic system of claim 12, wherein the controller is further configured to enter a training mode upon categorizing the action as being associated with the guidance for performing the action.

### Claim 14 (depends on 12)

The robotic system of claim 12, wherein the controller is further configured to:
receive an additional natural language command for performing the action within the real world environment;
parse the additional natural language command; and
categorize the action in the additional natural language command as a learned subroutine.

### Claim 15 (depends on 14)

The robotic system of claim 14, wherein the learned subroutine is at least one of a navigation operation or a cleaning operation.

### Claim 16 (depends on 14)

The robotic system of claim 14, wherein the controller is further configured to automatically perform the action within the real world environment independent of the guidance.

### Claim 17 (depends on 12)

The robotic system of claim 12, wherein the controller is further configured to:
receive an additional natural language command for performing an additional action within the real world environment; and
categorize the additional action as an engineered subroutine based on preprogrammed capabilities specific to the robot.

### Claim 18 (depends on 17)

The robotic system of claim 17, wherein the engineered subroutine is at least one of a grasping operation, a placing operation, a wiping operation, or a tool change operation.

### Claim 19 (depends on 12)

The robotic system of claim 12, wherein the controller is further configured to:
receive an additional natural language command for performing a plurality of actions according to a sequence, the sequence including a first task that is an engineered subroutine and a second task that is a learned subroutine.

### Claim 20 (depends on 19)

The robotic system of claim 19, wherein the controller is further configured to automatically perform each of the plurality of actions according to the sequence.
