# US 2023/0405812 A1 · Determining and utilizing corrections to robot actions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0405812 A1
- Document ID: US20230405812A1
- Status: Published application
- Kind: A1
- Application number: 18/239,735
- Filed: 2023-08-29
- Published: 2023-12-21
- Earliest claimed: 2017-07-03
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Nicolas Hudson; Devesh Yamparala
- Web page: https://patentgrep.com/patent/US20230405812A1

## CPC

- B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor}
- B25J 9/1602: {characterised by the control system, structure, architecture}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1656: {characterised by programming, planning systems for manipulators}
- B25J 9/1697: {Vision controlled systems}
- G05B 13/027: {using neural networks only}
- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G05B 2219/33036: Error back propagation
- G05B 2219/33037: Learn parameters of network offline, not while controlling system
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, apparatus, and computer-readable media for determining and utilizing human corrections to robot actions. In some implementations, in response to determining a human correction of a robot action, a correction instance is generated that includes sensor data, captured by one or more sensors of the robot, that is relevant to the corrected action. The correction instance can further include determined incorrect parameter(s) utilized in performing the robot action and/or correction information that is based on the human correction. The correction instance can be utilized to generate training example(s) for training one or model(s), such as neural network model(s), corresponding to those used in determining the incorrect parameter(s).

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
receiving, via at least one microphone of a robot, an instance of spoken natural language user interface input provided by a user;
processing the instance of spoken natural language user interface input to determine that the instance of spoken natural language user interface input conforms to a sequence of robotic actions;
in response to determining that the spoken natural language user interface input conforms to the sequence of robotic actions:
providing control commands to one or more actuators of the robot to autonomously perform each of the robotic actions of the sequence, wherein each of the robotic actions comprises autonomously moving one or more corresponding components of the robot; and
during autonomous performance of each of the robotic actions of the sequence, rendering, via at least one user interface output device of the robot, corresponding natural language user interface output that describes the robotic action being performed.

### Claim 2 (depends on 1)

The method of claim 1, wherein the rendering comprises visually rendering the corresponding natural language user interface output via a display of the at least one user interface output device of the robot.

### Claim 3 (depends on 2)

The method of claim 2, wherein the rendering further comprises speaking the corresponding natural language user interface output via a speaker of the at least one user interface output device of the robot.

### Claim 4 (depends on 1)

The method of claim 1, wherein the rendering comprises speaking the corresponding natural language user interface output via a speaker of the at least one user interface output device of the robot.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
receiving, in response to providing one of the corresponding natural language user interface outputs, user interface input;
determining that the user interface input indicates a correction to the robotic action being performed when the user interface input was received; and
in response to determining that the user interface input indicates the correction:
generating a training example that is based on the correction.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
training one or more neural network models based on the training example.

### Claim 7 (depends on 1)

The method of claim 1, wherein the robotic actions of the sequence include a picking action directed to an object and a placing action directed to the object.

### Claim 8 (depends on 7)

The method of claim 7, wherein the rendering comprises speaking the corresponding natural language user interface output via a speaker of the at least one user interface output device of the robot.

### Claim 9

A robot comprising:
at least one microphone;
at least one user interface output device;
actuators;
memory storing instructions;
one or more processors configured to execute the instructions to:
receive, via the at least one microphone, an instance of spoken natural language user interface input provided by a user;
process the instance of spoken natural language user interface input to determine that the instance of spoken natural language user interface input conforms to a sequence of robotic actions;
in response to determining that the spoken natural language user interface input conforms to the sequence of robotic actions:
provide control commands to one or more of the actuators to autonomously perform each of the robotic actions of the sequence, wherein each of the robotic actions comprises autonomously moving one or more corresponding components of the robot; and
during autonomous performance of each of the robotic actions of the sequence, render, via the user interface output device, corresponding natural language user interface output that describes the robotic action being performed.

### Claim 10 (depends on 9)

The robot of claim 9, wherein the at least one user interface output device includes a display and wherein the corresponding natural language user interface output is rendered via the display.

### Claim 11 (depends on 10)

The robot of claim 10, wherein the at least one user interface output device further includes a speaker and wherein the corresponding natural language user interface output is also rendered via the speaker.

### Claim 12 (depends on 9)

The robot of claim 9, wherein the at least one user interface output device includes a speaker and wherein the corresponding natural language user interface output is rendered via the speaker.

### Claim 13 (depends on 9)

The robot of claim 9, wherein one or more of the processors are further configured to execute the instructions to:
receive, in response to providing one of the corresponding natural language user interface outputs, user interface input;
determine that the user interface input indicates a correction to the robotic action being performed when the user interface input was received; and
in response to determining that the user interface input indicates the correction:
generate a training example that is based on the correction.

### Claim 14 (depends on 13)

The robot of claim 13, wherein one or more of the processors are further configured to execute the instructions to:
train one or more neural network models based on the training example.

### Claim 15 (depends on 9)

The robot of claim 9, wherein the robotic actions of the sequence include a picking action directed to an object and a placing action directed to the object.

### Claim 16 (depends on 15)

The robot of claim 15, wherein the at least one user interface output device includes a speaker and wherein the corresponding natural language user interface output is rendered via the speaker.

### Claim 17

A system comprising:
memory storing instructions;
one or more processors configured to execute the instructions to:
receive, via at least one microphone of a robot, an instance of spoken natural language user interface input provided by a user;
process the instance of spoken natural language user interface input to determine that the instance of spoken natural language user interface input conforms to a sequence of robotic actions;
in response to determining that the spoken natural language user interface input conforms to the sequence of robotic actions:
provide control commands to one or more actuators of the robot to autonomously perform each of the robotic actions of the sequence, wherein each of the robotic actions comprises autonomously moving one or more corresponding components of the robot; and
during autonomous performance of each of the robotic actions of the sequence, render, via a user interface output device of the robot, corresponding natural language user interface output that describes the robotic action being performed.
