US 12,746,667 B2Granted

Actuation of a robot to perform complex tasks

AssigneeRobert Bosch GmbH

InventorsAndras Gabor Kupcsik, Christian Graf, Christian Rauch, Patrick Kesper

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method for actuating a robot to perform a task. In the method, at least one image of a scene is provided; object types are ascertained for object(s) in the image; the task is fed with the object types to a trained language model, the trained language model outputs a plurality of candidate actions; the candidate actions are evaluated with a progress metric to determine the extent to which the performance of the candidate action promises progress with regard to the predetermined task, and evaluated with a predetermined success metric to determine the probability with which an attempt to perform the candidate action will be successful; the values of the progress metric and the success metric are merged into an overall rating of the candidate action; a candidate action having the best overall rating is selected; and the robot is actuated to perform the selected candidate action.

Claims17

Dependent claims

A method for actuating a robot to perform a predetermined task, the method comprising the following steps:

providing at least one image of a scene in which the task is to be performed;
ascertaining object types for one or more objects in the image;
feeding the task in combination with the object types to a trained language model, whereupon the trained language model outputs a plurality of scene-specific candidate actions describing alternative ways of interacting with the objects in the scene;
based on the current scene and the object types ascertained therefrom:
evaluating each of the candidate actions with a predetermined progress metric that quantifies an expected degree to which execution of the respective candidate action advances completion of the predetermined task irrespective of feasibility of execution, and
evaluating each of the candidate actions with a predetermined success metric that quantifies a likelihood that execution of the respective candidate action will succeed in the current scene given physical, geometric, or operational constraints;
for each of the candidate actions, merging respective values of the progress metric and the success metric relating to the candidate action into a respective overall rating of the respective candidate action, wherein the merging comprises:
semantically evaluating the progress metric of the respective candidate action in accordance with one or more progress-related conditions that are specific to advancement of predetermined task;
semantically evaluating the success metric of the respective candidate action in accordance with one or more success-related conditions that are specific to likelihood of successful execution of the respective candidate action in the current scene;
and
jointly determining the overall rating of the respective candidate action based on outcomes of the progress-related evaluation and the success-related evaluation, such that failure to satisfy either the progress-related conditions or the success-related conditions is sufficient to disfavor or exclude the respective candidate action;
comparing the overall ratings of the candidate actions;
based on a result of the comparison, selecting one of the candidate actions as having a best overall rating;
and
actuating the robot to perform the selected candidate action.

12 more elements

A non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions for actuating a robot to perform a predetermined task, the instructions, when executed by one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps:

providing at least one image of a scene in which the task is to be performed;
ascertaining object types for one or more objects in the image;
feeding the task in combination with the object types to a trained language model, whereupon the trained language model outputs a plurality of scene-specific candidate actions describing alternative ways of interacting with the objects in the scene;
based on the current scene and the object types ascertained therefrom:
evaluating each of the candidate actions with a predetermined progress metric that quantifies an expected degree to which execution of the respective candidate action advances completion of the predetermined task irrespective of feasibility of execution, and
evaluating each of the candidate actions with a predetermined success metric that quantifies a likelihood that execution of the respective candidate action will succeed in the current scene given physical, geometric, or operational constraints;
for each of the candidate actions, merging respective values of the progress metric and the success metric relating to the candidate action into a respective overall rating of the respective candidate action, wherein the merging comprises:
semantically evaluating the progress metric of the respective candidate action in accordance with one or more progress-related conditions that are specific to advancement of predetermined task;
semantically evaluating the success metric of the respective candidate action in accordance with one or more success-related conditions that are specific to likelihood of successful execution of the respective candidate action in the current scene;
and
jointly determining the overall rating of the respective candidate action based on outcomes of the progress-related evaluation and the success-related evaluation, such that failure to satisfy either the progress-related conditions or the success-related conditions is sufficient to disfavor or exclude the respective candidate action;
comparing the overall ratings of the candidate actions;
based on a result of the comparison, selecting one of the candidate actions as having a best overall rating;
and
actuating the robot to perform the selected candidate action.

12 more elements

One or more computers and/or compute instances having a non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions for actuating a robot to perform a predetermined task, the instructions, when executed by the one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps:

providing at least one image of a scene in which the task is to be performed;
ascertaining object types for one or more objects in the image;
feeding the task in combination with the object types to a trained language model, whereupon the trained language model outputs a plurality of scene-specific candidate actions describing alternative ways of interacting with the objects in the scene;
based on the current scene and the object types ascertained therefrom:
evaluating each of the candidate actions with a predetermined progress metric that quantifies an expected degree to which execution of the respective candidate action advances completion of the predetermined task irrespective of feasibility of execution, and
evaluating each of the candidate actions with a predetermined success metric that quantifies a likelihood that execution of the respective candidate action will succeed in the current scene given physical, geometric, or operational constraints;
for each of the candidate actions, merging respective values of the progress metric and the success metric relating to the candidate action into a respective overall rating of the respective candidate action, wherein the merging comprises:
semantically evaluating the progress metric of the respective candidate action in accordance with one or more progress-related conditions that are specific to advancement of predetermined task;
semantically evaluating the success metric of the respective candidate action in accordance with one or more success-related conditions that are specific to likelihood of successful execution of the respective candidate action in the current scene;
and
jointly determining the overall rating of the respective candidate action based on outcomes of the progress-related evaluation and the success-related evaluation, such that failure to satisfy either the progress-related conditions or the success-related conditions is sufficient to disfavor or exclude the respective candidate action;
comparing the overall ratings of the candidate actions;
based on a result of the comparison, selecting one of the candidate actions as having a best overall rating;
and
actuating the robot to perform the selected candidate action.

12 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/738,113Filed 10 Jun 2024
Priority: DE 10 2023 205 539.214 Jun 2023
Published: US 2024/0416518 A119 Dec 2024
Granted: 29 Sep 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification7 codes

B25J 9/1661
Characterised by programming, planning systems for manipulators: characterised by task planning, object-oriented languages
B25J 9/1671
Characterised by programming, planning systems for manipulators: characterised by simulation, either to verify existing program or to create and…
B25J 9/1697
Characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor…:…
G05B 19/4155
Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to…