US 11,731,271 B2 · Verbal-based focus-of-attention task model encoder

Abstract

Traditionally, robots may learn to perform tasks by observation in clean or sterile environments. However, robots are unable to accurately learn tasks by observation in real environments (e.g., cluttered, noisy, chaotic environments). Methods and systems are provided for teaching robots to learn tasks in real environments based on input (e.g., verbal or textual cues). In particular, a verbal-based Focus-of-Attention (FOA) model receives input, parses the input to recognize at least a task and a target object name. This information is used to spatio-temporally filter a demonstration of the task to allow the robot to focus on the target object and movements associated with the target object within a real environment. In this way, using the verbal-based FOA, a robot is able to recognize “where and when” to pay attention to the demonstration of the task, thereby enabling the robot to learn the task by observation in a real environment.

Claims19

Dependent claims

A computer-implemented method for teaching a robot a task in a cluttered environment, comprising:

receiving an input;
parsing the input to identify a task and a target object name;
receiving a set of time-series images;
detecting a plurality of objects within the set of time-series images, wherein the set of time-series images depicts a demonstration of the task associated with a target object;
based on the target object name, identifying the target object among the plurality of objects within the set of time-series images;
generating a spatially filtered set of time-series images by spatially filtering the set of time-series images based on the target object;
identifying a timing of at least one physical human movement for performing the task associated with the target object within the spatially filtered set of time-series images;
generating a spatio-temporal filtered set of time-series images by temporally filtering the spatially filtered set of time-series images based on the timing of the at least one physical human movement;
and
evaluating the spatio-temporal filtered set of time-series images to isolate one or more skill parameters associated with performing the task.

7 more elements

A system comprising:

at least one processor;
and
at least one memory communicatively coupled to the at least one processor and having computer-executable instructions stored thereon, the computer-executable instructions when executed by the at least one processor causing the system to:
receive a verbal cue;
parse the verbal cue to identify a task and a target object name;
receive a set of time-series images;
detect a plurality of objects within the set of time-series images, wherein the set of time-series images depicts a demonstration of the task associated with a target object;
based on the target object name, identify the target object from among the plurality of objects within the set of time-series images;
generate a spatially filtered set of time-series images by spatially filtering the set of time-series images based on the target object;
identify a timing of at least one physical human movement for performing the task associated with the target object within the spatially filtered set of time-series images;
generate a spatio-temporal filtered set of time-series images by temporally filtering the spatially filtered set of time-series images based on the timing of the at least one physical human movement;
and
evaluate the spatio-temporal filtered set of time-series images to identify one or more skill parameters associated with performing the task.

10 more elements

A computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions when executed by a processor causing a computer system to:

receive an input;
parse the input to identify a task and a target object name;
receive a set of time-series images;
detect a plurality of objects within the set of time-series images, wherein the set of time-series images depicts a demonstration of the task associated with a target object;
based on the target object name, identify the target object among the plurality of objects within the set of time-series images;
generate a spatially filtered set of time-series images by spatially filtering the set of time-series images based on the target object;
identify a timing of at least one physical human movement for performing the task associated with the target object within the spatially filtered set of time-series images;
generate a spatio-temporal filtered set of time-series images by temporally filtering the spatially filtered set of time-series images based on the timing of the at least one physical human movement;
evaluate the spatio-temporal filtered set of time-series images to identify one or more skill parameters associated with performing the task;
and
encode at least the one or more skill parameters as a task model.

8 more elements

Dependencies are inferred from the claim text.

Verbal-based focus-of-attention task model encoder

Life of this patent

Verbal-based focus-of-attention task model encoder

Life of this patent

Abstract

Claims19