# US 11,298,825 B2 · Systems, apparatus, and methods for robotic learning and execution of skills

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,298,825 B2
- Document ID: US11298825B2
- Status: Granted
- Kind: B2
- Application number: 17/504,130
- Filed: 2021-10-18
- Published: 2022-02-03
- Granted: 2022-04-12
- Earliest claimed: 2017-02-25
- Assignees: Diligent Robotics, Inc.
- Applicants: Diligent Robotics, Inc.
- Inventors: Vivian Yaw-Wen Chu; Shuai Li; Forrest Green; Peter Worsnop; Andrea Lockerd Thomaz
- Web page: https://patentgrep.com/patent/US11298825B2

## CPC

- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 11/00: Manipulators not otherwise provided for
- B25J 5/00: Manipulators mounted on wheels or on carriages (B25J1/00 takes precedence; program-controlled manipulators B25J9/00)
- B25J 9/0081: {with leader teach-in means}
- B25J 9/06: characterised by multi-articulated arms
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1694: {characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion}
- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- B25J 15/00: Gripping heads {and other end effectors (grippers used in machine tools B23Q7/04; gripping members fitted on cranes B66C1/42, B66C1/44; gripping means used for mounting electrical components H05K13/04; gripping means used in the manufacture of semiconductors H10P72/7602)}
- B25J 19/02: Sensing devices
- G05B 2219/40116: Learn by operator observation, symbiosis, show, watch
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Systems, apparatus, and methods are described for robotic learning and execution of skills. A robotic apparatus can include a memory, a processor, sensors, and one or more movable components (e.g., a manipulating element and/or a transport element). The processor can be operatively coupled to the memory, the movable elements, and the sensors, and configured to obtain information of an environment, including one or more objects located within the environment. In some embodiments, the processor can be configured to learn skills through demonstration, exploration, user inputs, etc. In some embodiments, the processor can be configured to execute skills and/or arbitrate between different behaviors and/or actions. In some embodiments, the processor can be configured to learn an environmental constraint. In some embodiments, the processor can be configured to learn using a general model of a skill.

## Claims

### Claim 1

A robotic device, comprising:
a memory;
a processor;
a manipulating element; and
a set of sensors,
the processor operatively coupled to the memory, the manipulating element, and the set of sensors, the processor configured to:
obtain, via a first subset of sensors from the set of sensors, a representation of an environment including at least one physical object;
obtain, via a second subset of sensors from the set of sensors, sensory information at each keyframe from a plurality of keyframes during a demonstration of a skill by a user in which the user demonstrates a physical interaction between the manipulating element and the at least one physical object, the set of sensory information including information of a set of features associated with the manipulating element and the environment, each keyframe of the plurality of keyframes representing a discrete point during the demonstration of the skill with at least two adjacent keyframes being separated from one another by a time interval;
present, via a user interface and to the user, the set of features such that the user can select a subset of the set of features that is relevant to learning the skill from the set of features; and
generate a model for executing the skill using a subset of the set of sensory information that is associated with the subset of features selected by the user.

### Claim 2 (depends on 1)

The robotic device of claim 1, wherein the processor is further configured to:
identify a set of markers in the representation of the environment, at least one marker from the set of markers being associated with the at least one physical object;
present, via the user interface and to the user, the set of markers in the representation of the environment; and
in response to the user accepting the set of markers, store a corresponding location for each marker in the set of markers in a map of the environment.

### Claim 3 (depends on 1)

The robotic device of claim 1, wherein the second subset of sensors is different from the first subset of sensors.

### Claim 4 (depends on 1)

The robotic device of claim 1, wherein:
the manipulating element includes a plurality of movable components joined via a plurality of joints,
the set of sensors includes a sensor configured to measure a force acting on a joint from the plurality of joints; and
the processor is configured to generate the model for executing the skill using at least sensory information indicative of a force measured by the sensor.

### Claim 5 (depends on 1)

The robotic device of claim 1, wherein:
the manipulating element includes (i) a plurality of joints, and (ii) an end effector configured to move a physical object from the set of physical objects,
the set of sensors includes a sensor configured to measure a force placed on at least one of the end effector or a joint from the plurality of joints coupled to the end effector when the end effector is moving the physical object, and
the processor is configured to generate the model for executing the skill using at least sensory information indicative of a force measured by the sensor.

### Claim 6 (depends on 1)

The robotic device of claim 1, wherein the processor is configured to generate the model for executing the skill by:
selecting a Hidden Markov Model (HMM) or support vector machine (SVM) algorithm from a machine learning library stored in the memory; and
setting one or more parameters of the HMM or SVM algorithm using the subset of sensory information that is associated with the subset of features selected by the user.

### Claim 7 (depends on 1)

The robotic device of claim 1, wherein the representation of the environment is a first representation of the environment,
the processor further configured to, after generating the model for executing the skill:
obtain, via the first subset of sensors, a second representation of the environment;
identify, via visual markers or object recognition based on object information, a plurality of markers in the second representation of the environment, each marker from the plurality of markers associated with a physical object from a set of physical objects located in the environment; and
generate, using the model and a set of markers from the plurality of markers, a trajectory to execute the skill using at least the manipulating element.

### Claim 8 (depends on 7)

The robotic device of claim 7, wherein the processor is further configured to:
present, via the user interface and to the user, the trajectory such that the user can accept or reject the trajectory; and
in response to receiving an input from the user indicating a rejection of the trajectory, modify one or more parameters of the model.

### Claim 9 (depends on 7)

The robotic device of claim 7, wherein the processor is further configured to:
execute the skill using at least the manipulating element;
obtain, using the set of sensors, additional sensory information during the execution of the skill;
determine whether the execution of the skill was successful based on the additional sensory information; and
in response to determining that the execution of the skill was not successful, modifying one or more parameters of the model.

### Claim 10 (depends on 7)

The robotic device of claim 7, wherein the processor is further configured to:
execute the skill using at least the manipulating element;
receive, via the user interface and from the user, an input indicating whether the execution of the skill was successful; and
in response to the input indicating that the execution of the skill was not successful, modifying one or more parameters of the model.

### Claim 11 (depends on 1)

The robotic device of claim 1, wherein the processor is further configured to:
when the user moves the manipulating element through the demonstration of the skill, monitor changes to the manipulating element; and
in response to detecting a directional change in the manipulating element, select a point associated with the directional change as a keyframe.

### Claim 12

A robotic device, comprising:
a memory;
a processor;
a manipulating element;
a transport element configured to move along a surface; and
a set of sensors,
the processor operatively coupled to the memory, the manipulating element, the transport element, and the set of sensors, the processor configured to:
obtain, via the set of sensors, information of an environment;
select, based on an arbitration algorithm or an input from a user, a skill to execute from a plurality of skills, the execution of the skill including a physical interaction between the manipulating element and a physical object in the environment;
generate, using a model for executing the skill, a first plan to execute the skill based on the information of the environment;
move the manipulating element and the transport element based on the first plan to execute at least a first portion of the skill while obtaining additional information of the environment;
in response to detecting a predefined change in the environment based on the additional information obtained of the environment, interrupt the execution of the skill;
prompt a user to provide an input based on the predefined change in the environment; and
generate a second plan to execute a second portion of the skill based on the input.

### Claim 13 (depends on 12)

The robotic device of claim 12, wherein the processor is further configured to move the manipulating element and the transport element based on the second plan to execute the second portion of the skill.

### Claim 14 (depends on 12)

The robotic device of claim 12, wherein the physical object is a human, and the skill includes an interaction with the human,
the processor configured to move the manipulating element and the transport element to execute at least a portion of the social behavior near the human.

### Claim 15 (depends on 12)

The robotic device of claim 12, wherein the processor is configured to prompt the user to provide the input by requesting a demonstration from the user, the processor further configured to:
obtain, for each keyframe from a plurality of keyframes associated with the demonstration, information collected by the set of sensors and associated with at least one of the environment, the manipulating element, or the transport element; and
generate, using the model and the information associated with the at least one of the environment, the manipulating element, or the transport element, a specialized model for executing at least the second portion of the skill in the environment.

### Claim 16 (depends on 15)

The robotic device of claim 15, wherein the processor is configured to generate the second plan using the specialized model.

### Claim 17 (depends on 12)

The robotic device of claim 12, wherein the processor is further configured to:
obtain, via the set of sensors, information associated with at least one of the environment, the manipulating element, or the transport element during movement of the manipulating element and the transport element; and
update a map of the environment based on the information associated with the at least one of the environment, the manipulating element, or the transport element.

### Claim 18 (depends on 17)

The robotic device of claim 17, wherein the information associated with the at least one of the environment, the manipulating element, or the transport element includes information of one or more objects in the environment, and the map of the environment includes at least a semantic layer and a contextual layer, the processor being configured to update the map of the environment by:
storing the information associated with the one or more objects in the environment in the semantic layer;
determining, based on the information associated with the one or more objects, one or more behaviors to execute near the one or more objects; and
storing the one or more behaviors in the contextual layer.

### Claim 19 (depends on 12)

The robotic device of claim 12, wherein the processor is further configured to:
display to the user the second plan to execute the second portion of the skill; and
receive, after the displaying, another input from the user indicating confirmation to proceed with executing the second plan,
the processor configured to move at least one of the manipulating element or the transport element in response to receiving the other input.

### Claim 20 (depends on 12)

The robotic device of claim 12, wherein the processor is further configured to:
display to the user the second plan to execute the second portion of the skill;
receive, after the displaying, another input from the user indicating a change to the second plan; and
modify the second plan based on the other input,
the processor configured to move at least one of the manipulating element or the transport element based on the second plan after modifying the second plan.

### Claim 21

A robotic device, comprising:
a memory;
a processor;
a manipulating element;
a transport element configured to move along a surface; and
a set of sensors,
the processor operatively coupled to the memory, the manipulating element, the transport element, and the set of sensors, the processor configured to:
obtain, via the set of sensors, information of a first portion of an environment;
select, based on an arbitration algorithm or an input from a user, a skill to execute from a plurality of skills, the execution of the skill including a physical interaction between the manipulating element and a physical object in the environment;
generate, using a model for executing at least a first portion of the skill, a first plan to execute the first portion of the skill based on the information of the first portion of the environment;
move the manipulating element and the transport element based on the first plan to execute the first portion of the skill;
obtain, via the set of sensors, information of a second portion of an environment that is different from the first portion of the environment;
in response to determining that user input is required to execute a second portion of the skill in the second portion of the environment, prompt the user to provide at least one input associated with executing the second portion of the skill; and
generate a second plan to execute the second portion of the skill based on the user input.

### Claim 22 (depends on 21)

The robotic device of claim 21, wherein the processor is further configured to move the manipulating element and the transport element based on the second plan to execute the second portion of the skill.

### Claim 23 (depends on 21)

The robotic device of claim 21, wherein the physical object is a human, and the skill includes an interaction with the human, the processor configured to move the manipulating element and the transport element to execute the social behavior near the human.

### Claim 24 (depends on 21)

The robotic device of claim 21, wherein the processor is further configured to:
obtain, via the set of sensors, information of at least one of the environment, the manipulating element, or the transport element during the movement of the manipulating element and the transport element;
determine, based on the information associated with the at least one of the environment, the manipulating element, or the transport element, that the movement of the manipulating element and the transport element failed to meet one or more success criteria;
modify the first plan to execute the first portion of the skill in response to determining that the movement failed to meet the one or more success criteria; and
move the manipulating element and the transport element based on the first plan after modifying the first plan.

### Claim 25 (depends on 21)

The robotic device of claim 21, wherein the processor is further configured to:
obtain, via the set of sensors, information associated with at least one of the environment, the manipulating element, or the transport element during the movement of the manipulating element and the transport element; and
update a map of the environment based on the information associated with the at least one of the environment, the manipulating element, or the transport element.

### Claim 26 (depends on 25)

The robotic device of claim 25, wherein the information associated with the at least one of the environment, the manipulating element, or the transport element includes information of one or more objects in the environment, and the map of the environment includes at least a semantic layer and a contextual layer, the processor being configured to update the map of the environment by:
storing the information associated with the one or more objects in the environment in the semantic layer;
determining, based on the information associated with the one or more objects, one or more behaviors to execute near the one or more objects; and
storing the one or more behaviors in the contextual layer.

### Claim 27

An apparatus, comprising:
a manipulating element;
a transport element configured to move along a surface;
a set of sensors;
a memory configured to store a map of an environment, the map including a static layer identifying locations of a first plurality of constraints in the environment and a dynamic layer identifying movements or behaviors of a second plurality of constraints in the environment, each constraint from the first and second plurality of constraints being associated with a physical object in the environment; and
a processor operatively coupled to the memory, the manipulating element, the transport element, and the set of sensors, the processor configured to:
obtain, via the set of sensors, information of a portion of the environment;
select a skill to execute from a plurality of skills;
generate, using a model for executing the skill, a plan to execute the skill based on the information of the portion of the environment;
move the manipulating element and the transport element based on the plan to execute the skill;
obtain, after executing at least a portion of the skill, information of a constraint in the environment via the set of sensors;
determine a type of the constraint based on an input from a user or information of a set of physical objects stored in the memory; and
store the information of the constraint and the type of constraint by updating the static and dynamic layers of the map.

### Claim 28 (depends on 27)

The apparatus of claim 27, wherein the type of constraint includes at least one of a first feature of an object of the set of physical objects or a second feature of the environment.

### Claim 29 (depends on 27)

The apparatus of claim 27, wherein the processor is further configured to:
obtain, via the set of sensors, information associated with at least one of the environment, the manipulating element, or the transport element during the movement of the manipulating element and the transport element; and
update the map of the environment based on the information associated with the at least one of the environment, the manipulating element, or the transport element.

### Claim 30 (depends on 27)

The apparatus of claim 27, wherein the processor is further configured to:
receive an input associated with at least one of the environment, the manipulating element, or transport element from the user; and
update the map of the environment based on the input.