# US 12,454,053 B2 · Control device, robot system, control method, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,454,053 B2
- Document ID: US12454053B2
- Status: Granted
- Kind: B2
- Application number: 18/197,747
- Filed: 2023-05-16
- Published: 2023-11-23
- Granted: 2025-10-28
- Earliest claimed: 2022-05-19
- Assignees: Honda Motor Co., Ltd.
- Applicants: Honda Motor Co., Ltd.
- Inventors: Randy Gomez
- Web page: https://patentgrep.com/patent/US12454053B2

## CPC

- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 9/1697: {Vision controlled systems}
- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 20/20: in augmented reality scenes
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/172: {Classification, e.g. identification}
- G05B 2219/33056: Reinforcement learning, agent acts, receives reward, emotion, action selective
- G05B 2219/36488: Record motion and emotion, mimics
- G05B 2219/40411: Robot assists human in non-industrial environment like home or office
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A control unit is configured to monitor knowledge data in which element information relating to an operation environment of a robot is integrated and, when the element information of a predetermined item changes, identify a salient feature relating to the element information of the item, and operate the robot on the basis of an operation plan relating to a salient feature identified from related data representing at least a relevance between the salient feature and the operation plan of the robot.

## Claims

### Claim 1

A control device comprising:
a control unit configured to monitor knowledge data in which element information relating to an operation environment of a robot is integrated and, when the element information of a predetermined item changes, identify a salient feature relating to the element information of the item, and operate the robot on the basis of an operation plan relating to a salient feature identified from related data representing at least a relevance between the salient feature and the operation plan of the robot,
wherein, in the related data, a constraint condition is associated with at least some of the operation plans, and
wherein the control unit determines whether or not the element information of the predetermined item satisfies a constraint condition corresponding to an operation plan relating to the identified salient feature and, when the constraint condition is not satisfied, does not operate the robot on the basis of the operation plan relating to the salient feature,
wherein the knowledge data includes element information of a plurality of items, and the element information of the plurality of items is associated on the basis of attributes of the items, and
wherein the control unit converts detection information detected from the operation environment into perception information of an attribute set in advance and updates element information of which an attribute is common to the perception information on the basis of the perception information, and
wherein a class and event of interest information are associated for each salient feature in the related data, and
wherein the control unit
reads a common class for the salient feature,
generates an instance by substituting the event of interest information into the read class,
executes the generated instance,
reads the operation plan corresponding to the event of interest information, and
operates the robot in accordance with the read operation plan.

### Claim 2 (depends on 1)

The control device according to claim 1,
wherein, in the related data, with element information of a specific item, a rule representing an estimation condition of other element information relating to the element information is associated, and
wherein, when the element information of the specific item satisfies the estimation condition, the control unit adds the other element information to the knowledge data as element information relating to the element information of the specific item.

### Claim 3 (depends on 2)

The control device according to claim 2, wherein, when at least a part of the element information of the specific item is updated, the control unit determines whether or not the estimation condition is satisfied.

### Claim 4 (depends on 1)

The control device according to claim 1, wherein, when a new person is detected on the basis of the perception information, the control unit generates a new item for each attribute relating to the person and adds element information, which relates to the person, of an attribute corresponding to the generated items to the knowledge data.

### Claim 5 (depends on 1)

A robot system comprising:
a robot; and
the control device according to claim 1.

### Claim 6 (depends on 1)

A computer-readable non-transitory storage medium storing a program for causing a computer to function as the control device according to claim 1.

### Claim 7

A control method for a control device, the control method using the control device comprising:
monitoring knowledge data in which element information relating to an operation environment of a robot is integrated and, when the element information of a predetermined item changes, identifying a salient feature relating to the element information of the item;
operating the robot on the basis of an operation plan relating to a salient feature identified from related data representing at least a relevance between the salient feature and the operation plan of the robot,
wherein, in the related data, a constraint condition is associated with at least some of the operation plans;
determining whether or not the element information of the predetermined item satisfies a constraint condition corresponding to an operation plan relating to the identified salient feature and, when the constraint condition is not satisfied, not operating the robot on the basis of the operation plan relating to the salient feature,
wherein the knowledge data includes element information of a plurality of items, and the element information of the plurality of items is associated on the basis of attributes of the items;
converting detection information detected from the operation environment into perception information of an attribute set in advance and updates element information of which an attribute is common to the perception information on the basis of the perception information,
wherein a class and event of interest information are associated for each salient feature in the related data;
reading a common class for the salient feature;
generating an instance by substituting the event of interest information into the read class;
executing the generated instance;
reading the operation plan corresponding to the event of interest information; and
operating the robot in accordance with the read operation plan.