# US 12,005,579 B2 · Robot reacting on basis of user behavior and control method therefor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,005,579 B2
- Document ID: US12005579B2
- Status: Granted
- Kind: B2
- Application number: 16/768,474
- Filed: 2018-09-12
- Published: 2021-06-10
- Granted: 2024-06-11
- Earliest claimed: 2018-01-29
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Jin-won Kim; Jung-gap Kuk
- Web page: https://patentgrep.com/patent/US12005579B2

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1697: {Vision controlled systems}
- B25J 11/001: {with emotions simulating means}
- B25J 11/0015: {Face robots, animated artificial faces for imitating human expressions}
- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0499: Feedforward networks
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- B25J 13/003: {by means of an audio-responsive input (audible safety signals B25J19/061)}
- G05B 2219/40408: Intention learning
- G05B 2219/40411: Robot assists human in non-industrial environment like home or office
- G05B 2219/40414: Man robot interface, exchange of information between operator and robot
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A robot for outputting various reactions according to user behaviors is disclosed. A control method for a robot using an artificial intelligence model, according to the present disclosure, comprises the steps of: acquiring data related to at least one user; inputting the data related to the at least one user into the artificial intelligence model as learning data so as to learn a user state for each user of which there is at least one; determining representative reactions corresponding to the user states learned on the basis of the data related to the at least one user; and inputting the input data into the artificial intelligence model so as to determine a user state of a first user and controlling the robot on the basis of a representative reaction corresponding to the determined user state, when input data related to the first user among the users, of which there is a least one, is acquired.

## Claims

### Claim 1

A control method for a robot using an artificial intelligence model, the control method comprising:
acquiring user data related to at least one user;
training the artificial intelligence model to acquire an emotional state for each of the at least one user by inputting the user data related to the at least one user to the artificial intelligence model, the user data comprising at least one image data;
acquiring first data related to a first user, the first data including information regarding a first schedule of the first user;
determining, based on the first data, a first emotional state of the first user by inputting the first data to the trained artificial intelligence model;
controlling the robot to output a first reaction based on the first data and the first emotional state;
determining a character of the robot based on a character area of a character model including a first axis indicating whether an emotion of the robot is positive or negative and a second axis indicating an activity of the robot, wherein the character area of the character model is determined based on a feedback of the first user for the first reaction, a number of interactions with the first user and a service completion rate representing an accuracy of the first reaction of the robot;
acquiring second data related to the first user after determining the character of the robot;
determining, based on acquiring the second data related to the first user, a second emotional state of the first user by inputting the second data to the trained artificial intelligence model; and
controlling the robot to output a second reaction based on the second emotional state and the character of the robot.

### Claim 2 (depends on 1)

The control method of claim 1, further comprising:
detecting, by the robot, a second user;
determining a third emotional state by inputting third data related to the second user to the artificial intelligence model; and
controlling the robot based on a representative reaction corresponding to the determined third emotional state.

### Claim 3 (depends on 1)

The control method of claim 1, wherein training the artificial intelligence model comprises:
clustering a plurality of learning data into a plurality of clusters by inputting the plurality of learning data into the artificial intelligence model; and
labeling emotional states corresponding to respective clusters of the plurality of clusters.

### Claim 4 (depends on 3)

The control method of claim 3, further comprising:
acquiring a variance value of learning data included in a cluster generated by the clustering; and
determining a representative reaction of the cluster by analyzing the acquired variance value.

### Claim 5 (depends on 4)

The control method of claim 4, wherein the representative reaction includes at least one of a facial expression reaction, a motion reaction, or a voice reaction.

### Claim 6 (depends on 4)

The control method of claim 4, wherein controlling the robot to output the first reaction comprises:
determining a robot reaction corresponding to the representative reaction; and
outputting the robot reaction.

### Claim 7 (depends on 6)

The control method of claim 6, wherein the character of the robot is determined according to a number of interactions with the first user, the first emotional state, and a degree of completion of the robot reaction.

### Claim 8 (depends on 1)

The control method of claim 1, wherein the learning data includes at least one of the image data, voice data, data about peripheral users, or data received from peripheral devices.

### Claim 9 (depends on 1)

The control method of claim 1, wherein the artificial intelligence model is learned using unsupervised learning, and
wherein the unsupervised learning uses at least one of a K-mean clustering technique or a gaussian mixture model (GMM) technique.

### Claim 10

A robot using an artificial intelligence model, the robot comprising:
a processor configured to:
acquire user data related to at least one user;
train the artificial intelligence model to acquire an emotional state for each of the at least one user by inputting the user data related to the at least one user to the artificial intelligence model as learning data, the user data comprising at least one image data;
acquire first data related to a first user, the first data including information regarding a first schedule of the first user;
determine, based on the first data, a first emotional state of the first user by inputting the first data to the trained artificial intelligence model;
output a first reaction based on the first data and the first emotional state;
determine a character of the robot based on a character area of a character model including a first axis indicating whether an emotion of the robot is positive or negative and a second axis indicating an activity of the robot, wherein the character area of the character model is determined based on a feedback of the first user for the first reaction, a number of interactions with the first user and a service completion rate representing an accuracy of the first reaction of the robot;
acquiring second data related to the first user after the character of the robot is determined;
determine, based on acquiring the second data related to the first user, a second emotional state of the first user by inputting the second data to the trained artificial intelligence model; and
output a second reaction based on the second emotional state and the character of the robot.

### Claim 11 (depends on 10)

The robot of claim 10, wherein the processor is further configured to:
detect a second user;
determine a third emotional state by inputting third data related to the second user to the artificial intelligence model; and
control the robot based on a representative reaction corresponding to the determined third emotional state.

### Claim 12 (depends on 10)

The robot of claim 10, wherein the processor is further configured to:
cluster a plurality of learning data into a plurality of clusters by inputting the plurality of learning data into the artificial intelligence model; and
label emotional states corresponding to respective clusters of the plurality of clusters.

### Claim 13 (depends on 12)

The robot of claim 12, wherein the processor is further configured to:
acquire a variance value of learning data included in a cluster generated by the clustering; and
determine a representative reaction of the cluster by analyzing the acquired variance value.

### Claim 14 (depends on 13)

The robot of claim 13, wherein the representative reaction includes at least one of a facial expression reaction, a motion reaction, or a voice reaction.

### Claim 15 (depends on 13)

The robot of claim 13, wherein the processor is further configured to:
determine a robot reaction corresponding to the representative reaction; and
control output of the robot reaction.