# US 10,482,886 B2 · Interactive robot and human-robot interaction method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,886 B2
- Document ID: US10482886B2
- Status: Granted
- Kind: B2
- Application number: 15/853,918
- Filed: 2017-12-25
- Published: 2019-02-07
- Granted: 2019-11-19
- Earliest claimed: 2017-08-05
- Assignees: Fu Tai Hua Industry (Shenzhen) Co., Ltd.; Hon Hai Precision Industry Co., Ltd.
- Applicants: Fu Tai Hua Industry (Shenzhen) Co., Ltd.; Hon Hai Precision Industry Co., Ltd.
- Inventors: Xue-Qin Zhang
- Web page: https://patentgrep.com/patent/US10482886B2

## CPC

- G10L 17/22: Interactive procedures; Man-machine interfaces
- B25J 9/0003: {Home robots, i.e. small robots for domestic use}
- B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor}
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/174: {Facial expression recognition}
- G10L 17/18: Artificial neural networks; Connectionist approaches
- G10L 25/63: for estimating an emotional state
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 17/00: Speaker identification or verification techniques
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An interactive robot includes an image capturing device, an audio capturing device, an output device, and a processor. The processor is configured to obtain audio information captured by the audio capturing device and image information captured by the image capturing device, recognize a target from the audio information and the image information, confirm basic information and event information of the target and link the basic information with the event information, obtain key information from the event information of the target, implement a neural network analysis algorithm on the key information to confirm an emotion type of the target, search a preset public knowledge database according to the key information to obtain a relevant result, apply a deep learning algorithm on the relevant result and the emotion type of the target to determine a response, and execute the response through the output device.

## Claims

### Claim 1

An interactive robot comprising:
an image capturing device;
an audio capturing device;
at least one output device; and
at least one processor configured to:
obtain audio information captured by the audio capturing device and image information captured by the image capturing device;
recognize a target from the audio information and the image information;
confirm basic information and event information of the target and link the basic information with the event information;
obtain key information from the event information of the target;
implement a neural network analysis algorithm on the key information to confirm an emotion type of the target, search a preset public knowledge database according to the key information to obtain a relevant result, apply a deep learning algorithm on the relevant result and the emotion type of the target to determine a response; and
execute the response through the output device;
wherein the response is a series of instructions for controlling the interactive robot to interact with the target.

### Claim 2 (depends on 1)

The interactive robot of claim 1, wherein the processor recognizes a voiceprint from the audio information and recognizes a faceprint from the image information, and recognizes the target from the voiceprint and the faceprint.

### Claim 3 (depends on 1)

The interactive robot of claim 1, wherein the processor controls the audio capturing device to set the audio information of the target as the event information of the target.

### Claim 4 (depends on 3)

The interactive robot of claim 3, wherein the processor recognizes the audio information, converts the audio information into text data, and obtains key information from the text data; the key information from the text data is the key information of the event information.

### Claim 5 (depends on 1)

The interactive robot of claim 1, wherein the processor controls the image capturing device to set the image information of the target as the event information of the target.

### Claim 6 (depends on 5)

The interactive robot of claim 5, wherein the processor obtains facial expression information and limb movement information from the image information, obtains facial expression parameters from the facial expression information, obtains limb movement parameters from the limb movement information, and sets the facial expression parameters and the limb movement parameters as the key information of the event information.

### Claim 7 (depends on 1)

The interactive robot of claim 1, wherein the processor is further configured to set emotions of the interactive robot; the processor implements a neural network analysis algorithm on the key information to confirm an emotion type of the target, searches a preset public knowledge database according to the key information to obtain a relevant result, applies a deep learning algorithm on the relevant result, the emotion type of the target, and the set emotions of the interactive robot to determine the response.

### Claim 8 (depends on 1)

The interactive robot of claim 1, wherein the output device includes an audio output device and an expression output device; the processor controls the audio output device to output audio information as the response and controls the expression output device to output an expression to execute the response.

### Claim 9

A human-robot interacting method implemented in an interactive robot, the method comprising:
obtaining audio information captured by an audio capturing device and image information captured by an image capturing device;
recognizing a target from the audio information and the image information;
confirming basic information and event information of the target and linking the basic information with the event information;
obtaining key information from the event information of the target;
implementing a neural network analysis algorithm on the key information to confirm an emotion type of the target, searching a preset public knowledge database according to the key information to obtain a relevant result, applying a deep learning algorithm on the relevant result and the emotion type of the target to determine a response; and
executing the response through the output device;
wherein the response is a series of instructions for controlling the interactive robot to interact with the target.

### Claim 10 (depends on 9)

The method of claim 9, wherein the target is recognized from the audio information and the image information by recognizing a voiceprint from the audio information and recognizing a faceprint from the image information.

### Claim 11 (depends on 9)

The method of claim 9, wherein the basic information and the event information are confirmed and linked together by setting the audio information captured by the audio capturing device as the event information.

### Claim 12 (depends on 11)

The method of claim 11, wherein the audio information is recognized and converted into text data, key information is obtained from the text data, and the key information from the text data is set as the key information of the event information.

### Claim 13 (depends on 9)

The method of claim 9, wherein the basic information and the event information are confirmed and linked together by setting the image information captured by the image capturing device as the event information.

### Claim 14 (depends on 13)

The method of claim 13, wherein facial expression information and limb movement information are obtained from the image information, facial expression parameters are obtained from the facial expression information, limb movement parameters are obtained from the limb movement information, and the facial expression parameters and the limb movement parameters are set as the key information of the event information.

### Claim 15 (depends on 9)

The method of claim 9, wherein an audio output device is controlled to output audio information as the response, and an expression output device is controlled to output an expression to execute the response.

### Claim 16

A non-transitory storage medium having stored thereon instructions that, when executed by at least one processor of an interactive robot, causes the least one processor to execute instructions of a method for human-robot interaction, the method comprising:
obtaining audio information captured by an audio capturing device and image information captured by an image capturing device;
recognizing a target from the audio information and the image information;
confirming basic information and event information of the target and linking the basic information with the event information;
obtaining key information from the event information of the target;
implementing a neural network analysis algorithm on the key information to confirm an emotion type of the target, searching a preset public knowledge database according to the key information to obtain a relevant result, applying a deep learning algorithm on the relevant result and the emotion type of the target to determine a response; and
executing the response through the output device;
wherein the response is a series of instructions for controlling the interactive robot to interact with the target.