# US 2019/0240842 A1 · Robot assisted interaction system and method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0240842 A1
- Document ID: US20190240842A1
- Status: Published application
- Kind: A1
- Application number: 16/261,574
- Filed: 2019-01-30
- Published: 2019-08-08
- Earliest claimed: 2018-02-08
- Assignees: YongLin Biotech Corp.
- Applicants: YongLin Biotech Corp.
- Inventors: MING-HSUN HSU; CHENG-YAN GUO; YAO-TSUNG CHANG; CHIA-HUNG KUO
- Web page: https://patentgrep.com/patent/US20190240842A1

## CPC

- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- B25J 9/1694: {characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion}
- B25J 9/1697: {Vision controlled systems}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
- H04W 4/80: Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
- H04W 76/10: Connection setup
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

A robot assisted interactive system comprises: a mobile device having a display unit for displaying visual content, a touch control unit for receiving user input, a camera unit for obtaining user reaction information, a communication unit for transmitting the use reaction information to a backend server, and a processing unit for controlling the abovementioned units; a robot that includes a gesture module for generating gesture output according to the visual content, a speech module for generating speech output according to the visual content, a communication module for establishing connection with a backend server, and a control module for controlling the abovementioned modules; and a backend server configured to generate a feedback signal in accordance with the reaction information and send it to the mobile device; so as to update the visual content on the mobile device, and cause the robot to generate updated gesture output and speech output.

## Claims

### Claim 1

A robot assisted interactive system, comprising:
a robot, a mobile device, a backend server, and a wearable device,
wherein the mobile device includes:
a display unit configured to display a visual content;
a touch control unit configured to receive user input;
a camera unit configured to obtain user reaction information;
a communication unit configured to establish data connection with the backend server and transmit the use reaction information thereto; and
a processing unit coupled to and configured to control the display unit, the touch control unit, the camera unit, and the communication unit; wherein the robot includes:
a gesture module configured to generate a gesture output according to the visual content;
a speech module configured to generate a speech output according to the visual content;
a communication module configured to establish data connection with the backend server; and
a control module coupled to and configured to control the gesture module, the speech module, and the communication module;
wherein the wearable device comprises a sensor configured to obtain a physiological information of a wearer;
wherein the backend server is configured to:
generate a feedback signal in accordance with the reaction information and send the feedback signal to the mobile device; so as to
cause the mobile device to update the visual content based on the feedback signal, and
cause the robot to generate updated gesture output and speech output based on the updated visual content.

### Claim 2 (depends on 1)

The system of claim 1, wherein the wearable device comprises:
a transceiver configured to send the physiological information to the mobile device; and
a processer coupled to and configured to control the sensor and the transceiver.

### Claim 3 (depends on 1)

The system of claim 1, wherein the wearable device is a brainwave detection and analysis device, and the sensor is a brainwave detection electrode module.

### Claim 4 (depends on 1)

The system of claim 1, wherein the robot further comprises a camera module configured to obtain user reaction information.

### Claim 5

A method of robot assisted interaction using a robot assisted interaction system that includes a robot, a mobile device, and a backend server, the method comprising:
establishing data connection between the mobile device and the robot;
displaying, on the mobile device, a first visual content, wherein the robot generates a first speech output and a first gesture output based on the first visual content;
determining, by the mobile device, a receipt of a user input signal; upon receipt of the user input signal by the mobile device, displaying on the mobile device a second visual content based on the user input signal, wherein the robot generates a second speech output and a second gesture output based on the second visual content;
obtaining a use reaction information;
sending the user reaction information to the backend server;
generating a feedback signal, by the backend server, based on the user reaction information, and sending the feedback signal to the mobile device; and
displaying, on the mobile device, a third visual content, wherein the robot generates a third speech output and a third gesture output based on the third visual content.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
providing a wearable device to obtain a user physiological information;
sending the user physiological information to the backend server, and
generating, by the backend server, the feedback signal based on the user physiological information.

### Claim 7 (depends on 5)

The method of claim 5, wherein the physiological information includes a user image information and a user voice information.

### Claim 8 (depends on 5)

The method of claim 5, wherein establishing data connection between the mobile device and the robot comprises:
receiving a user information from the backend server by the mobile device.

### Claim 9

A robot interaction system, comprising:
a wearable device, adapted to be worn by a user, configured to detect a physiological information of the user;
a mobile device configured to execute an application and receive an interaction information of the user;
a backend server that generates a feedback signal based on the interaction information; and
a robot coupled to the mobile device;
wherein when the mobile device receives the feedback signal, the application changes a display or speech output of the mobile device, and robot generates a corresponding gesture output based on the feedback signal.

### Claim 10 (depends on 9)

The system of claim 9, wherein the mobile device further comprises a camera unit configured to obtain user reaction information, wherein the interaction information is generated based on the reaction information.

### Claim 11 (depends on 9)

The system of claim 9, wherein the application operates to generate a visual content on the mobile device, wherein the robot is configured to generate a gesture output and a speech output according to the visual content.

### Claim 12 (depends on 9)

The system of claim 9, wherein the robot further comprises a camera module configured to obtain user reaction information and transmit the user reaction information to the mobile device, wherein the interaction information is generated based on the user reaction information.

### Claim 13 (depends on 9)

The system of claim 9, wherein the wearable device includes a brainwave detection and analysis device.

### Claim 14 (depends on 9)

The system of claim 9, wherein the robot further comprises a projection module configured to project an image.

### Claim 15 (depends on 9)

The system of claim 9, wherein the interaction information is generated based on a visual information or a speech information.
