# US 11,110,608 B2 · Robotic physical movement adjustment based on user emotional state > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,110,608 B2 - Document ID: US11110608B2 - Status: Granted - Kind: B2 - Application number: 15/858,400 - Filed: 2017-12-29 - Published: 2019-07-04 - Granted: 2021-09-07 - Earliest claimed: 2017-12-29 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Andrew Aaron; Hariklia Deligianni; David O. S. Melville; Mark E. Podlaseck; Hyun Kyu Seo - Web page: https://patentgrep.com/patent/US11110608B2 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 13/00: Controls for manipulators (program controls B25J9/16) - B25J 19/026: {Acoustical sensing devices} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - H04R 1/028: {associated with devices performing functions other than acoustics, e.g. electric candles} - G10L 25/63: for estimating an emotional state - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract A method controls a robot. One or more processors receive sensor readings from one or more sensors that are monitoring a human in real time, where the human is currently observing a robotic action by a robot, and where the robotic action is a physical movement performed by the robot. The processor(s) determine a cognitive state of the human while the human is observing the robotic action by the robot, and then adjust the robotic action being performed by the robot based on the cognitive state of the human. ## Claims ### Claim 1 A processor-implemented method comprising: receiving, by one or more processors, sensor readings from one or more sensors, wherein the one or more sensors are monitoring a human in real time, wherein the human is currently observing a robotic action by a robot, wherein the robotic action is a physical movement performed by the robot; determining, by one or more processors and based on the sensor readings, an emotional state of the human caused by the human observing the robotic action by the robot, wherein the emotional state of the human is fear; and in response to determining, by one or more processors and based on the sensor readings, that the emotional state of the human caused by the human observing the robotic action by the robot is fear, reducing, by one or more processors, a velocity of movements made by the robot while performing a physical task. ### Claim 2 (depends on 1) The processor-implemented method of claim 1, wherein the robotic action is a physical locomotion of the robot, and wherein the processor-implemented method further comprises: adjusting, by one or more processors, the physical locomotion of the robot to increase a physical distance between the robot and the human based on the emotional state of the human. ### Claim 3 (depends on 1) The processor-implemented method of claim 1, wherein the robotic action further comprises a generation of a visual image on a display on the robot, wherein the human is viewing the visual display on the robot, and wherein the processor-implemented method further comprises: adjusting, by one or more processors, the visual image on the display on the robot based on the emotional state of the human that is viewing the visual display on the robot. ### Claim 4 (depends on 1) The processor-implemented method of claim 1, wherein the robotic action further comprises a generation of a sound from a speaker on the robot, and wherein the processor-implemented method further comprises: adjusting, by one or more processors, a volume level of the sound from the speaker based on the emotional state of the human. ### Claim 5 (depends on 1) The processor-implemented method of claim 1, further comprising: detecting, by a chemical detector on the robot, a characteristic scent of the human; and identifying, by one or more processors, a presence of the human within a predefined distance of the robot based on the characteristic scent of the human that is detected by the chemical detector on the robot. ### Claim 6 (depends on 1) The processor-implemented method of claim 1, further comprising: determining, by one or more processors, the emotional state of the human based on an analysis of a speech pattern of the human, wherein the analysis of the speech pattern is performed using advanced natural language analytics, and wherein the speech pattern is indicative of the emotional state of the human. ### Claim 7 (depends on 1) The processor-implemented method of claim 1, further comprising: determining, by one or more processors, the emotional state of the human based on biometric sensor readings from a set of biometric sensors directed towards the human. ### Claim 8 (depends on 1) The processor-implemented method of claim 1, further comprising: determining, by one or more processors, the emotional state of the human based on an analysis of thermal imaging of a face of the human, wherein the thermal imaging detects a blood flow level in the human that is indicative of the emotional state of the human. ### Claim 9 (depends on 1) The processor-implemented method of claim 1, further comprising: modifying, by one or more processors, the robotic action based on heuristic machine learning by the robot, wherein the heuristic machine learning causes the robot to learn about an environment of the robot in order to adjust the robotic action. ### Claim 10 (depends on 1) The processor-implemented method of claim 1, further comprising: identifying, by one or more processors, objects within an area; identifying, by one or more processors, a type of area of the area based on the objects identified within the area; determining, by one or more processors, that the robot is prohibited from operating in the identified type of area; and in response to determining that the robot is prohibited from operating in the identified type of area, blocking, by one or more processors, the robot from being in the area. ### Claim 11 (depends on 1) The processor-implemented method of claim 1, further comprising: receiving, by one or more processors, location sensor readings from one or more location sensors, wherein the location sensor readings describe a type of physical location in which the human is located; detecting, by one or more processors, that the robot is within the type of physical location in which the human is located; determining, by one or more processors, that a presence of the robot within the type of physical location in which the human is located is affecting the emotional state of the human being; and in response to determining that the robot is within the type of physical location in which the human is located and in response to determining that the presence of the robot within the type of physical location in which the human is located is affecting the emotional state of the human being, directing, by one or more processors, the robot to leave the type of physical location in which the human is located. ### Claim 12 (depends on 1) The processor-implemented method of claim 1, further comprising: detecting, by one or more processors and based on positioning sensor readings from one or more positioning sensors, that the robot is within a predefined distance of an object that has been predetermined to be delicate; and directing, by one or more processors, the robot to move away from the object by more than the predefined distance. ### Claim 13 (depends on 1) The processor-implemented method of claim 1, wherein the robot includes a mechanical manipulator arm, wherein the mechanical manipulator arm is engineered to lift an object that is away from the robot, and wherein the processor-implemented method further comprises: detecting, by one or more processors and based on positioning sensor readings from one or more positioning sensors, that the robot is within a predefined distance of an object that has been predetermined to be delicate; and preventing, by one or more processors, the robot from moving the mechanical manipulator arm while the robot is within the predefined distance from the object. ### Claim 14 (depends on 1) The processor-implemented method of claim 1, wherein the human is observing, via a remote video feed, the robotic action of the robot at a location that is remote from a location of the robot. ### Claim 15 (depends on 1) The processor-implemented method of claim 1, wherein the physical movement performed by the robot is an initial movement, wherein the initial movement is a two step movement along two different directions that causes the robot to move indirectly towards an object, and wherein the adjusted robotic action is a one step movement along a single direction that causes the robot to move directly towards the object. ### Claim 16 A robot comprising: a sensor receiver for receiving sensor readings from a set of one or more sensors that detect an emotional state of a human who is currently observing a robotic action by a robot, wherein the robotic action is a physical movement performed by the robot, wherein the emotional state of the human is fear, and wherein the emotional state of fear in the human is caused by the human observing the robotic action by the robot; an instruction receiver for receiving a computer-executable program for causing the robot to modify a velocity of the robotic action based on the sensor readings that describe the emotional state of the human; and a robotic controller processor for: determining, based on the sensor readings, the emotional state of fear the human caused by the human observing the robotic action by the robot, and executing the computer-executable program in order to modify the velocity of the robotic action in response to determining, based on the sensor readings, the emotional state of fear in the human caused by the human observing the robotic action by the robot, wherein the robotic controller processor modifies the velocity of the robotic action by reducing a velocity of movements made by the robot while performing a physical task. ### Claim 17 (depends on 16) The robot of claim 16, further comprising: one or more processors configured to: receive location sensor readings from one or more location sensors, wherein the location sensor readings describe a type of physical location in which the human is located; detect that the robot is within the type of physical location in which the human is located; determine that a presence of the robot within the type of physical location in which the human is located is affecting the emotional state of the human being; in response to determining that the robot is within the type of physical location in which the human is located and in response to determining that the presence of the robot within the type of physical location in which the human is located is affecting the emotional state of the human being, direct the robot to leave the type of physical location in which the human is located. ### Claim 18 A computer program product for controlling a robot, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code readable and executable by a processor to perform a method comprising: receiving sensor readings from one or more sensors, wherein the one or more sensors are monitoring a human in real time, wherein the human is currently observing a robotic action by a robot, and wherein the robotic action is a physical movement performed by the robot; determining, based on the sensor readings, an emotional state of the human caused by the human observing the robotic action by the robot, wherein the emotional state of the human is fear; and in response to determining, based on the sensor readings, that the emotional state of the human caused by the human observing the robotic action by the robot is fear, reducing a velocity of movements made by the robot while performing a physical task. ### Claim 19 (depends on 18) The computer program product of claim 18, wherein the program instructions are provided as a service in a cloud environment.