# US 12,117,848 B2 · Control device and control method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,117,848 B2 - Document ID: US12117848B2 - Status: Granted - Kind: B2 - Application number: 17/052,753 - Filed: 2019-02-25 - Published: 2021-07-29 - Granted: 2024-10-15 - Earliest claimed: 2018-05-15 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Kuniaki Torii; Mikio Nakai; Yusuke Kudo - Web page: https://patentgrep.com/patent/US12117848B2 ## CPC - B25J 11/001: {with emotions simulating means} - G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras - G05D 1/247: using signals provided by artificial sources external to the vehicle, e.g. navigation beacons - G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06N 5/046: Forward inferencing; Production systems - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A control device includes a recognizing unit that recognizes the external environment based on sensing information; an action decision managing unit that, based on the recognized external environment, decides on the actions to be taken; a behavior managing unit that, based on at least either the difficulty level of recognition for the recognizing unit or the difficulty level of action decision for the action decision managing unit, determines about whether or not to exhibit emotional expressions; and a drive control unit that, based on the determination performed by the behavior control unit, controls the execution of the actions corresponding to emotional expressions. ## Claims ### Claim 1 A control device, comprising: a central processing unit (CPU) configured to: recognize an external environment based on sensing information; determine a difficulty level of the recognition of the external environment; determine, based on the recognized external environment, a first action to be taken; determine a difficulty level of the determination of the first action based on at least one of a number of options for the first action, or a complexity of the first action; determine, based on at least one of the difficulty level of the recognition of the external environment or the difficulty level of the determination of the first action whether to exhibit an emotional expression; and control, based on the determination to exhibit the emotional expression, execution of a second action corresponding to the emotional expression. ### Claim 2 (depends on 1) The control device according to claim 1, wherein the emotional expression implies an expression that causes a decline in an efficiency of the first action. ### Claim 3 (depends on 1) The control device according to claim 1, wherein, in a case where the difficulty level of the recognition high, the CPU is further configured to determine to exhibit the emotional expression. ### Claim 4 (depends on 1) The control device according to claim 1, wherein the CPU is further configured to determine the difficulty level of the recognition based on at least one of a count or types of sensors used in obtaining the sensing information, a complexity of the external environment, a recognition algorithm implemented for the recognition by the CPU, or a reliability or a required accuracy of the recognition. ### Claim 5 (depends on 1) The control device according to claim 1, wherein, in a case where the difficulty level of the determination of the first action is high, the CPU is further configured to determine to exhibit the emotional expression. ### Claim 6 (depends on 1) The control device according to claim 1, wherein the CPU is further configured to determine the difficulty level of the determination of the first action based on at least one of a presence or an absence of relative merits or history, or a limiting condition of the first action. ### Claim 7 (depends on 1) The control device according to claim 1, wherein the CPU is further configured to control a robot device that is used as a robotic pet or an ornamental robot. ### Claim 8 (depends on 7) The control device according to claim 7, wherein the emotional expression implies an expression indicating an emotion of the robot device. ### Claim 9 (depends on 7) The control device according to claim 7, wherein the emotional expression implies an expression of the robot device which evokes feelings of personal attachment in a user of the robot device. ### Claim 10 (depends on 7) The control device according to claim 7, wherein the CPU is further configured to determine, based on a presence of a user of the robot device, whether to exhibit the emotional expression. ### Claim 11 (depends on 10) The control device according to claim 10, wherein the CPU is further configured to determine, based on a degree of attention of the user toward the robot device, whether to exhibit the emotional expression. ### Claim 12 (depends on 11) The control device according to claim 11, wherein, in a case where the degree of attention of the user toward the robot device is high, the CPU is further configured to determine to exhibit the emotional expression. ### Claim 13 (depends on 11) The control device according to claim 11, wherein the CPU is further configured to determine the degree of attention of the user toward the robot device is based on at least one of image recognition of an image of the user, speech recognition or context recognition of a speech of the user, or an input from the user. ### Claim 14 (depends on 10) The control device according to claim 10, wherein the CPU is further configured to determine, based on an attribute of the user, whether to exhibit the emotional expression. ### Claim 15 (depends on 14) The control device according to claim 14, the CPU is further configured to modify, based on the attribute of the user, a determination criterion regarding the difficulty level of the recognition or regarding the difficulty level of the determination of the first action. ### Claim 16 (depends on 7) The control device according to claim 7, wherein the sensing information is obtained by a sensor unit installed in the robot device. ### Claim 17 A control method implemented in an arithmetic device, comprising: recognizing an external environment based on sensing information; determining a difficulty level of the recognition of the external environment; determining, based on the recognized external environment, a first action to be taken; determining a difficulty level of the determination of the first action based on at least one of a number of options for the first action, or a complexity of the first action; determining, based on at least one of the difficulty level of the recognition or the difficulty level of the determination of the first action, whether to exhibit emotional expression; and controlling, based on the determination to exhibit the emotional expression, execution of a second action corresponding to the emotional expression. ### Claim 18 A non-transitory computer-readable medium having stored thereon, computer-executable instructions which, when executed by a computer, cause the computer to execute operations, the operations comprising: recognizing an external environment based on sensing information; determining a difficulty level of the recognition of the external environment; determining, based on the recognized external environment, a first action to be taken; determining a difficulty level of the determination of the first action based on at least one of a number of options for the first action, or a complexity of the first action; determining, based on at least one of the difficulty level of the recognition or the difficulty level of the determination of the first action, whether to exhibit emotional expression; and controlling, based on the determination to exhibit the emotional expression, execution of a second action corresponding to the emotional expression.