# US 2023/0234232 A1 · Autonomous control system, autonomous control method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0234232 A1 - Document ID: US20230234232A1 - Status: Published application - Kind: A1 - Application number: 18/096,042 - Filed: 2023-01-12 - Published: 2023-07-27 - Earliest claimed: 2022-01-24 - Assignees: Honda Motor Co., Ltd. - Applicants: Honda Motor Co., Ltd. - Inventors: Akinobu Hayashi; Itoshi Naramura - Web page: https://patentgrep.com/patent/US20230234232A1 ## CPC - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1694: {characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion} - B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1697: {Vision controlled systems} - B25J 13/084: {Tactile sensors (in general G01L5/16, G01L5/22)} - G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program - G05B 2219/39369: Host and robot controller and vision processing - G05B 2219/40264: Human like, type robot arm - G05B 2219/40269: Naturally compliant robot arm - G05B 2219/40575: Camera combined with tactile sensors, for 3-D - G05B 2219/40625: Tactile sensor - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract An autonomous control system includes an acquirer configured to acquire state data of a robot, visual data of the robot, and tactile data of the robot and a processor configured to decide on an action of the robot capable of accomplishing a task given to the robot on the basis of the state data, the visual data, and the tactile data. The processor generates first compressed data having a smaller number of dimensions than data obtained by combining the visual data and the tactile data by fusing and dimensionally compressing the visual data and the tactile data. The processor generates second compressed data having a smaller number of dimensions than the tactile data by dimensionally compressing the tactile data. The processor decides on the action on the basis of combined state data obtained by combining the state data, the first compressed data, and the second compressed data into one. ## Claims ### Claim 1 An autonomous control system comprising: an acquirer configured to acquire state data of a robot, visual data of the robot, and tactile data of the robot; and a processor configured to decide on an action of the robot capable of accomplishing a task given to the robot on the basis of the state data, the visual data, and the tactile data, wherein the processor generates first compressed data having a smaller number of dimensions than data obtained by combining the visual data and the tactile data by fusing and dimensionally compressing the visual data and the tactile data, wherein the processor generates second compressed data having a smaller number of dimensions than the tactile data by dimensionally compressing the tactile data, and wherein the processor decides on the action on the basis of combined state data obtained by combining the state data, the first compressed data, and the second compressed data into one. ### Claim 2 (depends on 1) The autonomous control system according to claim 1, wherein the acquirer acquires depth image data generated by a camera that images a body of the robot and a target of the task as the visual data and acquires data in which a contact force detected by each tactile sensor is associated with a distribution of a plurality of tactile sensors arranged in the body as the tactile data, and wherein the processor generates the first compressed data by fusing and dimensionally compressing the distribution of the plurality of tactile sensors and the depth image data. ### Claim 3 (depends on 2) The autonomous control system according to claim 2, wherein the processor generates the second compressed data by dimensionally compressing the data in which the contact force detected by each tactile sensor is associated with the distribution of the plurality of tactile sensors. ### Claim 4 (depends on 1) The autonomous control system according to claim 1, wherein the processor generates the first compressed data from the visual data and the tactile data using a first encoder, and wherein the first encoder is a neural network trained on the basis of a training dataset in which a state of a correct answer of the target of the task is labeled for the visual data and the tactile data. ### Claim 5 (depends on 1) The autonomous control system according to claim 1, wherein the processor generates the first compressed data from the visual data and the tactile data using a first encoder, and wherein the first encoder is a neural network that converts input data into data having a smaller number of dimensions and outputs the data having the smaller number of dimensions and that is trained so that data input to the first encoder matches data output by a decoder in combination with the decoder that converts the input data into data having a larger number of dimensions and outputs the data having the larger number of dimensions. ### Claim 6 (depends on 1) The autonomous control system according to claim 1, wherein the processor generates the second compressed data from the tactile data using a second encoder, and wherein the second encoder is a neural network that converts input data into data having a smaller number of dimensions and outputs the data having the smaller number of dimensions and that is trained so that data input to the second encoder matches data output by a decoder in combination with the decoder that converts the input data into data having a larger number of dimensions and outputs the data having the larger number of dimensions. ### Claim 7 (depends on 1) The autonomous control system according to claim 1, wherein the processor decides on the action from the combined state data using reinforcement learning. ### Claim 8 (depends on 2) The autonomous control system according to claim 2, wherein the processor further decides on sensitivity of the tactile sensor on the basis of the combined state data. ### Claim 9 (depends on 2) The autonomous control system according to claim 2, wherein the processor further decides on an angle of the camera when the body and the target are imaged on the basis of the combined state data. ### Claim 10 An autonomous control method comprising: acquiring state data of a robot, visual data of the robot, and tactile data of the robot; deciding on an action of the robot capable of accomplishing a task given to the robot on the basis of the state data, the visual data, and the tactile data; generating first compressed data having a smaller number of dimensions than data obtained by combining the visual data and the tactile data by fusing and dimensionally compressing the visual data and the tactile data; generating second compressed data having a smaller number of dimensions than the tactile data by dimensionally compressing the tactile data; and deciding on the action on the basis of combined state data obtained by combining the state data, the first compressed data, and the second compressed data into one. ### Claim 11 A computer-readable non-transitory storage medium storing a program for causing a computer to: acquire state data of a robot, visual data of the robot, and tactile data of the robot; decide on an action of the robot capable of accomplishing a task given to the robot on the basis of the state data, the visual data, and the tactile data; generate first compressed data having a smaller number of dimensions than data obtained by combining the visual data and the tactile data by fusing and dimensionally compressing the visual data and the tactile data; generate second compressed data having a smaller number of dimensions than the tactile data by dimensionally compressing the tactile data; and decide on the action on the basis of combined state data obtained by combining the state data, the first compressed data, and the second compressed data into one.