# US 12,715,115 B2 · Information processing apparatus, information processing method, and information processing system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,715,115 B2 - Document ID: US12715115B2 - Status: Granted - Kind: B2 - Application number: 18/567,953 - Filed: 2021-07-29 - Published: 2024-08-15 - Granted: 2026-08-25 - Earliest claimed: 2021-06-15 - Assignees: Sony Group Corporation - Applicants: Sony Group Corporation - Inventors: Shunichi Sekiguchi; Kazumi Aoyama; Takayoshi Takayanagi; Hirotaka Suzuki - Web page: https://patentgrep.com/patent/US12715115B2 ## CPC - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1697: {Vision controlled systems} - G05B 2219/39539: Plan hand shape - G05B 2219/39543: Recognize object and plan hand shapes in grasping movements - G05B 2219/40584: Camera, non-contact sensor mounted on wrist, indep from gripper - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract An information processing apparatus according to an embodiment of the present technology includes a region calculation unit. The region calculation unit calculates, on the basis of operation information regarding an operation performed on a target object and an image of the target object, a hand area when the operation is performed on the target object by using an image generator or a predictor. This enables a high-precision operation. Moreover, a dataset used for the image generator and the predictor can be collected more easily. Therefore, a massive amount of datasets can be built and can be applied to various target objects. ## Claims ### Claim 1 An information processing method, comprising: by a computer system; receiving an image of a target object from a sensor via a network; setting operation information associated with an operation executable on the target object; calculating a hand area of a robotic machine based on the operation information and the image of the target object; and controlling, based on a neural network model, movement of the robotic machine to execute the operation on the target object, wherein the movement of the robotic machine is controlled based on the calculated hand area of the robotic machine, and the neural network model includes one of an image generator or a predictor. ### Claim 2 A non-transitory computer-readable medium having stored thereon, computer-executable instructions which, when executed by a computer, cause the computer to execute operations, the operations comprising: receiving an image of a target object from a sensor via a network; setting operation information associated with an operation executable on the target object; calculating a hand area of a robotic machine based on the operation information and the image of the target object; and controlling, based on a neural network model, movement of the robotic machine to execute the operation on the target object, wherein the movement of the robotic machine is controlled based on the calculated hand area of the robotic machine, and the neural network model includes one of an image generator or a predictor. ### Claim 3 An information processing apparatus, comprising a Central Processing Unit (CPU) configured to: receive an image of a target object from a sensor via a network; set operation information associated with an operation executable on the target object; calculate a hand area of a robotic machine based on the operation information and the image of the target object; and control, based on a neural network model, movement of the robotic machine to execute the operation on the target object, wherein the movement of the robotic machine is controlled based on the calculated hand area of the robotic machine, and the neural network model includes one of an image generator or a predictor. ### Claim 4 (depends on 3) The information processing apparatus according to claim 3, wherein the CPU is further configured to control the image generator to generate a pseudo-image of the operation executed on the target object, and the operation is executed on the target object based on the operation information and the image of the target object. ### Claim 5 (depends on 3) The information processing apparatus according to claim 3, wherein the CPU is further configured to control the image generator to set each of the operation information, the image of the target object, and a specific image as a first learning dataset, and the specific image indicates the operation executed on the target object. ### Claim 6 (depends on 4) The information processing apparatus according to claim 4, wherein the CPU is further configured to calculate the hand area based on a difference between the pseudo-image and the image of the target object. ### Claim 7 (depends on 3) The information processing apparatus according to claim 3, wherein the CPU is further configured to control the predictor to set each of a black and white image, the image of the target object, and the hand area as a second learning dataset, the black and white image indicates the operation information, and the hand area indicates an area of a hand at a time the operation is executed on the target object. ### Claim 8 (depends on 1) The information processing apparatus according to claim 1, wherein the operation corresponds to an operation executed in a state the target object is held. ### Claim 9 (depends on 3) The information processing apparatus according to claim 3, wherein the robotic machine is a robot arm, and the operation is executed by the robot arm. ### Claim 10 (depends on 1) The information processing apparatus according to claim 1, wherein the robotic machine is a robot arm, and the CPU is further configured to calculate a hold point of the robot arm based on the calculated hand area. ### Claim 11 (depends on 10) The information processing apparatus according to claim 10, wherein the hold point is a parameter regarding position and attitude of a holding part of the robot arm; and the holding part holds the target object. ### Claim 12 (depends on 11) The information processing apparatus according to claim 11, wherein the parameter includes x-coordinate, y-coordinate, z-coordinate, roll, pitch, and yaw. ### Claim 13 (depends on 10) The information processing apparatus according to claim 10, wherein the operation information includes at least one of instruction information or execution information, the instruction information includes an instruction to pass the target object to a user, and the CPU is further configured to: control the robot arm to execute the operation based on the execution information; and calculate the hold point based on the operation information. ### Claim 14 (depends on 13) The information processing apparatus according to claim 13, wherein the hold point includes a hold candidate point, the target object is held by the robot arm at the hold candidate point, and the CPU is further configured to: remove the hold candidate point included in the hand area; and calculate the hold point based on the instruction information and the removal of the hold candidate point. ### Claim 15 (depends on 13) The information processing apparatus according to claim 13, wherein the hold point includes a hold candidate point, the target object is held by the robot arm at the hold candidate point, and the CPU is further configured to calculate the hold candidate point included in the hand area as the hold point based on the execution information. ### Claim 16 An information processing system, comprising: a robot arm configured to execute an operation on a target object; and an information processing apparatus, wherein the information processing apparatus includes a Central Processing Unit (CPU) configured to: receive an image of the target object from a sensor via a network; set operation information associated with the operation executable on the target object; calculate a hand area of the robot arm based on the operation information and the image of the target object; and control, based on a neural network model, movement of the robot arm to execute the operation on the target object, wherein the movement of the robot arm is controlled based on the calculated hand area of the robot arm, and the neural network model includes one of an image generator or a predictor. ### Claim 17 (depends on 16) The information processing system according to claim 16, wherein the CPU is further configured to calculate a hold point of the robot arm based on the calculated hand area. ### Claim 18 (depends on 17) The information processing system according to claim 17, wherein the CPU is further configured to control the robot arm to execute the operation, wherein the operation is executed based on the calculated hold point, and the robot arm is controlled based on a control signal. ### Claim 19 (depends on 16) The information processing system according to claim 16, wherein the robot arm includes a sensor, and the sensor is configured to acquire the image of the target object.