# US 12,108,996 B2 · Method of remotely supporting surgery assistant robot and remote support system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,108,996 B2 - Document ID: US12108996B2 - Status: Granted - Kind: B2 - Application number: 18/166,066 - Filed: 2023-02-08 - Published: 2023-06-15 - Granted: 2024-10-08 - Earliest claimed: 2018-10-06 - Assignees: Sysmex Corporation - Applicants: Sysmex Corporation - Inventors: Masanao Ohashi; Takehiko Sudoh; Shoko Ito; Jota Ida; Yasuyuki Fukuda - Web page: https://patentgrep.com/patent/US12108996B2 ## CPC - A61B 34/30: Surgical robots - A61B 34/35: for telesurgery - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - A61B 90/37: {Surgical systems with images on a monitor during operation} - B25J 9/1697: {Vision controlled systems} - G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)} - G16H 30/40: for processing medical images, e.g. editing - G16H 40/67: for remote operation - H04N 23/815: {for controlling the resolution by using a single image} - A61B 2034/256: {having a database of accessory information, e.g. including context sensitive help or scientific articles} - G05B 2219/45117: Medical, radio surgery manipulator - H04N 23/555: {for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes} - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A method for remotely supporting a surgery assistant robot may include: receiving at least one piece of operation information concerning an operation of the surgery assistant robot, by a server device that performs a remote support for the surgery assistant robot; and transmitting, in response to a predetermined event, at least one of a sound, an image, or a text from the server device to at least one of the surgery assistant robot or a terminal device. ## Claims ### Claim 1 A remote assistance method for a surgical assistance robot that is provided in a medical facility and comprises a patient-side device to which a medical instrument and an endoscope are attached and a remote operation device configured to remotely operate the medical instrument attached to the patient-side device, the method comprising: performing, by a communication control device, image processing to visually obscure identifiable information that can identify a person on an image captured by at least one of the endoscope attached to the patient-side device of the surgical assistance robot in the medical facility and a camera that captures an image of a room in which the surgical assistance robot is installed, so as to obtain a processed image; transmitting the processed image from the communication control device to a server device; distributing the processed image from the server device to a second terminal device provided in a call center that is different from the medical facility and that does not provide a remote operation apparatus that is configured to remotely control the medical instrument attached to the patient-side device; transmitting, from the communication control device to the server device, a request for a bi-directional communication that is transmitted from at least one of the surgical assistance robot and a first terminal device used in the medical facility; establishing, by the server device, based on to the request, the bi-directional communication between (i) the at least one of the surgical assistance robot and the first terminal device in the medical facility and (ii) the second terminal device in the call center; and displaying, by the second terminal device in the call center, the processed image for a remote assistance for an abnormality occurred to the surgical assistance robot or in the room in which the surgical assistance robot is provided. ### Claim 2 (depends on 1) The method according to claim 1, further comprising determining whether the identifiable information appears in the captured image, and wherein the image processing is performed when it is determined that the identifiable information appears in the captured image. ### Claim 3 (depends on 1) The method according to claim 1, further comprising establishing, in response to detecting a predetermined event, the bi-directional communication between (i) the at least one of the surgical assistance robot and the first terminal device in the medical facility and (ii) the second terminal device in the call center. ### Claim 4 (depends on 3) The method according to claim 3, wherein the predetermined event is detected based on detection of an abnormal situation based on image analysis or voice analysis. ### Claim 5 (depends on 3) The method according to claim 3, wherein the predetermined event is detected by receiving a request of a bi-directional communication transmitted from the second terminal device. ### Claim 6 (depends on 3) The method according to claim 3, wherein the server device is configured to receive the processed image and distribute the processed image to at least the second terminal device in the call center when the bi-directional communication is established. ### Claim 7 (depends on 1) The method according to claim 1, wherein both of (i) a first processed image from the camera and (ii) a second processed image from an endoscope attached to the surgical assistance robot are transmitted to the second terminal device in the call center. ### Claim 8 A remote support system comprising: a first terminal device used in a medical facility in which a surgical assistance robot is provided, the surgical assistance robot comprising a patient-side device to which a medical instrument and an endoscope are attached and a remote operation device configured to remotely operate the medical instrument attached to the patient-side device; a second terminal device provided in a call center different from the medical facility, wherein the call center is not provided with a remote operation apparatus configured to remotely control the medical instrument attached to the patient-side device; a server device; and a communication control device configured to transmit to the server device an image captured by at least one of the endoscope attached to the patient-side device of the surgical assistance robot and a camera that captures an image of a room in which the surgical assistance robot is installed, and to receive information from the server device, wherein the communication control device is configured to perform image processing for visually obscuring identifiable information that can identify a person on the captured image to obtain a processed image, and transmit the processed image to the server device, the server device is configured to receive the processed image from the communication control device and to transmit the processed image to the second terminal device in the call center, the communication control device is configured to transmit, to the server device, a request requesting a bi-directional communication transmitted from at least one of the surgical assistance robot and the first terminal device in the medical facility, the server device is configured, in response to the request, to establish the bi-directional communication between (i) the at least one of the surgical assistance robot and the first terminal device in the medical facility and (ii) the second terminal device in the call center, and the second terminal device in the call center is configured to display the processed image on a monitor, for remote support for an abnormality occurred to the surgical assistance robot or in the room in which the surgical assistance robot is provided. ### Claim 9 (depends on 8) The system according to claim 8, wherein the communication control device is configured to determine whether the identifiable information appears in the captured image, and to perform the image processing when the identifiable information appears in the captured image. ### Claim 10 (depends on 8) The system according to claim 8, wherein the server device is configured to establish, in response to detecting a predetermined event, the bi-directional communication between (i) at least one of the surgical assistance robot and the first terminal device used in the medical facility and (ii) the second terminal device in the call center. ### Claim 11 (depends on 10) The system according to claim 10, wherein the server device is configured to detect the predetermined event based on detection of an abnormal situation based on image analysis of the image or audio analysis. ### Claim 12 (depends on 10) The system according to claim 10, wherein the server device is configured to detect the predetermined event by receiving the request requesting the bi-directional communication transmitted from the second terminal device. ### Claim 13 (depends on 10) The system according to claim 10, wherein the server device is configured to distribute the processed image received by the server device to at least the second terminal device when the bi-directional communication is established. ### Claim 14 (depends on 10) The system according to claim 10, wherein the server device is configured to distribute the image captured by the camera and received by the server device, and the image captured by an endoscope attached to the surgical assistance robot to at least the second terminal device. ### Claim 15 A surgical robot system comprising: a surgical assistance robot provided in a medical facility, the surgical assistance robot comprising a patient-side device to which a medical instrument and an endoscope are attached and a remote operation device configured to remotely operate the medical instrument attached to the patient-side device; a first terminal device in the medical facility; a second terminal device in a call center different from the medical facility, wherein the call center is not provided with a remote operation apparatus configured to remotely control the medical instrument attached to the patient-side device; a server device; and a communication control device configured to transmit to the server device an image captured by at least one of the endoscope attached to the patient-side device of the surgical assistance robot and a camera that captures an image of a room in which the surgical assistance robot is installed, and to receive information from the server device, wherein the communication control device is configured to perform image processing for visually obscuring identifiable information that can identify a person on the image to obtain a processed image, and transmit the processed image to the server device, the server device is configured to receive the processed image from the communication control device and transmit the processed image to the second terminal device in the call center, the communication control device is configured to transmit, to the server device, a request requesting a bi-directional communication transmitted from at least one of the surgical assistance robot and the first terminal device in the medical facility, the server device is configured, in response to the request, to establish the bi-directional communication between (i) the at least one of the surgical assistance robot and the first terminal device in the medical facility and (ii) the second terminal device in the call center, and the second terminal device in the call center is configured to display the processed image on a monitor, for remote support for an abnormality occurred to the surgical assistance robot or in the room in which the surgical assistance robot is provided. ### Claim 16 (depends on 15) The system according to claim 15, wherein the communication control device is configured to determine whether the identifiable information appears in the captured image, and to perform the image processing when the identifiable information appears in the captured image. ### Claim 17 (depends on 15) The system according to claim 15, wherein the server device is configured to establish, in response to detecting a predetermined event, the bi-directional communication between (i) at least one of the surgical assistance robot and the first terminal device used in the medical facility and (ii) the second terminal device in the call center. ### Claim 18 (depends on 15) The system according to claim 15, wherein the server device is configured to detect a predetermined event based on detection of an abnormal situation based on image analysis of the image or audio analysis. ### Claim 19 (depends on 15) The system according to claim 15, wherein the server device is configured to detect a predetermined event by receiving the request requesting the bi-directional communication transmitted from the second terminal device. ### Claim 20 (depends on 15) The system according to claim 15, wherein the server device is configured to distribute the processed image received by the server device to at least the second terminal device when the bi-directional communication is established. ### Claim 21 (depends on 15) The system according to claim 15, wherein the server device is configured to distribute the image captured by the camera and received by the server device, and the image captured by an endoscope attached to the surgical assistance robot to at least the second terminal device.