# US 12,599,444 B2 · Surgical image providing method using surgical robot, and information processing system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,599,444 B2
- Document ID: US12599444B2
- Status: Granted
- Kind: B2
- Application number: 18/333,335
- Filed: 2023-06-12
- Published: 2023-12-21
- Granted: 2026-04-14
- Earliest claimed: 2022-06-17
- Assignees: Sysmex Corporation
- Applicants: Sysmex Corporation
- Inventors: Masanao Ohashi; Yutaka Kitayama; Keiko Ishino; Arisa Ikeda
- Web page: https://patentgrep.com/patent/US12599444B2

## CPC

- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/30: Surgical robots
- A61B 17/29: Forceps for use in minimally invasive surgery
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 18/085: {Forceps, scissors}
- A61B 18/1442: {Probes having pivoting end effectors, e.g. forceps}
- A61B 2017/00119: {alarm; indicating an abnormal situation}
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2017/00973: {pedal-operated}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2059: {Mechanical position encoders}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2034/301: {for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes}
- A61B 2034/305: {Details of wrist mechanisms at distal ends of robotic arms}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

Disclosed is a surgical image providing method using a surgical robot for providing an image of a surgery performed by the surgical robot, the surgical robot including an arm device to which a surgical instrument is attached, and an operation device that an operator operates to drive the arm device. The method includes: obtaining a state log indicating a state of an operation target of the operation device, the state being changed according to an operation of the operator to the operation target; generating a reconstructed image visualizing the operation of the operator to the operation target, on the basis of the state log; and providing a surgical image in which a motion image showing a motion of the surgical robot or the surgical instrument is associated with the reconstructed image.

## Claims

### Claim 1

A surgical robot system comprising a surgical robot, a storage device and an information processing device, wherein
the surgical robot comprises:
a first arm to which a forceps is detachably attachable, the first arm having a motor operable to open and close the forceps;
a second arm to which an endoscope is detachably attachable, the endoscope being configured to capture a surgical motion image that shows operation of the forceps attached to the first arm;
an operation device including a hand controller comprising (i) a pair of movable members operable to change an angle between the pair of movable members to thereby open and close the forceps attached to the first arm, and (ii) a first detector configured to detect the angle between the pair of movable members; and
an arm control device configured to monitor the angles between the pair of movable members detected by the first detector and operate the motor of the first arm to open and close the forceps based on the detected angle between the pair of movable members,
the storage device is configured to communicate with the surgical robot and programmed to store a first set of log data representative of a change of the angle between the pair of movable members detected by the first detector during a first time interval and a series of the surgical motion images captured by the endoscope during a second time interval, wherein the first and second time intervals are at least partially co-temporal to each other; and
the information processing device is configured to communicate with the storage device and programmed to:
obtain, from the storage device, the series of surgical motion images and the first set of log data,
generate a first set of reconstructed motion images showing relative positions of the pair of movable members, based on the obtained the first set of log data; and
play the first set of reconstructed motion images and the surgical motion image in synchronism with each other.

### Claim 2 (depends on 1)

The system of claim 1, wherein the operation device further comprises a viewer screen configured to display the reconstructed motion images and the series of surgical motion images played by the information processing device.

### Claim 3 (depends on 1)

The system of claim 1, wherein the reconstructed motion images include a schematic motion image showing the relative positions of the pair of movable members.

### Claim 4 (depends on 3)

The system of claim 3, wherein the reconstructed motion images show, in a numerical form, the angle between the pair of the movable members.

### Claim 5 (depends on 1)

The system of claim 1, wherein
the first arm includes a second detector configured to detect a pinching angle indicative of a degree of pinching by the forceps,
the storage device is configured to store a second set of log data representative of a change of the pinching angle detected by the second detector during the first time interval, and
the information processing device is programmed to:
obtain, from the storage device, the second set of log data;
generate a second set of reconstructed motion images showing the degree of pinching by the forceps, based on the obtained second set of log data; and
play the first set of reconstructed motion images, the second set of reconstructed motion images and the surgical motion images in synchronism with each other.

### Claim 6 (depends on 5)

The system of claim 5, wherein the second set of reconstructed motion images shows a schematic motion image showing the degree of pinching by the forceps.

### Claim 7 (depends on 6)

The system of claim 6, wherein the second set of reconstructed motion images further shows, in a numerical form, the degree of pinching by the forceps.

### Claim 8 (depends on 1)

The system of claim 1, wherein
an electrocautery is detachably attached to the first arm,
the operation device further includes a foot pedal and a third detector configured to detect a stepping operation of the foot pedal, wherein the electrocautery is energized by operation of the foot pedal, and
the storage device is configured to store a third set of log data representative of a operation of the foot peal detected by the third detector during the first time interval, and
the information processing device is programmed to:
obtain, from the storage device, the third set of log data;
generate a third set of reconstructed motion images showing a operation of the electrocautery, based on the obtained third set of log data; and
play the first set of reconstructed motion image, the third set of reconstructed motion images, and the surgical motion images in synchronism with each other.

### Claim 9 (depends on 8)

The system of claim 8, wherein the information processing device is programmed to play the first set of reconstructed motion images and the third set of reconstructed motion images in such a manner that the first set of reconstructed motion images is displayed above the third set of reconstructed motion images on a viewer screen.

### Claim 10 (depends on 8)

The system of claim 8, wherein
the operation device further includes a fourth detector configured to detect a position of the foot of the operator on and near the foot pedal,
the storage device is configured to store a fourth set of log data representative of the position of the foot of the operator detected by the fourth detector during the first time interval, and
the information processing device is programmed to:
obtain, from the storage device, the fourth set of log data;
generate a fourth set of reconstructed motion images showing the position of the foot of the operator on and near the foot pedal based on the obtained fourth set of log data; and
play the first set of reconstructed motion images, the fourth set of reconstructed motion images, and the surgical motion images in such a manner that the first set of reconstructed motion images, the fourth set of reconstructed motion images, and the surgical motion images are shown in synchronism with each other.

### Claim 11 (depends on 1)

The system of claim 1, wherein
the operation device further includes a viewer screen configured to display the first set of reconstructed motion images and the surgical motion images and a fifth detector configured to detect a head position of the operator in front of the viewer screen, and
the storage device is configured to store a fifth set of log data representative of a movement of the head position detected by the fifth detector during the first time interval, and
the information processing device is programmed to:
obtain, from the storage device, the fifth set of log data; and
display a screen showing the movement of the head position based on the fifth set of log data on the viewer screen.

### Claim 12 (depends on 1)

The system of claim 1, wherein
the first arm includes a sixth detector configured to detect attachment and detachment of the forceps to and from the first arm,
the storage device is configured to store a sixth set of log data representative of the attachment and detachment of the forceps detected by the sixth detector during the first time interval, and
the information processing device is programmed to:
obtain, from the storage device, the sixth set of log data;
display a screen showing, based on the sixth set of log data, the attachment and detachment of the forceps to and from the first arm on a viewer screen operable to display the first set of reconstructed motion images and the surgical motion images.

### Claim 13 (depends on 12)

The system of claim 12, wherein
the arm control device is configured to determine a type of the forceps attached to the first arm,
the storage device is configured to store a seventh set of log data representative of the type of forceps attached to the first arm during the first time interval, and
the information processing device is programmed to:
obtain, from the storage device, the seventh set of log data; and
display on the viewer screen the type of forceps attached to the first arm based on the seventh set of log data.