# US 2010/0169815 A1 · Visual force feedback in a minimally invasive surgical procedure

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0169815 A1
- Document ID: US20100169815A1
- Status: Published application
- Kind: A1
- Application number: 12/428,142
- Filed: 2009-04-22
- Published: 2010-07-01
- Earliest claimed: 2008-12-31
- Assignees: Intuitive Surgical, Inc.
- Inventors: Wenyi Zhao; Tao Zhao; David Q. Larkin
- Web page: https://patentgrep.com/patent/US20100169815A1

## CPC

- B25J 9/1633: {compliant, force, torque control, e.g. combined with position control}
- A61B 34/30: Surgical robots
- A61B 34/35: for telesurgery
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 34/76: {Manipulators having means for providing feel, e.g. force or tactile feedback}
- B25J 9/1697: {Vision controlled systems}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 2034/102: {Modelling of surgical devices, implants or prosthesis}
- A61B 2090/065: {for measuring contact or contact pressure}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- Y10S 901/47: Optical
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS

## Abstract

Methods of and a system for providing a visual representation of force information in a robotic surgical system. A real position of a surgical end effector is determined. A projected position of the surgical end effector if no force were applied against the end effector is also determined. Images representing the real and projected positions are output superimposed on a display. The offset between the two images provides a visual indication of a force applied to the end effector or to the kinematic chain that supports the end effector. In addition, tissue deformation information is determined and displayed.

## Claims

### Claim 1

A method of providing a visual representation of force information in a robotic surgical system, the method comprising:
determining a force applied against a surgical end effector;
displaying a first image showing an actual position of the end effector under the applied force; and
superimposing on the first image a second image representing a projected position of the end effector offset from the first position, wherein the offset is a visual indication of the force.

### Claim 2 (depends on 1)

The method of claim 1, wherein said at least some force comprises force applied to the end effector as a result of movement of the end effector between a first position and a second position, wherein another force is applied to a linkage supporting the end effector, and wherein the offset between the projected position and the actual position disregards at least some of the other force.

### Claim 3 (depends on 1)

The method of claim 1, wherein the second image represents kinematic position information derived from joint states of a linkage supporting the end effector.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first image comprises a visual image of the end effector.

### Claim 5 (depends on 1)

The method of claim 1, wherein displaying comprises displaying on a viewer of a surgeon console of the robotic surgical system.

### Claim 6 (depends on 1)

The method of claim 1, wherein the second image comprises a synthetic representation of the tool showing a position of the tool, an orientation of the tool, or both.

### Claim 7 (depends on 6)

The method of claim 6, wherein the synthetic representation comprises a model of the tool.

### Claim 8 (depends on 7)

The method of claim 7, wherein the model comprises a three-dimensional model of the tool.

### Claim 9 (depends on 1)

The method of claim 1, wherein the force is derived from force information derived from active force sensors.

### Claim 10 (depends on 9)

The method of claim 9, wherein the active force sensors comprise force sensors located at the end effector.

### Claim 11 (depends on 10)

The method of claim 10, wherein the end effector is supported by a linkage having joints, and wherein the active force sensors comprise sensors located at the joints.

### Claim 12 (depends on 1)

The method of claim 1, wherein the end effector is supported by a linkage having joints, and further comprising:
determining joint errors from positions of the joints with respect to a commanded position of the joints;
correcting the joints toward the command position; and
wherein the force is derived from information derived from the correcting of the joints.

### Claim 13

A system of providing a visual representation of force information in a robotic system, the system comprising:
an image input for capturing an image of an end effector while a force is applied to the end effector;
a display coupled to the image input so as to present an actual position of the end effector under the applied force; and
a processor coupled to the display, the processor generating a second image representing a projected position of the end effector offset from the first position so as to visually indicate the force, the processor transmitting the second image to the display so that the second image is superimposed with the first image.

### Claim 14

A method of providing a visual representation of tissue deformation in a robotic surgical system, the method comprising:
applying a force against a tissue;
determining deformation of the tissue, the deformation being due to application of the force; and
displaying information about the deformation.

### Claim 15 (depends on 14)

The method of claim 14, wherein determining the deformation comprises utilizing the force to derive the deformation.

### Claim 16 (depends on 15)

The method of claim 15, wherein the force is derived from force information derived from active force sensors.

### Claim 17 (depends on 16)

The method of claim 16, wherein the active force sensors comprise force sensors located at the end effector.

### Claim 18 (depends on 16)

The method of claim 16, wherein the end effector is supported by a linkage having joints, and wherein the active force sensors comprise sensors located at the joints.

### Claim 19 (depends on 15)

The method of claim 15, wherein the end effector is supported by a linkage having joints, and further comprising:
determining joint errors from positions of the joints with respect to a commanded position of the joints;
correcting the joints toward the command position; and
wherein the force is derived from information derived from the correcting of the joints.

### Claim 20 (depends on 15)

The method of claim 15, wherein the force is derived from force information, the force information being provided by:
storing first kinematic position information and first actual position information for a first position of an end effector;
moving the end effector via the robotic surgical system from the first position a second position;
storing second kinematic position information and second actual position information; and
providing force information regarding force applied to the end effector at the second position utilizing the first actual position information, the second actual position information, the first kinematic position information, and the second kinematic position information.

### Claim 21 (depends on 14)

The method of claim 14, wherein displaying information about the deformation comprises displaying the tissue simultaneous with the information about the deformation.

### Claim 22 (depends on 21)

The method of claim 21, wherein the information about the deformation is superimposed over the tissue.

### Claim 23 (depends on 22)

The method of claim 22, wherein the information about the deformation comprises superimposing a color over the tissue.

### Claim 24 (depends on 23)

The method of claim 23, wherein the color changes based upon the amount of force applied to the tissue.

### Claim 25 (depends on 23)

The method of claim 23, wherein the color is varied based upon how far the tissue is from contact with the end effector.

### Claim 26 (depends on 14)

The method of claim 14, wherein the deformation information is combined with ultrasound imaging to provide elasticity imaging of tissues.
