# US 2018/0324414 A1 · Automated panning in robotic surgical systems based on tool tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0324414 A1
- Document ID: US20180324414A1
- Status: Published application
- Kind: A1
- Application number: 16/034,004
- Filed: 2018-07-12
- Published: 2018-11-08
- Earliest claimed: 2008-03-28
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: Brian D. Hoffman; Christopher J. Hasser; William C. Nowlin
- Web page: https://patentgrep.com/patent/US20180324414A1

## CPC

- H04N 13/366: using viewer tracking
- A61B 1/00042: {for mechanical operation}
- A61B 34/30: Surgical robots
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- H04N 13/156: Mixing image signals
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
- A61B 1/00193: {adapted for stereoscopic vision}
- A61B 1/042: {characterised by a proximal camera, e.g. a CCD camera}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 2017/00477: {Coupling (A61B2017/0046 takes precedence)}
- A61B 2090/371: {with simultaneous use of two cameras}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

In one embodiment, a digital zoom and panning system for digital video is disclosed including an image acquisition device to capture digital video images; an image buffer to store one or more frames of digital video images as source pixels; a display device having first pixels to display images; a user interface to accept user input including a source rectangle to select source pixels within frames of the digital video images, a destination rectangle to select target pixels within the display device to display images, and a region of interest within the digital video images to display in the destination rectangle; and a digital mapping and filtering device to selectively map and filter source pixels in the region of interest from the image buffer into target pixels of the display device in response to the user input.

## Claims

### Claim 1

A method comprising:
capturing video images of a site, each image including an array of source pixels having pixel information of a first resolution;
selecting first inner and outer source windows in the video images for a first array of source pixels in the first inner source window and a first set of source pixels between the first inner and outer source windows;
selecting first inner and outer target windows in target pixels of a first display device for a first array of target pixels in the first inner target window and a first set of target pixels between the first inner and outer target windows;
mapping the first array of source pixels into the first array of target pixels;
mapping the first set of source pixels into the first set of target pixels; and
displaying on the first display device the first array of target pixels in the first inner target window and the first set of target pixels between the first inner and outer target windows around the first array of target pixels.

### Claim 2 (depends on 1)

The method of claim 1, wherein
the mapping of the first array of source pixels into the first array of target pixels is a linear mapping; and
the mapping of the first set of source pixels into the first set of target pixels is a non-linear mapping.

### Claim 3 (depends on 2)

The method of claim 2, wherein
the linear mapping is a linear scaling factor function.
the non-linear mapping is a bilinear interpolation function.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
subdividing the first set of target pixels into a plurality of rectangular arrays of target pixels with four vertices of each being assigned a predetermined mapping from source pixels to target pixels, and
for the first set of target pixels in an interior of each of the plurality of rectangular arrays of target pixels, bilinearly interpolating between the first set of source pixels and the first set of target pixels.

### Claim 5 (depends on 2)

The method of claim 2, further comprising:
adjusting the selection of the first inner source window to digitally zoom a first portion of the video images mapped into the first inner target window.

### Claim 6 (depends on 5)

The method of claim 5, wherein
the size of the first inner source window is decreased to magnify the first portion of the video images mapped in the first inner target window.

### Claim 7 (depends on 5)

The method of claim 5, wherein
the size of the first inner source window is increased to demagnify the first portion of the video images mapped in the first inner target window.

### Claim 8 (depends on 2)

The method of claim 2, further comprising:
adjusting a position of the first inner source window with respect to the first outer source window to digitally pan over source pixels in the video images.

### Claim 9 (depends on 7)

The method of claim 7, wherein
if the first inner source window becomes near the outer source window, the method further includes
adjusting a position of the first outer source window to center the first inner source window within the first outer source window.

### Claim 10 (depends on 2)

The method of claim 2, wherein
the first inner and outer source windows are concentric, and
the method further includes
adjusting positions of the first inner and outer source windows together to digitally pan over source pixels in the video images mapped into the first array of target pixels in the first inner target window and the first set of target pixels between the first inner and outer target windows.

### Claim 11 (depends on 2)

The method of claim 2, wherein
the first inner and outer target windows are concentric, and
the method further includes
adjusting positions of the first inner and outer target windows together to reposition the first inner and outer target windows in the first display.

### Claim 12 (depends on 2)

The method of claim 2, further comprising:
adjusting a position of the first inner target window to digitally pan over target pixels in the first display device and reposition the first inner target window within the first outer target window.

### Claim 13 (depends on 1)

The method of claim 1, wherein
the mapping of the first array of source pixels into the first array of target pixels is a first linear mapping; and
the mapping of the first set of source pixels into the first set of target pixels is a second linear mapping.

### Claim 14 (depends on 13)

The method of claim 13, wherein
the first linear mapping is a linear scaling factor function.
the second linear mapping is a piece-wise linear mapping.

### Claim 15 (depends on 1)

The method of claim 1, further comprising:
selecting second inner and outer source windows in the video images for a second array of source pixels in the second inner source window and a second set of source pixels between the second inner and outer source windows;
selecting second inner and outer target windows in target pixels of a second display device for a second array of target pixels in the second inner target window and a second set of target pixels between the second inner and outer target windows;
mapping the second array of source pixels into the second array of target pixels;
mapping the second set of source pixels into the second set of target pixels; and
displaying on the second display device the second array of target pixels in the second inner target window and the second set of target pixels between the second inner and outer target windows around the second array of target pixels.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
displaying on a second display device the first array of target pixels in the first inner target window and the first set of target pixels between the first inner and outer target windows around the first array of target pixels.

### Claim 17 (depends on 1)

The method of claim 1, further comprising:
prior to displaying the first array of target pixels and the first set of target pixels, digitally filtering the first array of target pixels and the first set of target pixels.

### Claim 18 (depends on 1)

The method of claim 1, wherein
the site is a minimally invasive surgical site and the video images are captured by a robotic endoscopic camera.

### Claim 19 (depends on 9)

The method of claim 9, further comprising
detecting motion of a user to automatically adjust the positions of the first inner and outer source windows.

### Claim 20 (depends on 19)

The method of claim 19, wherein
the motion of the user to detect is one of a finger by a touch sensitive panel, a pair of hands by a motion sensing display device, a pair of eyes by a gaze detector, and a foot by a pedal.

### Claim 21

A digital zoom and panning system for digital video, the system comprising:
an image acquisition device to capture digital video images;
an image buffer coupled to the image acquisition device, the image buffer to store one or more frames of digital video images as source pixels;
a first display device having first pixels to display images; and
a first digital mapping and filtering device coupled to the image buffer, the first digital mapping and filtering device to selectively map and filter source pixels in a first region of interest from the image buffer into target pixels in a first destination rectangle of the first display device.

### Claim 22 (depends on 21)

The digital zoom and panning system of claim 21, further comprising:
a first user interface to accept a first user input including the first source rectangle to select first source pixels within frames of the digital video images and the first destination rectangle to select target pixels within the first display device to display images;
wherein the first digital mapping and filtering device to automatically select a first region of interest within the digital video images to display in the first destination rectangle; and
wherein the first digital mapping and filtering device selectively maps and filters the source pixels in the first region of interest into target pixels of the first display device.

### Claim 23 (depends on 22)

The digital zoom and panning system of claim 22, wherein
the automatic selection of the first region of interest is in response to a tool centroid, a depth of focus, a user focal-point, an average working envelope, or a combination thereof.

### Claim 24 (depends on 21)

The digital zoom and panning system of claim 21, further comprising:
a first user interface to accept a first user input including the first source rectangle to select first source pixels within frames of the digital video images, the first destination rectangle to select target pixels within the first display device to display images, and the first region of interest within the digital video images to display in the first destination rectangle; and
wherein the first digital mapping and filtering device selectively maps and filters the source pixels in the first region of interest into target pixels of the first display device in response to the first user input.

### Claim 25 (depends on 21)

The digital zoom and panning system of claim 21, further comprising:
a first display buffer coupled to and between the first display device and the digital mapping and filtering device, the first display buffer to store one or more frames of target pixels received from the digital mapping and filtering device and write one or more frames of target pixels to the display device to display images.

### Claim 26 (depends on 21)

The digital zoom and panning system of claim 21, further comprising:
a second display device having target pixels to display images; and
a second digital mapping and filtering device coupled to the image buffer and the user interface, the second digital mapping and filtering device to selectively map and filter source pixels in a second region of interest from the image buffer to digitally zoom into target pixels in a second destination rectangle of the second display device.

### Claim 27 (depends on 21)

The digital zoom and panning system of claim 21, further comprising:
a second user interface to accept second user input including the second source rectangle to select second source pixels within frames of the digital video images, the second destination rectangle to select target pixels within the second display device to display images, and the second region of interest within the digital video images to display in the second destination rectangle.

### Claim 28 (depends on 21)

The digital zoom and panning system of claim 21, further comprising:
a gaze detection system to detect the first region of interest within the digital video images to digitally pan and display into the first destination rectangle in response to a gaze detected from a user's left and right eyes.

### Claim 29 (depends on 21)

The digital zoom and panning system of claim 21, further comprising:
a tool tracking system to detect the first region of interest within the digital video images to digitally pan and display in the first destination rectangle in response to one or more tracked positions of one or more tools by the tool tracking system.

### Claim 30 (depends on 21)

The digital zoom and panning system of claim 21, wherein
the image acquisition device is a robotic endoscopic camera to capture digital video images of a surgical site and to be mechanically panned within the surgical site; and
in response to the detection of the first region of interest being outside the first source rectangle, the robotic endoscopic camera to mechanically pan within the surgical site and the first digital mapping and filtering device to digitally pan and display images in the first region of interest into the first destination rectangle.

### Claim 31

A method for a digital video system, the method comprising:
selecting an outer target window on a display device within which to display a fovea image and a surround image around the fovea image;
selecting an inner target window on the display device within the outer target window to display the fovea image;
linearly mapping source pixels of a digital video into target pixels in the inner target window to display the fovea image and non-linearly mapping source pixels of the digital video into target pixels between the inner target window and the outer target window to display the surround image around the fovea image.

### Claim 32 (depends on 31)

The method of claim 31, wherein
the display device includes a touch sensitive display panel; and
the outer target window and the inner target window are selected by a touching a finger on the touch sensitive display panel.

### Claim 33 (depends on 32)

The method of claim 32, further comprising:
touching the touch sensitive display panel with the finger to pan over the source pixels of the digital video for mapping into the inner target window of the display.

### Claim 34 (depends on 31)

The method of claim 31, further comprising:
touching one or more buttons to pan over the source pixels of the digital video for mapping into the inner target window of the display.

### Claim 35 (depends on 31)

The method of claim 31, further comprising:
adjusting the position of the display device to pan over the source pixels of the digital video for mapping into the inner target window of the display.

### Claim 36 (depends on 35)

The method of claim 35, further comprising:
sensing the motion of the display device with an inertia sensor to digitally pan the source pixels of the digital video.

### Claim 37 (depends on 35)

The method of claim 35, further comprising:
sensing the motion of the display device with one or more encoders at one or more joints of set-up arms supporting the display device.
