# US 12,178,399 B2 · System and method for view restoration > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,178,399 B2 - Document ID: US12178399B2 - Status: Granted - Kind: B2 - Application number: 17/599,989 - Filed: 2020-04-02 - Published: 2022-05-12 - Granted: 2024-12-31 - Earliest claimed: 2019-04-03 - Assignees: Intuitive Surgical Operations, Inc. - Applicants: Intuitive Surgical Operations, Inc. - Inventors: Brandon D. Itkowitz; Yiming Mei - Web page: https://patentgrep.com/patent/US12178399B2 ## CPC - A61B 1/00149: {using articulated arms} - A61B 1/00006: {of control signals} - A61B 1/00009: {of image signals during a use of endoscope} - A61B 1/00174: {characterised by the viewing angles} - A61B 1/00183: {for variable viewing angles} - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects - A61B 2034/2057: {Details of tracking cameras} - G06T 2207/10068: Endoscopic image - H04N 23/555: {for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes} - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system and method of view restoration include a computer-assisted device having an imaging device and a controller coupled to the imaging device. The controller is configured to record kinematic information, imaging information, or both the kinematic information and the imaging information before movement of the imaging device from a first repositionable arm to a second repositionable arm or from a first workspace port to a second workspace port; detect the movement of the imaging device from the first repositionable arm to the second repositionable arm or from the first workspace port to the second workspace port; determine, in response to the detection, a desired position and orientation of the imaging device based on the recorded kinematic information, the recorded imaging information, or both the recorded kinematic information and the recorded imaging information; and move the imaging device based on the desired position and orientation. ## Claims ### Claim 1 A computer-assisted device comprising: an imaging device; and a controller coupled to the imaging device and configured to: record kinematic information, imaging information, or both the kinematic information and the imaging information of the computer-assisted device before movement of the imaging device from being mounted to a first repositionable arm to being mounted to a second repositionable arm or from being inserted through a first workspace port to access a workspace to being inserted through a second workspace port to access the workspace; detect that the imaging device has moved from being mounted to the first repositionable arm to being mounted to the second repositionable arm or from being inserted through the first workspace port to access the workspace to being inserted through the second workspace port to access the workspace; determine, in response to the detection, a desired position and orientation of the imaging device based on the recorded kinematic information, the recorded imaging information, or both the recorded kinematic information and the recorded imaging information; and move the imaging device based on the desired position and orientation. ### Claim 2 (depends on 1) The computer-assisted device of claim 1, wherein the imaging device has a first view before the movement of the imaging device from being mounted to the first repositionable arm or from being inserted through the first workspace port to access the workspace, wherein the imaging device has a second view after the imaging device has moved to being mounted to the second repositionable arm or to being inserted through the second workspace port to access the workspace, and wherein the desired position and orientation cause the imaging device to have a third view that is more similar to the first view than the second view. ### Claim 3 (depends on 2) The computer-assisted device of claim 2, wherein the first view is determined based on one or more parameters selected from the group consisting of: a direction of view of the imaging device; a view up direction of the imaging device, a center of view of the imaging device, a depth of a point of interest or the center of view from the imaging device; a position of a distal portion of each of one or more instruments; a visibility of a point of interest; a position of the point of interest; and an orientation of the point of interest. ### Claim 4 (depends on 1) The computer-assisted device of claim 1, wherein the imaging device is an endoscope. ### Claim 5 (depends on 1) The computer-assisted device of claim 1, wherein the controller is further configured to determine one or more parameters selected from the group consisting of: a direction of view of the imaging device based on the recorded imaging information; a view up direction of the imaging device based on the recorded imaging information or a direction of gravity; a depth between a point of interest and the imaging device; and a center of view of the imaging device. ### Claim 6 (depends on 1) The computer-assisted device of claim 1, wherein the controller is configured to determine the desired position and orientation of the imaging device by using one or more parameters selected from the group consisting of: a desired center of view of the imaging device; a desired direction of view of the imaging device; a desired view up direction of the imaging device; and a desired depth from the imaging device to a point of interest or the desired center of view. ### Claim 7 (depends on 1) The computer-assisted device of claim 1, wherein: the controller is configured to determine the desired position and orientation of the imaging device by minimizing differences between first and second representative positions and orientations, the second representative position and orientation being after the imaging device has moved from being mounted to the first repositionable arm to being mounted to the second repositionable arm or from being inserted through the first workspace port to access the workspace to being inserted through the second workspace port to access the workspace, and the first representative position and orientation being before the movement of the imaging device from being mounted to the first repositionable arm to being mounted to the second repositionable arm or from being inserted through the first workspace port to access the workspace to being inserted through the second workspace port to access the workspace. ### Claim 8 (depends on 7) The computer-assisted device of claim 7, wherein each of the first and second representative positions and orientations comprise one or more parameters selected from the group consisting of: a position of a distal portion of each of one or more instruments; orientation axes for the distal portion of each of the one or more instruments; a visibility of one or more points of interest; a position of the one or more points of interest; and an orientation of the one or more points of interest. ### Claim 9 (depends on 7) The computer-assisted device of claim 7, wherein to determine the desired position and orientation of the imaging device, the controller is further configured to: determine impact on an objective function applied to the first and second representative positions and orientations due to incremental movements of a plurality of joints used to control the imaging device; estimate a gradient in a joint space of the plurality of joints based on the determined impact; and simulate movement of the imaging device based on the estimated gradient. ### Claim 10 (depends on 1) The computer-assisted device of claim 1, wherein to move the imaging device based on the desired position and orientation, the controller is configured to: determine whether the imaging device has an articulated wrist; and in response to a determination that the imaging device has an articulated wrist: position the imaging device using degrees of freedom of the first or second repositionable arm; and orient the imaging device using the articulated wrist and a roll degree of freedom of the first or second repositionable arm; or in response to a determination that the imaging device does not have an articulated wrist: position the imaging device using the degrees of freedom of the first or second repositionable arm; and orient the imaging device using the degrees of freedom of the first or second repositionable arm. ### Claim 11 A method comprising: recording, using a controller, kinematic information, imaging information, or both the kinematic information and the imaging information of a computer-assisted device before movement of an imaging device from being mounted to a first repositionable arm to being mounted to a second repositionable arm or from being inserted through a first workspace port to access a workspace to being inserted through a second workspace port to access the workspace; detecting, by the controller, that the imaging device has moved from being mounted to the first repositionable arm to being mounted to the second repositionable arm or from being inserted through the first workspace port to access the workspace to being inserted through the second workspace port to access the workspace; determining, by the controller in response to the detecting, a desired position and orientation of the imaging device based on the recorded kinematic information, the recorded imaging information, or both the recorded kinematic information and the recorded imaging information; and moving, by the controller, the imaging device based on the desired position and orientation. ### Claim 12 (depends on 11) The method of claim 11, wherein the imaging device has a first view before the movement of the imaging device from being mounted to the first repositionable arm or from being inserted through the first workspace port to access the workspace, wherein the imaging device has a second view after the imaging device has moved to being mounted to the second repositionable arm or to being inserted through the second workspace port to access the workspace, and wherein the desired position and orientation cause the imaging device to have a third view that is more similar to the first view than the second view. ### Claim 13 (depends on 12) The method of claim 12, wherein the first view is determined based on one or more parameters selected from the group consisting of: a direction of view of the imaging device; a view up direction of the imaging device, a center of view of the imaging device, a depth of a point of interest or the center of view from the imaging device; a position of a distal portion of each of one or more instruments; a visibility of a point of interest; a position of the point of interest; and an orientation of the point of interest. ### Claim 14 (depends on 11) The method of claim 11, further comprising determining one or more parameters selected from the group consisting of: a direction of view of the imaging device based on the recorded imaging information; a view up direction of the imaging device based on the recorded imaging information or a direction of gravity; a depth between a point of interest and the imaging device; and a center of view of the imaging device. ### Claim 15 (depends on 11) The method of claim 11, further comprising determining the desired position and orientation of the imaging device by using one or more parameters selected from the group consisting of: a desired center of view of the imaging device; a desired direction of view of the imaging device; a desired view up direction of the imaging device; and a desired depth from the imaging device to a point of interest or the desired center of view. ### Claim 16 (depends on 15) The method of claim 15, further comprising determining the desired position and orientation of the imaging device to reduce a weighted sum of a distance between an actual center of view and the desired center of view and a distance between an actual view up direction and the desired view up direction. ### Claim 17 (depends on 11) The method of claim 11, further comprising: determining the desired position and orientation of the imaging device by minimizing differences between first and second representative positions and orientations, the second representative position and orientation being after the imaging device has moved from being mounted to the first repositionable arm to being mounted to the second repositionable arm or from being inserted through the first workspace port to access the workspace to being inserted through the second workspace port to access the workspace and the first representative position and orientation being before the movement of the imaging device from being mounted to the first repositionable arm to being mounted to the second repositionable arm or from being inserted through the first workspace port to access the workspace to being inserted through the second workspace port to access the workspace; wherein each of the first and second representative positions and orientations comprise one or more parameters selected from the group consisting of: a position of a distal portion of each of one or more instruments; orientation axes for the distal portion of each of the one or more instruments; a visibility of one or more points of interest; a position of the one or more points of interest; and an orientation of the one or more points of interest. ### Claim 18 A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which when executed by one or more processors associated with a computer-assisted device are adapted to cause the one or more processors to perform a method comprising: recording kinematic information, imaging information, or both the kinematic information and the imaging information of the computer-assisted device before movement of an imaging device from being mounted to a first repositionable arm to being mounted to a second repositionable arm or from being inserted through a first workspace port to access a workspace to being inserted through a second workspace port to access the workspace; detecting that the imaging device has moved from being mounted to the first repositionable arm to being mounted to the second repositionable arm or from being inserted through the first workspace port to access the workspace to being inserted through the second workspace port to access the workspace; determining, in response to the detecting, a desired position and orientation of the imaging device based on the recorded kinematic information, the recorded imaging information, or both the recorded kinematic information and the recorded imaging information; and moving the imaging device based on the desired position and orientation. ### Claim 19 (depends on 18) The non-transitory machine-readable medium of claim 18, wherein the imaging device has a first view before the movement of the imaging device from being mounted to the first repositionable arm or from being inserted through the first workspace port to access the workspace, wherein the imaging device has a second view after the imaging device has moved to being mounted to the second repositionable arm or to being inserted through the second workspace port to access the workspace, and wherein the desired position and orientation cause the imaging device to have a third view that is more similar to the first view than the second view. ### Claim 20 (depends on 19) The non-transitory machine-readable medium of claim 19, wherein the method further comprises determining the first view based on one or more parameters selected from the group consisting of: a direction of view of the imaging device; a view up direction of the imaging device, a center of view of the imaging device, a depth of a point of interest or the center of view from the imaging device; a position of a distal portion of each of one or more instruments; a visibility of a point of interest; a position of the point of interest; and an orientation of the point of interest.