# US 11,439,473 B2 · Surgical control apparatus, surgical control method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,439,473 B2 - Document ID: US11439473B2 - Status: Granted - Kind: B2 - Application number: 15/764,916 - Filed: 2016-09-27 - Published: 2018-09-13 - Granted: 2022-09-13 - Earliest claimed: 2015-10-09 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Takeshi Maeda; Seiji Wada; Tatsumi Sakaguchi; Kana Matsuura - Web page: https://patentgrep.com/patent/US11439473B2 ## CPC - A61B 34/70: {Manipulators specially adapted for use in surgery} - A61B 34/25: {User interfaces for surgical systems} - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - A61B 90/50: Supports for surgical instruments, e.g. articulated arms - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - A61B 2017/00203: {with speech control or speech recognition} - A61B 2017/00216: {with eye tracking or head position tracking control} - A61B 2034/2055: {Optical tracking systems} - A61B 2034/254: {being adapted depending on the stage of the surgical procedure} - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - A61B 2090/367: {creating a 3D dataset from 2D images using position information} - A61B 2090/372: {Details of monitor hardware} - A61B 2090/3937: {Visible markers} - A61B 2090/502: {Headgear, e.g. helmet, spectacles} - G05B 2219/35444: Gesture interface, controlled machine observes operator, executes commands - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract The present invention relates to a surgical control apparatus, a surgical control method, and a program that can prevent erroneous operations of surgical apparatuses when the surgical apparatuses are controlled with contactless inputs. A state estimation block (64) estimates a state of an operator recognized by a recognition block (61) on the basis of at least one type of contactless input sent from the operator. In accordance with the state estimated by the state estimation block (64), a command block (62) restricts a control operation of a surgical camera (11), a camera arm (12), or an image processing block (66) based on at least one type of contactless input sent from the operator recognized by the recognition block (61). The present invention is applicable to a surgical system and the like that have a surgical camera (11), a camera arm (12), an action recognition camera (13), a display (14), a control apparatus (15), a pair of glasses (17), a microphone (18), a marker, a foot switch (20), etc., thereby enabling the treatment realized by referencing images. ## Claims ### Claim 1 A surgical control apparatus comprising: circuitry configured to recognize a first type of contactless input from a user as first input information; recognize a second type of contactless input from the user as second input information, the second type of contactless input being different from the first type of contactless input; estimate a state of the user from a plurality of predetermined user states, on the basis of the first input information and the second input information; set at least one command type from a plurality of command types as a permitted command type and set other commands from the plurality of command types as restricted command type, in accordance with the state of the user estimated, the at least one permitted command type including one or more commands for controlling a control operation of a surgical apparatus, each command having an associated recognition condition; and output one of the one or more commands on the basis of whether the first input information and the second input information satisfy a recognition condition of one of the one of the one or more commands. ### Claim 2 (depends on 1) The surgical control apparatus according to claim 1, wherein the first and second types of contactless input include a voice input, a line of sight input, a movement input, or a gesture input of the user. ### Claim 3 (depends on 1) The surgical control apparatus according to claim 1, wherein the control operation is executed on the basis of the first and second types of contactless input from the user recognized by the second contactless input recognition circuit and an input by contact from the user recognized by a contact input recognition circuit. ### Claim 4 (depends on 1) The surgical control apparatus according to claim 1, wherein the circuitry is configured to estimate the state of the user as an action-other-than-surgical-procedure state, a downward viewing state, a close watching state, or an observation state. ### Claim 5 (depends on 4) The surgical control apparatus according to claim 4, wherein the control operation of the surgical apparatus includes a menu display control operation of a display control apparatus, an annotation display control operation of a display control apparatus, an imaging control operation of a surgical imaging apparatus for taking a surgical field image, and an arm driving control operation for holding the surgical imaging apparatus. ### Claim 6 (depends on 5) The surgical control apparatus according to claim 5, wherein if the state of the user is estimated to be an action-other-than-surgical-procedure state, the circuitry is configured to restrict set the command type for controlling the menu display control operation of the display control apparatus as the permitted command type. ### Claim 7 (depends on 5) The surgical control apparatus according to claim 5, wherein if the state of the user is estimated to be a downward viewing state, the circuitry is configured to set the command type for controlling operation of the annotation display control operation of the display control apparatus as the permitted command type. ### Claim 8 (depends on 5) The surgical control apparatus according to claim 5, wherein if the state of the user is estimated to be a close watching state, the circuitry is configured to set the command type for controlling operation of the imaging control operation of the surgical imaging apparatus as the permitted command type. ### Claim 9 (depends on 5) The surgical control apparatus according to claim 5, wherein if the state of the user is estimated to be an observation state, the circuitry is configured to restrict the command type for controlling operation of the driving control operation of an arm for holding the surgical imaging apparatus as the permitted command type. ### Claim 10 (depends on 1) The surgical control apparatus according to claim 1, wherein the circuitry is configured to estimate the state of the user if the operation mode is a mode for controlling the surgical apparatus on the basis of at least one type of contactless input from the user. ### Claim 11 A surgical control method comprising: recognizing a first type of contactless input from a user as first input information; recognizing a second type of contactless input from the user as second input information, the second type of contactless input being different from the first type of contactless input; estimating a state of the user, from a plurality of predetermined user states, on the basis of the first input information and the second input information; setting, in accordance with the state of the user estimated, at least one command type from a plurality of command types as a permitted command type and set other commands from the plurality of command types as restricted command type, the at least one command type including one or more commands for controlling a control operation of a surgical apparatus, each command having an associated recognition condition; and outputting one of the one or more commands on the basis of whether the first input information and the second input information satisfy a recognition condition of one of the one of the one or more commands. ### Claim 12 A non-transitory computer readable medium having stored thereon a program that, when executed by a computer, causes the computer to execute processing, the processing comprising: recognizing a first type of contactless input from a user as first input information; recognizing a second type of contactless input from the user as second input information, the second type of contactless input being different from the first type of contactless input; estimating a state of the user, from a plurality of predetermined user states, on the basis of the first input information and the second input information; setting, in accordance with the state of the user estimated, at least one command type from a plurality of command types as a permitted command type and set other commands from the plurality of command types as restricted command type, the at least one command type including one or more commands for controlling a control operations of a surgical apparatus, each command having an associated recognition condition; and outputting one of the one or more commands on the basis of whether the first input information and the second input information satisfy a recognition condition of one of the one of the one or more commands. ### Claim 13 (depends on 1) The surgical control apparatus according to claim 1, wherein, the circuitry is configured to set the command type to at least one of a menu display control, an annotation display control, an imaging control, and a camera arm control as permitted command types. ### Claim 14 (depends on 13) The surgical control apparatus according to claim 13, wherein, when the state of the user indicates the user is not looking at a display, the circuitry is configured to set the command type to the menu display control as the permitted command type. ### Claim 15 (depends on 13) The surgical control apparatus according to claim 13, wherein, when the state of the user indicates the user is looking at a display, the circuitry is configured to set the menu display control and the annotation display control as permitted command types. ### Claim 16 (depends on 1) The surgical control apparatus according to claim 1, wherein, when the command type is a camera arm control for holding a surgical imaging apparatus, the circuitry is configured to set the command types for controlling operation based on at least one type of contactless input from the user recognized by a third contactless input recognition. ### Claim 17 (depends on 16) The surgical control apparatus according to claim 16, wherein the circuitry is configured to set the command type for controlling a camera arm to change imaging angles without changing an imaging center when line-of-sight information of the user is unchanged and the user travels as the permitted command type. ### Claim 18 (depends on 16) The surgical control apparatus according to claim 16, wherein the circuitry is configured to set the command type for controlling a camera arm to change an imaging center when line-of-sight information of the user and the user rotates as the permitted command type. ### Claim 19 (depends on 3) The surgical control apparatus according to claim 3, wherein, the circuitry is configured to set the command type for controlling the operation to at least one of a menu display control, an annotation display control, and a camera arm control as the permitted command type based on the input by contact from the user recognized by a contact input recognition circuit. ### Claim 20 (depends on 1) The surgical control apparatus according to claim 1, wherein, when the second contactless input recognition circuit recognizes a gesture by the user that is not a registered gesture or recognizes a volume of the user is greater than a predetermined value, the circuitry is configured to determine an emergency state.