# US 10,863,971 B2 · Touchless input ultrasound control > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,863,971 B2 - Document ID: US10863971B2 - Status: Granted - Kind: B2 - Application number: 16/206,806 - Filed: 2018-11-30 - Published: 2020-06-04 - Granted: 2020-12-15 - Earliest claimed: 2018-11-30 - Assignees: Fujifilm Sonosite, Inc. - Applicants: Fujifilm Sonosite, Inc. - Inventors: Davinder S. Dhatt - Web page: https://patentgrep.com/patent/US10863971B2 ## CPC - A61B 8/467: {characterised by special input means} - A61B 8/468: {allowing annotation or message recording} - A61B 8/54: {Control of the diagnostic device} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06V 40/174: {Facial expression recognition} - G06V 40/176: {Dynamic expression} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/084: Backpropagation, e.g. using gradient descent - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method and apparatus are disclosed herein for controlling an ultrasound machine using one or more touchless inputs. In one embodiment, the method for controlling operation of the ultrasound machine comprises obtaining one or more touchless inputs; determining one or more operations to control the ultrasound machine based on the one or more touchless inputs and machine state of the ultrasound machine; and controlling the ultrasound machine using the one or more operations. ## Claims ### Claim 1 A method for controlling operation of an ultrasound machine, the method comprising: obtaining one or more touchless inputs, including lip reading by capturing images of lips, analyzing and matching the images of lips to a specific word or sequence of words; determining one or more operations to control the ultrasound machine based on the one or more touchless inputs, including the specific word or sequence of words from the lip reading, and machine state of the ultrasound machine, wherein determining the one or more operations is further based on captured audio information of the individual and comprises dynamically adjusting, based on an environment in which the ultrasound machine resides, weights associated with results of lip reading recognition and audio recognition to determine the one or more operations to control the ultrasound machine; and controlling the ultrasound machine using at least one of the one or more operations. ### Claim 2 (depends on 1) The method defined in claim 1 further comprising: capturing lip movements of an individual using at least one camera. ### Claim 3 (depends on 1) The method defined in claim 1 wherein determining the one or more operations is further based on ultrasound data. ### Claim 4 (depends on 1) The method defined in claim 1 wherein ultrasound data comprises one or more of ultrasound image data, an examination type being performed with the ultrasound machine, and a list of operations being performed with the ultrasound machine. ### Claim 5 (depends on 1) The method defined in claim 1 wherein determining the one or more operations comprises predicting an operation that the individual is likely to perform based on historical data. ### Claim 6 (depends on 1) The method defined in claim 1 wherein performing lip reading is triggered in response to determining the individual is looking directly at one camera of the one or more cameras or in response to determining the individual has closed at least one eye for a predetermined period of time, winked, nodded, or performed another facial cue or other gesture. ### Claim 7 (depends on 1) The method defined in claim 1 further comprising identifying one or more image characteristics, and wherein determining the one or more operations is further based on a correlation of an operating parameter to the one or more image characteristics and further wherein controlling the ultrasound machine using the at least one of the one or more operations comprises changing the operating parameter to change an image. ### Claim 8 (depends on 1) The method defined in claim 1 further comprising receiving, at a neural network, one or more of feedback of ultrasound data from an ultrasound imaging subsystem, feedback of machine state from an ultrasound control subsystem, at least one of the one or more touchless inputs, and wherein determining the one or more operations to control the ultrasound machine based on the one or more touchless inputs and the machine state of the ultrasound machine is performed by the neural network. ### Claim 9 (depends on 1) The method defined in claim 1 wherein determining one or more operations to control the ultrasound machine comprises generating and displaying one or more selectable commands; capturing information from the individual; and interpreting the captured information as confirming selection of at least one of the one or more selectable commands prior to controlling the ultrasound machine using the at least one operation. ### Claim 10 (depends on 1) The method defined in claim 1 wherein the one or more operations comprise at least one selected from a group consisting of adjusting gain, adjusting depth, freezing an image being displayed by the ultrasound machine, saving an image being displayed by the ultrasound machine, adding an annotation at a user-specified location on an image being displayed by the ultrasound machine, and creating a report with one or more images displayed by the ultrasound machine. ### Claim 11 (depends on 1) The method defined in claim 1 wherein determining one or more operations to control the ultrasound machine comprises placing an annotation on an image being displayed by the ultrasound machine based on at least one of the one or more touchless inputs. ### Claim 12 (depends on 11) The method defined in claim 11 wherein determining one or more operations to control the ultrasound machine further comprises recognizing a start of annotation command, an end of annotation command and one or more annotation movement commands based on at least one touchless input. ### Claim 13 (depends on 11) The method defined in claim 11 wherein the one or more annotation movement commands are recognized from eye tracking information. ### Claim 14 (depends on 1) The method defined in claim 1 further comprising: identifying the individual from a group of individuals located in proximity to the ultrasound machine based on user identification operation; and providing control of the ultrasound machine through use of touchless inputs to the identified individual. ### Claim 15 An apparatus comprising: a display screen; an ultrasound imaging subsystem coupled to the display to generate ultrasound images on the display screen; an ultrasound control subsystem coupled to control the imaging subsystem; one or more cameras to capture images of lip movements of an individual; a microphone to capture audio; a recognizer coupled to the one or more cameras to execute a lip recognition routine on captured images from the one or more cameras to perform lip reading and matching the lip reading to a specific word or sequence of words; and a command generator coupled to the recognizer and the control subsystem to determine one or more operations based on one or more touchless inputs from the recognizer, including the specific word or sequence of words, and microphone and machine state of the ultrasound machine received from the ultrasound control subsystem and send at least one of the one or more operations to control the ultrasound control subsystem, wherein to determine the one or more operations is further based on captured audio information of the individual and comprises dynamically adjusting, based on an environment in which the ultrasound machine resides, weights associated with results of lip reading recognition and audio recognition to determine the one or more operations to control the ultrasound machine. ### Claim 16 (depends on 15) The apparatus defined in claim 15 wherein the command generator comprises a neural network. ### Claim 17 (depends on 15) The apparatus defined in claim 15 wherein the command generator is operable to determine the one or more operations based further on ultrasound data. ### Claim 18 (depends on 17) The apparatus defined in claim 17 wherein the ultrasound data comprises one or more of ultrasound image data, an examination type being performed with the ultrasound machine, and a list of operations being performed by the ultrasound machine. ### Claim 19 (depends on 15) The apparatus defined in claim 15 wherein the command generator is operable to determine the one or more operations by predicting an operation that the individual is likely to perform based on historical data. ### Claim 20 (depends on 15) The apparatus defined in claim 15 wherein the recognizer is operable to trigger touchless input control for use by the individual in response to a determination that the individual is looking at one camera of the one or more cameras or in response to a facial cue, wherein the facial cue comprises one of a group consisting of: closing at least one eye for a predetermined period of time, winking, nodding. ### Claim 21 (depends on 15) The apparatus defined in claim 15 wherein the command generator comprises a neural network that is operable to receive feedback of ultrasound data from the ultrasound imaging subsystem, feedback of machine state from the ultrasound control subsystem, and at least one of the one or more touchless inputs of the individual from the recognizer, and, in response thereto, the command generator is operable to determine the one or more operations to control the ultrasound control subsystem. ### Claim 22 (depends on 15) The apparatus defined in claim 15 wherein the one or more operations comprise at least one selected from a group consisting of adjusting gain, adjusting depth, freezing an image being displayed by the ultrasound machine, saving an image being displayed by the ultrasound imaging subsystem, adding an annotation at a user-specified location on an image being displayed by the ultrasound imaging subsystem, and creating a report with one or more images displayed by the ultrasound imaging subsystem. ### Claim 23 (depends on 15) The apparatus defined in claim 15 wherein the one or more operations comprise placing an annotation on an image being displayed by the ultrasound imaging subsystem based on the one or more touchless inputs, including recognizing a start of annotation command, an end of annotation command and one or more annotation movement commands based on information captured from the individual. ### Claim 24 An article of manufacture having one or more computer readable storage media storing instructions thereon which, when executed by a system, cause the system to perform a method for controlling operation of an ultrasound machine, the method comprising: obtaining one or more touchless inputs, including lip reading by capturing images of lips, analyzing and matching the images of lips to a specific word or sequence of words; determining one or more operations to control the ultrasound machine based on the one or more touchless inputs, including the specific word or sequence of words from the lip reading, and machine state of the ultrasound machine, wherein determining the one or more operations is further based on captured audio information of the individual and comprises dynamically adjusting, based on an environment in which the ultrasound machine resides, weights associated with results of lip reading recognition and audio recognition to determine the one or more operations to control the ultrasound machine; and controlling the ultrasound machine using at least one of the one or more operations.