# US 2018/0322949 A1 · Video content searches in a medical context > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322949 A1 - Document ID: US20180322949A1 - Status: Published application - Kind: A1 - Application number: 15/735,073 - Filed: 2016-06-09 - Published: 2018-11-08 - Earliest claimed: 2015-06-09 - Assignees: Intuitive Surgical Operations, Inc. - Applicants: Intuitive Surgical Operations, Inc. - Inventors: Catherine J. Mohr; Brian E Miller - Web page: https://patentgrep.com/patent/US20180322949A1 ## CPC - G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS - G06F 16/71: Indexing; Data structures therefor; Storage structures - G06T 7/0012: {Biomedical image inspection} - G16H 30/40: for processing medical images, e.g. editing - G16H 40/63: for local operation - G06N 20/00: Machine learning - G06T 2207/10016: Video; Image sequence - G06T 2207/30004: Biomedical image processing - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method of searching video content for specific subject matter of interest augments traditional image data analysis methods with analysis of contemporaneously gathered non-image data. One application involves video recordings of medical procedures performed using a medical device. When a user desires to locate portions of video recordings showing a medical event of interest, one or more medical device system events likely to correspond to the occurrence of the medical event of interest are identified from one or more procedure event logs. Timestamps associated with these system events are used to identify candidate video clips from the procedure video recordings. Image data analysis is performed on the candidate video clips to determine whether each candidate video clip contains the medical event of interest. ## Claims ### Claim 1 A method comprising: receiving a user command to locate one or more video clips showing a medical procedure event of interest, that is a portion of a medical procedure performed on a medical system, from one or more medical procedure video recordings of the medical procedure: identifying, from one or more medical system event logs that record the occurrence of medical system events, during performance of the medical procedure on the medical system, and the time of occurrence of medical system events on the medical system, one or more medical system events occurring on the medical system during the medical procedure likely to correspond to the occurrence of the medical procedure event of interest; identifying one or more candidate video clips from the one or more medical procedure video recordings, using one or more timestamps corresponding to each of the one or more identified medical system events occurring on the medical system; determining, by analyzing the image data of the one or more candidate video clips, whether each of the one or more candidate video segments is an identified video segment that contains the medical procedure event of interest; and presenting to a user at least one identified video segment. ### Claim 2 (depends on 1) The method of claim 1, wherein receiving the user command to locate one or more video clips showing a medical event of interest comprises: presenting to a plurality of video clips of medical procedure events of interest to choose from via a user interface; receiving a user's selection of one of the presented plurality of video clips of medical procedure events of interest. ### Claim 3 (depends on 1) The method of claim 1, wherein receiving a user command to locate one or more video clips showing a medical event of interest comprises receiving the user command intraoperatively while the user is performing a medical procedure. ### Claim 4 (depends on 1) The method of claim 1, wherein receiving a user command to locate one or more video clips showing a medical event of interest comprises receiving the user command preoperatively or postoperatively. ### Claim 5 (depends on 1) The method of claim 1, wherein identifying one or more system events likely to correspond to the occurrence of the medical event of interest comprises searching a look-up table that records the occurrence of events and the time of occurrence of medical system events on the medical system. ### Claim 6 (depends on 5) The method of claim 5, wherein the look-up table is periodically updated automatically to account for associations between medical system events and medical events of interest learned through machine learning. ### Claim 7 (depends on 1) The method of claim 1, wherein the one or more medical system events likely to correspond to the occurrence of the medical event of interest comprises the occurrence of an individual system event. ### Claim 8 (depends on 1) The method of claim 1, wherein the one or more medical system events likely to correspond to the occurrence of the medical event of interest comprises occurrence of two or more system events accordingly to a particular pattern. ### Claim 9 (depends on 1) The method of claim 1, wherein the one or more medical system events likely to correspond to the occurrence of the medical event of interest comprises the occurrence of an individual system event for a prolonged duration. ### Claim 10 (depends on 1) The method of claim 1, wherein identifying one or more candidate video segments using one or more timestamps corresponding to each of the one or more identified medical system events comprises identifying a video segment of a procedure video recording taking place between two timestamps. ### Claim 11 (depends on 1) The method of claim 1, wherein identifying one or more candidate video segments using one or more timestamps corresponding to each of the one or more identified medical system events comprises identifying a video segment of a procedure video recording taking place within a time window of a timestamp. ### Claim 12 (depends on 1) The method of claim 1, wherein presenting to the user at least one identified video segment comprises playing to the user the at least one identified video segment on a display. ### Claim 13 A device for performing a video search, comprising: a user interface; and a processing unit operatively coupled to the user interface, the processing unit including one or more processors, wherein the processing unit is configured to: receiving, via the user interface, a user command to locate one or more video clips showing a medical procedure event of interest, that is a portion of a medical procedure performed on a medical system, from one or more medical procedure video recordings of the medical procedure; identifying, from one or more medical system event logs, that record the occurrence of medical system events. during performance of the medical procedure on the medical system,. and the time of occurrence of medical system events on the medical system, one or more medical system events occurring on the medical system during the medical procedure likely to correspond to the occurrence of the medical procedure event of interest; identifying one or more candidate video clips from the one or more medical procedure video recordings, using one or more timestamps corresponding to each of the one or more identified medical system events occurring on the medical system; determining, by analyzing the image data of the one or more candidate video clips, whether each of the one or more candidate video segments is an identified video segment that contains the medical procedure event of interest; and presenting to a user at least one identified video segment. ### Claim 14 (depends on 13) The device of claim 13, wherein the user interface comprises a user interface of a personal computing device. ### Claim 15 (depends on 13) The device of claim 13, wherein the user interface comprises a user interface of a teleoperated surgical system. ### Claim 16 (depends on 13) The device of claim 13, wherein receiving the user command to locate one or more video clips showing a medical procedure event of interest comprises: presenting to a plurality of video clips of medical procedure events of interest to choose from via a user interface; receiving a user's selection of one of the presented plurality of medical procedure events of interest. ### Claim 17 (depends on 11) The device of claim 11, wherein identifying one or more medical system events likely to correspond to the occurrence of the medical procedure event of interest comprises searching a look-up table that records the occurrence of events and the time of occurrence of medical system events on the medical system. ### Claim 18 (depends on 17) The device of claim 17, wherein the look-up table is periodically updated automatically to account for associations between medical system events and medical procedure events of interest learned through machine learning. ### Claim 19 (depends on 13) The device of claim 13, wherein the one or more medical system events likely to correspond to the occurrence of the medical procedure event of interest comprises the occurrence of an individual medical system event. ### Claim 20 (depends on 13) The device of claim 13, wherein the one or more medical system events likely to correspond to the occurrence of the medical procedure event of interest comprises occurrence of two or more system events accordingly to a particular pattern. ### Claim 21 (depends on 13) The device of claim 13, wherein the one or more medical system events likely to correspond to the occurrence of the medical procedure event of interest comprises the occurrence of an individual medical system event for a prolonged duration. ### Claim 22 (depends on 13) The device of claim 13, wherein identifying one or more candidate video segments using one or more timestamps corresponding to each of the one or more identified medical system events comprises identifying a video segment of a medical procedure video recording taking place between two timestamps. ### Claim 23 (depends on 13) The device of claim 13, wherein identifying one or more candidate video segments using one or more timestamps corresponding to each of the one or more identified medical system system events comprises identifying a video segment of a medical procedure video recording taking place within a time window of a timestamp. ### Claim 24 (depends on 13) The device of claim 13, wherein presenting to the user at least one identified video segment comprises playing to the user the at least one identified video segment on a display.