# US 11,026,585 B2 · System and method for intraoperative video processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,026,585 B2
- Document ID: US11026585B2
- Status: Granted
- Kind: B2
- Application number: 16/432,404
- Filed: 2019-06-05
- Published: 2019-12-05
- Granted: 2021-06-08
- Earliest claimed: 2018-06-05
- Assignees: Synaptive Medical Inc.
- Applicants: Bradley Allan Fernald; Gal Sela; Neil Jeffrey Witcomb
- Inventors: Bradley Allan Fernald; Gal Sela; Neil Jeffrey Witcomb
- Web page: https://patentgrep.com/patent/US11026585B2

## CPC

- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/004: {adapted for image acquisition of a particular organ or body part (A61B5/0082 takes precedence; arrangements for optical scanning A61B5/0062)}
- A61B 5/02042: {Determining blood loss or bleeding, e.g. during a surgical procedure}
- A61B 5/1032: {Determining colour of tissue for diagnostic purposes}
- A61B 5/7282: {Event detection, e.g. detecting unique waveforms indicative of a medical condition (cough events A61B5/0823; seizures A61B5/4094; sleep apnoea A61B5/4818)}
- A61B 5/7425: {Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G06F 18/211: Selection of the most significant subset of features
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 7/90: Determination of colour characteristics
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/56: relating to colour
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04N 5/445: for displaying additional information (H04N5/50 takes precedence)
- A61B 34/25: {User interfaces for surgical systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2505/05: Surgical care
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30008: Bone
- G06T 2207/30096: Tumor; Lesion
- G06T 2207/30101: Blood vessel; Artery; Vein; Vascular
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H 30/40: for processing medical images, e.g. editing
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A monitoring method and system for providing visual enhancements during a medical procedure is described. The method includes capturing current visual information of a site during the medical procedure in real time, storing at least a portion of the captured visual information as stored visual information, identifying a feature of interest in at least one of the current visual information and the stored visual information; generating feedback data associated with the feature of interest, and displaying a virtual representation of the feedback data overlaid on the current visual information.

## Claims

### Claim 1

A monitoring system for providing visual enhancements during a medical procedure, the system comprising:
a sensor for capturing visual information of a site during the medical procedure in real time, the sensor being a camera and the visual information being a video of a field-of-view (FOV) of the site during the medical procedure, the visual information comprising current visual information as a current frame of the video taken at a first point in time during the medical procedure;
a memory for storing at least a portion of the visual information as stored visual information, the stored visual information having been captured at a second point in time during the medical procedure prior to the first point in time, the stored visual information being one or more past frames of the video;
a display for displaying the current visual information; and
a processor coupled to receive the current and stored visual information from the sensor and the memory, and coupled to transmit output data for display on the display, the processor being configured to:
identify and compare a feature of interest in the current visual information at the first point in time and the stored visual information at the second point in time;
generate feedback data associated with the feature of interest using the current visual information and the stored visual information; and
cause the display to display a virtual representation of the feedback data overlaid on the current visual information.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processor is further configured to analyze the current or past frames according to at least one of colour, texture, physical shape, and location within the FOV for identifying the feature of interest.

### Claim 3 (depends on 2)

The system of claim 2, wherein the processor is configured to segment an area in the at least one of the current and past frames according to the video analysis, and identify the segmented area as indicative of the feature of interest.

### Claim 4 (depends on 2)

The system of claim 2, wherein the processor is configured to analyze the current frame to detect a colour indicative of a pool of blood, and the stored visual information is analyzed to identify the feature of interest related to the pool of blood.

### Claim 5 (depends on 4)

The system of claim 4, wherein the feature of interest is a bleed point, and the feedback data indicates a location of the bleed point.

### Claim 6 (depends on 3)

The system of claim 3, wherein the processor is configured to analyze the current or past frames to identify the segmented area as representing an anatomical structure.

### Claim 7 (depends on 6)

The system of claim 6, wherein the anatomical structure is at least one of a blood vessel, a nerve, a ligament, a bone structure and a tumour.

### Claim 8 (depends on 7)

The system of claim 7, wherein the virtual representation displayed includes a virtual representation of the segmented anatomical structure.

### Claim 9 (depends on 1)

A system for reacting to an event during a medical procedure, the system comprising:
an implementation system configured to perform the medical procedure and to perform predetermined steps in response to the event; and
the monitoring system of claim 1 coupled to the implementation system, wherein the feedback data includes indication that the event has occurred; and
the monitoring system configured to:
analyze the current visual information to detect occurrence of the event;
signal the implementation system to pause the medical procedure and initiate the predetermined steps in response to the event; and
when the event is no longer detected, signal the implementation system to resume the medical procedure.

### Claim 10

A monitoring method for use during a medical procedure, the method comprising:
capturing visual information of a site during the medical procedure in real-time, the visual information being a video of a field-of-view (FOV) of the site during the medical procedure, the visual information comprising current visual information as a current frame of the video taken at a first point in time during the medical procedure;
storing at least a portion of the visual information as stored visual information, the stored visual information having been captured at a second point in time during the medical procedure prior to the first point in time, the stored visual information being one or more past frames of the video;
identifying and comparing a feature of interest in the current visual information at the first point in time and the stored visual information at the second point in time;
generating feedback data associated with the feature of interest using the current visual information and the stored visual information; and
displaying a virtual representation of the feedback data overlaid on the current visual information.

### Claim 11 (depends on 10)

The method of claim 10, further comprising analyzing the current or past frames according to at least one of colour, texture, physical shape, and location within the FOV for identifying the feature of interest.

### Claim 12 (depends on 11)

The method of claim 11, further comprising segmenting an area in the at least one of the current and past frames according to the video analysis; and wherein the identifying comprises identifying the segmented area as indicative of the feature of interest.

### Claim 13 (depends on 11)

The method of claim 11, wherein the analyzing comprises detecting, in the current frame, a colour indicative of a pool of blood, and the stored visual information is analyzed to identify the feature of interest related to the pool of blood.

### Claim 14 (depends on 13)

The method of claim 13, wherein the feature of interest is a bleed point, and the feedback data generated indicates a location of the bleed point.

### Claim 15 (depends on 12)

The method of claim 12, wherein the segmented area is identified as representing an anatomical structure, the anatomical structure being the feature of interest.

### Claim 16 (depends on 15)

The method of claim 15, wherein the displaying includes displaying a virtual representation of the anatomical structure.

### Claim 17 (depends on 10)

A method for reacting to an event during a medical procedure, the method comprising:
performing the method of claim 10, wherein the feedback data includes an indication that the event has occurred;
the feedback data causing a pause to the medical procedure when the event is detected;
initiating a response procedure; and
resuming the medical procedure when the event is no longer detected.

### Claim 18 (depends on 17)

The method of claim 17, wherein the event is a bleed and the response procedure is a bleed response procedure, the bleed response procedure comprises:
changing the FOV and/or magnification of the video to focus on an identified bleed point, as the feature of interest, during the bleed response procedure.