# US 2025/0143807 A1 · Provision of surgical guidance based on audiovisual data and instrument data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0143807 A1
- Document ID: US20250143807A1
- Status: Published application
- Kind: A1
- Application number: 18/833,570
- Filed: 2023-01-28
- Published: 2025-05-08
- Earliest claimed: 2022-01-28
- Assignees: Covidien LP; Digital Surgery Limited
- Applicants: Covidien LP; Digital Surgery Limited
- Inventors: Robert M. Nostrant; Susan L. Roweton; Petros Giataganas; Danail V. Stoyanov; Carole RJ Addis; Tom K. Matthews; Anthony L. Ceniccola; Christopher K. Evans; Kasey A. Grim; Drew Robert Seils; Christopher Switalski; Benjamin J. Noe; Patrick Dale Mozdzierz; Michael S. Gallie; Scott J. Prior; Heather I. Tuccolo; Sheldon K. Hall; James G. Pappas; Scott M. Kilcoyne
- Web page: https://patentgrep.com/patent/US20250143807A1

## CPC

- A61B 34/25: {User interfaces for surgical systems}
- A61B 1/00045: {Display arrangement}
- A61B 1/04: combined with photographic or television appliances
- A61B 17/00234: {for minimally invasive surgery (A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence)}
- A61B 17/068: Surgical staplers {, e.g. containing multiple staples or clamps}({staplers containing only one staple A61B17/10; magazines or containers for staples A61B17/105;} for performing anastomosis A61B17/115; {staplers in general B25C5/00})
- G06N 20/00: Machine learning
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/60: for the operation of medical equipment or devices
- G16H 40/63: for local operation
- A61B 2017/00411: {actuated by application of energy from an energy source outside the body}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2090/0803: {Counting the number of times an instrument is used}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A system is provided that includes a memory device and one or more processors coupled with the memory device. The one or more processors are configured to determine, autonomously, one or more phases in a surgical procedure based on a video stream of the surgical procedure. The one or more processors are further configured to identify one or more usages of a surgical instrument used during the surgical procedure. The one or more processors are configured to display a chart of the one or more usages. The chart divides the one or more usages according to the one or more phases respectively, and a representation of each of the one or more usages indicates a duration of each usage.

## Claims

### Claim 1

A system comprising:
a memory device; and
one or more processors coupled with the memory device, the one or more processors configured to:
determine, autonomously, one or more phases in a surgical procedure based on a video stream of the surgical procedure;
identify one or more usages of a surgical instrument used during the surgical procedure; and
display a chart of the one or more usages, wherein the chart divides the one or more usages according to the one or more phases respectively, and a representation of each of the one or more usages indicates a duration of each usage.

### Claim 2 (depends on 1)

The system of claim 1, wherein the video stream of the surgical procedure is analyzed by a first device to determine and output the one or more phases in the surgical procedure and wherein the one or more usages of the surgical instrument are identified by a second device based on electrical energy applied to the surgical instrument.

### Claim 3 (depends on 1, 2)

The system of claim 1 or claim 2, wherein the one or more usages are identified based on an amount of electrical energy provided to the surgical instrument.

### Claim 4

The system of any preceding claim, wherein the video stream of the surgical procedure captured by an endoscopic camera from inside a body of a subject of the surgical procedure.

### Claim 5

The system of any preceding claim, wherein a visual attribute of the representation of each of the one or more usages is based on a type of the one or more usages.

### Claim 6

The system of any preceding claim, wherein a type of the one or more usages is selected from a group consisting of energy activation, reloading, firing, incision, clamping, dividing, and stapling.

### Claim 7

The system of any preceding claim, wherein the chart is user-interactive, and wherein an interaction with a first representation corresponding to a first usage displays a video segment of the surgical procedure comprising the first usage being performed.

### Claim 8

The system of any preceding claim, wherein the one or more processors are further configured to playback the video stream of the surgical procedure, and wherein a user-interface element displays a timeline depicting one or more timepoints in the video stream at which the one or more usages are performed.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one or more timepoints are rendered based on a type of the one or more usages respectively.

### Claim 10 (depends on 8)

The system of claim 8, wherein audio data corresponding to the one or more usages is generated artificially during the playback of the video stream.

### Claim 11

The system of any preceding claim, wherein the one or more processors are further configured to:
display a list of the one or more phases in the surgical procedure, wherein an entry corresponding to a first phase from the one or more phases includes a user-interface element comprising a timeline depicting the one or more usages performed for the first phase.

### Claim 12

The system of any preceding claim, wherein the representation of each of the one or more usages indicates a user that performed the usage.

### Claim 13

The system of any preceding claim, wherein the representation depicts a comparison of usages performed by a first user and a second user.

### Claim 14

The system of any preceding claim, wherein the representation of each of the one or more usages indicates an anatomical attribute of the subject of the surgical procedure, the anatomical attribute comprising a body mas index, a tissue thickness, and a gender.

### Claim 15

A method comprising:
determining, autonomously, one or more phases in a surgical procedure based on a video stream of the surgical procedure;
identifying one or more usages of a surgical instrument used during the surgical procedure based on energy supplied to the surgical instrument; and
displaying a chart of the one or more usages and a user-interaction with a representation of each of the one or more usages causes a corresponding portion of the video stream to be played back.

### Claim 16 (depends on 15)

The method of claim 15, wherein the chart groups the one or more usages according to the one or more phases respectively.

### Claim 17

A computer program product comprising a memory device with computer-readable instructions stored thereon, wherein executing the computer-readable instructions by one or more processing units causes the one or more processing units to perform a method comprising:
determining, autonomously, a stapling being performed in a surgical procedure based on a video stream of the surgical procedure;
identifying one or more usages of a surgical stapler used during the surgical procedure based on energy supplied to the surgical stapler; and
displaying a chart of the one or more usages of the surgical stapler, wherein a user-interaction with a representation of each of the one or more usages causes a corresponding portion of the video stream with the use of the surgical stapler to be played back.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the chart displays an amount of energy used during each of the one or more usages of the surgical stapler.
