# US 2024/0203552 A1 · Video surgical report generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203552 A1
- Document ID: US20240203552A1
- Status: Published application
- Kind: A1
- Application number: 18/539,090
- Filed: 2023-12-13
- Published: 2024-06-20
- Earliest claimed: 2022-12-16
- Assignees: Stryker Corporation
- Applicants: Stryker Corporation
- Inventors: Jose GEORGE; Sanskruti MASKE
- Web page: https://patentgrep.com/patent/US20240203552A1

## CPC

- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G16H 30/40: for processing medical images, e.g. editing
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Disclosed herein are methods for generating a video surgical report using a machine learning pipeline. The machine learning pipeline may include one or more machine learning models, each of which may support a particular aspect of a video surgical report generation process. For example, one or more images of a surgical procedure may be obtained. Using one or more machine learning models, a set of images from the one or more images may be selected based on the surgical procedure. A video surgical report may be generated for the surgical procedure, which may include at least some of the set of images. The machine learning pipeline can offload work typically performed by a user (e.g., surgeon, medical staff, etc.) to create the video surgical report, thereby saving significant time and/or resources.

## Claims

### Claim 1

A computer-implemented method, comprising:
obtaining one or more images of a surgical procedure;
determining, using one or more machine learning models, a set of images from the one or more images based on the surgical procedure; and
generating a video surgical report for the surgical procedure comprising at least some of the set of images.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the video surgical report comprises:
generating, using the one or more machine learning models, text describing the at least some of the set of images, wherein the video surgical report comprises at least some of the text corresponding to the at least some of the set of images.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the video surgical report comprises:
generating a virtual character programmed to output audio associated with the at least some of the set of images, the video surgical report comprising the virtual character.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining the set of images comprises:
determining, using the one or more machine learning models, at least one of: a phase of the surgical procedure, a surgical activity being performed during the phase, or information related to the set of images; and selecting the set of images from the one or more images based on the at least one of the phase, the surgical activity, or the information related to the set of images.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
training the one or more machine learning models to analyze content depicted by the set of images to determine the at least one of the phase, the surgical activity, or the information related to the set of images.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more images include a first image and a second image captured during a same phase of the surgical procedure, determining the set of images comprises:
computing, using the one or more machine learning models, a first classification score and a second classification score respectively associated with the first image and the second image; and
adding at least one of the first image or the second image to the set of images based on the first classification score and the second classification score.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
identifying at least one image from the one or more images based on preoperative information related to the surgical procedure, wherein the set of images comprises the at least one image.

### Claim 8 (depends on 1)

The method of claim 1, wherein obtaining the one or more images comprises:
identifying, using the one or more machine learning models, a subset of frames from a video of the surgical procedure that depict one or more objects associated with the surgical procedure, wherein the one or more images comprise the subset of frames.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
detecting, using the one or more machine learning models, one or more objects associated with the surgical procedure within the one or more images, wherein the at least some of the set of images are selected based on the one or more objects.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving preoperative information related to at least one of the surgical procedure or a patient associated with the surgical procedure; and
generating content to be included in the video surgical report based on the received preoperative information.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
generating, using the one or more machine learning models, text associated with the set of images; and
associating one or more portions of the text with the set of images.

### Claim 12 (depends on 1)

The method of claim 1, wherein generating the video surgical report comprises:
generating, using the one or more machine learning models, text for the at least some of the set of images; and
generating, using the one or more machine learning models, audio based on the text, the video surgical report comprising the audio.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
receiving, using one or more audio sensors, audio captured during the surgical procedure, wherein the video surgical report comprises at least some of the audio corresponding to the at least some of the set of images.

### Claim 14 (depends on 1)

The method of claim 1, wherein generating the video surgical report comprises:
obtaining pre-generated text associated with content depicted by the at least some of the set of images;
obtaining user-provided text of audio captured during the surgical procedure; and
generating text for the video surgical report based on the pre-generated text and the user-provided text.

### Claim 15 (depends on 1)

The method of claim 1, wherein generating the video surgical report comprises:
generating audio based on data stored in an audio profile of a user, the data comprising at least one of a pitch, a timbre, a loudness, or a modulation associated with the user.

### Claim 16 (depends on 1)

The method of claim 1, wherein obtaining the one or more images comprises:
accessing video captured during the surgical procedure; and
extracting at least one video snippet from the video based on the surgical procedure, wherein the one or more images comprise the at least one video snippet.

### Claim 17 (depends on 1)

The method of claim 1, further comprising:
obtaining one or more additional images captured subsequent to the surgical procedure; and
generating an updated video surgical report comprising at least some of the one or more additional images.

### Claim 18 (depends on 1)

The method of claim 1, wherein generating the video surgical report comprises:
adding one or more additional images to the video surgical report based on a similarity between content depicted by the one or more additional images and content depicted by at least one of the set of images, wherein the one or more additional images are captured prior to the surgical procedure.

### Claim 19

A non-transitory computer-readable medium storing computer program instructions that, when executed by one or more processors, effectuate a method comprising:
obtaining one or more images of a surgical procedure;
determining, using one or more machine learning models, a set of images from the one or more images based on the surgical procedure; and
generating a video surgical report for the surgical procedure comprising at least some of the set of images.

### Claim 20

A system comprising:
memory storing computer program instructions; and
one or more processors configured to execute the computer program instructions to cause the one or more processors to perform a method comprising:
obtaining one or more images of a surgical procedure;
determining, using one or more machine learning models, a set of images from the one or more images based on the surgical procedure; and
generating a video surgical report for the surgical procedure comprising at least some of the set of images.
