Automatic annotation of endoscopic videos
AssigneeGyrus ACMI, Inc. d/b/a Olympus Surgical Technologies America
InventorDawei Liu
Life of this patent
Estimating the term…
Abstract
A method for automatic annotation of individual frames of procedural videos can include receiving, with processing circuitry of a controller, a video stream captured by an endoscopic camera during an endoscopic procedure. The video stream can include a first timestamp. The method can also include, receiving an audio recording captured during the endoscopic procedure. The audio recording can include a second timestamp. The method can also include, receiving a transcribed text from the audio recording. The transcribed text can also the second timestamp. The method can also include, annotating the video stream with the transcribed text by corresponding the transcribed audio and the video stream when the first timestamp and the second timestamp agree.
Claims20
A method for automatic annotation of individual frames of procedural videos, the method comprising:
- receiving, with processing circuitry of a controller, a video stream captured by an endoscopic camera during an endoscopic procedure, the video stream including a first timestamp;
- receiving an audio recording captured during the endoscopic procedure, the audio recording including a second timestamp;
- receiving a transcribed text from the audio recording, the transcribed text including the second timestamp;
- and
- annotating the video stream with the transcribed text by corresponding the transcribed text and the video stream when the first timestamp and the second timestamp agree.
A system for automatic annotation of individual frames of procedural videos, the system comprising:
- an endoscope comprising:
- an elongated member including a distal portion, the elongated member comprising:
- a camera attached to the distal portion, the camera capturing a video stream during a procedure, the video stream including a first timestamp;
- a microphone configured to capture an audio recording of sounds around the system during the procedure, the audio recording including a second timestamp;
- a natural language processor configured to receive the audio recording and a transcribed audio recording, the transcribed audio recording including the second timestamp;
- a memory including instructions;
- and
- a controller including processing circuitry that, when in operation, is configured by the instructions to:
- receive the video stream from the camera;
- receive the audio recording from the microphone;
- receive a transcribed text from the natural language processor;
- and
- annotate the video stream with the transcribed text by corresponding the transcribed audio and the video stream when the first timestamp and the second timestamp agree.
Dependencies are inferred from the claim text.