Back to results
US 2026/0066090 A1Published application
Information processing system and methods for clinical video retrieval
OwnerPraveen Rao, Eduardo J. Simoes +3inventors
InventorsPraveen Rao, Eduardo J. Simoes, Mihail Popescu, Mirna Becevic, Zhandi Liu
Life of this patent
Estimating the term…
Abstract
The present disclosure generally relates to an integrated approach for retrieving biomedical information from clinical video presentations. In particular, the present disclosure is directed to video retrieval systems and methods of text-video retrieval from clinical video presentations.
Claims17
A computer-implemented clinical video retrieval system for searching and retrieving clinical videos and clinical text comprising:
- a data repository comprising clinical video recordings;
- an automatic speech recognition module configured to generate timestamped text transcriptions from the audio content of the clinical video recordings;
- an indexing module configured to (i) receive the timestamped text transcriptions, (ii) generate a record comprising a unique identifier, filename, timestamp, transcribed text, and a dense embedding vector of the transcribed text, and (iii) store the record in an index comprising an inverted index and a vector index;
- a query processing module configured to receive a text query from a user and generate a dense embedding vector of the text query using a transformer-based embedding model, wherein the query processing module optionally comprises a large language model configured to generate a set of dynamically rephrased query variants of the text query;
- a retrieval module configured to perform a keyword-based search using a ranking function on the inverted index and perform a semantic search using k-nearest neighbor (kNN) retrieval on the vector index, wherein results from the keyword-based search and the semantic search are combined using a weighted scoring function, and wherein candidate transcript segments are retrieved for each query variant;
- a cross-encoder reranking module configured to apply a cross-encoder reranker to the candidate transcript segments, user query and query variants to compute contextualized similarity scores and sort the candidate transcript segments based on the contextualized similarity scores to produce a ranked list of results;
- a video clip generating module configured to extract, for each ranked transcript segment, a video clip from the corresponding clinical video recording, the video clip beginning at the timestamp associated with the transcript segment and having a predefined duration;
- and
- a display module configured to present the ranked list of video clips and associated clinical text segments.
6 more elements
A computer-aided method for searching and retrieving clinical video clips and clinical text, the method comprising:
- (a) receiving a user query comprising a text input;
- (b) generating a dense embedding of the user query using at least one transformer-based embedding model;
- (c) retrieving, from a repository of clinical video recordings, a plurality of transcript segments, each transcript segment comprising a timestamped transcription of a portion of a clinical video, wherein the retrieval comprises:
- (i) performing a keyword-based search using a lexical ranking function to identify transcript segments relevant to the user query;
- (ii) performing a semantic search by computing similarity between the dense embedding of the user query and precomputed dense embeddings of the transcript segments to identify semantically relevant transcript segments;
- (iii) combining results from the keyword-based search and the semantic search using a weighted scoring function to generate a set of candidate transcript segments;
- (d) reranking the set of candidate transcript segments using a cross-encoder model to compute contextualized similarity scores between the user query and each candidate transcript segment, and sorting the candidate transcript segments based on the contextualized similarity scores;
- (e) extracting, for each reranked transcript segment, a corresponding clinical video clip from the clinical video recordings, wherein the video clip is generated based on the timestamp associated with the transcript segment;
- (f) outputting to a user interface the corresponding clinical video clips as video results, the reranked transcript segments as clinical text results, and combinations thereof.
6 more elements
A computer-aided process for indexing clinical video recordings, the process comprising:
- receiving a plurality of clinical video recordings, the clinical video recordings comprising audio, video, and presentation materials;
- transcribing audio of each clinical video recording of the plurality of clinical video recordings using automatic speech recognition (ASR) software to generate timestamped text transcripts corresponding to segments of the video recordings;
- generating dense vector embeddings for each segment of the transcribed text using a transformer-based deep learning model, wherein each dense vector embedding represents semantic content of the corresponding transcript segment;
- constructing an index in a search engine system, the index comprising an inverted index for keyword-based search, a vector index, and metadata for each transcript segment including a unique identifier, a filename, a timestamp, transcribed text, and the corresponding dense embedding vector;
- storing the index;
- and
- enabling retrieval of relevant video clips.
4 more elements
Dependencies are inferred from the claim text.