# US 2026/0294550 A1 · Method and apparatus for processing surgical information based on artificial intelligence > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2026/0294550 A1 - Document ID: US20260294550A1 - Status: Published application - Kind: A1 - Application number: 19/569,588 - Filed: 2026-03-17 - Published: 2026-10-01 - Earliest claimed: 2025-03-28 - Assignees: Korea Institute of Science and Technology - Applicants: Korea Institute of Science and Technology - Inventors: Juyoun PARK; Taemin CHOI; Yong Seok IHN - Web page: https://patentgrep.com/patent/US20260294550A1 ## CPC - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - G06F 40/279: Recognition of textual entities - G06T 7/0012: {Biomedical image inspection} - G06T 7/73: using feature-based methods - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking - G06V 10/764: using classification, e.g. of video objects - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/50: Context or environment of the image - G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients - A61B 2034/2055: {Optical tracking systems} - A61B 2034/2065: {Tracking using image or pattern recognition} - A61B 2090/0805: {automatically, e.g. by means of magnetic, optical or photoelectric detectors} - G06T 2207/30004: Biomedical image processing - G06V 2201/034: of medical instruments - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract The present disclosure relates to an image processing technology, and a method of processing surgical information includes receiving a surgical image and surgical-related text, extracting an image feature vector from the received surgical image and extracting at least two text feature vectors from the received text, generating a composite feature vector from the image feature vector and the text feature vector, and detecting an instrument present in the surgical image using the generated composite feature vector. ## Claims ### Claim 1 A method of processing, by an apparatus for processing surgical information including at least one processor, surgical information, the method comprising: receiving a surgical image and surgical-related text; extracting an image feature vector from the received surgical image and extracting at least two text feature vectors from the received text; generating a composite feature vector from the image feature vector and the text feature vector; and detecting an instrument present in the surgical image using the generated composite feature vector. ### Claim 2 (depends on 1) The method of claim 1, wherein the generating of the composite feature vector includes: generating a first composite feature vector from the image feature vector and a first text feature vector related to an instrument to generate a first mask for the instrument determined to be present in the surgical image; and generating a second composite feature vector from the image feature vector and a second text feature vector related to a location of the instrument determined to be present in the surgical image to generate a second mask for the instrument determined to be present in the surgical image. ### Claim 3 (depends on 2) The method of claim 2, wherein the generating of the first mask includes: generating a first composite feature vector by performing multi-modal fusion of a first text feature vector extracted based on at least one text input related to a name or attribute of an instrument and the image feature vector; and generating a segmentation mask corresponding to an instrument whose probability of being present in the surgical image is greater than or equal to a first reference value, among instruments input via text by referencing the generated first composite feature vector. ### Claim 4 (depends on 3) The method of claim 3, wherein the generating of the first mask includes: independently projecting, by an encoder, the image feature vector representing visual features and the first text feature vector representing linguistic features based on a prompt input indicating the name or attribute of the instrument into a feature space and generating the first composite feature vector using multi-head cross attention (MHCA); and calculating, by a decoder, a presence probability for each class of all objects designated by the text input by referencing the first composite feature vector and generating the first mask representing an initial segmentation region for objects with a presence probability greater than or equal to the first reference value. ### Claim 5 (depends on 2) The method of claim 2, wherein the generating of the second mask includes: generating a second composite feature vector by performing multi-modal fusion of the second text feature vector extracted based on text input regarding the location of the instrument derived using the first mask and the image feature vector, and generating a segmentation mask corresponding to an instrument whose probability of being present in the surgical image is greater than or equal to a second reference value, among instruments input via text by referencing the generated second composite feature vector. ### Claim 6 (depends on 5) The method of claim 5, wherein the generating of the second mask includes: determining, by an encoder, the location of the instrument within the image for the object detected from the first mask, independently projecting the image feature vector representing visual features and the second text feature vector representing linguistic features based on the prompt input indicating the location of the instrument into a feature space, and generating the second composite feature vector using MHCA; calculating, by a decoder, a presence probability for each class of objects located based on the text input by referencing the second composite feature vector and generating the second mask representing a refined segmentation region for objects with a presence probability greater than or equal to the second reference value. ### Claim 7 (depends on 1) The method of claim 1, wherein the detecting of the instrument present in the surgical image includes: calculating a single mask by fusing a plurality of masks generated for an instrument determined to be present in the surgical image using the generated composite feature vector, and outputting a segmentation map for the instrument present in the surgical image based on the calculated single mask. ### Claim 8 (depends on 1) The method of claim 1, further comprising: acquiring time-series feature information by extracting feature vectors from individual surgical images constituting a surgical video; and classifying a current surgical procedure from the acquired time-series feature information. ### Claim 9 (depends on 8) The method of claim 8, wherein the surgical procedure follows a hierarchical structure of a stage, a phase, and a step according to a classification granularity and is classified using a single hierarchy or a combination of two or more, and the classifying of the current surgical procedure includes using at least one hierarchical classifier based on a fusion model of spatial visual information and temporal flow information within the image to output a surgical procedure with a highest probability value from the time-series feature information as a classification result. ### Claim 10 (depends on 9) The method of claim 9, further comprising receiving the acquired time-series feature information and the classified current surgical procedure and predicting a next surgical procedure using at least one of locations, distances, and relationships among instruments detected in a current surgical image. ### Claim 11 (depends on 10) The method of claim 10, wherein the predicting of the next surgical procedure includes referencing the fusion model to determine a surgical procedure having a highest probability of appearing temporally consecutively to the current surgical procedure as the next surgical procedure and outputting at least one of the determined next surgical procedure or a remaining time until the next surgical procedure. ### Claim 12 (depends on 8) The method of claim 8, further comprising: deriving feature information for each of a plurality of surgical procedures constituting a surgery and storing the feature information together with temporal information in advance as a standard operating procedure for a surgical procedure; and comparing the time-series feature information acquired from the surgical video and the classified current surgical procedure with the pre-stored standard operating procedure to evaluate at least one of precision, skill, completion, efficiency, and safety of a surgery. ### Claim 13 A non-transitory computer-readable medium storing one or more instructions for processing surgical information, wherein the one or more instructions executable by one or more processors comprise: receiving a surgical image and surgical-related text; extracting an image feature vector from the received surgical image and extracting at least two text feature vectors from the received text; generating a composite feature vector from the image feature vector and the text feature vector; and detecting an instrument present in the surgical image using the generated composite feature vector. ### Claim 14 An apparatus for processing task information, comprising: a memory configured to store a program for processing task information; and a processor configured to execute the program stored in the memory, wherein the program includes instructions to: receive a task image and task-related text; extract an image feature vector from the received task image and extract at least two text feature vectors from the received text; generate a composite feature vector from the image feature vector and the text feature vector, and detect an instrument present in the task image using the generated composite feature vector. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the program performs instructions to generate the composite feature vector by: generating a first composite feature vector from the image feature vector and a first text feature vector related to an instrument to generate a first mask for an instrument determined to be present in the task image; and generating a second composite feature vector from the image feature vector and the second text feature vector relating to a location of the instrument determined to be present in the task image to generate a second mask for the instrument determined to be present in the task image. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the program performs instructions to generate the first mask by: generating the first composite feature vector by performing multi-modal fusion of a first text feature vector extracted based on at least one text input related to a name or attribute of an instrument and the image feature vector; and generating a segmentation mask corresponding to an instrument whose probability of presence in the task image is greater than or equal to a first reference value among instruments input via text by referencing the generated first composite feature vector. ### Claim 17 (depends on 15) The apparatus of claim 15, wherein the program performs instructions to generate the second mask by: generating a second composite feature vector by performing multi-modal fusion of the second text feature vector extracted based on text input regarding the location of the instrument derived using the first mask and the image feature vector, and generating a segmentation mask corresponding to an instrument whose probability of presence in the task image is greater than or equal to a second reference value among instruments input via text by referencing the generated second composite feature vector. ### Claim 18 (depends on 14) The apparatus of claim 14, wherein the program performs instructions to detect the instrument present in the task image by: calculating a single mask by fusing a plurality of masks generated for an instrument determined to be present in the task image using the generated composite feature vector, and outputting a segmentation map for the instrument present in the task based on the calculated single mask. ### Claim 19 (depends on 14) The apparatus of claim 14, wherein the program further includes instructions to: acquire time-series feature information by extracting feature vectors from individual task images constituting a task video; and classify a current task process from the acquired time-series feature information. ### Claim 20 (depends on 19) The apparatus of claim 19, wherein the program further includes instructions to receive the acquired time-series feature information and the classified current task process and predict a next task procedure using at least one of locations, distances, and relationships among instruments detected in a current task image.