Multithreading video processing with cloud services to detect and analyze landmarks and objects during a clinical procedure

A system, comprising:

• a computing infrastructure for processing video streams simultaneously captured from a plurality of medical procedure rooms, the computing infrastructure having a plurality of local processing systems, each configured to support one of the plurality of medical procedure rooms, wherein each local processing system is networked to a shared cloud service via a set of REP sockets that receive requests in a round-robin fashion and wherein each REP socket only allows alternating requests and replies, and wherein each local processing system includes:
• a memory;
• and
• a processor coupled to the memory and configured to process image data streamed from a scope during a clinical procedure according to process that includes:
• receiving and forwarding image data to a display as the scope is inserted into a patient;
• forwarding the image data to a first selected REP socket in the shared cloud service that is configured to identify a landmark in the image data using a first machine learning model, wherein the landmark comprises a specific anatomical feature that serves as a reference point during the clinical procedure;
• receiving landmark details from the first selected REP socket in the shared cloud service in response to a detected landmark and outputting the image data with landmark annotations via the display;
• receiving and forwarding further image data to the display as the scope is further manipulated within the patient;
• forwarding the further image data to a second selected REP socket in the cloud service that is configured to detect lesions using a second machine learning model;
• and
• receiving lesion details from the second selected cloud service in response to a detected lesion and outputting the further image data with lesion annotations via the display.

A method, comprising:

• simultaneously processing video streams captured from a plurality of medical procedure rooms with a plurality of local processing systems, wherein each local processing system is networked to a shared cloud service via a set of REP sockets that receive requests in a round-robin fashion and wherein each REP socket only allows alternating requests and replies, and wherein each local processing system includes:
• receiving image data streamed from a scope at a first local processing system during a clinical procedure;
• forwarding the image data to a display as the scope is inserted into a patient;
• forwarding the image data to a first selected REP socket at the shared cloud service that is configured to identify a landmark in the image data using a first machine learning model, wherein the landmark comprises a specific anatomical feature that serves as a reference point during the clinical procedure;
• receiving landmark details from the first selected REP socket at the shared cloud service in response to a detected landmark and outputting the image data with landmark annotations via the display;
• receiving and forwarding further image data to the display as the scope is further manipulated within the patient;
• forwarding the further image data to a second selected REP socket at the shared cloud service that is configured to detect lesions using a second machine learning model;
• and
• receiving lesion details from the second selected shared cloud service in response to a detected lesion and outputting the further image data with lesion annotations via the display.