System and method of providing real-time dynamic imagery of a medical procedure site using multiple modalities

A system of providing real-time dynamic imagery of a medical procedure site, comprising:

• a control system, wherein the control system is adapted to:
• receive a first real-time dynamic video stream of a scene based on a first modality from a first medical device;
• receive a second real-time dynamic video stream of the scene based on a second modality from a second medical device;
• obtain at least one of first real-time spatial state data associated with the first medical device or second real-time spatial state data associated with the second medical device;
• determine a relative orientation and position of the first medical device to the second medical device based on the at least one of the first real-time spatial state data or the second real-time spatial state data;
• spatially register the first real-time dynamic video stream and the second real-time dynamic video stream based on the determined relative orientation and position of the first medical device to the second medical device, wherein the first real-time dynamic video stream and the second real-time dynamic video stream align to form a composite representation of the scene;
• and
• generate a composite real-time dynamic video stream of the scene from the composite representation.

A computer-readable non-transitory storage medium comprising computer-executable instructions that when executed by one or more processors, cause the one or more processors to:

• receive a first real-time dynamic video stream of a scene based on a first modality from a first medical device;
• receive a second real-time dynamic video stream of the scene based on a second modality from a second medical device;
• obtain at least one of first real-time spatial state data associated with the first medical device or second real-time spatial state data associated with the second medical device;
• determine a relative orientation and position of the first medical device to the second medical device based on the at least one of the first real-time spatial state data or the second real-time spatial state data;
• spatially register the first real-time dynamic video stream and the second real-time dynamic video stream based on the determined relative orientation and position of the first medical device to the second medical device, wherein the first real-time dynamic video stream and the second real-time dynamic video stream align to form a composite representation of the scene;
• and
• generate a composite real-time dynamic video stream of the scene from the composite representation.