AI-based ultrasound navigation system for navigating to target positions defined by text or images

An apparatus comprising:

• means for jointly training a machine learning based image encoder and a machine learning based text encoder using one or more training target images depicting a particular target position and one or more training text-based instructions for navigating to the particular target position, wherein image features extracted from the one or more training target images by the machine learning based image encoder correspond to text features extracted from the one or more training text-based instructions by the machine learning based text encoder;
• means for receiving 1) an initial image depicting a current position of a medical image acquisition device and 2) at least one of a target image depicting a specific target position of the medical image acquisition device or text-based instructions for navigating to the specific target position of the medical image acquisition device;
• means for extracting features from the at least one of the target image or the text-based instructions respectively using at least one of the trained machine learning based image encoder or the trained machine learning based text encoder;
• means for determining one or more actions for navigating the medical image acquisition device from the current position towards the specific target position using a machine learning based policy network, wherein the machine learning based policy network receives as input the initial image and the extracted features, and wherein the one or more actions comprise a change in one or more of a position, location, or orientation of the medical image acquisition device;
• and
• means for outputting, by the machine learning based policy network, the one or more actions.

A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out operations comprising:

• jointly training a machine learning based image encoder and a machine learning based text encoder using one or more training target images depicting a particular target position and one or more training text-based instructions for navigating to the particular target position, wherein image features extracted from the one or more training target images by the machine learning based image encoder correspond to text features extracted from the one or more training text-based instructions by the machine learning based text encoder;
• receiving 1) an initial image depicting a current position of a medical image acquisition device and 2) at least one of a target image depicting a specific target position of the medical image acquisition device or text-based instructions for navigating to the specific target position of the medical image acquisition device;
• extracting features from the at least one of the target image or the text-based instructions respectively using at least one of the trained machine learning based image encoder or the trained machine learning based text encoder;
• determining one or more actions for navigating the medical image acquisition device from the current position towards the specific target position using a machine learning based policy network, wherein the machine learning based policy network receives as input the initial image and the extracted features, and wherein the one or more actions comprise a change in one or more of a position, location, or orientation of the medical image acquisition device;
• and
• outputting, by the machine learning based policy network, the one or more actions.