# US 2024/0189041 A1 · Virtual fiducial markings for automated planning in medical imaging

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0189041 A1
- Document ID: US20240189041A1
- Status: Published application
- Kind: A1
- Application number: 18/286,156
- Filed: 2022-03-30
- Published: 2024-06-13
- Earliest claimed: 2021-04-13
- Assignees: Koninklijke Philips N.V.
- Applicants: Koninklijke Philips N.V.
- Inventors: Manikanda Krishnan V; Srinivasa Rao Kundeti; Sumit Sharma
- Web page: https://patentgrep.com/patent/US20240189041A1

## CPC

- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 5/11: Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb {(A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721)}
- A61B 5/1128: {using image analysis (A61B5/1127 takes precedence)}
- A61B 90/39: {Markers, e.g. radio-opaque or breast lesions markers}
- A61B 5/113: occurring during breathing
- A61B 2017/00694: {with means correcting for movement of or for synchronisation with the body}
- A61B 2034/2046: {Tracking techniques}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2090/3966: {Radiopaque markers visible in an X-ray image}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

Disclosed herein is a medical system (100, 300, 500, 600) comprising: a medical imaging system (102, 302) configured to acquire medical imaging data (136) descriptive of a subject (110); a camera system (114) configured to acquire a subject image (138) of the subject; a memory (126) storing machine executable instructions (130), medical imaging system commands (134), and a coordinate system mapping, and an image marking neural network (132). Execution of the machine executable instruction by a computational system (120) causes the computational system to: acquire (200) the medical imaging data by controlling the medical imaging system with the medical imaging system commands; repeatedly (202) control the camera system to acquire the subject image during acquisition of the medical imaging data; repeatedly (204) receive camera system coordinates (142) of the virtual fiducial markers by inputting the subject image into the image marking neural network; and repeatedly (206) provide imaging system coordinates of the set of virtual fiducial markers by repeatedly converting the camera system coordinates of the virtual fiducial markers to the provided imaging system coordinates of the virtual fiducial markers using the coordinate system mapping.

## Claims

### Claim 1

A medical system comprising;
a medical imaging system configured to acquire medical imaging data descriptive of a subject from an imaging zone, wherein the medical imaging system has an image system coordinate system;
a camera system configured to acquire a subject image of the subject during acquisition of the medical imaging data, wherein the camera system has a camera coordinate system;
a memory configured to store machine executable instructions, medical imaging system commands, a coordinate system mapping, and an image marking neural network; wherein the coordinate system mapping is a mapping between the system coordinate system and the camera coordinate system; wherein the image marking neural network is configured to receive an input image descriptive of a predetermined anatomical region of the subject, wherein the image marking neural network is further configured to output camera system coordinates of a set of virtual fiducial markers in the input image in response to receiving the input image, wherein the medical imaging system commands are configured to control the medical imaging system to acquire the medical imaging data;
a computational system configured to control the medical system, wherein execution of the machine executable instruction causes the computational system to:
acquire the medical imaging data by controlling the medical imaging system with the medical imaging system commands;
repeatedly control the camera system to acquire the subject image during acquisition of the medical imaging data;
repeatedly receive the camera system coordinates of the virtual fiducial markers by inputting the subject image into the image marking neural network; and
repeatedly provide imaging system coordinates of the set of virtual fiducial markers by repeatedly converting the camera system coordinates of the virtual fiducial markers to the provided imaging system coordinates of the virtual fiducial markers using the coordinate system mapping.

### Claim 2 (depends on 1)

The medical system of claim 1, wherein execution of the machine executable instructions further causes the computational system to:
receive an initial medical image prior to beginning acquisition of the medical imaging data;
receive a chosen field of view identified in the initial medical image;
calculate a registration between the imaging system coordinates of the virtual fiducial markers and the initial medical image; and
configure the pulse sequence commands to acquire the medical imaging data from the chosen field of view using the registration.

### Claim 3 (depends on 2)

The medical system of claim 2, wherein the initial medical image is any one of the following: an anatomical atlas image, a scout scan of the subject, and a prior medical image of the subject.

### Claim 4 (depends on 2)

The medical system of claim 2, wherein execution of the machine executable instructions further causes the computational system to repeatedly adjust the medical imaging system commands to acquire the medical imaging data from the chosen field of view using the registration in response to a change in the imaging system coordinates of the virtual fiducial markers.

### Claim 5 (depends on 4)

The medical system of claim 4, further including at least one of the following:
the medical imaging system commands are adjusted such that the chosen field of view matches the most recent imaging system coordinates of the virtual fiducial markers; and
the pulse sequence commands are adjusted such that the chosen field of view matches predicted coordinates of the imaging system coordinates of the virtual fiducial markers determined using a velocity of imaging system coordinates of the virtual fiducial markers.

### Claim 6 (depends on 4)

The medical system of claim 4, wherein the medical system further comprises a subject mounted gyroscope configured for providing gyroscope data descriptive of subject motion, wherein execution of the machine executable instructions further causes the computational system to:
repeatedly receive the gyroscope data from the subject mounted gyroscope;
repeatedly determine a subject acceleration from the gyroscope data;
repeatedly calculate a predicted virtual fiducial marker velocity using the subject acceleration; and
repeatedly calculate a predicted virtual fiducial marker location using most recent imaging system coordinates of the virtual fiducial markers and the predicted virtual fiducial marker velocity.

### Claim 7 (depends on 1)

The medical system of claim 1, wherein the medical system further comprises a display, wherein execution of the machine executable instructions further causes the computational system to:
receive the camera system coordinates of the set of virtual fiducial markers at a beginning of the acquisition of the medical imaging data;
calculate a position of an initial subject location indicator using the camera system coordinates of the set of virtual fiducial markers at a beginning of the acquisition;
render the initial subject location indicator on the display persistently;
repeatedly calculate a position of a current location indicator using the camera system coordinates of the set of virtual fiducial markers; and
repeatedly render the current subject location indicator on the display.

### Claim 8 (depends on 7)

The medical system of claim 7, wherein any one of the following:
the initial subject location indicator is a rendering of the set of virtual fiducial markers positioned using the output camera system coordinates at the beginning of the acquisition superimposed on the subject image, wherein the current subject location indicator is a rendering of the set of virtual fiducial markers positioned using the camera system coordinates superimposed on the subject image; and
the initial subject location indicator is a first object positioned using a combination of the set of virtual fiducial markers in the camera system coordinates at the beginning of the acquisition, wherein the current subject location indicator is a second object positioned using a combination of the set of virtual fiducial markers in the camera system coordinates.

### Claim 9 (depends on 1)

The medical system of claim 1, wherein the medical imaging data is acquired in portions, wherein execution of the machine executable instructions further causes the computational system to correct each of the portions of the medical imaging data using the camera system coordinates of the virtual fiducial markers at the time each of the portions of the medical imaging data was acquired.

### Claim 10 (depends on 9)

The medical system of claim 9, wherein the correction to the portions of the medical imaging data are corrected using any one of the following methods: by performing a rigid body rotation and/or translation, and using a medical image data correcting neural network configured to output corrected medical imaging data in response to receiving one of the portions of medical imaging data and the output coordinates of the virtual fiducial markers at the time each of the portions of the medical imaging data was acquired.

### Claim 11 (depends on 1)

The medical system of claim 1, wherein execution of the machine executable instructions further causes the computational system to reconstruct a clinical medical image from the medical imaging data.

### Claim 12 (depends on 11)

The medical system of claim 11, wherein the medical system further comprises a radiotherapy system configured for irradiating a treatment zone, wherein the treatment zone is within the imaging zone, wherein execution of the machine executable instructions further causes the computational system to:
receive radiotherapy control commands configured to control the radiotherapy system to iridate the treatment zone;
register the imaging system coordinates of the virtual fiducial markers to the clinical medical image;
receive a location of the treatment zone in the clinical medical image; and
modify the radiotherapy control commands using the location of the treatment zone in the clinical medical image and the registration of the imaging system coordinates of the virtual fiducial markers to the clinical medical image.

### Claim 13 (depends on 1)

The medical system of claim 1, wherein the medical imaging data is any one of the following: a magnetic resonance imaging system, a computed tomography system, a positron emission tomography system, a single photon emission tomography system, a combined magnetic resonance imaging system and positron emission tomography system, and a combined computed tomography system and positron emission tomography system.

### Claim 14

A method of operating a medical system, wherein the medical imaging system is configured to acquire medical imaging data descriptive of a subject from an imaging zone, wherein the medical imaging system has an image system coordinate system, wherein the medical system further comprises a camera system configured to acquire a subject image of the subject during acquisition of the medical imaging data, wherein the camera system has a camera coordinate system, wherein the method comprises:
acquiring the medical imaging data by controlling the medical imaging system with medical imaging system commands;
repeatedly controlling the camera system to acquire the subject image during acquisition of the medical imaging data;
repeatedly receiving camera system coordinates of the virtual fiducial markers by inputting the subject image into the image marking neural network, wherein the image marking neural network is configured to receive an input image descriptive of a predetermined anatomical region of the subject, wherein the image marking neural network is further configured to output camera system coordinates of a set of virtual fiducial markers in the input image in response to receiving the input image; and
repeatedly providing imaging system coordinates of the virtual fiducial markers by repeatedly converting the camera system coordinates of the set of virtual fiducial markers to the provided imaging system coordinates of the virtual fiducial markers using a coordinate system mapping, wherein the coordinate system mapping is a mapping between the imaging system coordinate system and the camera coordinate system.

### Claim 15

A computer program comprising machine executable instructions for execution by a computational system controlling a medical system, wherein the medical system comprises a medical imaging system is configured to acquire medical imaging data descriptive of a subject from an imaging zone, wherein the medical imaging system has an image system coordinate system, wherein the medical system further comprises a camera system configured to acquire a subject image of a subject during acquisition of the medical imaging data, wherein the camera system has a camera coordinate system, wherein execution of the machine executable instruction causes the computational system to:
acquire the medical imaging data by controlling the medical imaging system with medical imaging system commands;
repeatedly control the camera system to acquire the subject image during acquisition of the medical imaging data;
repeatedly receive camera system coordinates of the virtual fiducial markers by inputting the subject image into the image marking neural network, wherein the image marking neural network is configured to receive an input image descriptive of a predetermined anatomical region of a subject, wherein the image marking neural network is further configured to output camera system coordinates of a set of virtual fiducial markers in the input image in response to receiving the input image; and
repeatedly provide imaging system coordinates of the virtual fiducial markers by repeatedly converting the camera system coordinates of the set of virtual fiducial markers to the provided imaging system coordinates of the virtual fiducial markers using a coordinate system mapping (140), wherein the coordinate system mapping is a mapping between the imaging system coordinate system and the camera coordinate system.
