# US 2023/0410291 A1 · Method and system for segmenting and identifying at least one tubular structure in medical images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0410291 A1 - Document ID: US20230410291A1 - Status: Published application - Kind: A1 - Application number: 18/031,101 - Filed: 2021-10-19 - Published: 2023-12-21 - Earliest claimed: 2020-10-22 - Assignees: Strasbourg - Applicants: Visible Patient - Inventors: Adrien HEITZ; Julien WEINZORN; Luc SOLER - Web page: https://patentgrep.com/patent/US20230410291A1 ## CPC - G06T 7/0012: {Biomedical image inspection} - G06T 7/11: Region-based segmentation - G06T 7/13: Edge detection - G06T 7/174: involving the use of two or more images - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/10088: Magnetic resonance imaging [MRI] - G06T 2207/10108: Single photon emission computed tomography [SPECT] - G06T 2207/10136: 3D ultrasound image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30056: Liver; Hepatic - G06T 2207/30061: Lung - G06T 2207/30101: Blood vessel; Artery; Vein; Vascular - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract This invention concerns a computer implemented method for segmenting and identifying at least one tubular structure, having a 3D tree layout and located in at least one body part of a subject, in medical images showing a volume region of interest of said subject containing said body part(s), and for providing a labelled 3D image of said structure(s), said method mainly comprising the steps of: providing a set of 2D medical images corresponding to respective mutually distinct sectional views across said region of interest containing said body part(s), the planes of said medical images being all perpendicular to a given direction or all mutually intersecting at a given straight line, segmenting the visible section(s) of the concerned body part(s) present in each one of said 2D medical images, which comprises in particular the complete linear outline or external boundary of said body part(s) visible in the considered 2D image, and creating a corresponding 2D body part masking image, pre-processing each 2D medical image, by applying the corresponding body part masking image to it and so producing processed images containing only the image data of the original 2D image which are related to said body part(s), segmenting the tubular structure(s) in said resulting pre-processed images, possibly by segmenting tubular structures of different kinds in differentiated segmentation processes, performing the previous steps with at least one other set of 2D medical images corresponding to other respective distinct sectional views, along other mutually parallel or intersecting planes, of said same volume region of interest containing said same body part(s), merging the results of the tubular structure segmentations of the different sets of pre-processed images, in order to provide a labelled 3D image of said tubular structure(s) of one or different kind(s). ## Claims ### Claim 1 A computer implemented method for segmenting and identifying at least one tubular structure, having a 3D tree layout and located in at least one body part of a subject, in medical images showing a volume region of interest of said subject containing said body part(s), and for providing a labelled 3D image of said structure(s), said method mainly comprising the steps of: providing a set of 2D medical images corresponding to respective mutually distinct sectional views across said region of interest containing said body part(s), the planes of said medical images being all perpendicular to a given direction or all mutually intersecting at a given straight line, segmenting the visible section(s) of the concerned body part(s) present in each one of said 2D medical images, which comprises in particular the complete linear outline or external boundary of said body part(s) visible in the considered 2D image, and creating a corresponding 2D body part masking image, pre-processing each 2D medical image, by applying the corresponding body part masking image to it and so producing processed images containing only the image data of the original 2D image which are related to said body part(s), segmenting the tubular structure(s) in said resulting pre-processed images, possibly by segmenting tubular structures of different kinds in differentiated segmentation processes, performing the previous steps with at least one other set of 2D medical images corresponding to other respective distinct sectional views, along other mutually parallel or intersecting planes, of said same volume region of interest containing said same body part(s), and merging the results of the tubular structure segmentations of the different sets of pre-processed images, in order to provide a labelled 3D image of said tubular structure(s) of one or different kind(s). ### Claim 2 (depends on 1) The method according to claim 1, wherein the step of segmenting the tubular structure(s) of a considered pre-processed image comprises taking also into account the image data of at least one other pre-processed sectional view, of the same set of images, which is adjacent said considered pre-processed image, for example of at least the nearest neighboring sectional view(s) located on each side of said considered pre-processed image. ### Claim 3 (depends on 1) The method according to claim 1, wherein the body part segmenting steps are performed by using a 2D neural network and the tubular structure segmenting steps are performed by using a 2,5D neural network, said neural networks being preferably U-Net neural networks. ### Claim 4 (depends on 1) The method according to claim 1, wherein a dedicated neural network, previously trained on data labelled by expert(s), is used for the body part segmenting and pre-processing steps of each set of 2D medical images, a dedicated and specifically trained neural network being also used for segmenting the tubular structure(s) in each set of pre-processed images, possibly one specific neural network for each kind of tubular structure. ### Claim 5 (depends on 4) The method according to claim 4, wherein, in a preliminary preparatory phase, the final training parameters values of the neural network intended to handle a first set of medical images are used as starting parameter values for the training of at least one other neural network intended to handle another set of medical images, in the pre-processing phase and/or in the tubular structure segmenting phase. ### Claim 6 (depends on 1) The method according to claim 1, wherein a first set of medical images correspond to axial views of the subject and wherein second and third sets of medical images correspond respectively to sagittal and coronal views. ### Claim 7 (depends on 1) The method according to claim 1, wherein the steps of segmenting the concerned body part(s) in each one of said 2D medical images and creating a corresponding body part masking image consist in determining the outline(s) and/or internal area(s) of said body parts(s) and in locating and identifying the connection site(s) of the internal tubular structure(s) with at least one other body part or body component, said latter being or not part of the masking image. ### Claim 8 (depends on 1) The method according to claim 1, wherein, before the steps of segmenting the concerned body part(s) in each one of the 2D medical images of a given set and creating a corresponding body part masking image, said images are undergoing operations of an initial treatment workflow comprising at least retrimming and resizing operations, and possibly also a normalizing operation. ### Claim 9 (depends on 1) The method according to claim 1, wherein, during the pre-processing phase, the modified pre-processed image resulting from applying the body part masking image to an original 2D medical image, is submitted, before applying the step of segmenting the tubular structure(s), to isotropic resampling and rescaling operations. ### Claim 10 (depends on 1) The method according to claim 1, wherein any medical image for which the corresponding body part masking image, in anyone of the sets of medical images, is empty and any pre-processed image which does not show any part of the considered body part(s) are disregarded in view of further processing. ### Claim 11 (depends on 1) The method according to claim 1, wherein the medical images are images of the thoracic cavity of a human subject, wherein the concerned body parts are the lungs, wherein the segmenting step also includes identifying the left lung and the right lung and wherein at least certain body part masking images also contain the representation of the trachea and the connection area between pulmonary arteries and veins and the heart. ### Claim 12 (depends on 11) The method according to claim 11, wherein two different types of tubular structures are segmented in parallel, namely the bronchus tree and the pulmonary vascular tree, the arteries and veins being labelled within the vascular tree in a further step. ### Claim 13 (depends on 11) The method according to claim 11, wherein three different types of tubular structures are segmented in parallel, namely the bronchus tree, the pulmonary artery tree and the pulmonary vein tree. ### Claim 14 (depends on 1) The method according to claim 1, wherein the medical images are abdominal images, wherein the concerned body part is the liver and wherein the vascular system to be segmented and labelled comprises the portal vein, the hepatic vein and the hepatic artery. ### Claim 15 (depends on 1) The method according to claim 1, wherein the final merging step is performed by means of a fusion operation consisting of a type of operation chosen among fusion by union, fusion by majority voting, fusion by logits averaging, fusion by neural network and fusion by simultaneous truth and performance level estimation. ### Claim 16 An image treatment system able to fully automatically perform organ and inside tubular structure segmentation and labeling, in particular lungs, bronchus and pulmonary arteries and veins segmentation, said system relying on cascaded convolutional neural networks, said image treatment system comprising: a first component which is dedicated to organ segmentation, for example right and left lung segmentation, and is based on a slightly modified 2D U-Net architecture and a second component based on a three-paths 2.5D fully convolutional networks along axial, coronal and sagittal slices, fed with pre-processed 2D images of the first component and configured to perform tubular structures and elements extraction and labelling. ### Claim 17 (depends on 1, 16) The image treatment system according to claim 16, characterized in that it is configured and laid out to perform the method according to claim 1.