# US 11,304,762 B2 · Mapping image display control device, method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,304,762 B2 - Document ID: US11304762B2 - Status: Granted - Kind: B2 - Application number: 15/901,056 - Filed: 2018-02-21 - Published: 2018-08-30 - Granted: 2022-04-19 - Earliest claimed: 2017-02-24 - Assignees: FUJIFILM Corporation - Applicants: FUJIFILM Corporation - Inventors: Kenta Yamada - Web page: https://patentgrep.com/patent/US11304762B2 ## CPC - A61B 34/25: {User interfaces for surgical systems} - A61B 5/004: {adapted for image acquisition of a particular organ or body part (A61B5/0082 takes precedence; arrangements for optical scanning A61B5/0062)} - A61B 5/08: Measuring devices for evaluating the respiratory organs - A61B 5/4887: {Locating particular structures in or on the body} - A61B 6/466: {adapted to display 3D data} - A61B 6/481: {involving the use of contrast agents} - A61B 6/50: specially adapted for specific body parts; specially adapted for specific clinical applications - A61B 6/5217: {extracting a diagnostic or physiological parameter from medical diagnostic data} - A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - G06T 7/0012: {Biomedical image inspection} - G06T 7/11: Region-based segmentation - G06T 7/187: involving region growing; involving region merging; involving connected component labelling - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - A61B 5/05: Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves (measuring movement of the entire body or parts thereof A61B5/11; detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof A61B5/24) - A61B 6/032: {Transmission computed tomography [CT]} - A61B 6/4405: {the apparatus being movable or portable, e.g. handheld or mounted on a trolley} - A61B 2017/00809: {Lung operations} - A61B 2034/104: {Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring} - A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones} - A61B 2034/2057: {Details of tracking cameras} - A61B 2034/2065: {Tracking using image or pattern recognition} - G06T 2200/04: involving 3D image data - G06T 2207/10024: Color image - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/10088: Magnetic resonance imaging [MRI] - G06T 2207/30061: Lung - G06T 2207/30096: Tumor; Lesion - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An arrival position information estimation unit that estimates arrival position information at which an extension line of a peripheral branch included in a bronchial region arrives at a surface of a lung region, and estimates assumed mapping information other than the arrival position information of a specific peripheral branch on the basis of a preset condition in a case where the specific peripheral branch is set as a target branch, and a display control unit that generates a mapping image obtained by mapping the arrival position information of the specific peripheral branch and the assumed mapping information to a surface of the lung region and displays the mapping image on a display unit are included. ## Claims ### Claim 1 A mapping image display control device comprising: a processor configured to: extract a lung region included in a three-dimensional image; extract a bronchial region included in the lung region; acquire information on a branch position of the bronchial region; estimate arrival position information at which an extension line of a specific peripheral branch included in the bronchial region as a target branch arrives at a surface of the lung region on a basis of the information on the branch position; set a condition of an erroneous insertion of a catheter inserting into a bronchus erroneously; estimate, on the basis of the condition of the erroneous insertion, assumed mapping information at which an extension line of another peripheral branch that is erroneously inserted with the catheter arrives at the surface of the lung region, the another peripheral branch being different from the specific peripheral branch and the another peripheral branch extending from a branch position that is present between a proximal end portion of the bronchial region and the specific peripheral branch, and the arrival position information and the assumed mapping information representing a point or a range at which a dye arrives when the dye is sprayed from within the bronchial region toward the specific peripheral branch; and generate a mapping image obtained by mapping the arrival position information of the specific peripheral branch and the assumed mapping information to the surface of the lung region and display the mapping image on a display. ### Claim 2 (depends on 1) The mapping image display control device according to claim 1, wherein the another peripheral branch extends from a first branch position or a second branch position from a terminal of the specific peripheral branch. ### Claim 3 (depends on 1) The mapping image display control device according to claim 1, wherein the processor is further configured to estimate a straight line set on the basis of the information on the branch position and the specific peripheral branch as the extension line of the branch. ### Claim 4 (depends on 3) The mapping image display control device according to claim 3, wherein the processor is further configured to estimate a straight line set on the basis of a terminal of the specific peripheral branch and information on a first branch position from the terminal as the extension line of the specific peripheral branch. ### Claim 5 (depends on 1) The mapping image display control device according to claim 1, wherein the processor is further configured to perform spline interpolation using a point on the specific peripheral branch and a point specified on the basis of the information on the first branch position from the terminal of the specific peripheral branch, and estimate a curve obtained by the spline interpolation as the extension line of the specific peripheral branch. ### Claim 6 (depends on 1) The mapping image display control device according to claim 1, wherein the processor is further configured to specify the specific peripheral branch on the basis of the information on the branch position, specify a dominant region of the specific peripheral branch in the lung region, and estimate a straight line connecting a center of gravity of the dominant region to a terminal of the specific peripheral branch as the extension line of the specific peripheral branch. ### Claim 7 (depends on 1) The mapping image display control device according to claim 1, wherein the processor is further configured to specify the specific peripheral branch on the basis of the information on the branch position, specify a dominant region of the specific peripheral branch in the lung region, and estimate a center of gravity of a region that is a surface of the lung region in a surface of the dominant region as the arrival position information. ### Claim 8 (depends on 1) The mapping image display control device according to claim 1, wherein the processor is further configured to extract a blood vessel region included in the lung region, and estimate the extension line of the specific peripheral branch on the basis of information on the blood vessel region and the branch position. ### Claim 9 (depends on 8) The mapping image display control device according to claim 8, wherein the processor is further configured to extract at least one of a pulmonary vein region or a pulmonary artery region as the blood vessel region. ### Claim 10 (depends on 1) The mapping image display control device according to claim 1, wherein the processor is further configured to set a region on the basis of the arrival position information of the specific peripheral branch and the assumed mapping information, generate a mapping image obtained by mapping the region on the surface of the lung region, and display the mapping image on the display. ### Claim 11 A mapping image display control method comprising: extracting a lung region included in a three-dimensional image; extracting a bronchial region included in the lung region; acquiring information on a branch position of the bronchial region; estimating arrival position information at which an extension line of a specific peripheral branch included in the bronchial region as a target branch arrives at a surface of the lung region on a basis of the information on the branch position; setting a condition of an erroneous insertion of a catheter inserting into a bronchus erroneously; estimating, on the basis of the condition of the erroneous insertion, assumed mapping information at which an extension line of another peripheral branch that is erroneously inserted with the catheter arrives at the surface of the lung region, the another peripheral branch being different from the specific peripheral branch and the arrival position information and the assumed mapping information representing a point or a range at which a dye arrives when the dye is sprayed from within the bronchial region toward the specific peripheral branch; and generating a mapping image obtained by mapping the arrival position information of the specific peripheral branch and the assumed mapping information to the surface of the lung region and displaying the mapping image on a display. ### Claim 12 (depends on 11) A non-transitory computer-readable storage medium storing therein a mapping image display control program that causes a computer to implement the method of claim 11. ### Claim 13 A mapping image display control device, comprising: a processor configured to: extract a lung region included in a three-dimensional image; extract a bronchial region included in the lung region; acquire information on a branch position of the bronchial region; estimate arrival position information at which an extension line of a specific peripheral branch included in the bronchial region as a target branch arrives at a surface of the lung region on a basis of the information on the branch position; set a condition of a bronchus deformation of the bronchus in a case where a bronchoscope is inserted into the bronchus; estimate assumed mapping information at which an extension line of a deformed peripheral branch in which the specific peripheral branch is deformed according to the condition of the bronchus deformation arrives at the surface of the lung region, the arrival position information and the assumed mapping information representing a point or a range at which a dye arrives when the dye is sprayed from within the bronchial region toward the specific peripheral branch; and generate a mapping image obtained by mapping the arrival position information of the specific peripheral branch and the assumed mapping information to the surface of the lung region and display the mapping image on a display, wherein the bronchus deformation is determined based on deformation conditions of branch positions between a terminal of the specific peripheral branch and the proximal end portion of the bronchus, the deformation conditions being set for the respective branch position of the bronchial region in advance of the bronchoscope being inserted into a patient. ### Claim 14 A mapping image display control method comprising: extracting a lung region included in a three-dimensional image; extracting a bronchial region included in the lung region; acquiring information on a branch position of the bronchial region; estimating arrival position information at which an extension line of a specific peripheral branch included in the bronchial region as a target branch arrives at a surface of the lung region on a basis of the information on the branch position; setting a condition of a bronchus deformation of the bronchus in a case where a bronchoscope is inserted into the bronchus; estimating assumed mapping information at which an extension line of a deformed peripheral branch in which the specific peripheral branch is deformed according to the condition of the bronchus deformation arrives at the surface of the lung region, the arrival position information and the assumed mapping information representing a point or a range at which a dye arrives when the dye is sprayed from within the bronchial region toward the specific peripheral branch; and generating a mapping image obtained by mapping the arrival position information of the specific peripheral branch and the assumed mapping information to the surface of the lung region and display the mapping image on a display, wherein the bronchus deformation is determined based on deformation conditions of branch positions between a terminal of the specific peripheral branch and the proximal end portion of the bronchus, the deformation conditions being set for the respective branch position of the bronchial region in advance of the bronchoscope being inserted into a patient. ### Claim 15 (depends on 14) A non-transitory computer-readable storage medium storing therein a mapping image display control program that causes a computer to implement the method of claim 14.