# US 2018/0020992 A1 · Systems and methods for medical visualization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0020992 A1 - Document ID: US20180020992A1 - Status: Published application - Kind: A1 - Application number: 15/549,851 - Filed: 2016-02-15 - Published: 2018-01-25 - Earliest claimed: 2015-02-16 - Assignees: Dimensions and Shapes, LLC - Applicants: Dimensions and Shapes, LLC - Inventors: Yanhui GUO - Web page: https://patentgrep.com/patent/US20180020992A1 ## CPC - A61B 5/7425: {Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image} - A61B 5/0015: {characterised by features of the telemetry system} - A61B 5/061: {Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body (A61B1/00158 takes precedence)} - A61B 6/032: {Transmission computed tomography [CT]} - A61B 6/12: Arrangements for detecting or locating foreign bodies - A61B 6/466: {adapted to display 3D data} - A61B 8/00: Diagnosis using ultrasonic, sonic or infrasonic waves - A61B 8/0841: {for locating instruments} - A61B 8/466: {adapted to display 3D data} - A61B 8/483: {involving the acquisition of a 3D volume of data} - A61B 90/00: Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges - G02B 27/017: {Head mounted} - G02B 27/0172: {characterised by optical features} - G06T 7/0012: {Biomedical image inspection} - G06T 7/11: Region-based segmentation - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G16H 50/50: for simulation or modelling of medical disorders - G16Z 99/00: Subject matter not provided for in other main groups of this subclass - A61B 5/064: {using markers (A61B5/062 takes precedence)} - A61B 5/066: {Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging} - A61B 5/7445: {Display arrangements, e.g. multiple display units} - A61B 8/523: {for generating planar views from image data in a user selectable plane not corresponding to the acquisition plane} - A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches - G02B 27/01: Head-up displays - G02B 2027/0134: {of stereoscopic type} - G02B 2027/014: {comprising information/image processing systems} - G06T 2207/10081: Computed x-ray tomography [CT] - G06T 2207/10088: Magnetic resonance imaging [MRI] - G06T 2207/30096: Tumor; Lesion - G06T 2210/41: Medical - G06T 2219/008: Cut plane or projection plane definition - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - G16Z: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS, NOT OTHERWISE PROVIDED FOR ## Abstract A visualization system includes an imaging device, a processing circuit, and a display device. The imaging device is configured to acquire image data relating to an object. The processing circuit is configured to generate a three-dimensional object image of the object based on the image data and generate a two-dimensional object image of the object by projecting a cross-section of the three-dimensional object image onto a plane. The display device is configured to display the three-dimensional object image of the object and display the two-dimensional object image of the object on the plane. ## Claims ### Claim 1 A visualization system, comprising: an imaging device configured to acquire image data relating to an object; a processing circuit configured to: generate a three-dimensional object image of the object based on the image data; and generate a two-dimensional object image of the object by projecting a cross-section of the three-dimensional object image onto a plane; and a display device configured to: display the three-dimensional object image of the object; and display the two-dimensional object image of the object on the plane. ### Claim 2 (depends on 1) The visualization system of claim 1, wherein the two-dimensional object image projected onto the plane includes at least one of a first two-dimensional object image of the object projected onto a first plane, a second two-dimensional object image of the object projected onto a second plane, and a third two-dimensional object image of the object projected onto a third plane. ### Claim 3 (depends on 2) The visualization system of claim 2, wherein: the imaging device is configured to acquire second image data relating to a tool; the processing circuit is configured to: generate a three-dimensional tool image of the tool based on the second image data; and generate a two-dimensional tool image of the tool by projecting a cross-section of the three-dimensional tool image onto the plane; and the display device is configured to: display the three-dimensional tool image of the tool; and display the two-dimensional tool image of the tool on the plane. ### Claim 4 (depends on 3) The visualization system of claim 3, wherein the two-dimensional tool image projected onto the plane includes at least one of a first two-dimensional tool image of the tool projected onto the first plane, a second two-dimensional tool image of the tool projected onto the second plane, and a third two-dimensional tool image of the tool projected onto the third plane. ### Claim 5 (depends on 4) The visualization system of claim 4, wherein the display device is configured to display at least one of the three-dimensional object image, the first two-dimensional object image, the second two-dimensional object image, and the third two-dimensional object image in a different color than at least one of the three-dimensional tool image, the first two-dimensional tool image, the second two-dimensional tool image, and the third two-dimensional tool image. ### Claim 6 (depends on 3) The visualization system of claim 3, wherein the processing circuit is configured to: determine whether a portion of the tool is located inside a surface; and provide a command to the display device to change a color of the two-dimensional tool image to a different color when it is determined that the portion of the tool is located inside the surface. ### Claim 7 (depends on 3) The visualization system of claim 3, wherein the processing circuit is configured to mark a lower end position of the three-dimensional tool image of the tool with a line for display by the display device. ### Claim 8 (depends on 1) The visualization system of claim 1, wherein: the imaging device is configured to acquire second image data relating to a tool which identifies a presence of the tool and its location; the processing circuit is configured to: acquire a three-dimensional model of the tool; and generate a first two-dimensional tool image of the tool and a second two-dimensional tool image of the tool by projecting a first cross-section of the three-dimensional model onto a first plane and a second cross-section of the three-dimensional model onto a second plane; and the display device is configured to: display the three-dimensional model of the tool between the first plane and the second plane; and display the first two-dimensional tool image and the second two-dimensional tool image on the first plane and the second plane, respectively. ### Claim 9 (depends on 1) The visualization system of claim 1, wherein the processing circuit is configured to: detect a region of interest based on a preoperative analysis; and construct a three-dimensional representation of the region of interest including the three-dimensional object image and the two-dimensional object image; wherein the display device is configured to display the three-dimensional representation of the region of interest. ### Claim 10 (depends on 1) The visualization system of claim 1, wherein the visualization system is an augmented reality based visualization system. ### Claim 11 (depends on 1) The visualization system of claim 1, further comprising an input device configured to facilitate at least one of rotation, translation, and magnification of the three-dimensional object image. ### Claim 12 A method for visualizing objects using an augmented reality based visualization system comprising: acquiring, by an imaging device, image data relating to an object; generating, by a processing circuit, a three-dimensional object image of the object based on the image data; projecting, by the processing circuit, a two-dimensional object image of the object onto a plane; and displaying, by a display device, at least one of the three-dimensional object image and the two-dimensional object image. ### Claim 13 (depends on 12) The method of claim 12, wherein projecting the two-dimensional object image onto the plane includes: projecting, by the processing circuit, a first two-dimensional object image of the object onto a first plane; and projecting, by the processing circuit, a second two-dimensional object image of the object onto a second plane; wherein the three-dimensional object image is displayed between the first plane and the second plane. ### Claim 14 (depends on 13) The method of claim 13, further comprising: acquiring, by the imaging device, second image data relating to a tool; generating, by the processing circuit, a three-dimensional tool image of the tool based on the second image data; projecting, by the processing circuit, a first two-dimensional tool image onto the first plane and a second two-dimensional tool image onto the second plane; displaying, by the display device, the three-dimensional tool image between the first plane and the second plane; and displaying, by the display device, the first two-dimensional tool image and the second two-dimensional tool image on the first plane and the second plane, respectively. ### Claim 15 (depends on 14) The method of claim 14, wherein at least one of the first two-dimensional object image and the second two-dimensional object image are displayed in a different color than at least one of the first two-dimensional tool image and the second two-dimensional tool image. ### Claim 16 (depends on 14) The method of claim 14, wherein the step of displaying the first two-dimensional tool image and the second two-dimensional tool image includes: determining, by the processing circuit, whether a portion of the tool is located inside a surface; and changing, by the processing circuit, a color of at least one of the first two-dimensional tool image and the second two-dimensional tool image to a different color when it is determined that the portion of the tool is located inside the surface. ### Claim 17 (depends on 14) The method of claim 14, wherein the step of displaying the three-dimensional tool image includes marking, by the processing circuit, a lower end position of the three-dimensional tool image with an indicator. ### Claim 18 (depends on 12) The method of claim 12, further comprising steps of: acquiring, by the imaging device, second image data relating to a tool, wherein the second image data spatially orients the tool in regards to the first image data regarding the object; acquiring, by the processing circuit, a three-dimensional model of the tool based on the second image data; projecting, by the processing circuit, a first two-dimensional tool image and a second two-dimensional tool image of the tool onto a first plane and a second plane, respectively; displaying, by the display device, the three-dimensional model of the tool in relation the three-dimensional object image; and displaying, by the display device, the first two-dimensional tool image and the second two-dimensional tool image on the first plane and the second plane, respectively. ### Claim 19 (depends on 12) The method of claim 12, further comprising: detecting, by the imaging device, a region of interest based on preoperative analysis; constructing, by the processing circuit, a three-dimensional representation of the region of interest including at least one of the three-dimensional object image and the two-dimensional object image; and displaying, by the display device, the three-dimensional representation of the region of interest. ### Claim 20 A visualization system, comprising: a processing circuit configured to: receive image data from an imaging device regarding an object; generate a three-dimensional image of the object based on the image data; set a first plane and a second plane intersecting the first plane; generate a first two-dimensional image of the object by projecting a first cross-section of the three-dimensional image of the object onto the first plane; generate a second two-dimensional image of the object by projecting a second cross-section of the three-dimensional image of the object onto the second plane; and provide a command to a display device to display the three-dimensional image of the object between the first plane and the second plane, the first two-dimensional image of the object on the first plane, and the second two-dimensional image of the object on the second plane.