# US 10,120,451 B1 · Systems and user interfaces for dynamic interaction with two- and three-dimensional medical image data using spatial positioning of mobile devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,120,451 B1 - Document ID: US10120451B1 - Status: Granted - Kind: B1 - Application number: 14/593,330 - Filed: 2015-01-09 - Granted: 2018-11-06 - Earliest claimed: 2014-01-09 - Assignees: D.R. Systems, Inc. - Applicants: DR Systems, Inc. - Inventors: Evan K. Fram - Web page: https://patentgrep.com/patent/US10120451B1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - A61B 6/463: {characterised by displaying multiple images or images and diagnostic data on one display} - G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06F 3/0485: Scrolling or panning - G06T 7/0012: {Biomedical image inspection} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Embodiments of the present disclosure relate to systems and techniques for accessing data stores of medical images and displaying the medical images in substantially real-time to provide information in an interactive user interface. Systems are disclosed that may advantageously provide highly efficient, intuitive, and rapid dynamic interaction with two- and three-dimensional medical image data using spatial positioning of mobile devices. The systems may include interactive user interfaces that are dynamically updated to provide a virtual viewbox or window to two- and/or three-dimensional image data. A user may use the systems described herein to more quickly, thoroughly, and efficiently interact with image data including two-dimensional images, three-dimensional image data, and/or series of image data, as compared to previous systems. The systems described may also enable a user to efficiently view and interact with such image data using a small display, such as the display of a smartphone or tablet. ## Claims ### Claim 1 A medical image computing system comprising: a mobile computing device including at least an electronic display; one or more sensors configured to provide spatial position and/or motion information associated with the mobile computing device; a storage device configured to store electronic software instructions; and one or more computer processors in communication with the electronic display, the one or more sensors, and the storage device, the one or more computer processors configured to execute the stored software instructions to cause the computing system to: provide a user interface on the electronic display, the user interface configured to display medical image data on the electronic display; determine, based on the spatial position and/or motion information provided by the one or more sensors, a first spatial position of the mobile computing device in a physical space; determine a fixed virtual spatial position of the medical image data in a virtual space corresponding to the physical space, wherein the fixed virtual spatial position of the medical image data corresponds to a fixed physical spatial position in the physical space; receive, via the mobile computing device, a user input initially selecting the fixed virtual spatial position, wherein the fixed virtual spatial position is fixed relative to the physical space; determine, based on the first spatial position of the mobile computing device relative to the fixed virtual spatial position of the medical image data, at least a first portion of the medical image data viewable via the electronic display; update the user interface to include the first portion of the medical image data such that the first portion of the medical image data is displayed on the electronic display; receive an indication from the one or more sensors of a movement of the mobile computing device from the first spatial position and relative to the fixed virtual spatial position of the medical image data initially selected via the user input; calculate, based on the spatial position and/or motion information provided by the one or more sensors, a position difference between the first spatial position and a second spatial position of the mobile computing device; determine, based on the position difference, the second spatial position of the mobile computing device in the physical space and relative to the fixed virtual spatial position of the medical image data initially selected via the user input; automatically determine, based on the calculated second position difference and/or the second spatial position of the mobile computing device relative to the fixed virtual spatial position of the medical image data initially selected via the user input, at least a second portion of the medical image data viewable via the electronic display; and update the user interface to include the second portion of the medical image data such that the second portion of the medical image data is displayed on the electronic display, wherein the second portion of the medical image data is virtually positioned relative to the fixed virtual spatial position of the medical image data initially selected via the user input. ### Claim 2 (depends on 1) The computing system of claim 1, wherein the medical image data comprises an array of medical images. ### Claim 3 (depends on 2) The computing system of claim 2, wherein the array of medical images comprises a two-dimensional array of medical images, and wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to the position difference indicating a lateral movement, determine the second portion of the medical image data to be at a corresponding lateral position on a plane of the two-dimensional array of medical images relative to the first portion. ### Claim 4 (depends on 3) The computing system of claim 3, wherein the first and second portions of the medical image data comprise first and second medical images of the two-dimensional array. ### Claim 5 (depends on 4) The computing system of claim 4, wherein updating the user interface to include the second portion of the medical image data includes snapping to the second medical image. ### Claim 6 (depends on 4) The computing system of claim 4, wherein updating the user interface to include the second portion of the medical image data includes smoothly panning from the first medical image to the second medical image. ### Claim 7 (depends on 3) The computing system of claim 3, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to determining that the second spatial position is to a right of the first spatial position, determine the second portion of the medical image data to be to a right of the first portion of the medical image data; in response to determining that the second spatial position is to a left of the first spatial position, determine the second portion of the medical image data to be to a left of the first portion of the medical image data; in response to determining that the second spatial position is above the first spatial position, determine the second portion of the medical image data to be above the first portion of the medical image data; and in response to determining that the second spatial position is below the first spatial position, determine the second portion of the medical image data to be below the first portion of the medical image data. ### Claim 8 (depends on 3) The computing system of claim 3, wherein translational and rotational motion of the mobile computing device is not used in determining the second portion of the medical image data. ### Claim 9 (depends on 2) The computing system of claim 2, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to the position difference indicating an angular movement, determine the second portion of the medical image data to be at a corresponding position on the array of medical images relative to the first portion. ### Claim 10 (depends on 2) The computing system of claim 2, wherein the array of medical images comprises a curved array of medical images. ### Claim 11 (depends on 1) The computing system of claim 1, wherein the medical image data comprises a two-dimensional array of a plurality of series of medical images, wherein the first portion of the medical image data comprises a first of the plurality of series of medical images, and wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to the position difference indicating at least one of an angular movement, a lateral movement, a translational movement, or a rotational movement, determine the second portion of the medical image data to be a second of the plurality of series of medical images, the second of the plurality of series of medical images positioned at a corresponding position on the two-dimensional array relative to the first portion. ### Claim 12 (depends on 11) The computing system of claim 11, wherein each of the plurality of series of medical images is associated with an exam obtained at a different time or via a different modality. ### Claim 13 (depends on 11) The computing system of claim 11, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to an input from a user of the mobile computing device, updating the user interface to display a particular image of the second of the plurality of series of medical images. ### Claim 14 (depends on 13) The computing system of claim 13, wherein, during the input from the user, other movement of the mobile computing device that would otherwise be used to select a different of the plurality of series of medical images is ignored. ### Claim 15 (depends on 13) The computing system of claim 13, wherein the input comprises at least one of a touch input or a combination of a touch input and a movement of the mobile computing device. ### Claim 16 (depends on 1) The computing system of claim 1, wherein the medical image data comprises a plurality of two-dimensional medical images ordered one after another front to back in a virtual space, wherein the first portion comprises a first medical image of the plurality of two-dimensional medical images, and wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to the position difference indicating a translational movement, determine the second portion of the medical image data to be at a corresponding second medical image of the plurality of two-dimensional medical images relative to the first medical image. ### Claim 17 (depends on 1) The computing system of claim 1, wherein: the medical image data comprises a plurality of two-dimensional series of medical images, each of the plurality of two-dimensional series shows a same anatomical part from differing orientations, the first and second portions of the medical image data comprise medical images selected from the plurality of two-dimensional series of medical images, the first portion of the medical image data comprises a first medical image from a first two-dimensional series, and the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to the second spatial position comprising a rotation of the mobile computing device with reference to the first spatial position and relative to the fixed virtual spatial position of the medical image data initially selected via the user input, determine the second portion of the medical image data to be a second medical image from a second two-dimensional series showing the anatomical part from a corresponding rotated position. ### Claim 18 (depends on 1) The computing system of claim 1, wherein: the medical image data comprises a plurality of two-dimensional series of medical images, at least two of the plurality of two-dimensional series shows a same anatomical part from a same orientation, the first and second portions of the medical image data comprise medical images selected from the plurality of two-dimensional series of medical images, and the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to the second portion of the medical image data comprising a medical image from one of the at least two of the plurality of two-dimensional series, allowing a user of the mobile computing device to select one of the at least two of the plurality of two-dimensional series and determining the second portion of the medical image data to be a corresponding medical image from the selected on of the at least two of the plurality of two-dimensional series. ### Claim 19 (depends on 18) The computing system of claim 18, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: notify the user of the existence of the at least two of the plurality of two-dimensional series before allowing the user to select the one of the at least two of the plurality of two-dimensional series. ### Claim 20 (depends on 1) The computing system of claim 1, wherein the medical image data comprises three-dimensional medical image data. ### Claim 21 (depends on 20) The computing system of claim 20, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: utilize the second spatial position of the mobile computing device as a rendering parameter for rendering a view of the three-dimensional medical image data corresponding to the second spatial position. ### Claim 22 (depends on 20) The computing system of claim 20, wherein the first spatial position, the second spatial position, the position difference, the fixed virtual spatial position, and the fixed physical spatial position include at least one of location or orientation information. ### Claim 23 (depends on 20) The computing system of claim 20, wherein the first and second portions of the medical image data comprise rendered slices of the three-dimensional medical image data. ### Claim 24 (depends on 23) The computing system of claim 23, wherein the rendered slices of the three-dimensional medical image data are rendered in response to the received indication of the movement of the mobile computing device. ### Claim 25 (depends on 20) The computing system of claim 20, wherein the first and second portions of the medical image data comprise rendered volumes of the three-dimensional medical image data. ### Claim 26 (depends on 20) The computing system of claim 20, wherein the first and second portions of the medical image data comprise two-dimensional images of sections of the three-dimensional medical image data. ### Claim 27 (depends on 20) The computing system of claim 20, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to the position difference indicating a translational movement, determine the second portion of the medical image data to be at a corresponding translated position in the three-dimensional medical image data relative to the first portion. ### Claim 28 (depends on 20) The computing system of claim 20, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to the position difference indicating a rotational movement, determine the second portion of the medical image data to be at a corresponding rotated position in the three-dimensional medical image data relative to the first portion and relative to the fixed virtual spatial position of the medical image data initially selected via the user input. ### Claim 29 (depends on 20) The computing system of claim 20, wherein the one or more computer processors are configured to execute the stored software instructions to further cause the computing system to: in response to the position difference indicating an arbitrary movement, determine the second portion of the medical image data to be at a corresponding arbitrary position in the three-dimensional medical image data relative to the first portion. ### Claim 30 (depends on 1) The computing system of claim 1, wherein the medical image data comprises at least one of a single medical image, an array of medical images, a series of medical images, two-dimensional medical image data, three-dimensional medical image data, a three-dimensional medical image, a three-dimensional volume, rendered portions of three-dimensional medical image data, or combinations thereof. ### Claim 31 (depends on 1) The computing system of claim 1, wherein the mobile computing device comprises at least one of a smartphone, a tablet, a smartwatch, or a wearable mobile computing device. ### Claim 32 (depends on 1) The computing system of claim 1, wherein the one or more sensors are housed in the mobile computing device and comprise at least one of an accelerometer, a gyroscope, or a Global Positioning System transceiver. ### Claim 33 (depends on 1) The computing system of claim 1, wherein the one or more sensors are located external to the mobile computing device and comprise at least one of a video camera or an infrared camera. ### Claim 34 (depends on 1) The computing system of claim 1, wherein the fixed virtual spatial position of the medical image data in the virtual space is determined based on user preferences.