# US 2022/0035489 A1 · Machine learning methods for monitoring a user's interaction with 3d medical images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0035489 A1 - Document ID: US20220035489A1 - Status: Published application - Kind: A1 - Application number: 16/942,812 - Filed: 2020-07-30 - Published: 2022-02-03 - Earliest claimed: 2020-07-30 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: FLORA GILBOA-SOLOMON; AVIAD ZLOTNICK - Web page: https://patentgrep.com/patent/US20220035489A1 ## CPC - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06V 10/82: using neural networks - G06F 18/22: Matching criteria, e.g. proximity measures - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/09: Supervised learning - G06V 10/40: Extraction of image or video features - G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS - G16H 30/40: for processing medical images, e.g. editing - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract There is provided a method for monitoring interaction with 3D medical images, comprising: dividing the 3D image into a sequence of a 2D images, arranging the sequence into slabs each including at least one 2D image, computing, for each respective slab, a minimal amount of viewing time a user is predicted to spend viewing the respective slab, monitoring, while the 3D medical image is presented on a display, an amount of time a user actually spent viewing portions of the 3D medical image corresponding to each of the of slabs, in response to the amount of time spent viewing a certain portion of the 3D medical image being less than the computed minimal amount of viewing time of a certain slab corresponding to the certain portion, generating instructions for implementation by a user interface indicative of the amount of time spent being less than the computed minimal amount of time. ## Claims ### Claim 1 A computer implemented method for monitoring a user's interaction with 3D medical images, comprising: receiving a 3D medical image; dividing the 3D medical image into a sequence of a plurality 2D images; arranging the sequence into a plurality of slabs each including at least one 2D image; computing, for each respective slab, a minimal amount of viewing time a user is predicted to spend viewing the respective slab; monitoring, while the 3D medical image is presented on a display, an amount of time a user actually spent viewing portions of the 3D medical image corresponding to each of the plurality of slabs; in response to the amount of time spent viewing a certain portion of the 3D medical image being less than the computed minimal amount of viewing time of a certain slab corresponding to the certain portion, generating instructions for implementation by a user interface indicative of the amount of time spent being less than the computed minimal amount of time. ### Claim 2 (depends on 1) The method of claim 1, wherein arranging comprises: computing a similarity dataset indicative of an amount of similarity between each pair of the plurality of 2D images; segmenting the similarity dataset into a plurality of groups by minimizing the amount of similarity between consecutive groups and maximizing the amount of similarity within each group, wherein the plurality of slabs correspond to the plurality of groups. ### Claim 3 (depends on 2) The method of claim 2, wherein the similarity dataset is a matrix of size N×N wherein N denotes a number of the plurality of 2D images. ### Claim 4 (depends on 2) The method of claim 2, further comprising: inputting each 2D image into a neural network trained to analyze a target 2D image and output an indication of a target visual feature being depicted within the target 2D image; extracting, for each 2D image, a feature vector from the neural network; wherein the amount of similarity is computed for the feature vectors of each pair. ### Claim 5 (depends on 4) The method of claim 4, wherein the amount of similarity is computed for the feature vectors of each pair using a cosine similarity. ### Claim 6 (depends on 4) The method of claim 4, wherein the feature vector is selected from the group consisting of: embeddings obtained from hidden layers of the neural network, an output of an autoencoder implementation of the neural network, and the feature vector is an output of the neural network. ### Claim 7 (depends on 6) The method of claim 6, further comprising selecting the neural network from a plurality of neural networks each trained to output an indication of a different visual feature, according to the visual feature, wherein the user is viewing the 3D medical image to search for the visual feature. ### Claim 8 (depends on 1) The method of claim 1, wherein a first subset of the plurality of slabs having fewer than a first threshold number of slabs are designated as small slabs, and a second subset of the plurality of slabs having more than a second threshold number of slabs are designated as large slabs, wherein monitoring comprises monitoring the amount of time corresponding to each small slab and each large slab, and wherein the amount of time spent viewing the certain portion of the 3D medical image being less than the computed minimal amount of viewing time of a certain slab corresponding to the certain portion comprises the amount of time spent viewing a certain small slab is less than a viewing threshold computed based on the amount of time spent viewing a certain large slab. ### Claim 9 (depends on 8) The method of claim 8, wherein the viewing threshold is selected from a group consisting of: a statistical average and distribution of time spent viewing each 2D images of the large slabs and the time viewing time of the certain small slab comprises the average viewing time of each 2D image of the small slabs, and a ratio between viewing time per 2D image of the large slab and viewing time per 2D image of the small slab. ### Claim 10 (depends on 1) The method of claim 1, wherein the plurality of slabs are computed by inputting the sequential 2D images into a video scene analysis process that divides a video into scenes of frames, the 2D images corresponding to frames of the video and the scenes corresponding to slabs. ### Claim 11 (depends on 1) The method of claim 1, wherein the minimal amount of viewing time the user is predicted to spend viewing the respective slab is computed based on an analysis of historical data including monitored amount of time the user and/or other users spent viewing sample 2D images of a plurality of sample 3D images of a plurality of subjects. ### Claim 12 (depends on 11) The method of claim 11, wherein the minimal amount of time the user is predicted to spend viewing the respective slab is an outcome of a time classifier that receives the slab as input, wherein the time classifier trained on a training dataset of the plurality of 2D images of the plurality of sample 3D images of the plurality of subjects labeled with amount of time spent viewing each 2D images presented on a display. ### Claim 13 (depends on 1) The method of claim 1, wherein the minimal amount of viewing time for a respective slab is computed based on a number of 2D images included in the respective slab, wherein the minimal amount of time is inversely related to the number of 2D images. ### Claim 14 (depends on 1) The method of claim 1, wherein the 3D medical image is presented on a display within a graphical user interface (GUI) of a medical image viewing application, and the 2D images of the certain slab are presented within the GUI in response to the amount of time spent viewing a certain portion of the 3D medical image being less than the computed minimal amount of viewing time of the certain slab corresponding to the certain portion. ### Claim 15 (depends on 1) The method of claim 1, wherein the generating instructions is selected from a group consisting of: presenting the 2D images of the certain slab on the display for an additional view by the user, generating a sound signal played by a microphone, generating a visual signal presented on a display, generating a haptic signal implemented on a haptic device, and adding the 2D images of the certain slab to a second viewing list and presenting an indication of the second viewing list on the display. ### Claim 16 (depends on 1) The method of claim 1, wherein the 3D medical images are divided into a sequence of a plurality 2D images according to a slice orientation and/or a slice thickness defined by the user viewing the plurality of 2D images and corresponds to the slice orientation and/or slice thickness when the user views the plurality of 2D images. ### Claim 17 A computer implemented method for monitoring a user's interaction with real time 2D medical images, comprising: in a plurality of iterations: receiving a 2D medical image captured in real time; analyzing the 2D medical image to determine whether the 2D medical image represents an end of a slab including at least one of a previously obtained sequence of 2D images; computing a minimal amount of viewing time a user should spend viewing the slab; monitoring, while the 2D medical image and the previously obtained sequence of 2D images are presented on a display, an amount of time a user actually spent viewing portions of the 2D medical image and the previously obtained sequence of 2D images corresponding to the slab; in response to the amount of time spent actually spent being less than the computed minimal amount of viewing time, presenting instructions on the display to spend additional time viewing additional 2D images captured from an anatomical region of a subject corresponding to the slab. ### Claim 18 (depends on 17) The method of claim 17, further comprising: in response to the 2D medical image not representing end of the slab, including the 2D medical image as part of the slab, and performing another iteration by receiving another 2D medical image. ### Claim 19 (depends on 17) The method of claim 17, wherein the 2D medical image and the sequence of 2D images are captured by an imaging device selected from the group consisting of: colonoscope, endoscope, bronchoscope, and 2D ultrasound. ### Claim 20 (depends on 17) The method of claim 17, further comprising: inputting the 2D image into a neural network trained to analyze a target 2D image and output an indication of a target visual feature being depicted within the target 2D image; extracting, for the 2D image, a feature vector from the neural network; computing an amount of similarity between the feature vector of the 2D image and feature vectors of previously obtained sequence of 2D images; in response to the amount of similarity being below a threshold, determining that the 2D medical image represents the end of the slab.