# US 2021/0341997 A1 · Method and system for improving the visual exploration of an image during a target search

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0341997 A1
- Document ID: US20210341997A1
- Status: Published application
- Kind: A1
- Application number: 14/390,544
- Filed: 2013-04-05
- Published: 2021-11-04
- Earliest claimed: 2012-04-05
- Assignees: Sr Labs S.R.L.
- Applicants: SR Labs S.R.L.
- Inventors: Paolo Invernizzi
- Web page: https://patentgrep.com/patent/US20210341997A1

## CPC

- G06T 7/73: using feature-based methods
- G06V 40/18: Eye characteristics, e.g. of the iris
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0213: {Arrangements providing an integrated pointing device in a keyboard, e.g. trackball, mini-joystick (for pointing devices integrated in a laptop computer G06F1/169; joysticks G05G9/047; constructional details of pointing devices G06F3/033)}
- G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)}
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/63: for local operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- A61B 6/463: {characterised by displaying multiple images or images and diagnostic data on one display}
- A61B 6/467: {characterised by special input means}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

Object of the present invention are a method and a system for monitoring a user's attention distribution in the course of a digital image exploration, adapted to produce feedback that give the user himself an indication of the efficacy of the exploration performed, even on the basis of the relevance that the various zones of the image have in relation to a certain goal, relevance that can possibly be determined by an expert system responsible for examining the image as a function of the above-mentioned objective.

## Claims

### Claim 1

A system for improving the analysis of digital images by a user comprising electronic data processing means, comprising data storage means, digital image display means and interface means, adapted to allow said electronic data processing means to collect data from a user, wherein said electronic data processing means are adapted to modify the display of said digital images during the exploration thereof by the user so as to add indicators adapted to mark the zones of said digital images deemed important and which have not yet been explored by the user with the necessary care.

### Claim 2 (depends on 1)

The system according to claim 1, wherein said interface means comprise devices selected from the group comprising: alphanumeric keyboards, mouse pointers and buttons and eye-tracker devices.

### Claim 3 (depends on 1)

The system according to claim 1, wherein said electronic data processing means comprising data storage means further comprise a first processing module comprising a first digital image analysis expert system adapted to interact with appropriate databases so as to associate a relevance index to each point of the analysed digital image as a function of a certain predetermined target, and a second processing module comprising a second digital image analysis expert system, adapted to supply a feedback to both said first digital image analysis expert system and to said digital image display means.

### Claim 4 (depends on 3)

The system according to claim 3, wherein said feedback supplied to said first digital image analysis expert system comprises an update of the value of the relevance index associated to each point of said digital image.

### Claim 5 (depends on 3)

The system according to claim 3, wherein said feedback supplied to said digital image display means comprises highlighter markers superimposed on said image and adapted to direct the user's attention to a certain zone of said image.

### Claim 6 (depends on 5)

The system according to claim 5, wherein said dynamic highlighter markers are such as to dissolve after a certain interval of time so as to allow display of the original image without any superimposed modification.

### Claim 7 (depends on 5)

The system according to claim 5, wherein said dynamic highlighter markers are selected from the group comprising: zones of said image with contrasting colours, moving shapes superimposed on said image and adapted to attract the user's attention towards the zones of interest of said image, direction indicators facing towards the zones of interest of said image, zones of said image which intermittently blink.

### Claim 8 (depends on 3)

The system according to claim 3, wherein said second digital image analysis expert system is adapted to process the data received from said eye-tracker device so as to distinguish the user's saccades from fixations; to analyse and categorise said fixations into a certain number of cases which are distinguished in relation to predetermined thresholds and related to the relevance indexes of the points of said image fixed by the user; to update the value of the relevance index associated to each point of said digital image and to transmit said updated relevance index value to said first digital image analysis expert system; to calculate an optimal scanpath of said image; to generate dynamic markers, on the basis of said optimal scanpath, that are adapted to highlight appropriate zones of said image; to send said dynamic markers to said graphic interface of said data processing means so that they are appropriately superimposed on said image and suggest said optimal scanpath to the user.

### Claim 9

A method for improving the analysis of digital images by a user comprising: displaying a digital image to a user on appropriate display means; acquiring the user's scanpath of said image by means of an eye-tracker device; acquiring point markers of said image made by said user; comparing the user's scanpath of said image and the points of said image marked by the user with data relating to the relevance index of the points of said image originating from a digital image analysis expert system; producing a first feedback adapted to update said data relating to the relevance index of the points of said image originating from a digital image analysis expert system.

### Claim 10 (depends on 9)

The method according to claim 9, wherein comparison of the points of said image marked by the user with said data relating to the relevance index of the points of said image originating from a digital image analysis expert system is carried out by establishing a certain number of distinction thresholds and ranges with respect to the possible values of said relevance index within which said points of said image marked by the user are categorised.

### Claim 11 (depends on 9)

The method according to claim 9, comprising: calculating an optimal scanpath of said image; producing a second feedback adapted to generate dynamic markers adapted to highlight appropriate zones of said image; sending said dynamic markers to said appropriate display means so that said dynamic markers are displayed superimposed on said image and suggest said optimal scanpath to the user.

### Claim 12 (depends on 9)

The method according to claim 9, wherein said first and second feedbacks are produced according to the following steps: establishing four distinction ranges by dividing the range of values of said relevance index, comprised between 0 and 1, into four sub-ranges: a) for relevance indexes between 0 and 0.2, b) between 0.2 and 0.5, c) between 0.5 and 0.8, d) between 0.8 and 1; evaluating the point fixed by the user and the type of fixation acquired with regards to two time thresholds, a first threshold S1 and a second threshold S2>S1; in the case that fixation has a greater duration than both said first threshold S1 and said second threshold S2, the display of said digital image is optimised to promote the user's inspection thereof; if said fixation has a longer duration than said first threshold S1 but lesser than said second threshold S2, checks for the presence of a manual input by the user corresponding to the user's intention to mark the concerned point as important are carried out, and if present the relevance index is compared with said distinction thresholds, if said index belongs to said range c) or to said range d), no feedback is produced, if said index belongs to said range b) feedback is produced by adding the point fixed by the user to the interest points of said image by updating the set of positive target points, if said index belongs to said range a) a graphic type feedback is provided by means of a dynamic marker superimposed on the display of said graphic image; in the case that said manual input by the user is not present, the relevance index is compared with said distinction thresholds, if said index belongs to said range c) or to said range d), a feedback is generated that is adapted to insert the point fixed by the user into the points which are not of interest of said image, if said index belongs to said ranges a) or b), no feedback is produced; in the case in which said fixation has a shorter duration than said first threshold S1, checks for the presence of manual input by the user are carried out, if present, the relevance index is compared with said distinction thresholds: if said index belongs to said range c) or to said range d), no feedback is produced, if said index belongs to said ranges a) or b), feedback is generated that is adapted to add the point of said digital image currently being examined to the set of positive points, removing it from the set of negative points; in the case that said manual input by the user is not present, the relevance index is compared with said distinction thresholds: if said index belongs to said range d), user confirmation that the examined point is a positive target is requested: if the user agrees, then it is confirmed that the point is a positive target, otherwise, if the user does not confirm, the point is removed from the set of positive targets and added to the set of negative targets, if said index does not belong to said range d), no feedback is produced.

### Claim 13 (depends on 9)

A computer program comprising code programming means adapted to execute the steps of the method described in claim 9 when said program is executed by a computer.

### Claim 14 (depends on 9)

A computer-readable recording medium having a program recorded thereon, said readable recording medium comprising computer program code means adapted to implement all the steps of claim 9, when said computer program is executed on the computer.

### Claim 15 (depends on 11)

The method according to claim 11, wherein said second feedback is produced according to the following steps: establishing four distinction ranges by dividing the range of values of said relevance index, comprised between 0 and 1, into four sub-ranges: a) for relevance indexes between 0 and 0.2, b) between 0.2 and 0.5, c) between 0.5 and 0.8, d) between 0.8 and 1; evaluating the point fixed by the user and the type of fixation acquired with regards to two time thresholds, a first threshold S1 and a second threshold S2>S1; in the case that fixation has a greater duration than both said first threshold S1 and said second threshold S2, the display of said digital image is optimised to promote the user's inspection thereof; if said fixation has a longer duration than said first threshold S1 but lesser than said second threshold S2, checks for the presence of a manual input by the user corresponding to the user's intention to mark the concerned point as important are carried out, and if present the relevance index is compared with said distinction thresholds, if said index belongs to said range c) or to said range d), no feedback is produced, if said index belongs to said range b) feedback is produced by adding the point fixed by the user to the interest points of said image by updating the set of positive target points, if said index belongs to said range a) a graphic type feedback is provided by means of a dynamic marker superimposed on the display of said graphic image; in the case that said manual input by the user is not present, the relevance index is compared with said distinction thresholds, if said index belongs to said range c) or to said range d), a feedback is generated that is adapted to insert the point fixed by the user into the points which are not of interest of said image, if said index belongs to said ranges a) or b), no feedback is produced; in the case in which said fixation has a shorter duration than said first threshold Si, checks for the presence of manual input by the user are carried out, if present, the relevance index is compared with said distinction thresholds: if said index belongs to said range c) or to said range d), no feedback is produced , if said index belongs to said ranges a) or b), feedback is generated that is adapted to add the point of said digital image currently being examined to the set of positive points, removing it from the set of negative points; in the case that said manual input by the user is not present, the relevance index is compared with said distinction thresholds: if said index belongs to said range d), user confirmation that the examined point is a positive target is requested: if the user agrees, then it is confirmed that the point is a positive target, otherwise, if the user does not confirm, the point is removed from the set of positive targets and added to the set of negative targets, if said index does not belong to said range d), no feedback is produced.