# US 12,001,607 B2 · System and method for iterative classification using neurophysiological signals > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,607 B2 - Document ID: US12001607B2 - Status: Granted - Kind: B2 - Application number: 18/107,037 - Filed: 2023-02-08 - Published: 2023-06-15 - Granted: 2024-06-04 - Earliest claimed: 2016-12-21 - Assignees: InnerEye Ltd. - Applicants: InnerEye Ltd. - Inventors: Amir B. Geva; Eitan Netzer; Ran El Manor; Sergey Vaisman; Leon Y. Deouell; Uri Antman - Web page: https://patentgrep.com/patent/US12001607B2 ## CPC - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/23: Clustering techniques - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 18/2433: Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/091: Active learning - G06N 20/20: Ensemble learning - G06T 7/0012: {Biomedical image inspection} - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher} - G06V 10/82: using neural networks - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30016: Brain - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An image classification neural network is trained based on images that are the presented to an observer as a visual stimulus while collecting neurophysiological signals from a brain of the observer. The neurophysiological signals are processes to identify a neurophysiological event indicative of a detection of a target by the observer in one or more of the images, and the image classification neural network is trained to identify the target in the image based on the identification of the neurophysiological event. ## Claims ### Claim 1 A method of training an image classification neural network, the method comprising: presenting to an observer a stimulus describing a target; following said presentation of said stimulus describing a target, presenting a first plurality of images to an observer as a visual stimulus, while collecting neurophysiological signals from a brain of said observer; processing said neurophysiological signals to identify a neurophysiological event indicative of a detection of a target by said observer in at least one image of said first plurality of images; training the image classification neural network to identify the target in the image, based on said identification of said neurophysiological event; and storing said trained image classification neural network in a non-transitory computer-readable storage medium. ### Claim 2 (depends on 1) The method according to claim 1, further comprising applying unsupervised clustering to a second plurality of images, and selecting said first plurality of images from said second plurality of images based on said unsupervised clustering. ### Claim 3 (depends on 1) The method according to claim 1, comprising randomly selecting said first plurality of images from a second plurality of images. ### Claim 4 (depends on 1) The method according to claim 1, comprising receiving a reference image including said target, and selecting said first plurality of images from a second plurality of images responsively to said reference image. ### Claim 5 (depends on 1) The method according to claim 1, wherein said image classification neural network is a convolutional neural network. ### Claim 6 (depends on 2) The method according to claim 2, wherein said image classification neural network comprises a first neural subnetwork configured for receiving and processing said neurophysiological data, a second neural subnetwork configured for receiving and processing said second plurality of images, and a shared subnetwork having a neural network layer receiving and combining outputs from both said first neural subnetwork and said second neural subnetwork. ### Claim 7 (depends on 6) The method according to claim 6, wherein said image classification neural network is a convolutional neural network and at least one of said first and said second neural subnetworks is a convolutional neural subnetwork. ### Claim 8 (depends on 6) The method according to claim 6, wherein said image classification neural network comprises a first separate output layer for said first neural subnetwork outputting a first score, and second separate output layer for said second neural subnetwork outputting a second score, wherein the method comprises combining said first score with said second score as a weighted sum of said first and said second score, labeling said image with said weighted sum, and using said label in at least one iteration of said training. ### Claim 9 (depends on 1) The method according to claim 1, wherein said image classification neural network comprises an autoencoder subnetwork for unsupervised feature learning. ### Claim 10 (depends on 9) The method according to claim 9, wherein said autoencoder subnetwork is used for selecting said first plurality of images. ### Claim 11 (depends on 9) The method according to claim 9, wherein said autoencoder subnetwork comprises a convolutional neural network (CNN). ### Claim 12 (depends on 1) The method according to claim 1, further comprising scoring said neurophysiological event using said neurophysiological signals, wherein said training is based at least in part on said score. ### Claim 13 (depends on 12) The method according to claim 12, further comprising using said score for determining a level of similarity of said target to an object in an image observed by said observer. ### Claim 14 (depends on 1) The method according to claim 1, wherein said stimulus comprises at least one stimulus selected from the group consisting of a visual stimulus, a textual stimulus, an auditory stimulus and an olfactory stimulus. ### Claim 15 (depends on 1) The method according to claim 1, further comprising presenting to the observer a stimulus complementary to said target prior to said presentation of said first plurality of images. ### Claim 16 (depends on 15) The method according to claim 15, wherein said wherein complementary stimulus comprises at least one stimulus selected from the group consisting of a visual stimulus, a textual stimulus, an auditory stimulus and an olfactory stimulus. ### Claim 17 A system for training an image classification neural network, comprising: a neurophysiological signal collection system, configured for collecting neurophysiological signals from a brain of an observer; and a data processor, communicating with said neurophysiological signal collection system and being configured for: presenting a first plurality of images to an observer as a visual stimulus, while collecting neurophysiological signals from a brain of said observer; processing said neurophysiological signals to identify a neurophysiological event indicative of a detection of a target by said observer in at least one image of said first plurality of images; training the image classification neural network to identify the target in the image, based on said identification of said neurophysiological event; and storing said trained image classification neural network in a non-transitory computer-readable storage medium; wherein said data processor is configured to apply unsupervised clustering to a second plurality of images, and to select said first plurality of images from said second plurality of images based on said unsupervised clustering; wherein said image classification neural network comprises: a first neural subnetwork configured for receiving and processing said neurophysiological data, a first separate output layer for said first neural subnetwork outputting a first score, a second neural subnetwork configured for receiving and processing said second plurality of images, second separate output layer for said second neural subnetwork outputting a second score, and a shared subnetwork having a neural network layer receiving and combining outputs from both said first neural subnetwork and said second neural subnetwork; and wherein the data processor is also configured to combine said first score with said second score as a weighted sum of said first and said second score, to label said image with said weighted sum, and to use said label in at least one iteration of said training. ### Claim 18 A method of image classification, comprising: presenting to an observer a stimulus describing a target; following said presentation of said stimulus describing a target, training an image classification neural network to identify said target in images; applying said trained image classification neural network to the image to detect therein candidate image regions suspected as being occupied by said target; presenting to an observer each candidate image region as a visual stimulus, while collecting neurophysiological signals from a brain of said observer; and determining an existence of said target in the image is based, at least in part, on said identification of said neurophysiological event. ### Claim 19 A system for image classification, comprising: a neurophysiological signal collection system, configured for collecting neurophysiological signals from a brain of an observer; and a data processor, communicating with said neurophysiological signal collection system and being configured for: applying a trained image classification neural network to the image to detect therein candidate image regions suspected as being occupied by a target; presenting to an observer each candidate image region as a visual stimulus, while collecting neurophysiological signals from a brain of said observer; and determining an existence of said target in the image is based, at least in part, on said identification of said neurophysiological event; wherein said trained image classification neural network comprises an autoencoder subnetwork for unsupervised feature learning. ### Claim 20 A method of training an image classification neural network, the method comprising: presenting a first plurality of images to an observer as a visual stimulus, while collecting neurophysiological signals from a brain of said observer; processing said neurophysiological signals to identify a neurophysiological event indicative of a detection of a target by said observer in at least one image of said first plurality of images; training the image classification neural network to identify the target in the image, based on said identification of said neurophysiological event; and storing said trained image classification neural network in a non-transitory computer-readable storage medium; wherein said image classification neural network comprises an autoencoder subnetwork for unsupervised feature learning. ### Claim 21 A method of training an image classification neural network, the method comprising: presenting a first plurality of images to an observer as a visual stimulus, while collecting neurophysiological signals from a brain of said observer; processing said neurophysiological signals to identify a neurophysiological event indicative of a detection of a target by said observer in at least one image of said first plurality of images; training the image classification neural network to identify the target in the image, based on said identification of said neurophysiological event; and storing said trained image classification neural network in a non-transitory computer-readable storage medium; wherein the method comprises applying unsupervised clustering to a second plurality of images, and selecting said first plurality of images from said second plurality of images based on said unsupervised clustering; and wherein said image classification neural network comprises: a first neural subnetwork configured for receiving and processing said neurophysiological data, a second neural subnetwork configured for receiving and processing said second plurality of images, a shared subnetwork having a neural network layer receiving and combining outputs from both said first neural subnetwork and said second neural subnetwork, a first separate output layer for said first neural subnetwork outputting a first score, and a second separate output layer for said second neural subnetwork outputting a second score; and wherein the method comprises combining said first score with said second score as a weighted sum of said first and said second score, labeling said image with said weighted sum, and using said label in at least one iteration of said training.