# US 12,217,487 B2 · Method, apparatus, electronic device, and medium for generating image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,487 B2 - Document ID: US12217487B2 - Status: Granted - Kind: B2 - Application number: 17/896,818 - Filed: 2022-08-26 - Published: 2024-01-25 - Granted: 2025-02-04 - Earliest claimed: 2022-07-22 - Assignees: Dell Products L.P. - Applicants: Dell Products L.P. - Inventors: Zhisong Liu; Zijia Wang; Zhen Jia - Web page: https://patentgrep.com/patent/US12217487B2 ## CPC - G06F 18/22: Matching criteria, e.g. proximity measures - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/253: {of extracted features} - G06V 10/40: Extraction of image or video features - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 10/778: Active pattern-learning, e.g. online learning of image or video features - G06V 10/806: {of extracted features} - G06V 10/82: using neural networks - G06F 2218/00: Aspects of pattern recognition specially adapted for signal processing - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In an illustrative embodiment, a method is disclosed for generating an image from fMRI data and EEG data. The method includes extracting a first feature map of the fMRI data, the first feature map being multidimensional data having a temporal dimension related to a sample collection time of the fMRI data, and converting the first feature map based on the temporal dimension of the first feature map and a first attention model. The method further includes extracting a second feature map of the EEG data, the second feature map being multidimensional data having a spatial dimension related to an electrode for collecting the EEG data, and converting the second feature map based on the spatial dimension of the second feature map and a second attention model. The method further includes generating an image based on the converted first feature map and the converted second feature map. ## Claims ### Claim 1 A method for generating an image from functional magnetic resonance imaging (fMRI) data and electroencephalogram (EEG) data, comprising: extracting a first feature map of the fMRI data, the first feature map being multidimensional data having a temporal dimension related to a sample collection time of the fMRI data; converting the first feature map based on the temporal dimension of the first feature map and a first attention model; extracting a second feature map of the EEG data, the second feature map being multidimensional data having a spatial dimension related to an electrode for collecting the EEG data; converting the second feature map based on the spatial dimension of the second feature map and a second attention model; and generating an image based on the converted first feature map and the converted second feature map. ### Claim 2 (depends on 1) The method according to claim 1, wherein the fMRI data and the EEG data are collected in the same time period, extracting the first feature map of the fMRI data comprises: performing temporal position encoding on the fMRI data; and generating the first feature map based on a first convolution model and the position-encoded fMRI data; and extracting the second feature map of the EEG data comprises: performing temporal position encoding on the EEG data; shaping the position-encoded EEG data; and generating the second feature map based on a second convolution model and the shaped position-encoded EEG data. ### Claim 3 (depends on 1) The method according to claim 1, wherein converting the first feature map based on the temporal dimension and the first attention model comprises: reforming the first feature map based on the temporal dimension; generating a reformed first feature map with temporal correlation information based on the first attention model and the reformed first feature map; and shaping the reformed first feature map with temporal correlation information into the converted first feature map, the converted first feature map and the first feature map having the same size. ### Claim 4 (depends on 1) The method according to claim 1, wherein converting the second feature map based on the spatial dimension and the second attention model comprises: reforming the second feature map based on the spatial dimension; generating a reformed second feature map with spatial correlation information based on the second attention model and the reformed second feature map; and shaping the reformed second feature map with spatial correlation information into the converted second feature map, the converted second feature map and the second feature map having the same size. ### Claim 5 (depends on 1) The method according to claim 1, wherein the first feature map and the second feature map have the same size in spatial dimension, at least one feature dimension, and temporal dimension. ### Claim 6 (depends on 5) The method according to claim 5, wherein generating the image based on the converted first feature map and the converted second feature map comprises: generating a spliced feature map by splicing the converted first feature map and the converted second feature map, the spliced feature map providing a combined feature map; converting, by eliminating the temporal dimension, the combined feature map into a third feature map having the spatial dimension and the at least one feature dimension; up-sampling the third feature map; and generating the image based on a third convolution model and the up-sampled third feature map. ### Claim 7 (depends on 1) The method according to claim 1, wherein the fMRI data comprises multidimensional data with a width of an fMRI image, a height of the fMRI image, a color channel, and a sampling time as dimensions, and the EEG data comprises multidimensional data with a signal intensity of an EEG signal, an electrode for collecting the EEG signal, and a sampling time as dimensions. ### Claim 8 (depends on 1) An electronic device comprising: at least one processing unit; and at least one memory that is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, wherein the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to claim 1. ### Claim 9 (depends on 1) A non-transitory computer-readable storage medium, comprising machine-executable instructions that, when executed by a device, cause the device to perform the method according to claim 1. ### Claim 10 A method for training a model, wherein the model comprises: a first feature extraction model configured to extract, from functional magnetic resonance imaging (fMRI) data, a first feature map of the fMRI data, the first feature map being multidimensional data having a temporal dimension related to a sample collection time of the fMRI data; a first conversion model configured to convert the first feature map based on the temporal dimension, the first conversion model comprising a first attention model; a second feature extraction model configured to extract, from electroencephalogram (EEG) data, a second feature map of the EEG data, the second feature map being multidimensional data having a spatial dimension related to an electrode for collecting the EEG data; a second conversion model configured to convert the second feature map based on the spatial dimension, the second conversion model comprising a second attention model; and a generation model configured to generate an image based on the converted first feature map and the converted second feature map, and the method comprises: generating a training dataset; and updating the model based on the training dataset, wherein the training dataset comprises: a first target image and first fMRI data and first EEG data corresponding to the first target image; a second target image and second fMRI data and second EEG data corresponding to the second target image; and a third target image and third fMRI data and third EEG data corresponding to the third target image, wherein the third target image is a spatial combination of the first target image and the second target image. ### Claim 11 (depends on 10) The method according to claim 10, wherein updating the model based on the training dataset comprises updating the model using a loss function, the loss function comprises a pixel comparison result between a predicted image generated by the model from fMRI data and EEG data and a corresponding target image, a semantic comparison result between the predicted image and the target image, and a structure comparison result between the predicted image and the target image. ### Claim 12 (depends on 10) The method according to claim 10, wherein updating the model comprises: updating the model while keeping the first feature extraction model and the first conversion model from being updated; and updating the model while keeping the second feature extraction model and the second conversion model from being updated. ### Claim 13 (depends on 10) A non-transitory computer-readable storage medium, comprising machine-executable instructions that, when executed by a device, cause the device to perform the method according to claim 10. ### Claim 14 An apparatus for generating an image from functional magnetic resonance imaging (fMRI) data and electroencephalogram (EEG) data, comprising: a first feature extraction unit configured to extract a first feature map of the fMRI data, the first feature map being multidimensional data having a temporal dimension related to a sample collection time of the fMRI data; a first conversion unit configured to convert the first feature map based on the temporal dimension of the first feature map and a first attention model; a second feature extraction unit configured to extract a second feature map of the EEG data, the second feature map being multidimensional data having a spatial dimension related to an electrode for collecting the EEG data; a second conversion unit configured to convert the second feature map based on the spatial dimension of the second feature map and a second attention model; and a generation unit configured to generate an image based on the converted first feature map and the converted second feature map. ### Claim 15 (depends on 14) The apparatus according to claim 14, wherein the fMRI data and the EEG data are collected in the same time period, the first feature extraction unit is further configured to: perform temporal position encoding on the fMRI data; and generate the first feature map based on a first convolution model and the position-encoded fMRI data; and the second feature extraction unit is further configured to: perform temporal position encoding on the EEG data; shape the position-encoded EEG data; and generate the second feature map based on a second convolution model and the shaped position-encoded EEG data. ### Claim 16 (depends on 14) The apparatus according to claim 14, wherein the first conversion unit is further configured to: reform the first feature map based on the temporal dimension; generate a reformed first feature map with temporal correlation information based on the first attention model and the reformed first feature map; and shape the reformed first feature map with temporal correlation information into the converted first feature map, the converted first feature map and the first feature map having the same size. ### Claim 17 (depends on 14) The apparatus according to claim 14, wherein the second conversion unit is further configured to: reform the second feature map based on the spatial dimension; generate a reformed second feature map with spatial correlation information based on the second attention model and the reformed second feature map; and shape the reformed second feature map with spatial correlation information into the converted second feature map, the converted second feature map and the second feature map having the same size. ### Claim 18 (depends on 14) The apparatus according to claim 14, wherein the first feature map and the second feature map have the same size in spatial dimension, at least one feature dimension, and temporal dimension. ### Claim 19 (depends on 18) The apparatus according to claim 18, wherein the generation unit is further configured to: generate a spliced feature map by splicing the converted first feature map and the converted second feature map, the spliced feature map providing a combined feature map; convert, by eliminating the temporal dimension, the combined feature map into a third feature map having the spatial dimension and the at least one feature dimension; and up-sample the third feature map; and generate the image based on a third convolution model and the up-sampled third feature map. ### Claim 20 (depends on 14) The apparatus according to claim 14, wherein the fMRI data comprises multidimensional data with a width of an fMRI image, a height of the fMRI image, a color channel, and a sampling time as dimensions, and the EEG data comprises multidimensional data with a signal intensity of an EEG signal, an electrode for collecting the EEG signal, and a sampling time as dimensions.