# US 2020/0401858 A1 · Image recognition method and device, and image presentation time adjustment method and device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401858 A1 - Document ID: US20200401858A1 - Status: Published application - Kind: A1 - Application number: 17/007,869 - Filed: 2020-08-31 - Published: 2020-12-24 - Earliest claimed: 2018-03-02 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Hui YANG; Peng YUAN; Weidong TANG; Shuaihua PENG - Web page: https://patentgrep.com/patent/US20200401858A1 ## CPC - G06F 18/25: Fusion techniques - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation - G06F 18/251: {of input or preprocessed data} - G06F 18/256: {of results relating to different input data, e.g. multimodal recognition} - G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)} - G06F 2218/00: Aspects of pattern recognition specially adapted for signal processing - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract This application provides an image recognition method and device. The image recognition method includes: setting a presentation time sequence corresponding to an image sequence includes N images, the presentation time sequence includes unequal presentation times, a difference between any two presentation times of the unequal presentation times is k×Δ, k is a positive integer, and Δ is a preset time period value; processing the image sequence by using a computer vision algorithm, to obtain a computer vision signal corresponding to each image in the image sequence; obtaining a feedback signal that is corresponding to each image in the image sequence generated when an observation object watches the image sequence displayed in the presentation time sequence; and fusing, for each image in the image sequence, a corresponding computer vision signal and a corresponding feedback signal to obtain a target recognition signal of each image in the image sequence. ## Claims ### Claim 1 A brain-computer combination image recognition method based on image sequence presentation, comprising: setting a presentation time sequence corresponding to an image sequence, wherein the image sequence comprises N images, N is a positive integer, the presentation time sequence comprises a presentation time of each image in the image sequence, a presentation time of an image i is used to indicate a time period from a presentation start moment of the image i to a presentation start moment of a next adjacent image, the image i is any image in the image sequence, the presentation time sequence comprises at least two unequal presentation times, a difference between any two presentation times of the at least two unequal presentation times is k×Δ, k is a positive integer, and Δ is a preset time period value; processing the image sequence by using a computer vision algorithm, to obtain a computer vision signal corresponding to each image in the image sequence; obtaining a feedback signal that is generated when an observation object watches the image sequence displayed in the presentation time sequence and that corresponds to each image in the image sequence, wherein the feedback signal is used to indicate a reaction of the observation object to the watched image; and fusing, for each image in the image sequence, a corresponding computer vision signal and a corresponding feedback signal to obtain a target recognition signal of each image in the image sequence, wherein the target recognition signal is used for image recognition. ### Claim 2 (depends on 1) The method according to claim 1, wherein before the setting of the presentation time sequence corresponding to the image sequence, the method further comprises: receiving M images from a camera device, wherein M is an integer greater than 1; and selecting N images from the M images as the image sequence, wherein N is less than or equal to M. ### Claim 3 (depends on 1) The method according to claim 1, wherein the setting of the presentation time sequence corresponding to the image sequence comprises: determining a corresponding presentation time for each image in the image sequence based on a duration impact parameter, to obtain the presentation time sequence corresponding to the image sequence; wherein the duration impact parameter comprises at least one of the first recognition probability or a fatigue state parameter; wherein the first recognition probability is used to indicate a probability that an image comprises a preset image feature, the probability is obtained using the computer vision algorithm; wherein the fatigue state parameter is used to indicate a fatigue degree of the observation object when the observation object observes an image; and wherein the presentation time is inversely correlated with the first recognition probability, and positively correlated with the fatigue state parameter. ### Claim 4 (depends on 3) The method according to claim 3, wherein the duration impact parameter comprises the fatigue state parameter, and before the determining of the corresponding presentation time for each image in the image sequence based on a duration impact parameter, the method further comprises: predicting, according to a fatigue rule, the fatigue state parameter corresponding to each image in the image sequence, wherein the fatigue rule is used to indicate a change rule of a fatigue degree of the observation object. ### Claim 5 (depends on 3) The method according to claim 3, wherein the obtaining of a feedback signal that is generated when an observation object watches the image sequence displayed in the presentation time sequence and that corresponds to each image in the image sequence comprises: in a process of displaying the image sequence in the presentation time sequence, obtaining the fatigue state parameter corresponding to an image j; adjusting, based on the fatigue state parameter corresponding to the image j, a presentation time of the presentation time sequence, wherein the image corresponds to an image to be displayed after the image j in the image sequence, wherein the image j is any image in the image sequence. ### Claim 6 (depends on 5) The method according to claim 5, wherein the obtaining of the fatigue state parameter corresponding to the image j comprises: obtaining the fatigue state parameter based on fatigue state information that is sent by a sensor and that is obtained when the observation object watches the image j. ### Claim 7 (depends on 3) The method according to claim 3, wherein the determining of the corresponding presentation time for each image in the image sequence based on the duration impact parameter comprises: for each image in the image sequence, finding a presentation time corresponding to the duration impact parameter from a first mapping table, wherein the first mapping table comprises a plurality of duration impact parameters and presentation times respectively corresponding to the plurality of duration impact parameters. ### Claim 8 (depends on 3) The method according to claim 3, further comprising: in response detecting that a corresponding fatigue state parameter obtained when the observation object observes an image q is greater than or equal to a first fatigue threshold, controlling to stop displaying images to be displayed after the image q in the image sequence, and obtaining an image whose corresponding first recognition probability is greater than or equal to a first probability threshold in the images to be displayed after the image q, wherein the image q is any image in the image sequence; and in response to detecting that the fatigue state parameter of the observation object is less than or equal to a second fatigue threshold, controlling to sequentially display the image whose first recognition probability is greater than or equal to the first probability threshold in the images to be displayed after the image q. ### Claim 9 (depends on 1) The method according to claim 1, wherein the fusing, for each image in the image sequence, of the corresponding computer vision signal and the corresponding feedback signal to obtain a target recognition signal of each image in the image sequence comprises: fusing, for each image in the image sequence, a corresponding computer vision signal and at least two corresponding feedback signals to obtain a target recognition signal of each image in the image sequence. ### Claim 10 (depends on 9) The method according to claim 9, wherein the fatigue state parameter comprises at least two fatigue state parameters respectively generated when at least two observation objects observe a same image. ### Claim 11 (depends on 3) The method according to claim 3, wherein the fusing, for each image in the image sequence, of the corresponding computer vision signal and the corresponding feedback signal to obtain a target recognition signal of each image in the image sequence comprises: determining, for each image in the image sequence based on at least one of the first recognition probability, the fatigue state parameter, or the presentation time, a first weight corresponding to each image in the image sequence, wherein the first weight is a weight used when the corresponding feedback signal is used to determine the target recognition signal, the first weight is inversely correlated with the first recognition probability, the first weight is inversely correlated with the fatigue state parameter, and the first weight is positively correlated with the presentation time; and fusing, for each image in the image sequence based on a corresponding first weight, a corresponding computer vision signal and a corresponding feedback signal to obtain the target recognition signal of each image in the image sequence. ### Claim 12 (depends on 1) The method according to claim 1, wherein the computer vision signal is a first recognition probability determined by using the computer vision algorithm; before the fusing, for each image in the image sequence, of the corresponding computer vision signal and the corresponding feedback signal to obtain a target recognition signal of each image in the image sequence, the method further comprises: calculating, for each image in the image sequence, a second recognition probability of each image in the image sequence based on a corresponding feedback signal, wherein the second recognition probability is used to indicate a probability that the observation object determines that the image comprises the preset image feature; and wherein the fusing, for each image in the image sequence, a corresponding computer vision signal and a corresponding feedback signal to obtain a target recognition signal of each image in the image sequence comprises: calculating, for each image in the image sequence, a target recognition probability of each image in the image sequence based on the corresponding first recognition probability and the corresponding second recognition probability. ### Claim 13 An image recognition device, comprising: a processor; a memory; and an interface circuit, wherein the memory, the interface circuit and the processor are interconnected, wherein the memory stores program instructions, which, when executed by the processor, cause the processor to: set a presentation time sequence corresponding to an image sequence, wherein the image sequence comprises N images, N is a positive integer, the presentation time sequence comprises a presentation time of each image in the image sequence, a presentation time of an image i is used to indicate a time period from a presentation start moment of the image i to a presentation start moment of a next adjacent image, the image i is any image in the image sequence, the presentation time sequence comprises at least two unequal presentation times, a difference between any two presentation times of the at least two unequal presentation times is k×Δ, k is a positive integer, and Δ is a preset time period value, process the image sequence by using a computer vision algorithm, to obtain a computer vision signal corresponding to each image in the image sequence, cause the interface circuit to obtain a feedback signal that is generated when an observation object watches the image sequence displayed in the presentation time sequence and that corresponds to each image in the image sequence, wherein the feedback signal is used to indicate a reaction of the observation object to the watched image, and fuse, for each image in the image sequence, a corresponding computer vision signal and a corresponding feedback signal to obtain a target recognition signal of each image in the image sequence, wherein the target recognition signal is used for image recognition. ### Claim 14 (depends on 13) The device according to claim 13, wherein: the interface circuit is configured to receive M images from a camera device, wherein M is an integer greater than 1; and the processor further is configured to select N images from the M images as the image sequence, wherein N is less than or equal to M. ### Claim 15 (depends on 13) The device according to claim 13, wherein the processor is configured to: determine a corresponding presentation time for each image in the image sequence based on a duration impact parameter, to obtain the presentation time sequence corresponding to the image sequence; wherein the duration impact parameter comprises at least one of the first recognition probability or a fatigue state parameter, the first recognition probability is used to indicate a probability, obtained by using the computer vision algorithm, that an image comprises a preset image feature, the fatigue state parameter is used to indicate a fatigue degree of the observation object when the observation object observes an image, the presentation time is inversely correlated with the first recognition probability, and the presentation time is positively correlated with the fatigue state parameter. ### Claim 16 (depends on 15) The device according to claim 15, wherein the duration impact parameter comprises the fatigue state parameter, and the device further comprises a prediction unit; and wherein the processor is configured to predict, according to a fatigue rule, the fatigue state parameter corresponding to each image in the image sequence, wherein the fatigue rule is used to indicate a change rule of a fatigue degree of the observation object. ### Claim 17 (depends on 15) The device according to claim 15, wherein the interface circuit is further configured to: in a process of displaying the image sequence in the presentation time sequence, obtain the fatigue state parameter corresponding to an image j, and adjust, based on the fatigue state parameter corresponding to the image j, a presentation time, in the presentation time sequence, corresponding to an image to be displayed after the image j in the image sequence, wherein the image j is any image in the image sequence. ### Claim 18 (depends on 17) The device according to claim 17, wherein the obtaining unit is configured to obtain the fatigue state parameter based on fatigue state information that is sent by a sensor and that is obtained when the observation object watches the image j. ### Claim 19 (depends on 15) The device according to claim 15, wherein the processor is configured to: for each image in the image sequence, find a presentation time corresponding to the duration impact parameter from a first mapping table, wherein the first mapping table comprises a plurality of duration impact parameters and presentation times respectively corresponding to the plurality of duration impact parameters. ### Claim 20 A computer readable storage medium, wherein the storage medium is configured to store program instructions, which, when executed by a processor, cause the processor to perform the operations of recognizing a brain-computer combination image based on image sequence presentation, the operations comprising: setting a presentation time sequence corresponding to an image sequence, wherein the image sequence comprises N images, N is a positive integer, the presentation time sequence comprises a presentation time of each image in the image sequence, a presentation time of an image i is used to indicate a time period from a presentation start moment of the image i to a presentation start moment of a next adjacent image, the image i is any image in the image sequence, the presentation time sequence comprises at least two unequal presentation times, a difference between any two presentation times of the at least two unequal presentation times is k×Δ, k is a positive integer, and Δ is a preset time period value; processing the image sequence by using a computer vision algorithm, to obtain a computer vision signal corresponding to each image in the image sequence; obtaining a feedback signal that is generated when an observation object watches the image sequence displayed in the presentation time sequence and that corresponds to each image in the image sequence, wherein the feedback signal is used to indicate a reaction of the observation object to the watched image; and fusing, for each image in the image sequence, a corresponding computer vision signal and a corresponding feedback signal to obtain a target recognition signal of each image in the image sequence, wherein the target recognition signal is used for image recognition.