# US 2021/0240761 A1 · Method and device for cross-modal information retrieval, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240761 A1 - Document ID: US20210240761A1 - Status: Published application - Kind: A1 - Application number: 17/239,974 - Filed: 2021-04-26 - Published: 2021-08-05 - Earliest claimed: 2019-01-31 - Applicants: Shenzhen Sensetime Technology Co., Ltd. - Inventors: Zihao WANG; Jing Shao; Hongsheng Li; Junjie Yan; Xiaogang Wang; Lv Sheng - Web page: https://patentgrep.com/patent/US20210240761A1 ## CPC - G06F 40/216: using statistical methods - G06F 16/3347: {using vector based model} - G06F 16/56: having vectorial format - G06F 16/5854: {using shape and object relationship} - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/253: {of extracted features} - G06F 40/194: Calculation of difference between files - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/30: Semantic analysis - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method for cross-modal information retrieval is as follows. First modal information and second modal information are acquired. A first semantic feature of the first modal information and a first attention feature of the first modal information are determined according to a modal feature of the first modal information. A second semantic feature of the second modal information and a second attention feature of the second modal information are determined according to a modal feature of the second modal information. A similarity between the first modal information and the second modal information are determined based on the first attention feature, the second attention feature, the first semantic feature, and the second semantic feature. ## Claims ### Claim 1 A method for cross-modal information retrieval, comprising: acquiring first modal information and second modal information; determining a first semantic feature of the first modal information and a first attention feature of the first modal information according to a modal feature of the first modal information; determining a second semantic feature of the second modal information and a second attention feature of the second modal information according to a modal feature of the second modal information; and determining a similarity between the first modal information and the second modal information based on the first attention feature, the second attention feature, the first semantic feature, and the second semantic feature. ### Claim 2 (depends on 1) The method of claim 1, wherein the first semantic feature comprises a first branch semantic feature and a first overall semantic feature, wherein the first attention feature comprises a first branch attention feature and a first overall attention feature, wherein the second semantic feature comprises a second branch semantic feature and a second overall semantic feature, wherein the second attention feature comprises a second branch attention feature and a second overall attention feature. ### Claim 3 (depends on 2) The method of claim 2, wherein determining the first semantic feature of the first modal information and the first attention feature of the first modal information according to the modal feature of the first modal information comprises: dividing the first modal information into at least one information unit; determining a first modal feature of each information unit of the at least one information unit by performing first modal feature extraction on the each information unit; extracting the first branch semantic feature in a semantic feature space based on the first modal feature of the each information unit; and extracting the first branch attention feature in an attention feature space based on the first modal feature of the each information unit. ### Claim 4 (depends on 3) The method of claim 3, further comprising: determining the first overall semantic feature of the first modal information according to the first branch semantic feature of the each information unit; and determining the first overall attention feature of the first modal information according to the first branch attention feature of the each information unit. ### Claim 5 (depends on 2) The method of claim 2, wherein determining the second semantic feature of the second modal information and the second attention feature of the second modal information according to the modal feature of the second modal information comprises: dividing the second modal information into at least one information unit; determining a second modal feature of each information unit of the at least one information unit by performing second modal feature extraction on the each information unit; extracting the second branch semantic feature in a semantic feature space based on the second modal feature of the each information unit; and extracting the second branch attention feature in an attention feature space based on the second modal feature of the each information unit. ### Claim 6 (depends on 5) The method of claim 5, further comprising: determining the second overall semantic feature of the second modal information according to the second branch semantic feature of the each information unit; and determining the second overall attention feature of the second modal information according to the second branch attention feature of the each information unit. ### Claim 7 (depends on 2) The method of claim 2, wherein determining the similarity between the first modal information and the second modal information based on the first attention feature, the second attention feature, the first semantic feature, and the second semantic feature comprises: determining first attention information according to the first branch attention feature of the first modal information and the first branch semantic feature of the first modal information and the second overall attention feature of the second modal information; determining second attention information according to the second branch attention feature of the second modal information and the second branch semantic feature of the second modal information and the first overall attention feature of the first modal information; and determining the similarity between the first modal information and the second modal information according to the first attention information and the second attention information. ### Claim 8 (depends on 7) The method of claim 7, wherein determining the first attention information according to the first branch attention feature of the first modal information and the first branch semantic feature of the first modal information and the second overall attention feature of the second modal information comprises: determining attention information of the second modal information for each information unit of the first modal information according to the first branch attention feature of the first modal information and the second overall attention feature of the second modal information; and determining the first attention information of the second modal information for the first modal information according to the attention information of the second modal information for the each information unit of the first modal information and the first branch semantic feature of the first modal information. ### Claim 9 (depends on 7) The method of claim 7, wherein determining the second attention information according to the second branch attention feature of the second modal information and the second branch semantic feature of the second modal information and the first overall attention feature of the first modal information comprises: determining attention information of the first modal information for each information unit of the second modal information according to the second branch attention feature of the second modal information and the first overall attention feature of the first modal information; and determining the second attention information of the first modal information for the second modal information according to the attention information of the first modal information for the each information unit of the second modal information and the second branch semantic feature of the second modal information. ### Claim 10 (depends on 1) The method of claim 1, wherein the first modal information is to-be-retrieved information of a first mode, wherein the second modal information is pre-stored information of a second mode, wherein the method further comprises: in response to the similarity meeting a preset condition, determining the second modal information as a retrieval result for the first modal information. ### Claim 11 (depends on 10) The method of claim 10, wherein there are multiple pieces of the second modal information, wherein in response to the similarity meeting the preset condition, determining the second modal information as the retrieval result for the first modal information comprises: acquiring a ranking result by ranking the multiple pieces of the second modal information according to a similarity between the first modal information and each of the multiple pieces of the second modal information; determining second modal information meeting the preset condition according to the ranking result; and determining the second modal information meeting the preset condition as the retrieval result for the first modal information. ### Claim 12 (depends on 11) The method of claim 11, wherein the preset condition comprises any one of: the similarity being greater than a preset value, or a rank of the similarity acquired by ranking similarities in an ascending order being greater than a preset rank. ### Claim 13 (depends on 10) The method of claim 10, further comprising: after determining the second modal information as the retrieval result for the first modal information, outputting the retrieval result to a user terminal. ### Claim 14 (depends on 1) The method of claim 1, wherein the first modal information comprises any one of text information or image information, wherein the second modal information comprises any one of the text information or the image information. ### Claim 15 (depends on 1) The method of claim 1, wherein the first modal information is training sample information of a first mode, wherein the second modal information is training sample information of a second mode, wherein each piece of the training sample information of the first mode and each piece of the training sample information of the second mode form a training sample pair. ### Claim 16 A device for cross-modal information retrieval, comprising a processor and memory, wherein the memory is adapted to storing an instruction executable by the processor, wherein the processor is adapted to executing the executable instruction stored in the memory to implement: acquiring first modal information and second modal information; determining a first semantic feature of the first modal information and a first attention feature of the first modal information according to a modal feature of the first modal information; determining a second semantic feature of the second modal information and a second attention feature of the second modal information according to a modal feature of the second modal information; and determining a similarity between the first modal information and the second modal information based on the first attention feature, the second attention feature, the first semantic feature, and the second semantic feature. ### Claim 17 (depends on 16) The device of claim 16, wherein the first semantic feature comprises a first branch semantic feature and a first overall semantic feature, wherein the first attention feature comprises a first branch attention feature and a first overall attention feature, wherein the second semantic feature comprises a second branch semantic feature and a second overall semantic feature, wherein the second attention feature comprises a second branch attention feature and a second overall attention feature. ### Claim 18 (depends on 17) The device of claim 17, wherein the processor is adapted to determining the first semantic feature of the first modal information and the first attention feature of the first modal information according to the modal feature of the first modal information, by: dividing the first modal information into at least one information unit; determining a first modal feature of each information unit of the at least one information unit by performing first modal feature extraction on the each information unit; extracting the first branch semantic feature in a semantic feature space based on the first modal feature of the each information unit; and extracting the first branch attention feature in an attention feature space based on the first modal feature of the each information unit. ### Claim 19 (depends on 18) The device of claim 18, wherein the processor is adapted to: determining the first overall semantic feature of the first modal information according to the first branch semantic feature of the each information unit; and determining the first overall attention feature of the first modal information according to the first branch attention feature of the each information unit. ### Claim 20 A non-transitory computer-readable storage medium, having stored therein computer program instructions which, when executed by a processor, implement: acquiring first modal information and second modal information; determining a first semantic feature of the first modal information and a first attention feature of the first modal information according to a modal feature of the first modal information; determining a second semantic feature of the second modal information and a second attention feature of the second modal information according to a modal feature of the second modal information; and determining a similarity between the first modal information and the second modal information based on the first attention feature, the second attention feature, the first semantic feature, and the second semantic feature.