# US 12,183,450 B2 · Constructing trained models to associate object in image with description in sentence where feature amount for sentence is derived from structured information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,450 B2 - Document ID: US12183450B2 - Status: Granted - Kind: B2 - Application number: 17/884,525 - Filed: 2022-08-09 - Published: 2023-02-23 - Granted: 2024-12-31 - Earliest claimed: 2021-08-17 - Assignees: FUJIFILM Corporation - Applicants: FUJIFILM Corporation - Inventors: Akimichi Ichinose - Web page: https://patentgrep.com/patent/US12183450B2 ## CPC - G16H 30/40: for processing medical images, e.g. editing - G06T 7/0012: {Biomedical image inspection} - G06T 7/60: Analysis of geometric attributes - G06T 7/74: {involving reference images or patches} - G06T 7/75: {involving models} - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients - G06T 2207/10072: Tomographic images - G06T 2207/10116: X-ray image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30004: Biomedical image processing - G06T 2207/30064: Lung nodule - G06T 2207/30096: Tumor; Lesion - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A processor derives a first feature amount for an object included in an image by a first neural network, structures a sentence including description of the object included in the image to derive structured information for the sentence, and derives a second feature amount for the sentence from the structured information by a second neural network. The processor trains the first neural network and the second neural network such that, in a feature space to which the first feature amount and the second feature amount belong, a distance between the derived first feature amount and second feature amount is reduced in a case in which the object included in the image and the object described in the sentence correspond to each other. ## Claims ### Claim 1 A learning device comprising: at least one processor, wherein the processor derives a first feature amount for an object included in an image by a first neural network, structures a sentence including description of the object included in the image to derive structured information for the sentence, derives a second feature amount for the sentence from the structured information by a second neural network, and constructs a first derivation model that derives a feature amount for the object included in the image and a second derivation model that derives a feature amount for the sentence including the description of the object by training the first neural network and the second neural network such that, in a feature space to which the first feature amount and the second feature amount belong, a distance between the derived first feature amount and second feature amount is smaller in a case in which the object included in the image and the object described in the sentence correspond to each other than a case in which the object included in the image and the object described in the sentence do not correspond to each other. ### Claim 2 (depends on 1) The learning device according to claim 1, wherein the processor trains the first neural network and the second neural network such that, in the feature space, the distance between the derived first feature amount and second feature amount is larger in a case in which the object included in the image and the object described in the sentence do not correspond to each other than a case in which the object included in the image and the object described in the sentence correspond to each other. ### Claim 3 (depends on 1) The learning device according to claim 1, wherein the processor extracts one or more unique expressions for the object from the sentence and determines factuality for the unique expression to derive the unique expression and a determination result of the factuality as the structured information. ### Claim 4 (depends on 3) The learning device according to claim 3, wherein the unique expression represents at least one of a position, an opinion, or a size of the object, and the determination result of the factuality represents any of positivity, negativity, or suspicion for the opinion. ### Claim 5 (depends on 3) The learning device according to claim 3, wherein, in a case in which a plurality of the unique expressions are extracted, the processor further derives a relationship between the unique expressions as the structured information. ### Claim 6 (depends on 5) The learning device according to claim 5, wherein the relationship represents whether or not the plurality of unique expressions are related to each other. ### Claim 7 (depends on 3) The learning device according to claim 3, wherein the processor normalizes the unique expression and the factuality to derive normalized structured information. ### Claim 8 (depends on 1) The learning device according to claim 1, wherein the image is a medical image, the object included in the image is a lesion included in the medical image, and the sentence is an opinion sentence in which an opinion about the lesion is described. ### Claim 9 (depends on 1) An information processing apparatus comprising: at least one processor, wherein the processor derives a first feature amount for one or more objects included in a target image by the first derivation model constructed by the learning device according to claim 1, structures one or more target sentences including description of the object to derive structured information for the target sentence, derives a second feature amount for the target sentence from the structured information for the target sentence by the second derivation model constructed by the learning device according to claim 1, specifies the first feature amount corresponding to the second feature amount based on a distance between the derived first feature amount and second feature amount in a feature space, and displays the object from which the specified first feature amount is derived, in distinction from other regions in the target image. ### Claim 10 (depends on 1) An information processing apparatus comprising: at least one processor, wherein the processor receives input of a target sentence including description of an object, structures the target sentence to derive structured information for the target sentence, derives a second feature amount for the input target sentence from the structured information for the target sentence by the second derivation model constructed by the learning device according to claim 1, refers to a database in which a first feature amount for one or more objects included in a plurality of reference images, which is derived by the first derivation model constructed by the learning device according to claim 1, is associated with each of the reference images, to specify at least one first feature amount corresponding to the second feature amount based on a distance between the first feature amounts for the plurality of reference images and the derived second feature amount in a feature space, and specifies the reference image associated with the specified first feature amount. ### Claim 11 (depends on 9) The information processing apparatus according to claim 9, wherein the processor gives a notification of a unique expression that contributes to association with the first feature amount. ### Claim 12 A learning method comprising: deriving a first feature amount for an object included in an image by a first neural network; structuring a sentence including description of the object included in the image to derive structured information for the sentence; deriving a second feature amount for the sentence from the structured information by a second neural network; and constructing a first derivation model that derives a feature amount for the object included in the image and a second derivation model that derives a feature amount for the sentence including the description of the object by training the first neural network and the second neural network such that, in a feature space to which the first feature amount and the second feature amount belong, a distance between the derived first feature amount and second feature amount is smaller in a case in which the object included in the image and the object described in the sentence correspond to each other than a case in which the object included in the image and the object described in the sentence do not correspond to each other. ### Claim 13 (depends on 1) An information processing method comprising: deriving a first feature amount for one or more objects included in a target image by the first derivation model constructed by the learning device according to claim 1; structuring one or more target sentences including description of the object to derive structured information for the target sentence; deriving a second feature amount for the target sentence from the structured information for the target sentence by the second derivation model constructed by the learning device according to claim 1; specifying the first feature amount corresponding to the second feature amount based on a distance between the derived first feature amount and second feature amount in a feature space; and displaying the object from which the specified first feature amount is derived, in distinction from other regions in the target image. ### Claim 14 (depends on 1) An information processing method comprising: receiving input of a target sentence including description of an object; structuring the target sentence to derive structured information for the target sentence; deriving a second feature amount for the input target sentence from the structured information for the target sentence by the second derivation model constructed by the learning device according to claim 1; referring to a database in which a first feature amount for one or more objects included in a plurality of reference images, which is derived by the first derivation model constructed by the learning device according to claim 1, is associated with each of the reference images, to specify at least one first feature amount corresponding to the second feature amount based on a distance between the first feature amounts for the plurality of reference images and the derived second feature amount in a feature space; and specifying the reference image associated with the specified first feature amount. ### Claim 15 A non-transitory computer-readable storage medium that stores a learning program causing a computer to execute: a procedure of deriving a first feature amount for an object included in an image by a first neural network; a procedure of structuring a sentence including description of the object included in the image to derive structured information for the sentence; a procedure of deriving a second feature amount for the sentence from the structured information by a second neural network; and a procedure of constructing a first derivation model that derives a feature amount for the object included in the image and a second derivation model that derives a feature amount for the sentence including the description of the object by training the first neural network and the second neural network such that, in a feature space to which the first feature amount and the second feature amount belong, a distance between the derived first feature amount and second feature amount is smaller in a case in which the object included in the image and the object described in the sentence correspond to each other than a case in which the object included in the image and the object described in the sentence do not correspond to each other. ### Claim 16 (depends on 1) A non-transitory computer-readable storage medium that stores an information processing program causing a computer to execute: a procedure of deriving a first feature amount for one or more objects included in a target image by the first derivation model constructed by the learning device according to claim 1; a procedure of structuring one or more target sentences including description of the object to derive structured information for the target sentence; a procedure of deriving a second feature amount for the target sentence from the structured information for the target sentence by the second derivation model constructed by the learning device according to claim 1; a procedure of specifying the first feature amount corresponding to the second feature amount based on a distance between the derived first feature amount and second feature amount in a feature space; and a procedure of displaying the object from which the specified first feature amount is derived, in distinction from other regions in the target image. ### Claim 17 (depends on 1) A non-transitory computer-readable storage medium that stores an information processing program causing a computer to execute: a procedure of receiving input of a target sentence including description of an object; a procedure of structuring the target sentence to derive structured information for the target sentence; a procedure of deriving a second feature amount for the input target sentence from the structured information for the target sentence by the second derivation model constructed by the learning device according to claim 1; a procedure of referring to a database in which a first feature amount for one or more objects included in a plurality of reference images, which is derived by the first derivation model constructed by the learning device according to claim 1, is associated with each of the reference images, to specify at least one first feature amount corresponding to the second feature amount based on a distance between the first feature amounts for the plurality of reference images and the derived second feature amount in a feature space; and a procedure of specifying the reference image associated with the specified first feature amount.