# US 12,217,170 B2 · Data handling and machine learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,170 B2 - Document ID: US12217170B2 - Status: Granted - Kind: B2 - Application number: 17/216,057 - Filed: 2021-03-29 - Published: 2021-09-30 - Granted: 2025-02-04 - Earliest claimed: 2020-03-27 - Assignees: Battelle Memorial Institute - Applicants: Battelle Memorial Institute - Inventors: Anthony George; Nicholas Darby; Jeremy Bellay; David Collins; Katie Liszewski; Amir Rahimi - Web page: https://patentgrep.com/patent/US12217170B2 ## CPC - G06F 21/71: to assure secure computing or processing of information - G06N 3/08: Learning methods - G06F 21/562: {Static detection} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06N 3/096: Transfer learning - G01N 2021/8845: {Multiple wavelengths of illumination or detection} - G01N 2021/8883: {involving the calculation of gauges, generating models} - G01N 2021/95638: {for PCB's} - G01N 2201/1296: using neural networks - G01R 31/281: {Specific types of tests or tests for a specific type of fault, e.g. thermal mapping, shorts testing (G01R31/2818 takes precedence)} - G01R 31/2837: {Characterising or performance testing, e.g. of frequency response (transient response G01R27/28)} - G01R 31/2846: {using hard- or software simulation or using knowledge-based systems, e.g. expert systems, artificial intelligence or interactive algorithms} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q) - G01R: MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES (indicating correct tuning of resonant circuits H03J3/12) ## Abstract A method implemented by a software for a multimodal evaluation engine stored on a memory is provided herein. The software is executable by a processor coupled to the memory to cause the method. The method includes receiving multimodal signatures of an object of interest from inspection elements and processing the multimodal signatures to transform the multimodal signatures into formats. The method also includes generating data representations of the formats and detecting whether anomalies are present within the object of interest based on the data representations. ## Claims ### Claim 1 A method implemented by a software for a multimodal evaluation engine stored on a memory and executable by one or more processors coupled to the memory, the method comprising: receiving, by the multimodal evaluation engine, a plurality of multimodal signatures of an object of interest from one or more inspection elements, wherein the plurality of multimodal signatures comprise visible light, infrared, electromagnetic interference, and laser profilometry; processing, by the multimodal evaluation engine, the plurality of multimodal signatures to transform the plurality of multimodal signatures into one or more formats; generating, by the multimodal evaluation engine, data representations of the one or more formats; and detecting, by the multimodal evaluation engine, whether one or more anomalies are present within the object of interest based on the data representations. ### Claim 2 (depends on 1) The method of claim 1, wherein processing the plurality of multimodal signatures comprises executing a design information extraction operation. ### Claim 3 (depends on 1) The method of claim 1, wherein processing the plurality of multimodal signatures comprises executing a design information recovery operation. ### Claim 4 (depends on 1) The method of claim 1, wherein processing the plurality of multimodal signatures comprises executing a spatial risk scoring operation. ### Claim 5 (depends on 1) The method of claim 1, wherein the data representations include hyperspectral-multimodal scans of the object of interest, assessments of a bill of materials of the object of interest, determinations of how components are connected within the object of interest, or vulnerability information for the object of interest. ### Claim 6 (depends on 1) The method of claim 1, wherein processing the plurality of multimodal signatures comprises labeling a first subset of an unlabeled dataset of the plurality of multimodal signatures and training an artificial neural network on the labeled first subset. ### Claim 7 (depends on 1) The method of claim 1, wherein processing the plurality of multimodal signatures comprises: generating a plurality of labeled signatures from the plurality of multimodal signatures; grouping each of the plurality of labeled signatures into training tiles of a fixed physical size; and training an artificial neural network to identify components of the object of interest based on the training tiles. ### Claim 8 (depends on 1) The method of claim 1, wherein processing the plurality of multimodal signatures comprises: selecting training data having m modalities from the plurality of multimodal signatures; grouping the training data into training tiles of a fixed physical size; and training m conditional generative adversarial networks to generate candidate tiles for each of the m modalities. ### Claim 9 A system comprising: a memory configured to store a software for a multimodal evaluation engine; and one or more processors coupled to the memory, the one or more processors configured to execute the software for the multimodal evaluation engine to cause the system to perform: receiving a plurality of multimodal signatures of an object of interest from one or more inspection elements, wherein the plurality of multimodal signatures comprise visible light, infrared, electromagnetic interference, and laser profilometry; processing the plurality of multimodal signatures to transform the plurality of multimodal signatures into one or more formats; generating data representations of the one or more formats; and detecting whether one or more anomalies are present within the object of interest based on the data representations. ### Claim 10 (depends on 9) The system of claim 9, wherein processing the plurality of multimodal signatures comprises executing a design information extraction operation. ### Claim 11 (depends on 9) The system of claim 9, wherein processing the plurality of multimodal signatures comprises executing a design information recovery operation. ### Claim 12 (depends on 9) The system of claim 9, wherein processing the plurality of multimodal signatures comprises executing a spatial risk scoring operation. ### Claim 13 (depends on 9) The system of claim 9, wherein the data representations include hyperspectral-multimodal scans of the object of interest, assessments of a bill of materials of the object of interest, determinations of how components are connected within the object of interest, or vulnerability information for the object of interest. ### Claim 14 (depends on 9) The system of claim 9, wherein processing the plurality of multimodal signatures comprises labeling a first subset of an unlabeled dataset of the plurality of multimodal signatures and training an artificial neural network on the labeled first subset. ### Claim 15 (depends on 9) The system of claim 9, wherein processing the plurality of multimodal signatures comprises: generating a plurality of labeled signatures from the plurality of multimodal signatures; breaking/grouping each of the plurality of labeled signatures into training tiles of a fixed physical size; and training an artificial neural network to identify components of the object of interest based on the training tiles. ### Claim 16 (depends on 9) The system of claim 9, wherein processing the plurality of multimodal signatures comprises: selecting training data having m modalities from the plurality of multimodal signatures; breaking/grouping the training data into training tiles of a fixed physical size; and training m conditional generative adversarial networks to generate candidate tiles for each of the m modalities. ### Claim 17 A computer readable medium storing a software for a multimodal evaluation engine, the software being executable by one or more processors to cause the multimodal evaluation engine to perform: receiving a plurality of multimodal signatures of an object of interest from one or more inspection elements, wherein the plurality of multimodal signatures comprise visible light, infrared, electromagnetic interference, and laser profilometry; processing the plurality of multimodal signatures to transform the plurality of multimodal signatures into one or more formats; generating data representations of the one or more formats; and detecting whether one or more anomalies are present within the object of interest based on the data representations.