# US 2026/0030724 A1 · Image reconstruction with multimodal fusion and physics-informed neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0030724 A1
- Document ID: US20260030724A1
- Status: Published application
- Kind: A1
- Application number: 19/275,418
- Filed: 2025-07-21
- Published: 2026-01-29
- Earliest claimed: 2024-07-23
- Assignees: University of Florida Research Foundation, Inc.
- Applicants: University of Florida Research Foundation, Incorporated
- Inventors: Navid Asadi-Zanjani; Nitin Varshney; Shajib Ghosh; Antika Roy; Patrick Craig; Md Mahfuz Al Hasan
- Web page: https://patentgrep.com/patent/US20260030724A1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 12/20: Inverse problem, i.e. transformations from projection space into object space
- G06T 2207/10024: Color image
- G06T 2207/10116: X-ray image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30148: Semiconductor; IC; Wafer
- G06T 2210/41: Medical
- G06T 2211/421: Filtered back projection [FBP]
- G06T 2211/441: AI-based methods, deep learning or artificial neural networks
- G06T 2211/452: involving suppression of scattered radiation or scatter correction
- G06T 2211/464: Dual or multimodal imaging, i.e. combining two or more imaging modalities
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method comprising receiving a plurality of images from a multi-modal imaging system; generating a plurality of filtered measurements by performing multi-modal spectral fusion of the plurality of images; and generating, using a physics-informed neural network (PINN) trained based on one or more physical principles associated with X-ray attenuation or scattering, a reconstructed object image based on the plurality of filtered measurements, wherein generating the reconstructed object image comprises (i) generating, using the PINN, a system matrix for an X-ray imaging forward model by refining one or more coefficients of the system matrix based on a physics-informed loss function, and (ii) generating, using the X-ray imaging forward model and based on the plurality of filtered measurements, the reconstructed object image.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving, by one or more processors, a plurality of images from a multi-modal imaging system;
generating, by the one or more processors, a plurality of filtered measurements by performing multi-modal spectral fusion of the plurality of images; and
generating, by the one or more processors and using a physics-informed neural network (PINN) trained based on one or more physical principles associated with X-ray attenuation or scattering, a reconstructed object image based on the plurality of filtered measurements, wherein generating the reconstructed object image comprises (i) generating, using the PINN, a system matrix for an X-ray imaging forward model by refining one or more coefficients of the system matrix based on a physics-informed loss function, and (ii) generating, using the X-ray imaging forward model and based on the plurality of filtered measurements, the reconstructed object image.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the multi-modal imaging system is configured to provide the plurality of images via optical acquisition and X-ray acquisition.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein (i) a first set of one or more images from the plurality of images comprises one or more red, green, and blue (RGB) images that are provided by the optical acquisition and (ii) a second set of one or more images from the plurality of images comprises one or more X-ray images that are provided by the X-ray acquisition.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein performing the multi-modal spectral fusion comprises decomposing the plurality of images into one or more corresponding low-frequency components (LFCs) and one or more corresponding high-frequency components (HFCs).

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein (i) the one or more corresponding LFCs comprise one or more general shapes and (ii) the one or more corresponding HFCs comprise one or more fine details.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein generating the plurality of filtered measurements further comprises filtering, using an attentional high-frequency feature fusion network, a first set of one or more HFCs associated with a first set of one or more images corresponding to X-ray acquisition from the plurality of images with a second set of one or more HFCs associated with a second set of one or more images corresponding to optical acquisition from the plurality of images.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein generating the reconstructed object image further comprises supervising based on a ground truth generated by one or more model-based iterative reconstruction (MBIR) or filtered back projection algorithms.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more coefficients of the system matrix corresponds to one or more of geometric efficiency, detector sensitivity, electronic efficiency, an attenuation term, a scatter correction factor, a beam hardening correction factor, or a detector noise factor.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the physics-informed loss function comprises one or more of a geometric efficiency-related adjustment or a scatter correction element.

### Claim 10

A system comprising
one or more processors and
at least one memory storing processor-executable instructions that, when executed by any of the one or more processors, causes the one or more processors to perform operations comprising:
receiving a plurality of images from a multi-modal imaging system;
generating a plurality of filtered measurements by performing multi-modal spectral fusion of the plurality of images; and
generating, using a physics-informed neural network (PINN) trained based on one or more physical principles associated with X-ray attenuation or scattering, a reconstructed object image based on the plurality of filtered measurements, wherein generating the reconstructed object image comprises (i) generating, using the PINN, a system matrix for an X-ray imaging forward model by refining one or more coefficients of the system matrix based on a physics-informed loss function, and (ii) generating, using the X-ray imaging forward model and based on the plurality of filtered measurements, the reconstructed object image.

### Claim 11 (depends on 10)

The system of claim 10, wherein the multi-modal imaging system is configured to provide the plurality of images via optical acquisition and X-ray acquisition.

### Claim 12 (depends on 11)

The system of claim 11, wherein (i) a first set of one or more images from the plurality of images comprises one or more red, green, and blue (RGB) images that are provided by the optical acquisition and (ii) a second set of one or more images from the plurality of images comprises one or more X-ray images that are provided by the X-ray acquisition.

### Claim 13 (depends on 10)

The system of claim 10, wherein performing the multi-modal spectral fusion comprises decomposing the plurality of images into one or more corresponding low-frequency components (LFCs) and one or more corresponding high-frequency components (HFCs).

### Claim 14 (depends on 13)

The system of claim 13, wherein (i) the one or more corresponding LFCs comprise one or more general shapes and (ii) the one or more corresponding HFCs comprise one or more fine details.

### Claim 15 (depends on 13)

The system of claim 13, wherein generating the plurality of filtered measurements further comprises filtering, using an attentional high-frequency feature fusion network, a first set of one or more HFCs associated with a first set of one or more images corresponding to X-ray acquisition from the plurality of images with a second set of one or more HFCs associated with a second set of one or more images corresponding to optical acquisition from the plurality of images.

### Claim 16 (depends on 10)

The system of claim 10, wherein generating the reconstructed object image further comprises supervising based on a ground truth generated by one or more model-based iterative reconstruction (MBIR) or filtered back projection algorithms.

### Claim 17 (depends on 10)

The system of claim 10, wherein the one or more coefficients of the system matrix corresponds to one or more of geometric efficiency, detector sensitivity, electronic efficiency, an attenuation term, a scatter correction factor, a beam hardening correction factor, or a detector noise factor.

### Claim 18 (depends on 10)

The system of claim 10, wherein the physics-informed loss function comprises one or more of a geometric efficiency-related adjustment or a scatter correction element.

### Claim 19

A computer-implemented method comprising:
receiving, by one or more processors, an input scanning acoustic microscopy (SAM) image; and
generating, by the one or more processors and using a physics-informed neural network (PINN) trained based on one or more acoustic wave physics constraints, an enhanced output image of the input SAM image in accordance with a hybrid loss function, wherein the hybrid loss function comprises a physics loss function and a self-consistency loss function.

### Claim 20 (depends on 19)

The computer-implemented method of claim 19, wherein the physics loss function is associated with (i) a density of an integrated circuit advanced packaging material, (ii) a wave propagation speed, and (iii) one or more second-order derivatives that corresponds to a wavefront with respect to time and spatial coordinates.
