# US 2021/0241478 A1 · Deep Photometric Learning (DPL) Systems, Apparatus and Methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241478 A1
- Document ID: US20210241478A1
- Status: Published application
- Kind: A1
- Application number: 17/166,976
- Filed: 2021-02-03
- Published: 2021-08-05
- Earliest claimed: 2020-02-03
- Assignees: Nanotronics Imaging, Inc.
- Applicants: Nanotronics Imaging, Inc.
- Inventors: Matthew C. Putman; Vadim Pinskiy; Tanaporn Na Narong; Denis Sharoukhov; Tonislav Ivanov
- Web page: https://patentgrep.com/patent/US20210241478A1

## CPC

- G06T 7/586: from multiple light sources, e.g. photometric stereo
- G02B 21/367: {providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- H04N 23/56: provided with illuminating means
- G06T 2207/10056: Microscopic image
- G06T 2207/10152: Varying illumination
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An imaging system is disclosed herein. The imaging system includes an imaging apparatus and a computing system. The imaging apparatus includes a plurality of light sources positioned at a plurality of positions and a plurality of angles relative to a stage configured to support a specimen. The imaging apparatus is configured to capture a plurality of images of a surface of the specimen. The computing system in communication with the imaging apparatus. The computing system configured to generate a 3D-reconstruction of the surface of the specimen by receiving, from the imaging apparatus, the plurality of images of the surface of the specimen, generating, by the imaging apparatus via a deep learning model, a height map of the surface of the specimen based on the plurality of images, and outputting a 3D-reconstruction of the surface of the specimen based on the height map generated by the deep learning model.

## Claims

### Claim 1

An imaging system, comprising:
an imaging apparatus comprising a plurality of light sources positioned at a plurality of positions and a plurality of angles relative to a stage configured to support a specimen, wherein the imaging apparatus is configured to capture a plurality of images of a surface of the specimen; and
a computing system in communication with the imaging apparatus, the computing system configured to generate a 3D-reconstruction of the surface of the specimen by:
receiving, from the imaging apparatus, the plurality of images of the surface of the specimen;
generating, by the imaging apparatus via a deep learning model, a height map of the surface of the specimen based on the plurality of images; and
outputting a 3D-reconstruction of the surface of the specimen based on the height map generated by the deep learning model.

### Claim 2 (depends on 1)

The imaging system of claim 1, wherein the deep learning model comprises a convolutional neural network.

### Claim 3 (depends on 2)

The imaging system of claim 2, wherein the convolutional neural network is configured to receive, as input, the plurality of images and output a single image.

### Claim 4 (depends on 1)

The imaging system of claim 1, wherein the computing system is further configured to train the deep learning model by:
generating a plurality of training data sets, wherein the plurality of training data sets comprises a plurality of synthetic height maps; and
training the deep learning model using the plurality of training data sets.

### Claim 5 (depends on 4)

The imaging system of claim 4, wherein the plurality of training data sets comprises:
a first training data set without noise and without tilt;
a second training data set comprising a fixed amount of noise and no tilt; and
a third training data set comprising tilt and no noise.

### Claim 6 (depends on 1)

The imaging system of claim 1, wherein the computing system is further configured to test the deep learning model by:
generating a plurality of microscopic images of an example specimen; and
testing an accuracy of the deep learning model using the plurality of microscopic images.

### Claim 7 (depends on 6)

The imaging system of claim 6, wherein the plurality of microscopic images comprise images taken with a fixed camera angle and varying illumination directions.

### Claim 8

A method of generating a 3D-reconstruction of a surface of a specimen, comprising:
receiving, by a computing system from an imaging apparatus, a plurality of images of a surface of a specimen positioned on a stage of the imaging apparatus;
generating, by the computing system via a deep learning model, a height map of the surface of the specimen based on the plurality of images; and
outputting, by the computing system, a 3D-reconstruction of the surface of the specimen based on the height map generated by the deep learning model.

### Claim 9 (depends on 8)

The method of claim 8, wherein the deep learning model comprises a convolutional neural network.

### Claim 10 (depends on 9)

The method of claim 9, wherein the convolutional neural network is configured to receive, as input, the plurality of images and output a single image.

### Claim 11 (depends on 8)

The method of claim 8, further comprising generating, by the computing system, a plurality of training data sets, wherein the plurality of training data sets comprises a plurality of synthetic height maps; and
training, by the computing system, the deep learning model using the plurality of training data sets.

### Claim 12 (depends on 11)

The method of claim 11, wherein the plurality of training data sets comprises:
a first training data set without noise and without tilt;
a second training data set comprising a fixed amount of noise and no tilt; and
a third training data set comprising tilt and no noise.

### Claim 13 (depends on 8)

The method of claim 8, further comprising generating, by the computing system, a plurality of microscopic images of an example specimen; and
testing, by the computing system, an accuracy of the deep learning model using the plurality of microscopic images.

### Claim 14 (depends on 13)

The method of claim 13, wherein the plurality of microscopic images comprises images taken with a fixed camera angle and varying illumination directions.

### Claim 15

A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by one or more processors, cause a computing system to perform operations, comprising:
receiving, by the computing system from an imaging apparatus, a plurality of images of a surface of a specimen positioned on a stage of the imaging apparatus;
generating, by the computing system via a deep learning model, a height map of the surface of the specimen based on the plurality of images; and
outputting, by the computing system, a 3D-reconstruction of the surface of the specimen based on the height map generated by the deep learning model.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the deep learning model comprises a convolutional neural network.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, further comprising
generating, by the computing system, a plurality of training data sets, wherein the plurality of training data sets comprises a plurality of synthetic height maps; and
training, by the computing system, the deep learning model using the plurality of training data sets.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein the plurality of training data sets comprises:
a first training data set without noise and without tilt;
a second training data set comprising a fixed amount of noise and no tilt; and
a third training data set comprising tilt and no noise.

### Claim 19 (depends on 15)

The non-transitory computer readable medium of claim 15, further comprising
generating, by the computing system, a plurality of microscopic images of an example specimen; and
testing, by the computing system, an accuracy of the deep learning model using the plurality of microscopic images.

### Claim 20 (depends on 19)

The non-transitory computer readable medium of claim 19, wherein the plurality of microscopic images comprises images taken with a fixed camera angle and varying illumination directions.