# US 2019/0378267 A1 · Automated material characterization system including conditional generative adversarial networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378267 A1
- Document ID: US20190378267A1
- Status: Published application
- Kind: A1
- Application number: 16/549,332
- Filed: 2019-08-23
- Published: 2019-12-12
- Earliest claimed: 2017-09-25
- Assignees: United Technologies Corporation
- Applicants: United Technologies Corporation
- Inventors: Michael J. Giering; Ryan B. Noraas; Kishore K. Reddy; Edgar A. Bernal
- Web page: https://patentgrep.com/patent/US20190378267A1

## CPC

- G06V 10/82: using neural networks
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24133: {Distances to prototypes}
- G06F 18/251: {of input or preprocessed data}
- G06T 7/0004: {Industrial image inspection}
- G06T 7/0006: {using a design-rule based approach}
- G06T 7/001: {using an image reference approach}
- G06T 7/12: Edge-based segmentation
- G06T 7/13: Edge detection
- G06T 7/174: involving the use of two or more images
- G06V 10/267: {by performing operations on regions, e.g. growing, shrinking or watersheds}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/776: Validation; Performance evaluation
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20152: Watershed segmentation
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30136: Metal
- G06T 2207/30164: Workpiece; Machine component
- G06T 2207/30242: Counting objects in image
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A material characterization system includes an imaging unit, a material characterization controller, and an imaging unit controller. The electronic imaging unit generates a test image of a specimen composed of a material. The electronic material characterization controller determines values of a plurality of parameters and maps the parameters to corresponding ground truth labeled outputs. The mapped parameters are applied to at least one test image to predict a presence of at least one target attribute of the specimen in response to applying the learned parameters. The test image is convert to a selected output image format so as to generate a synthetic image including the predicted at least one attribute. The electronic imaging unit controller performs a material characterization analysis that characterizes the material of the specimen based on the predicted at least one attribute included in the synthetic image.

## Claims

### Claim 1

A material characterization system comprising:
an electronic imaging unit configured to generate at least one test input image of a specimen composed of a material;
an electronic material characterization controller configured to determine values of a plurality of functional parameters that specify a functional mapping, to map the at least one test input image of the specimen to at least one output image based on the functional mapping specified by the determined parameters, to predict a presence of at least one target attribute of the specimen in response to applying the functional mapping and analyzing the at least one output image, and to convert the at least one output image to a selected output image format so as to generate a synthetic image including the predicted at least one attribute; and
an electronic imaging unit controller configured to perform a material characterization analysis that characterizes the material of the specimen based on the predicted at least one attribute included in the synthetic image;
the material characterization controller includes a plurality of conditional generative adversarial networks (CGANs) and, each CGAN and among the plurality of CGANs is configured to predict the presence of a respective attribute;
the plurality of CGANs includes:
a first CGAN configured to predict a presence of at least one material characteristic in the specimen, and to generate a first synthetic image including the predicted at least one material characteristic; and
a second CGAN configured to predict a physical characteristic of the at least one material characteristic, and to generate a second synthetic image including the predicted physical characteristic of the at least one material characteristic; and
the imaging unit controller includes an electronic image fusing unit that superimposes the second synthetic image with the first synthetic image to generate the fused synthetic image,
wherein the superimposition of the predicted physical characteristics with the at least one material characteristic identifies an actual material characteristic of the specimen.

### Claim 2 (depends on 1)

The material characterization system of claim 1, wherein the synthetic image is a fused synthetic image including a plurality of predicted attributes that are different from one another.

### Claim 3 (depends on 1)

The material characterization system of claim 1, wherein the material characteristic is at least one grain element in the specimen, and wherein the physical characteristic is an edge location of the at least one grain element.

### Claim 4 (depends on 3)

The material characterization system of claim 3, wherein the superimposition of predicted edge locations with predicted grain elements identifies at least one individually segmented grain in the specimen.

### Claim 5 (depends on 4)

The material characterization system of claim 4, wherein the imaging unit controller is configured to perform at least one post-processing operation on the fused synthetic image.

### Claim 6 (depends on 5)

The material characterization system of claim 5, wherein at least one post-processing operation on the fused synthetic image includes at least one of a watershed analysis, a post-segmentation operation, a synthetic image correcting operation, and a grain filtering operation.

### Claim 7 (depends on 5)

The material characterization system of claim 5, wherein the material characterization analysis includes characterizing at least one of a strength of the material, a fatigue rate of the material, and fracture growth rate.

### Claim 8 (depends on 5)

The material characterization system of claim 5, wherein the imaging unit controller generates characterized image data based on the fused synthetic image and the material characterization analysis.

### Claim 9 (depends on 8)

The material characterization system of claim 8, further comprising a display unit that displays a characterized image based on the characterized image data.

### Claim 10 (depends on 9)

The material characterization system of claim 9, wherein the characterized image includes at least one graphical indicator that indicates at least one of the strength of the material, the fatigue rate of the material, and fracture growth rate.

### Claim 11 (depends on 10)

The material characterization system of claim 10, wherein the material characterization controller executes a training process prior to generating the test image.

### Claim 12 (depends on 11)

The material characterization system of claim 11, wherein the material characterization controller learns the modeled parameters in response to identifying known attributes of a previously analyzed specimen based on a comparison between the real image and the known ground truth image.

### Claim 13 (depends on 12)

The material characterization system of claim 12, wherein the learned ground truth images include the known ground truth attributes.

### Claim 14

A method of characterizing a material of a specimen, the method comprising:
generating, via an electronic imaging unit, at least one test image of the specimen;
determining values of a functional plurality of parameters that specify a functional mapping, and mapping the at least one test input image of the specimen to at least one output image based on the functional mapping specified by the determined parameters;
predicting, via an electronic material characterization controller, a presence of at least one target attribute of the specimen in response to applying the functional mapping specified by the determined parameters to analyze the at least one output image; and
converting, via the electronic material characterization controller, the at least one output image to a selected output image format so as to generate a synthetic image that includes the predicted at least one attribute;
performing, via the electronic imaging unit controller, a material characterization analysis that determines at least one physical characterization of the material of the specimen based on the predicted at least one attribute included in the synthetic image;
generating, via the electronic imaging unit controller, characterized image data based on the fused synthetic image and the material characterization analysis; and
displaying, via a display unit, a characterized image based on the characterized image data, the characterized image including a graphical indicator indicating the at least one physical characterization.

### Claim 15 (depends on 14)

The method of claim 14, wherein generating the synthetic image includes fusing together a plurality of predicted attributes that are different from one another to generate a fused synthetic image.

### Claim 16 (depends on 15)

The method of claim 15, wherein predicting the presence of at least one target attribute of the specimen further comprises:
predicting, via a first conditional generative adversarial network (CGAN), a presence of at least one grain element in the specimen, and generating, via the first CGAN, a first synthetic image including the predicted at least one grain element; and
predicting, via a second CGAN, an edge location of the at least one grain element, and generating, via the second CGAN, a second synthetic image including the predicted edge location of the at least one grain element.

### Claim 17 (depends on 16)

The method of claim 16, wherein fusing together a plurality of predicted attributes includes superimposing the second synthetic image with the first synthetic image to generate the fused synthetic image.

### Claim 18 (depends on 17)

The method of claim 17, wherein predicting the at least one attribute further comprises identifying at least one individually segmented grain and included in the specimen represented by the test image based on the superimposition of the predicted edge location and the predicted at least one grain element.

### Claim 19

A method of training a material characterization system, the method comprising:
training a first conditional generative adversarial network (CGAN) network to detect a first target attribute;
training a second CGAN to detect a second target attribute different from the first target attribute;
storing, in a database, a plurality of previously generated real images and a plurality of previously analyzed ground truth images that correspond to a given real image, the ground truth images including ground truth attributes having a one-to-one mapping to real attributes included in a given real image among the previously generated real images; and
performing, via a material characterization controller, a plurality of iterations of comparing the ground truth images to a corresponding real image,
wherein the material characterization controller learns to distinguish target attributes from non-target attributes in response to performing the plurality of iterations.

### Claim 20 (depends on 19)

The method of claim 19, wherein the first target attribute is a grain element, and wherein the second target attribute is selected from a group comprising edge locations of a detected grain element and boundaries of a detected grain element.
