# US 11,907,812 B2 · Apparatus for determining 3-dimensional atomic level structure and method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,812 B2 - Document ID: US11907812B2 - Status: Granted - Kind: B2 - Application number: 17/132,762 - Filed: 2020-12-23 - Published: 2022-06-16 - Granted: 2024-02-20 - Earliest claimed: 2020-12-16 - Assignees: Korea Advanced Institute of Science and Technology - Applicants: Korea Advanced Institute of Science and Technology - Inventors: Yongsoo Yang; Juhyeok Lee; Chaehwa Jeong - Web page: https://patentgrep.com/patent/US11907812B2 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 20/00: Machine learning - G01N 23/04: and forming images of the material - G01N 23/046: using tomography, e.g. computed tomography [CT] - G01N 24/087: {Structure determination of a chemical compound, e.g. of a biomolecule such as a protein} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - H01J 37/26: Electron or ion microscopes; Electron or ion diffraction tubes - G01N 2223/102: beta or electrons - G01N 2223/401: image processing - G01N 2223/646: flaws, defects - G01N 2223/6462: microdefects - G06N 3/045: Combinations of networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q) - H01J: ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS (spark-gaps H01T; electric arc lamps H05B31/00; particle accelerators H05H) ## Abstract A data generating method includes: an atomic model generating step of generating one or more three-dimensional atomic models corresponding to a nanomaterial to be measured; a three-dimensional data generating step of generating three-dimensional atomic level structure volume data corresponding to the nanomaterial to be measured based on the one or more three-dimensional atomic model; a tilt series generating step of generating a tilt series by simulating three-dimensional tomography for a plurality of different angles in a predetermined angle range for at least some of the three-dimensional atomic level structure volume data; and a three-dimensional atomic structure tomogram volume data generating step of generating a three-dimensional atomic structure tomogram volume data set by performing three-dimensional reconstruction on at least some of the three-dimensional atomic level structure volume data based on the tilt series. ## Claims ### Claim 1 A data generating apparatus, comprising: an atomic model generator that generates one or more three-dimensional atomic models corresponding to a nanomaterial to be measured; a three-dimensional data generator that generates three-dimensional atomic level structure volume data corresponding to the nanomaterial to be measured based on the one or more three-dimensional atomic model; a tilt series generator that generates a tilt series by simulating three-dimensional tomography for a plurality of different angles in a predetermined angle range for at least some of the three-dimensional atomic level structure volume data; and a three-dimensional atomic structure tomogram volume data generator that generates a three-dimensional atomic structure tomogram volume data set by performing three-dimensional reconstruction on at least some of the three-dimensional atomic level structure volume data based on the tilt series. ### Claim 2 (depends on 1) The data generating apparatus of claim 1, wherein the atomic model generator generates a three-dimensional atomic model based on a theoretical crystal structure of the nanomaterial to be measured and a three-dimensional atomic model based on an amorphous structure. ### Claim 3 (depends on 1) The data generating apparatus of claim 1, wherein the atomic model generator performs at least one of actualization correction processes including random atomic defect provision, random rotation or movement of the model structure, spatial change, atomic density change, and rand shape cropping on the one or more three-dimensional atomic models. ### Claim 4 (depends on 1) The data generating apparatus of claim 1, wherein the three-dimensional data generator generates the three-dimensional atomic level structure volume data by deriving a three-dimensional volume of atomic potential based on the one or more three-dimensional models and acquiring a result of convolution of a Gaussian distribution having the atomic potential and a predetermined standard deviation at each atomic position. ### Claim 5 (depends on 1) The data generating apparatus of claim 1, wherein the tilt series generator generates the tilt series by acquiring a linear projection from a three-dimensional atomic potential volume at a plurality of different tilt angles within the predetermined angle range for at least some of the three-dimensional atomic level structure volume data. ### Claim 6 (depends on 1) The data generating apparatus of claim 1, wherein the three-dimensional atomic structure tomogram volume data generator three-dimensionally reconstructs on the tilt series using a predetermined algorithm, and selects axis convention during the three-dimensional reconstruction so that at least one of an x-direction axis, a y-direction axis, and a z-direction axis is parallel to a missing wedge direction. ### Claim 7 A data generating method, comprising: an atomic model generating step of generating one or more three-dimensional atomic models corresponding to a nanomaterial to be measured; a three-dimensional data generating step of generating three-dimensional atomic level structure volume data corresponding to the nanomaterial to be measured based on the one or more three-dimensional atomic model; a tilt series generating step of generating a tilt series by simulating three-dimensional tomography for a plurality of different angles in a predetermined angle range for at least some of the three-dimensional atomic level structure volume data; and a three-dimensional atomic structure tomogram volume data generating step of generating a three-dimensional atomic structure tomogram volume data set by performing three-dimensional reconstruction on at least some of the three-dimensional atomic level structure volume data based on the tilt series. ### Claim 8 (depends on 7) The data generating method of claim 7, wherein the atomic model generating step further includes generating a three-dimensional atomic model based on a theoretical crystal structure of the nanomaterial to be measured and a three-dimensional atomic model based on an amorphous structure. ### Claim 9 (depends on 7) The data generating method of claim 7, wherein the atomic model generating step further includes performing at least one of actualization correction processes including random atomic defect provision, random rotation or movement of the model structure, spatial change, atomic density change, and rand shape cropping on the one or more three-dimensional atomic models. ### Claim 10 (depends on 7) The data generating method of claim 7, wherein the three-dimensional data generating step further includes generating the three-dimensional atomic level structure volume data by deriving a three-dimensional volume of atomic potential based on the one or more three-dimensional models and acquiring a result of convolution of a Gaussian distribution having the atomic potential and a predetermined standard deviation at each atomic position. ### Claim 11 (depends on 7) The data generating method of claim 7, wherein the tilt series generating step further includes generating the tilt series by acquiring a linear projection from a three-dimensional atomic potential volume at a plurality of different tilt angles within the predetermined angle range for at least some of the three-dimensional atomic level structure volume data. ### Claim 12 (depends on 7) The data generating method of claim 7, wherein in the three-dimensional atomic structure tomogram volume data generating step, the three-dimensional reconstruction is performed on the tilt series using a predetermined algorithm, and an axis convention is selected during the three-dimensional reconstruction so that at least one of an x-direction axis, a y-direction axis, and a z-direction axis is parallel to a missing wedge direction. ### Claim 13 An apparatus for determining a three-dimensional atomic level structure, comprising: an atomic model generator that generates one or more three-dimensional atomic models corresponding to a nanomaterial to be measured; a three-dimensional learning data generator that generates three-dimensional atomic structure tomogram volume data corresponding to the nanomaterial to be measured as three-dimensional learning data by simulating and reconstructing three-dimensional tomography based on the one or more three-dimensional atomic model; a deep learning filter generator/learner that generates a deep learning filter by training a machine learning model using the three-dimensional learning data; and a three-dimensional atomic level structure determiner that generates second three-dimensional atomic structure tomogram volume data, in which an atomic arrangement region with insufficient data is corrected, by filtering first three-dimensional atomic structure tomogram volume data acquired through electron microscopy measurement of the nanomaterial to be measured with the deep learning filter, and determines a three-dimensional atomic level nanostructure of the nanomaterial to be measured based on the second three-dimensional atomic structure tomogram volume data. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the three-dimensional learning data generator includes: A three-dimensional data generator that generates three-dimensional atomic level structure volume data corresponding to the nanomaterial to be measured based on the one or more three-dimensional atomic model; a tilt series generator that generates a tilt series by simulating three-dimensional tomography for a plurality of different angles in a predetermined angle range for at least some of the three-dimensional atomic level structure volume data; and a three-dimensional atomic structure tomogram volume data generator that generates a three-dimensional atomic structure tomogram volume data set by performing three-dimensional reconstruction on at least some of the three-dimensional atomic level structure volume data based on the tilt series, and the reconstructed three-dimensional atomic structure tomogram volume data set forms a three-dimensional learning data set for the training the deep learning filter. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the three-dimensional data generator generates the three-dimensional atomic level structure volume data by deriving a three-dimensional volume of atomic potential based on the one or more three-dimensional models and acquiring a result of convolution of a Gaussian distribution having the atomic potential and a predetermined standard deviation at each atomic position. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the three-dimensional data generator generates, as a three-dimensional target data set, three-dimensional atomic level structure volume data acquired by deriving the three-dimensional volume of the atomic potential based on the one or more three-dimensional models and positioning a Gaussian distribution having a standard deviation equal to or different from the predetermined standard deviation at each atomic position in the three-dimensional volume. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the deep learning filter generator/learner calculates an error by comparing a result output from the deep learning filter through forward-propagation based on the three-dimensional learning input data set with the three-dimensional target data set, updates hyperparameters of the deep learning filter in a manner, in which the error is reduced, by calculating a gradient of the error in a reverse direction from an output layer to an input layer based on the error, and trains the deep learning filter and optimizes the parameters of the filter to determine the three-dimensional atomic level structure, by repeating the process. ### Claim 18 A method for determining a three-dimensional atomic level structure, the method comprising: an atomic model generating step of generating one or more three-dimensional atomic models corresponding to a nanomaterial to be measured; a three-dimensional learning data generating step of generating three-dimensional atomic level structure volume data corresponding to the nanomaterial to be measured as three-dimensional learning data by simulating and reconstructing three-dimensional tomography based on the one or more three-dimensional atomic model; a deep learning filter generating/learning step of generating a deep learning filter by training a machine learning model using the three-dimensional learning data; and a three-dimensional atomic level structure determining step of generating second three-dimensional atomic structure tomogram volume data, in which an atomic arrangement region with insufficient data is corrected, by filtering first three-dimensional atomic structure tomogram volume data acquired through electron microscopy measurement of the nanomaterial to be measured with the deep learning filter, and determining a three-dimensional atomic level nanostructure of the nanomaterial to be measured based on the second three-dimensional atomic structure tomogram volume data. ### Claim 19 (depends on 18) The method of claim 18, wherein the three-dimensional learning data generating step includes: a three-dimensional data generating step of generating three-dimensional atomic level structure volume data corresponding to the nanomaterial to be measured based on the one or more three-dimensional atomic model; a tilt series generating step of generating a tilt series by simulating three-dimensional tomography for a plurality of different angles in a predetermined angle range for at least some of the three-dimensional atomic level structure volume data; and a three-dimensional atomic structure tomogram volume data generating step of generating a three-dimensional atomic structure tomogram volume data set by performing three-dimensional reconstruction on at least some of the three-dimensional atomic level structure volume data based on the tilt series, and the reconstructed three-dimensional atomic structure tomogram volume data set forms a three-dimensional learning data set for the training the deep learning filter. ### Claim 20 (depends on 19) The method of claim 19, wherein the three-dimensional data generating step further includes generating the three-dimensional atomic level structure volume data by deriving a three-dimensional volume of atomic potential based on the one or more three-dimensional models and acquiring a result of convolution of a Gaussian distribution having the atomic potential and a predetermined standard deviation at each atomic position. ### Claim 21 (depends on 20) The method of claim 20, wherein the three-dimensional data generating step further includes generating, as a three-dimensional target data set, three-dimensional atomic level structure volume data acquired by deriving the three-dimensional volume of the atomic potential based on the one or more three-dimensional models and positioning a Gaussian distribution having a standard deviation equal to or different from the predetermined standard deviation at each atomic position in the three-dimensional volume. ### Claim 22 (depends on 21) The method of claim 21, wherein the deep learning filter generation/learning step further includes a step of optimizing parameters of the deep learning filter to determine a three-dimensional atomic level structure, and the optimizing step includes: calculating an error by comparing the result output from the deep learning filter through forward-propagation based on the three-dimensional learning input data set with the three-dimensional target data set; updating hyperparameters of the deep learning filter in a manner, in which the error is reduced, by calculating (back-propagation) a gradient of the error in a reverse direction (from an output layer to an input layer) based on the error; and training the deep learning filter and optimizing the parameters of the filter to determine the three-dimensional atomic level structure, by repeating the process.