# US 12,112,433 B2 · Apparatus and methods for image reconstruction using machine learning processes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,433 B2 - Document ID: US12112433B2 - Status: Granted - Kind: B2 - Application number: 17/714,743 - Filed: 2022-04-06 - Published: 2023-10-12 - Granted: 2024-10-08 - Earliest claimed: 2022-04-06 - Assignees: QUALCOMM Incorporated - Applicants: Qualcomm Incorporated - Inventors: Wei-Lun Chang; Michel Adib Sarkis; Chieh-Ming Kuo; Kuang-Man Huang - Web page: https://patentgrep.com/patent/US12112433B2 ## CPC - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06N 3/045: Combinations of networks - G06V 10/235: {based on user input or interaction} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods, systems, and apparatuses are provided to automatically reconstruct an image, such as a 3D image. For example, a computing device may obtain an image, and may apply a first trained machine learning process to the image to generate coefficient values characterizing the image in a plurality of dimensions. Further, the computing device may generate a mesh based on the coefficient values. The computing device may apply a second trained machine learning process to the coefficient values and the image to generate a displacement map. Based on the mesh and the displacement map, the computing device may generate output data characterizing an aligned mesh. The computing device may store the output data within a data repository. In some examples, the computing device provides the output data for display. ## Claims ### Claim 1 An apparatus comprising: a non-transitory, machine-readable storage medium storing instructions; and at least one processor coupled to the non-transitory, machine-readable storage medium, the at least one processor being configured to: apply a first trained machine learning process to an image and generate coefficient data characterizing the image in a plurality of dimensions based on the application of the first trained machine learning process to the image; generate first mesh data characterizing a first mesh of the image based on the coefficient data; apply a second trained machine learning process to the coefficient data and the image, and generate displacement data characterizing a displacement image based on the application of the second trained machine learning process to the coefficient data and the image; and generate second mesh data characterizing a second mesh of the image based on a combination of at least one vertex of the first mesh data and at least one vertex of the displacement data corresponding to a same coordinate. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein applying the second trained machine learning process comprises establishing a convolutional neural network. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the coefficient data comprises a plurality of texture values, each of the plurality of texture values corresponding to a texture coordinate. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the second trained machine learning process is applied to the plurality of texture values. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein the at least one processor is further configured to execute the instructions to: sample the displacement data to obtain a displacement value; and add the displacement value to a corresponding mesh value of the second mesh. ### Claim 6 (depends on 1) The apparatus of claim 1 comprising a camera configured to capture the image. ### Claim 7 (depends on 1) The apparatus of claim 1 comprising a display configured to display the second mesh data. ### Claim 8 (depends on 1) The apparatus of claim 1, wherein the at least one processor is further configured to execute the instructions to render a 3 dimensional model based on the second mesh data. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the 3 dimensional model includes a face. ### Claim 10 (depends on 1) The apparatus of claim 1, wherein the at least one processor is further configured to execute the instructions to output the second mesh data. ### Claim 11 (depends on 1) The apparatus of claim 1, wherein the at least one processor is further configured to execute the instructions to: receive a second image; apply the first trained machine learning process to the second image and generate second coefficient data characterizing the second image in the plurality of dimensions based on the application of the first trained machine learning process to the second image; generate third mesh data characterizing a first mesh of the second image based on the second coefficient data; apply the second trained machine learning process to the second coefficient data and the second image, and generate second displacement data characterizing a second displacement image based on the application of the second trained machine learning process to the second coefficient data and the second image; and generate fourth mesh data characterizing a second mesh of the second image based on the third mesh data and the second displacement data. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein the at least one processor is further configured to execute the instructions to output the second mesh data and the fourth mesh data. ### Claim 13 (depends on 1) The apparatus of claim 1, wherein the at least one processor is further configured to execute the instructions to: receive a first input from a user; and capture the image in response to the first input from the user. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the at least one processor is further configured to execute the instruction to: receive a second input from the user; determine a region of interest of the image based on the second input from the user; perform one or more of automatic focus, automatic exposure, automatic gain, and automatic white balance based on the region of interest; and capture the image based on the performance of the one or more of automatic focus, automatic exposure, automatic gain, and automatic white balance. ### Claim 15 A method comprising: applying a first trained machine learning process to an image and generate coefficient data characterizing the image in a plurality of dimensions based on the application of the first trained machine learning process to the image; generating first mesh data characterizing a first mesh of the image based on the coefficient data; applying a second trained machine learning process to the coefficient data and the image and generate displacement data characterizing a displacement image based on the application of the second trained machine learning process to the coefficient data and the image; and generating second mesh data characterizing a second mesh of the image based on a combination of at least one vertex of the first mesh data and at least one vertex of the displacement data corresponding to a same coordinate. ### Claim 16 (depends on 15) The method of claim 15, wherein applying the second trained machine learning process comprises establishing a convolutional neural network. ### Claim 17 (depends on 15) The method of claim 15, comprising: sampling the displacement data to obtain a displacement value; and adding the displacement value to a corresponding mesh value of the second mesh. ### Claim 18 (depends on 15) The method of claim 15, comprising: receiving a second image; applying the first trained machine learning process to the second image and generate second coefficient data characterizing the second image in the plurality of dimensions based on the application of the first trained machine learning process to the second image; generating third mesh data characterizing a first mesh of the second image based on the second coefficient data; applying the second trained machine learning process to the second coefficient data and the second image, and generate second displacement data characterizing a second displacement image based on the application of the second trained machine learning process to the second coefficient data and the second image; and generating fourth mesh data characterizing a second mesh of the second image based on the third mesh data and the second displacement data. ### Claim 19 A non-transitory, machine-readable storage medium storing instructions that, when executed by at least one processor, causes the at least one processor to perform operations that include: applying a first trained machine learning process to an image and generate coefficient data characterizing the image in a plurality of dimensions based on the application of the first trained machine learning process to the image; generating first mesh data characterizing a first mesh of the image based on the coefficient data; applying a second trained machine learning process to the coefficient data and the image and generate displacement data characterizing a displacement image based on the application of the second trained machine learning process to the coefficient data and the image; and generating second mesh data characterizing a second mesh of the image based on a combination of at least one vertex of the first mesh data and at least one vertex of the displacement data corresponding to a same coordinate. ### Claim 20 (depends on 19) The non-transitory, machine-readable storage medium of claim 19, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform operations that include: sampling the displacement data to obtain a displacement value; and adding the displacement value to a corresponding mesh value of the second mesh.