# US 2022/0284663 A1 · Method and apparatus with image processing and reconstructed image generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0284663 A1 - Document ID: US20220284663A1 - Status: Published application - Kind: A1 - Application number: 17/487,240 - Filed: 2021-09-28 - Published: 2022-09-08 - Earliest claimed: 2021-03-08 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Inwoo HA; Hyun Sung CHANG; Minjung SON - Web page: https://patentgrep.com/patent/US20220284663A1 ## CPC - G06T 15/506: {Illumination models} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 7/50: Depth or shape recovery - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 15/80: Shading - G06V 40/168: {Feature extraction; Face representation} - G06V 40/174: {Facial expression recognition} - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06T 2219/2021: Shape modification - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A processor-implemented method includes: determining albedo data in a canonical space and depth data in the canonical space based on input image data including an object, using one or more neural network-based extraction models; generating deformed albedo data and deformed depth data by applying a target shape deformation value respectively to the albedo data and the depth data; generating resultant shaded data by performing shading based on the deformed depth data and a target illumination value; generating intermediate image data based on the resultant shaded data and the deformed albedo data; and generating reconstructed image data from the intermediate image data and the deformed depth data based on a target pose value. ## Claims ### Claim 1 A processor-implemented method, the method comprising: determining albedo data in a canonical space and depth data in the canonical space based on input image data including an object, using one or more neural network-based extraction models; generating deformed albedo data and deformed depth data by applying a target shape deformation value respectively to the albedo data and the depth data; generating resultant shaded data by performing shading based on the deformed depth data and a target illumination value; generating intermediate image data based on the resultant shaded data and the deformed albedo data; and generating reconstructed image data from the intermediate image data and the deformed depth data based on a target pose value. ### Claim 2 (depends on 1) The method of claim 1, wherein the determining of the albedo data and the depth data comprises: determining the albedo data in the canonical space from the input image data using a neural network-based albedo extraction model; and determining the depth data in the canonical space from the input image data using a neural network-based depth extraction model. ### Claim 3 (depends on 1) The method of claim 1, wherein the albedo data in the canonical space corresponds to albedo data when the object is deformed into a canonical shape which is a reference, and the depth data in the canonical space corresponds to depth data when the object is deformed into the canonical shape. ### Claim 4 (depends on 1) The method of claim 1, wherein the generating of the deformed albedo data and the deformed depth data comprises: performing a backward warping operation on each of the albedo data and the depth data based on the target shape deformation value. ### Claim 5 (depends on 1) The method of claim 1, wherein the generating of the resultant shaded data comprises: extracting a surface normal element of the object from the deformed depth data; and generating the resultant shaded data by performing the shading based on the extracted surface normal element and the target illumination value. ### Claim 6 (depends on 1) The method of claim 1, wherein the generating of the resultant shaded data comprises: performing the shading using second-order spherical harmonics. ### Claim 7 (depends on 1) The method of claim 1, wherein the generating of the reconstructed image data comprises: generating the reconstructed image data by deforming a pose of the object in each of the intermediate image data and the deformed depth data based on the target pose value, and combining the intermediate image data in which the pose of the object is deformed and depth data in which the pose of the object is deformed. ### Claim 8 (depends on 1) The method of claim 1, wherein the target shape deformation value, the target illumination value, and the target pose value are values extracted from another input image data that is different from the input image data, and an object in the other input image data is the same as the object in the input image data. ### Claim 9 (depends on 1) The method of claim 1, wherein the target shape deformation value, the target illumination value, and the target pose value are values extracted from another image data including an object that is different from the object in the input image data. ### Claim 10 (depends on 1) The method of claim 1, wherein the target shape deformation value, the target illumination value, and the target pose value are values extracted from another input image data using a neural network-based extraction model other than the one or more neural network-based extraction models. ### Claim 11 (depends on 1) The method of claim 1, wherein the target shape deformation value, the target illumination value, and the target pose value are extracted from another image data, and the one or more extraction models are trained by updating parameters of the one or more extraction models based on the reconstructed image data and the other image data. ### Claim 12 (depends on 1) A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method of claim 1. ### Claim 13 An apparatus, the apparatus comprising: one or more processors configured to: determine albedo data in a canonical space and depth data in the canonical space based on input image data including an object, using one or more neural network-based extraction models; generate deformed albedo data and deformed depth data by applying a target shape deformation value respectively to the albedo data and the depth data; generate resultant shaded data by performing shading based on the deformed depth data and a target illumination value; generate intermediate image data based on the resultant shaded data and the deformed albedo data; and generate reconstructed image data from the intermediate image data and the deformed depth data based on a target pose value. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein, for the determining of the albedo data and the depth data, the one or more processors are configured to: determine the albedo data in the canonical space from the input image data using a neural network-based albedo extraction model; and determine the depth data in the canonical space from the input image data using a neural network-based depth extraction model. ### Claim 15 (depends on 13) The apparatus of claim 13, wherein, for the generating of the resultant shaded data, the one or more processors are configured to: extract a surface normal element of the object from the deformed depth data; and generate the resultant shaded data by performing the shading based on the extracted surface normal element and the target illumination value. ### Claim 16 (depends on 13) The apparatus of claim 13, wherein, for the generating of the reconstructed image data, the one or more processors are configured to: generate the reconstructed image data by deforming a pose of the object in each of the intermediate image data and the deformed depth data based on the target pose value, and combining the intermediate image data in which the pose of the object is deformed and depth data in which the pose of the object is deformed. ### Claim 17 (depends on 13) The apparatus of claim 13, wherein the target shape deformation value, the target illumination value, and the target pose value are values extracted from another input image data using a neural network-based extraction model other than the one or more neural network-based extraction models. ### Claim 18 (depends on 13) An electronic apparatus comprising the apparatus of claim 13 and a display. ### Claim 19 A processor-implemented method, the method comprising: determining albedo data in a canonical space and depth data in the canonical space based on first training image data using a neural network-based first extraction model; extracting a shape deformation value, an illumination value, and a pose value from second training image data; generating deformed albedo data and deformed depth data by applying the shape deformation value respectively to the albedo data and the depth data; generating resultant shaded data by performing shading based on the deformed depth data and the illumination value; generating intermediate image data based on the resultant shaded data and the deformed albedo data; generating reconstructed image data from the intermediate image data and the deformed depth data based on the pose value; and training the one or more extraction models by updating parameters of the one or more extraction models based on the reconstructed image data and the second training image data. ### Claim 20 (depends on 19) The method of claim 19, wherein the updating of the parameters of the first extraction model comprises: iteratively correcting the parameters of the first extraction model such that a difference between the reconstructed image data and the second training image data is reduced. ### Claim 21 (depends on 19) The method of claim 19, wherein the extracting of the shape deformation value, the illumination value, and the pose value comprises: extracting the shape deformation value, the illumination value, and the pose value from the second training image data using a neural network-based second extraction model, wherein the updating of the parameters of the first extraction model comprises iteratively correcting parameters of each of the first extraction model and the second extraction model such that a difference between the reconstructed image data and the second training image data is reduced. ### Claim 22 (depends on 19) The method of claim 19, further comprising using the trained one or more extraction models to generate reconstructed image data from input image data including an object. ### Claim 23 A processor-implemented method, the method comprising: decomposing an input image into an albedo component and a depth component using a trained neural network-based extraction model; deforming the albedo component and the depth component based on a target shape deformation value corresponding to a local geometric change of an object of the input image; shading the deformed depth component based on a target illumination value; generating an intermediate image by combining the deformed albedo component and the shaded deformed depth component; and adjusting a pose of the intermediate image based on the deformed depth component and a target pose value. ### Claim 24 (depends on 23) The method of claim 23, wherein the generating of the intermediate image comprises performing a vector dot product operation between the deformed albedo component and the shaded deformed depth component. ### Claim 25 (depends on 23) The method of claim 23, wherein the shading of the deformed depth component comprises: determining a surface normal of the deformed depth component through pixel-wise regression of local neighboring pixels of the deformed depth component; and applying an illumination element to the surface normal. ### Claim 26 (depends on 23) The method of claim 23, wherein the target shape deformation value corresponds to a facial expression of the object.