Learning 2D texture mapping in volumetric neural rendering
AssigneeAdobe Inc.
InventorsZexiang Xu, Yannick Hold-Geoffroy, Milos Hasan, Kalyan Sunkavalli, Fanbo Xiang
Life of this patent
Estimating the term…
Abstract
Embodiments are disclosed for neural texture mapping. In some embodiments, a method of neural texture mapping includes obtaining a plurality of images of an object, determining volumetric representation of a scene of the object using a first neural network, mapping 3D points of the scene to a 2D texture space using a second neural network, and determining radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a second neural network to generate a 3D appearance representation of the object.
Claims18
A computer-implemented method comprising:
- obtaining a plurality of images of a scene depicting at least one object;
- determining a volume density of the scene using a scene geometry network to generate a 3D geometric representation of the object;
- mapping 3D points of the scene to a 2D texture space using a texture mapping network, wherein the texture mapping network is trained on a cycle loss using an inverse texture mapping network that maps back from the 2D texture space to 3D points to train the texture mapping network and the inverse texture mapping network to enforce a cycle mapping between the 2D texture space and points on a surface of the scene;
- and
- determining radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a texture network to generate a 3D appearance representation of the object.
A non-transitory computer readable storage medium including instructions stored thereon which, when executed by a processor, cause the processor to:
- obtain a plurality of images of a scene depicting at least one object;
- determine a volume density of the scene using a scene geometry network to generate a 3D geometric representation of the object;
- map 3D points of the scene to a 2D texture space using a texture mapping network, wherein the texture mapping network is trained on a cycle loss using an inverse texture mapping network that maps back from the 2D texture space to 3D points to train the texture mapping network and the inverse texture mapping network to enforce a cycle mapping between the 2D texture space and points on a surface of the scene;
- and
- determine radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a third neural texture network to generate a 3D appearance representation of the object.
A computer-implemented method comprising:
- receiving a request to edit a texture map generated by a volumetric neural rendering system including a plurality of neural networks, wherein the volumetric neural rendering system generates the texture map representing an appearance of an object in a scene separately from a 3D geometric representation of the object;
- obtaining the texture map, wherein the texture map is generated at least in part using a texture mapping network, and wherein the texture mapping network is trained on a cycle loss using an inverse texture mapping network that maps back from a 2D texture space to 3D points to train the texture mapping network and the inverse texture mapping network to enforce a cycle mapping between the 2D texture space and points on a surface of the scene;
- and
- generating a modified texture map based on the texture map and the request.
Dependencies are inferred from the claim text.