Image generation system with controllable scene lighting

A method of operating an image generation system to generate an image of a given 3-dimensional (3D) object, the method comprising:

• with a camera, capturing an image of the given 3D object;
• conditioning the image generation system to generate an image of the given 3D object;
• and
• generating images of the given 3D object under different lighting conditions by:
• receiving lighting information about an environment in which the given 3D object is located and a perspective of the given 3D object in the environment;
• with a feature extractor, extracting corresponding orthogonal features based on the received lighting information and the perspective;
• and
• decoding diffuse and specular lighting parameters based on the orthogonal features.

A method of operating a light based image generation system comprising:

• with a feature extractor, receiving lighting information and pose information and extracting corresponding geometric features based on the lighting information and the pose information;
• decoding the extracted geometric features to produce diffuse and specular reflection parameters;
• with a volume renderer, rendering a set of images based on the diffuse and specular reflection parameters;
• generating a super resolution image from the set of images;
• and
• comparing the super resolution image to one or more ground truth images to adjust weights associated with the light based image generation system.