Real world object reconstruction and representation

A method comprising:

• calculating a camera image loss accumulated across a subset of camera rays using a plurality of color values and sensor data from a camera, the subset of camera rays comprising the camera ray and the plurality of color values comprising the color value;
• calculating a lidar loss accumulated across a subset of lidar rays using a plurality of lidar point values and sensor data from a lidar sensor;
• generating a total loss using the camera image loss and the lidar loss;
• sampling a first plurality of locations along a camera ray from a virtual camera to a target object to obtain a first sample set of the first plurality of locations along the camera ray;
• for each location of at least a subset of the first sample set:
• determining a position of the location with respect to the target object,
• executing, based on the position, a reflectance multilayer perceptron (MLP) model, to determine an albedo and material shininess for the location, and
• computing a radiance for the location and based on a viewing direction of the camera ray using the albedo and the material shininess;
• and
• rendering a color value for the camera ray by compositing the radiance across the first sample set.

A system comprising:

• memory;
• and
• a computer processor comprising computer readable program code for performing operations comprising:
• sampling a first plurality of locations along a camera ray from a virtual camera to a target object to obtain a first sample set of the first plurality of locations along the camera ray,
• for each location of at least a subset of the first sample set:
• determining a position of the location with respect to the target object,
• executing, based on the position, a reflectance multilayer perceptron (MLP) model, to determine an albedo and material shininess for the location, wherein the reflectance MLP model comprises an intensity model as an extra layer on the reflectance MLP model, and wherein the intensity model generates the material shininess, wherein the extra layer generates a plurality of predicted intensities, and
• computing a radiance for the location and based on a viewing direction of the camera ray using the albedo and the material shininess, and
• rendering a color value for the camera ray by compositing the radiance across the first sample set.

A system comprising:

• memory;
• and
• a computer processor comprising computer readable program code for performing operations comprising:
• calculating a camera image loss accumulated across a subset of camera rays using a plurality of color values and sensor data from a camera, the subset of camera rays comprising the camera ray and the plurality of color values comprising the color value,
• calculating a lidar loss accumulated across a subset of lidar rays using a plurality of lidar point values and sensor data from a lidar sensor,
• generating a total loss using the camera image loss and the lidar loss,
• sampling a first plurality of locations along a camera ray from a virtual camera to a target object to obtain a first sample set of the first plurality of locations along the camera ray,
• for each location of at least a subset of the first sample set:
• determining a position of the location with respect to the target object,
• executing, based on the position, a reflectance multilayer perceptron (MLP) model, to determine an albedo and material shininess for the location, and
• computing a radiance for the location and based on a viewing direction of the camera ray using the albedo and the material shininess, and
• rendering a color value for the camera ray by compositing the radiance across the first sample set.

A non-transitory computer readable medium comprising computer readable program code for performing operations comprising:

• sampling a first plurality of locations along a camera ray from a virtual camera to a target object to obtain a first sample set of the first plurality of locations along the camera ray;
• for each location of at least a subset of the first sample set:
• determining a position of the location with respect to the target object,
• executing, based on the position, a reflectance multilayer perceptron (MLP) model, to determine an albedo and material shininess for the location, wherein the reflectance MLP model comprises an intensity model as an extra layer on the reflectance MLP model, and wherein the intensity model generates the material shininess, wherein the extra layer generates a plurality of predicted intensities, and
• computing a radiance for the location and based on a viewing direction of the camera ray using the albedo and the material shininess;
• and
• rendering a color value for the camera ray by compositing the radiance across the first sample set.