Real-time interactive three-dimensional (3d) scene reconstruction and simulation using neural representations

One or more processors, comprising:

• one or more circuits to:
• obtain at least one object segmented from video data;
• densify the at least one object by:
• sampling a plurality of points on or approximately around the at least one object;
• generating a voxelized volume of the at least one object based at least in part on the plurality of points;
• updating the voxelized volume based at least in part on an occupancy state of at least one voxel of a plurality of voxels of the voxelized volume based at least in part on at least one rendered depth map;
• simulate one or more interactions of the voxelized volume of the at least one densified object to update at least one physical attribute of the at least one object;
• and
• display the at least one object.

A method, comprising:

• receiving, by one or more processors, segmentation data corresponding to at least one object segmented from video data;
• densifying, by the one or more processors, the at least one object by:
• sampling a plurality of points on or approximately around the at least one object;
• generating a voxelized volume of the at least one object based at least in part on the plurality of points;
• updating the voxelized volume based at least in part on an occupancy state of at least one voxel of a plurality of voxels of the voxelized volume based at least in part on at least one rendered depth map;
• simulating, by the one or more processors, one or more interactions of the voxelized volume of the at least one densified object to update at least one physical attribute of the at least one object;
• and
• displaying, by the one or more processors, the at least one object.