Video lighting using depth and virtual lights

A computing device comprising:

• one or more hardware processors;
• and
• a memory coupled to the one or more hardware processors, with instructions thereon, that when executed by the one or more hardware processors to perform operations comprising:
• receiving a frame of a video from a sender device that is in a video call with the computing device, wherein the frame includes depth data and color data for a plurality of pixels;
• segmenting the frame based on the depth data to classify each pixel as a foreground pixel or a background pixel;
• setting the depth value of each background pixel to a fixed depth value;
• smoothing depth values of the plurality of pixels classified as the foreground pixel;
• calculating surface normals based on the smoothed depth values of the plurality of pixels;
• creating a three-dimensional (3D) mesh based on the surface normals;
• and
• rendering a relighted frame by adding a virtual light based on the 3D mesh and the color data.