Video background subtraction using depth

A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations comprising:

• receiving a plurality of frames of a video, wherein each frame includes depth data and color data for a plurality of pixels;
• and
• for each frame:
• based on the depth data, generating an initial segmentation mask that categorizes each pixel of the frame as foreground pixel or background pixel;
• detecting a head bounding box is based on one or more of the color data or the initial segmentation mask, wherein the head bounding box includes pixels corresponding to head portions of the frame, wherein detecting the head bounding box comprises:
• detecting one or more valid faces in the frame, wherein each valid face is associated with a face area that includes face pixels of the face;
• and
• expanding the face area for each valid face to obtain a head area corresponding to the valid face, wherein the head bounding box includes the head area for each face determined as valid;
• determining, based on the head bounding box, a trimap that classifies each pixel of the frame as one of known background, known foreground, or unknown, based on a first trimap for non-head portions of the frame and a second trimap for head portions of the frame;
• for each pixel of the frame that is classified in the trimap as unknown, calculating a weight for the pixel and storing the weight in a weight map;
• and
• performing fine segmentation based on the color data, the trimap, and the weight map to obtain a binary mask for the frame.

A system comprising:

• one or more hardware processors;
• and
• a memory coupled to the one or more hardware processors, with instructions thereon, that when executed by the one or more hardware processors to perform operations comprising:
• receiving a plurality of frames of a video, wherein each frame includes depth data and color data for a plurality of pixels;
• and
• for each frame:
• based on the depth data, generating an initial segmentation mask that categorizes each pixel of the frame as foreground pixel or background pixel;
• detecting a head bounding box is based on one or more of the color data or the initial segmentation mask, wherein the head bounding box includes pixels corresponding to head portions of the frame, wherein detecting the head bounding box comprises:
• detecting one or more valid faces in the frame, wherein each valid face is associated with a face area that includes face pixels of the face;
• and
• expanding the face area for each valid face to obtain a head area corresponding to the valid face, wherein the head bounding box includes the head area for each face determined as valid;
• determining, based on the head bounding box, a trimap that classifies each pixel of the frame as one of known background, known foreground, or unknown, based on a first trimap for non-head portions of the frame and a second trimap for head portions of the frame;
• for each pixel of the frame that is classified in the trimap as unknown, calculating a weight for the pixel and storing the weight in a weight map;
• and
• performing fine segmentation based on the color data, the trimap, and the weight map to obtain a binary mask for the frame.