Segmentation of objects in videos using color and depth information

A system for object segmentation, the system comprising:

• a pre-processing circuit to receive a plurality of image frames, each comprising a color frame component and a depth frame component, wherein one of the image frames is a reference image frame;
• a training circuit to receive a mask to outline a region, in the reference image frame, containing an object to be segmented, the training circuit further to calculate a first set of Gaussian mixture models associated with the masked region and a second set of Gaussian mixture models associated with a background region, the background region external to the masked region, the first and second sets of Gaussian mixture models each comprising a model based on the color frame component and a model based on the depth frame component;
• and
• a segmentation circuit to segment the object, from a current frame of the image frames, based on modelling of pixels within an active area of the current frame as a Markov Random Field (MRF) of nodes for cost minimization, wherein the costs are based on the Gaussian mixture models, and the active area is based on segmentation of a previous frame of the image frames and on an optical flow estimation between the previous frame and the current frame.

At least one non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in the following operations for object segmentation, the operations comprising:

• receiving a plurality of image frames, each comprising a color frame component and a depth frame component, wherein one of the image frames is a reference image frame;
• receiving a mask to outline a region, in the reference image frame, containing an object to be segmented;
• calculating a first set of Gaussian mixture models associated with the masked region and a second set of Gaussian mixture models associated with a background region, the background region external to the masked region, the first and second sets of Gaussian mixture models each comprising a model based on the color frame component and a model based on the depth frame component;
• and
• segmenting the object, from a current frame of the image frames, based on modelling of pixels within an active area of the current frame as a Markov Random Field (MRF) of nodes for cost minimization, wherein the costs are based on the Gaussian mixture models, and the active area is based on segmentation of a previous frame of the image frames and on an optical flow estimation between the previous frame and the current frame.