System and method for generating a mask for object instances in an image

A method for training a neural network map to be used for generating a mask for object instances in an image, the method comprising:

• obtaining a plurality of training images each labelled by a map indicating for each image pixel whether it belongs to an object instance or to image background, and
• iteratively training the neural network for each training image by:
• generating pixel offset vectors for the pixels of an object instance configured to point towards a unique center of the object instance,
• placing a differentiable distribution function at the center of each object instance transforming the offset vectors into a probability of belonging to said object,
• optimizing parameters of said distribution function such that the probability of the pixels belonging to said object instance is increased and the probability of pixels belonging to the background is decreased, and
• optimizing the offset vectors of the pixels of said object instance for pointing to the center of the distribution function placed at the center of said object instance.