Flexible segmentation of images

A non-transitory machine-readable storage medium on which is stored a computer program including machine-readable instructions for determining a segmentation of an input image, the input image including pixels carrying pixel values, the segmentation assigning, to each pixel of the pixels of the input image, a class of an entity that has given rise to the pixel value of the pixel, the instructions, when executed by one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps:

• providing the input image to a first vision processing network that outputs masks designating sets of pixels belonging to different object types, as well as an associated weight matrix that is indicative of distinguishing features characterizing entities of different types in the input image;
• transforming, by an encoder network, the weight matrix in combination with the input image into at least one mask representation in a latent space that is a notion of assignments of classes to masks;
• processing the input image, together with the mask representation, by a second vision processing network, into a refinement for the masks and a refined weight matrix;
• transforming, by the encoder network, the refined weight matrix in combination with the input image into at least one refined mask representation in the latent space;
• and
• computing the segmentation:
• (i) from the masks and the at least one mask representation in the latent space, or (ii) from further refinements of the masks and the at least one mask representation in the latent space obtained by further passes through the second vision processing network and the encoder network.

One or more computers and/or compute instances with a non-transitory machine-readable storage medium on which is stored a computer program including machine-readable instructions for determining a segmentation of an input image, the input image including pixels carrying pixel values, the segmentation assigning, to each pixel of the pixels of the input image, a class of an entity that has given rise to the pixel value of the pixel, the instructions, when executed by the one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps:

• providing the input image to a first vision processing network that outputs masks designating sets of pixels belonging to different object types, as well as an associated weight matrix that is indicative of distinguishing features characterizing entities of different types in the input image;
• transforming, by an encoder network, the weight matrix in combination with the input image into at least one mask representation in a latent space that is a notion of assignments of classes to masks;
• processing the input image, together with the mask representation, by a second vision processing network, into a refinement for the masks and a refined weight matrix;
• transforming, by the encoder network, the refined weight matrix in combination with the input image into at least one refined mask representation in the latent space;
• and
• computing the segmentation:
• (i) from the masks and the at least one mask representation in the latent space, or (ii) from further refinements of the masks and the at least one mask representation in the latent space obtained by further passes through the second vision processing network and the encoder network.