Training a machine learnable model to estimate relative object scale

A computer-implemented method of estimating a relative scale of objects in an image, comprising the following steps:

• providing a feature extractor which is configured to:
• receive an image patch of the image as input,
• detect a plurality of features in the image patch to obtain a plurality of feature maps as output, wherein the plurality of features is associated with one or more objects in the image, wherein each respective feature map of the feature maps is generated by applying a filter to image data of the image patch, and
• wherein each respective feature map includes, along a scale dimension, a filter response across a set of different spatial scales;
• providing a scale estimator to process the output of the feature extractor,
• wherein the scale estimator includes a trained machine learnable model part, wherein the scale estimator is configured to:
• aggregate each respective feature map into a feature-level scale estimate, wherein the aggregation includes identifying a maximum filter response across the different spatial scales, thereby obtaining a plurality of feature-level scale estimates, and
• infer, with the machine learnable model part, a patch-level scale estimate from the plurality of feature-level scale estimates;
• applying the feature extractor and the scale estimator to at least one image patch of the image to obtain a patch-level scale estimate for the at least one image patch;
• and
• outputting a data representation of the patch-level scale estimate.

A non-transitory computer-readable medium on which is stored data representing instructions training a machine learnable model to estimate a relative scale of objects in an image, the instructions, when executed by a processor system, causing the processor system to perform the following steps:

• providing a feature extractor which is configured to:
• receive an image patch of the image as input, and
• detect a plurality of features in the image patch to obtain a plurality of feature maps as output, wherein the plurality of features is associated with one or more objects in the image, wherein each respective feature map of the feature maps is generated by applying a filter to image data of the image patch, wherein each respective feature map includes, along a scale dimension, a filter response across a set of different spatial scales;
• providing a scale estimator to process the output of the feature extractor, wherein the scale estimator includes a machine learnable model part and is configured to:
• aggregate each respective feature map into a feature-level scale estimate,
• wherein the aggregation includes identifying a maximum filter response across the different spatial scales, thereby obtaining a plurality of feature-level scale estimates, and
• infer, with the machine learnable model part, a patch-level scale estimate from the plurality of feature-level scale estimates;
• accessing training data including a set of training images;
• and
• training the machine learnable model part of the scale estimator on the training data to infer the patch-level scale estimate from the plurality of feature-level scale estimates, wherein the training includes:
• spatially scaling image data of each image patch of a training image by at least two known scale factors to obtain at least two further image patches,
• applying the feature extractor and the scale estimator to the at least two further image patches to obtain at least two patch-level scale estimates, and
• optimizing parameters of the machine learnable model part by minimizing an error term of a loss function, wherein the error term expresses a mismatch between an actual relative scale and an estimated relative scale,
• wherein the actual relative scale is determined as a difference between the two known scale factors and the estimated relative scale as a difference between the at least two patch-level scale estimates.

A system for training a machine learnable model to estimate a relative scale of objects in an image, comprising:

• an input interface subsystem configured to access:
• training data including a set of training images;
• a feature extractor which is configured to:
• receive an image patch of the image as input,
• detect a plurality of features in the image patch to obtain a plurality of feature maps as output, wherein the plurality of features is associated with one or more objects in the image, wherein a respective feature map is generated by applying a filter to image data of the image patch, and
• wherein a respective feature map includes, along a scale dimension, a filter response across a set of different spatial scales;
• a scale estimator configured to process the output of the feature extractor, wherein the scale estimator includes a machine learnable model part and is configured to:
• aggregate a respective feature map into a feature-level scale estimate, wherein the aggregation includes identifying a maximum filter response across the different spatial scales, thereby obtaining a plurality of feature-level scale estimates, and
• infer, with the machine learnable model part, a patch-level scale estimate from the plurality of feature-level scale estimates;
• a processor subsystem configured to train the machine learnable model part of the scale estimator on the training data to infer the patch-level scale estimate from the plurality of feature-level scale estimates, wherein the training includes:
• spatially scale image data of an image patch of a training image by at least two known scale factors to obtain at least two further image patches,
• apply the feature extractor and the scale estimator to the at least two further image patches to obtain at least two patch-level scale estimates,
• optimize parameters of the machine learnable model part by minimizing an error term of a loss function, wherein the error term expresses a mismatch between an actual relative scale and an estimated relative scale, wherein the actual relative scale is determined as a difference between the two known scale factors and the estimated relative scale as a difference between the at least two patch-level scale estimates.

A system for estimating a relative scale of objects in an image, comprising:

• an input interface subsystem configured to access:
• the image,
• a feature extractor which is configured to:
• receive an image patch of the image as input;
• detect a plurality of features in the image patch to obtain a plurality of feature maps as output, wherein the plurality of features is associated with one or more objects in the image, wherein a respective feature map is generated by applying a filter to image data of the image patch,
• wherein a respective feature map includes, along a scale dimension, a filter response across a set of different spatial scales;
• a scale estimator configured to process the output of the feature extractor, wherein the scale estimator includes a trained machine learnable model part, and wherein the scale estimator is configured to:
• aggregate a respective feature map into a feature-level scale estimate, wherein the aggregation includes identifying a maximum filter response across the different spatial scales, thereby obtaining a plurality of feature-level scale estimates, and
• infer, with the machine learnable model part, a patch-level scale estimate from the plurality of feature-level scale estimates;
• and
• a processor subsystem configured to:
• apply the feature extractor and the scale estimator to at least one image patch of the image to obtain a patch-level scale estimate for the at least one image patch, and
• output a data representation of the patch-level scale estimate.