Leveraging synthetic images to improve visual localization in case of extreme domain shifts

A system comprising:

• one or more data processors;
• and
• a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including:
• access an initial dataset of real images of an environment, wherein for an image of the initial dataset there exists at least one matching image in the initial dataset and a set of non-matching images, the matching image represents similar scene of the environment as in the image of the initial dataset;
• identify a plurality of domain shifts, wherein each domain shift of the plurality of domain shifts corresponds to a potential change in the environment;
• generate, for each image of the initial dataset and for each domain shift of the plurality of domain shifts, a synthetic image;
• identify a plurality of matching pairs in the initial dataset, wherein each matching pair of the plurality of matching pairs comprises of a first image and a second image and both images captures similar aspects or scene of the environment;
• determine, for each matching pair of the plurality of matching pairs in the initial dataset, a plurality of tentative pairs, wherein each tentative pair of the plurality of tentative pairs includes at least one synthetic image, the tentative pairs are generated by making image pairs of:
• the first image with the synthetic images of the second image, and
• the synthetic images of the first image with the second image;
• determine, for each tentative pair of the plurality of tentative pairs, geometric correspondences;
• determine, for each tentative pair of the plurality of tentative pairs, whether the tentative pair is a valid pair or not based on a degree of the geometric correspondences between images of the tentative pair inside an area of interest;
• generate an extended dataset by combining the initial dataset with the valid pairs;
• and
• train an image retrieval model using a contrastive loss function with weights assigned to each matching pair and corresponding valid pairs in the extended dataset.

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:

• accessing an initial dataset of real images of an environment, wherein for an image of the initial dataset there exists at least one matching image in the initial dataset and a set of non-matching images, the matching image represents similar scene of the environment as in the image of the initial dataset;
• identifying a plurality of domain shifts, wherein each domain shift of the plurality of domain shifts corresponds to a potential change in the environment;
• generating, for each image of the initial dataset and for each domain shift of the plurality of domain shifts, a synthetic image;
• identifying a plurality of matching pairs in the initial dataset, wherein each matching pair of the plurality of matching pairs comprises of a first image and a second image and both images captures similar aspects or scene of the environment;
• determining, for each matching pair of the plurality of matching pairs in the initial dataset, a plurality of tentative pairs, wherein each tentative pair of the plurality of tentative pairs includes at least one synthetic image, the tentative pairs are generated by making image pairs of:
• the first image with the synthetic images of the second image, and
• the synthetic images of the first image with the second image;
• determining, for each tentative pair of the plurality of tentative pairs, geometric correspondences;
• determining, for each tentative pair of the plurality of tentative pairs, whether the tentative pair is a valid pair or not based on a degree of the geometric correspondences between images of the tentative pair inside an area of interest;
• generating an extended dataset by combining the initial dataset with the valid pairs;
• and
• training an image retrieval model using a contrastive loss function with weights assigned to each matching pair and corresponding valid pairs in the extended dataset.