Method and device for training a neural network

A computer-implemented method for training a machine learning system, the machine learning system being trained to transfer images of a source domain into a target domain, the method comprising the following steps:

• ascertaining a plurality of source patches based on a plurality of source images of a source domain, and ascertaining a plurality of target patches based on a plurality of target images of a target domain, the source patches and the target patches each being assigned pixel-by-pixel pieces of meta-information;
• ascertaining a plurality of tuples, each tuple of the tuples including one source patch of the plurality of source patches and at least one target patch of the plurality of target patches, the at least one target patch characterizing a neighbor of the source patch nearest to k according to a similarity measure, k being a hyperparameter of the method and the similarity measure characterizing a similarity between a source patch and a target patch based on the pieces of pixel-by-pixel meta-information of the source patch and on the pixel-by-pixel pieces of meta-information of the target patch;
• and
• training the machine learning system based on the source patches of the tuples and on the target patches of the tuples, wherein the training step includes the following steps:
• selecting a source patch from the source patches of the tuples and a target patch from the target patches of the tuples;
• ascertaining a first generated image based on the selected source patch using a first generator of the machine learning system and ascertaining a first reconstruction based on the first generated image using a second generator of the machine learning system;
• ascertaining a second generated image based on the selected target patch using the second generator and ascertaining a second reconstruction based on the second generated image using the first generator;
• ascertaining a first loss value, the first loss value characterizing a first difference between the selected source patch and the first reconstruction, the first difference being weighted according to a first attention map, and ascertaining a second loss value, the second loss value characterizing a second difference between the selected target patch and the second reconstruction, the second difference being weighted according to a second attention map;
• and
• training the machine learning system by training the first generator and/or the second generator based on the first loss value and/or on the second loss value.

A computer-implemented method for training an object detector comprising the following steps:

• providing an input image and an annotation, the annotation characterizing a position of at least one object mapped in the input image;
• ascertaining an intermediate image using a first generator of a machine learning system, the machine learning system the machine learning system being trained to transfer images of a source domain into a target domain, including:
• ascertaining a plurality of source patches based on a plurality of source images of a source domain, and ascertaining a plurality of target patches based on a plurality of target images of a target domain, the source patches and the target patches each being assigned pixel-by-pixel pieces of meta-information;
• ascertaining a plurality of tuples, each tuple of the tuples including one source patch of the plurality of source patches and at least one target patch of the plurality of target patches, the at least one target patch characterizing a neighbor of the source patch nearest to k according to a similarity measure, k being a hyperparameter of the method and the similarity measure characterizing a similarity between a source patch and a target patch based on the pieces of pixel-by-pixel meta-information of the source patch and on the pixel-by-pixel pieces of meta-information of the target patch;
• and
• training the machine learning system based on the source patches of the tuples and on the target patches of the tuples, including:
• selecting a source patch from the source patches of the tuples and a target patch from the target patches of the tuples;
• ascertaining a first generated image based on the selected source patch using a first generator of the machine learning system and ascertaining a first reconstruction based on the first generated image using a second generator of the machine learning system;
• ascertaining a second generated image based on the selected target patch using the second generator and ascertaining a second reconstruction based on the second generated image using the first generator;
• ascertaining a first loss value, the first loss value characterizing a first difference between the selected source patch and the first reconstruction, the first difference being weighted according to a first attention map, and ascertaining a second loss value, the second loss value characterizing a second difference between the selected target patch and the second reconstruction, the second difference being weighted according to a second attention map;
• and
• training the machine learning system by training the first generator and/or the second generator based on the first loss value and/or on the second loss value;
• training the object detector, the object detector being trained in such a way that the object detector for the intermediate image predicts as input the at least one object, which are characterized by the annotation.

A computer-implemented method for ascertaining an activation signal for activating an actuator and/or a display device, comprising the following steps:

• providing an input image;
• ascertaining objects mapped on the input image using an object detector, the object detector having been trained by:
• providing a first input image and an annotation, the annotation characterizing a position of at least one object mapped in the first input image;
• ascertaining an intermediate image using a first generator of a machine learning system, the machine learning system the machine learning system being trained to transfer images of a source domain into a target domain, including:
• ascertaining a plurality of source patches based on a plurality of source images of a source domain, and ascertaining a plurality of target patches based on a plurality of target images of a target domain, the source patches and the target patches each being assigned pixel-by-pixel pieces of meta-information;
• ascertaining a plurality of tuples, each tuple of the tuples including one source patch of the plurality of source patches and at least one target patch of the plurality of target patches, the at least one target patch characterizing a neighbor of the source patch nearest to k according to a similarity measure, k being a hyperparameter of the method and the similarity measure characterizing a similarity between a source patch and a target patch based on the pieces of pixel-by-pixel meta-information of the source patch and on the pixel-by-pixel pieces of meta-information of the target patch;
• and
• training the machine learning system based on the source patches of the tuples and on the target patches of the tuples, including:
• selecting a source patch from the source patches of the tuples and a target patch from the target patches of the tuples;
• ascertaining a first generated image based on the selected source patch using a first generator of the machine learning system and ascertaining a first reconstruction based on the first generated image using a second generator of the machine learning system;
• ascertaining a second generated image based on the selected target patch using the second generator and ascertaining a second reconstruction based on the second generated image using the first generator;
• ascertaining a first loss value, the first loss value characterizing a first difference between the selected source patch and the first reconstruction, the first difference being weighted according to a first attention map, and ascertaining a second loss value, the second loss value characterizing a second difference between the selected target patch and the second reconstruction, the second difference being weighted according to a second attention map;
• and
• training the machine learning system by training the first generator and/or the second generator based on the first loss value and/or on the second loss value;
• training the object detector, the object detector being trained in such a way that the object detector for the intermediate image predicts as input the at least one object, which are characterized by the annotation;
• ascertaining the activation signal based on the ascertained objects;
• activating the actuator and/or the display device according to the activation signal.

A training device configured to train a machine learning system, the machine learning system being trained to transfer images of a source domain into a target domain, the training device configured to:

• ascertain a plurality of source patches based on a plurality of source images of a source domain, and ascertaining a plurality of target patches based on a plurality of target images of a target domain, the source patches and the target patches each being assigned pixel-by-pixel pieces of meta-information;
• ascertain a plurality of tuples, each tuple of the tuples including one source patch of the plurality of source patches and at least one target patch of the plurality of target patches, the at least one target patch characterizing a neighbor of the source patch nearest to k according to a similarity measure, k being a hyperparameter of the method and the similarity measure characterizing a similarity between a source patch and a target patch based on the pieces of pixel-by-pixel meta-information of the source patch and on the pixel-by-pixel pieces of meta-information of the target patch, wherein the pixel-by-pixel pieces of meta-information characterize a piece of depth information of pixels of the source patch or a piece of depth information of pixels of the target patch;
• and
• train the machine learning system based on the source patches of the tuples and on the target patches of the tuples.

A non-transitory machine-readable memory medium on which is stored a computer program for training a machine learning system, the machine learning system being trained to transfer images of a source domain into a target domain, the computer program, when executed by a processor, causing the processor to perform the following steps:

• ascertaining a plurality of source patches based on a plurality of source images of a source domain, and ascertaining a plurality of target patches based on a plurality of target images of a target domain, the source patches and the target patches each being assigned pixel-by-pixel pieces of meta-information;
• ascertaining a plurality of tuples, each tuple of the tuples including one source patch of the plurality of source patches and at least one target patch of the plurality of target patches, the at least one target patch characterizing a neighbor of the source patch nearest to k according to a similarity measure, k being a hyperparameter of the method and the similarity measure characterizing a similarity between a source patch and a target patch based on the pieces of pixel-by-pixel meta-information of the source patch and on the pixel-by-pixel pieces of meta-information of the target patch;
• and
• training the machine learning system based on the source patches of the tuples and on the target patches of the tuples.