Verifying the provenance of a digital object using watermarking and embeddings

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• processing an audio signal to generate a spectrogram of the audio signal;
• processing the spectrogram of the audio signal using a watermark generation neural network to generate a watermark for the audio signal;
• and
• generating a watermarked version of the audio signal using the watermark for the audio signal generated by the watermark generation neural network.

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• processing an audio signal to generate a spectrogram of the audio signal;
• and
• processing the spectrogram of the audio signal using a watermark decoding neural network to generate a watermarking signal for the audio signal, wherein the watermarking signal predicts whether or not the audio signal is watermarked;
• and
• determining, based on the watermarking signal generated by processing the spectrogram of the audio signal using the watermark decoding neural network, whether a watermark is present in the audio signal.

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• maintaining an object verification system comprising:
• a first interface to receive a digital object or a request to generate a digital object;
• a second interface to provide a watermarked digital object for use;
• an embedding neural network configured to process the digital object to generate an embedding of the digital object;
• and
• an object verification database configured to store at least the embedding of the digital object;
• wherein the operations further comprise:
• receiving a query digital object for verification wherein the query digital object comprises a digital audio object;
• processing the query digital object using a watermark decoding neural network to generate a watermarking signal for the query digital object by processing the digital audio object to generate a spectrogram of the digital audio object, and processing the spectrogram using the watermark decoding neural network to generate the watermarking signal for the query digital object;
• processing the query digital object using the embedding neural network to generate a query embedding of the query digital object;
• interrogating the object verification database using the query embedding to determine a set of one or more similarity scores for a corresponding set of one or more stored embeddings of digital objects that are similar to the query digital object;
• and
• verifying a provenance of the query digital object based on a combination of the watermarking signal and the set of one or more similarity scores.

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• for a plurality of audio data objects each comprising a representation of an audio signal:
• processing the audio data object to generate a spectrogram of the audio data object;
• processing the spectrogram using a watermark generation neural network to generate a watermark for the audio data object;
• combining the watermark for the audio data object and the spectrogram to obtain a watermarked spectrogram;
• applying an adversarial transformation to one or both of a watermarked version of the audio data object obtained by converting the watermarked spectrogram to a watermarked version of the audio data object, and the watermarked spectrogram, to generate a perturbed watermarked data object;
• applying an adversarial transformation to one or both of the audio data object and the spectrogram of the audio data object to generate a perturbed data object;
• processing the perturbed watermarked data object using a watermark decoding neural network to generate a first watermarking signal that indicates whether or not the perturbed watermarked data object is predicted to be watermarked;
• processing the perturbed data object using the watermark decoding neural network to generate a second watermarking signal that indicates whether or not the perturbed data object is predicted to be watermarked;
• jointly training the watermark decoding neural network and the watermark generation neural network, using the first watermarking signal and the second watermarking signal, to distinguish between the perturbed watermarked data object and the perturbed data object.