Inference-time control of transformers for audio generation
AssigneeMitsubishi Electric Research Laboratories, Inc.
InventorsGordon Wichern, Junghyun Koo, François G Germain, Sameer Khurana, Jonathan Le Roux
Life of this patent
Estimating the term…
Abstract
An audio system for synthesizing audio sounds having a desired audio trait executes an autoregressive generative audio transformer trained for generating the audio by processing inputs with multiple layers employing multi-head attention, and uses directional inference-time intervention (ITI) to push at least some outputs of at least some heads of the multi-head attention into a direction predetermined for the desired audio trait.
Claims19
An audio system for synthesizing audio sounds having a desired audio trait, comprising:
- a processor coupled to a memory storing instructions implementing executable modules of the audio system, the modules comprising:
- an input interface configured to collect inputs for synthesizing audio and data indicative of a desired audio trait in the synthesized audio;
- an autoregressive generative audio transformer trained for generating the audio by processing the inputs with one or more layers employing multi-head attention;
- an audio trait tuner configured to use directional inference-time intervention (ITI) to push at least some outputs of at least some heads of the multi-head attention into directions predetermined for each of the at least some heads according to the desired audio trait;
- and
- an output interface configured to output the audio generated by the autoregressive generative audio transformer;
- wherein the audio trait tuner is configured to:
- classify the outputs of the heads using classifiers trained in a supervised manner for corresponding heads, such that there is a one-to-one mapping between the heads and the classifiers;
- and
- select a subset of heads sensitive to the desired audio trait based on the classification;
- wherein the subset of heads is determined during the execution of the autoregressive generative audio transformer.
A method for synthesizing audio sounds having a desired audio trait, wherein the method uses a processor coupled with stored instructions implementing steps of the method, comprising:
- collecting inputs for synthesizing audio and data indicative of a desired audio trait in the synthesized audio;
- executing an autoregressive generative audio transformer trained for generating the audio by processing the inputs with multiple layers employing multi-head attention;
- using directional inference-time intervention (ITI) to push at least some outputs of at least some heads of the multi-head attention into a direction predetermined for the desired audio trait;
- and
- outputting the audio generated by the autoregressive generative audio transformer
- tuning one or more desired audio traits by classifying the outputs of the heads using classifiers trained in a supervised manner for corresponding heads, such that there is a one-to-one mapping between the heads and the classifiers;
- and
- selecting a subset of heads sensitive to the desired audio trait based on the classification;
- wherein the subset of heads is determined during the execution of the autoregressive generative audio transformer.
Dependencies are inferred from the claim text.