System and method for multi-conditioned audio generation
AssigneeRobert Bosch GmbH
InventorsPongtep Angkititrakul, Long Huang, Samarjit Das
Life of this patent
Estimating the term…
Abstract
A method for audio generation includes defining an audio input condition for an obtained input using an encoder, where the obtained input is indicative of one or more audio characteristics. The method further includes defining an audio style condition of a selected audio style profile employing an audio feature extraction neural network, and outputting a generated audio data indicative of a desired generated audio using a multi-conditioned latent diffusion model that employs the audio input condition and the audio style condition as adapters to the multi-conditioned latent diffusion model.
Claims17
A method for audio generation, comprising
- defining an audio input condition for an obtained input using an encoder, the obtained input being indicative of one or more audio characteristics;
- defining an audio style condition of a selected audio style profile employing an audio feature extraction neural network;
- and
- outputting a generated audio data indicative of a desired generated audio using a multi-conditioned latent diffusion model that employs the audio input condition and the audio style condition as adapters to the multi-conditioned latent diffusion model, the multi-conditioned latent diffusion model being trained using the audio style condition as a local control condition and the audio input condition as a global control condition concatenating with one or more text tokens associated with a text condition.
A system for multi-conditional audio generation comprising:
- one or more hardware computing devices configured to:
- define an audio input condition for an obtained input using an encoder, the obtained input being indicative of one or more audio characteristics;
- define an audio style condition of a selected audio style profile employing an audio feature extraction neural network;
- and
- output a generated audio data indicative of a desired generated audio using a multi-conditioned latent diffusion model that employs the audio input condition and the audio style condition as adapters to the multi-conditioned latent diffusion model, wherein the multi-conditioned latent diffusion model is trained using the audio style condition as a local control condition and the audio input condition as a global control condition concatenating with one or more text tokens associated with a text condition.
A non-transitory computer-readable medium comprising instructions for a multi-conditional audio generation system that, when executed by one or more hardware computing devices cause the one or more hardware computing devices to perform operations including to:
- define an audio input condition for an obtained input using an encoder, the obtained input being indicative of one or more audio characteristics;
- define an audio style condition of a selected audio style profile employing an audio feature extraction neural network;
- and
- output a generated audio data indicative of a desired generated audio using a multi-conditioned latent diffusion model that employs the audio input condition and the audio style condition as adapters to the multi-conditioned latent diffusion model, wherein the multi-conditioned latent diffusion model is trained using the audio style condition as a local control condition and the audio input condition as a global control condition concatenating with one or more text tokens associated with a text condition.
Dependencies are inferred from the claim text.