From voice to virtuosity: DDSP-based timbre transfer

Landarech Mayo, Iker

From voice to virtuosity: DDSP-based timbre transfer

License

Citation

Landarech Mayo, I. From voice to virtuosity: DDSP-based timbre transfer. 2024. handle: http://hdl.handle.net/10230/61316

This citation was generated automatically.

Abstract

Timbre is a crucial but elusive feature of music; it originates in the basic physical structure of sound waves but blossoms to touch upon a range of perceptual and social processes that are fundamental to how we derive meaning and emotion from music. This thesis explores timbre transfer with a new Differential Digital Signal Processing (DDSP)-based approach that leverages strong inductive biases without losing the expressive power of deep neural networks and end-to-end learning. By seamlessly integrating classic signal processing synthesizers as differentiable interpretable functions within a neural network, this method achieves high-fidelity generation without relying on large black-box autoregressive models or adversarial losses. This integration results in a domain-appropriate strong inductive bias, enabling a new interpretable representation and modular framework based on parametric signal processors for generative modeling. In this physically grounded approach, fundamental frequency and loudness envelopes are explicitly presented as domain-specific conditioning features to the model, which predicts the timbre via time-varying harmonic amplitudes and filtered noise, yielding an efficient end-to-end backpropagation and reducing model complexity and data requirements. This study presents state-of-the-art results through self-recorded trained models and demonstrates AuraVox, a virtual instrument implementation that enables real-time timbre transfer.