Math on Matias Di Bernardo

Generative AI, Flow Matching and TTS

Mon, 24 Mar 2025 00:00:00 +0000

At Intercambios Transorgánicos, we continuously work with state-of-the-art Deep Learning models. To achieve this, we stay up to date with the latest developments in the field. In this presentation, the objective was to explain the fundamentals of the F5-TTS model, a cutting-edge speech synthesis system. During the session, the concept of generative artificial intelligence and flow-based models was introduced, as F5-TTS is based on this technique. The purpose is to share this knowledge with the team and with anyone interested in getting started with Text-to-Speech (TTS) models.

Introduction - Generative AI and TTS

The class was structured as follows:

Introduction to TTS models: Description of the different models used, their main characteristics, and the reasons behind changing approaches.
Evolution of TTS models: A graph-based survey illustrating the progression of various TTS technologies.
Paradigm shift in language models: Explanation of how using large datasets has enabled better generalization in language models.
Example of generalization: Illustrative cases, such as generating an image of an astronaut riding a horse, audio examples, and synthesis of specific accents (e.g., an angry Cordoban speaker).
Change in dataset structuring:
- Traditional approach: Use of a single high-quality speaker with a large amount of data (model trained for each speaker).
- Current and future approach: Use of diverse datasets with labels that enable training models capable of understanding speech concepts and generalizing across multiple tasks (VoiceBox examples).
- Computational cost: Discussion of the increasing model size and training difficulty, similar to large-scale language models (LLMs).
Explanation of Generative AI:
- Introduction to generative models and the concept of underlying distribution.
- Explanation of Transformers and their function in classification with context. Clarification that while Transformers are Non-Autoregressive (NAR) during training, they operate Autoregressively (AR) in inference.
- Difference between classification models and generative models based on dimensionality reduction and latent variables.

TTS Models: VoiceBox, E2, and F5

This section details the key advancements in TTS models leading to the development of F5-TTS.

1. VoiceBox TTS - In Context Learning

The initial architecture of VoiceBox is introduced, focusing on generalization through In Context Learning. This approach allows the model to infer information based on prior examples. To achieve this, it is given an audio sample with a missing segment, forcing it to complete it coherently.

“We show that Voicebox’s text-guided speech infilling approach is much more scalable in terms of data while subsuming many common speech generative tasks.”

This method, introduced by Meta in VoiceBox, represents a paradigm shift by demonstrating that TTS models can benefit from techniques previously used in language and image models.

However, the model still relies on classical TTS techniques such as phoneme duration prediction and Forced Alignment to map phonemes to Mel spectrograms, as their lengths do not match those of text tokens.

2. E2 TTS - Filler Tokens

E2 TTS simplifies many of the architectural decisions of VoiceBox, particularly in handling the differing lengths of Mel spectrograms and text. Instead of using forced alignment, it introduces Filler Tokens, which allow text and audio dimensions to be equalized more efficiently. This approach makes it easier for the model to learn temporal associations between the two elements.

Another significant change is that the model directly processes text instead of phonemes. While this might seem disadvantageous (since homographic words can have different pronunciations), context provides enough information to infer the correct pronunciation, similar to how humans process natural language.

3. F5 TTS - Open Source

Although E2 TTS is a major step forward, its training is slow due to the need to infer associations that were previously guided. F5 TTS optimizes the previous model with the following improvements:

Use of ConvNeXt: Enhances text processing.
Architectural change: Replaces the U-Net network with DiT (Diffusion Transformer).
Sway Sampling: Optimizes the inference process.

Additionally, F5-TTS is open-source, facilitating adoption and improvement by the community.

Flow Matching - Theoretical Foundations

To understand how F5-TTS works, it is essential to review the models it is based on, known as Flow Matching Models. These models have evolved from previous techniques in probabilistic distribution modeling.

1. Normalizing Flow

Normalizing Flows are a family of probabilistic models that transform a simple distribution into a more complex one through a series of invertible and differentiable functions. They are used to model high-dimensional data distributions and generate realistic samples with precise control over probability.

2. Residual Flow

Residual Flows extend Normalizing Flows by allowing more flexible transformations without strictly adhering to the invertibility condition. This is achieved through residual networks that approximate transformation functions without structural constraints.

3. Continuous Normalizing Flows (CNF)

Continuous Normalizing Flows reformulate Normalizing Flows using ordinary differential equations (ODEs). Instead of applying discrete transformations in successive steps, these models learn a continuous dynamic in the latent space, allowing for greater expressiveness and computational efficiency.

4. Flow Matching

Flow Matching is a technique that optimizes the transformation of a reference distribution into a target distribution by minimizing a specific cost function. It differs from previous approaches by not requiring explicit density function calculations, making it particularly useful for large-scale generative models such as F5-TTS.

Online Presentation

All of this sections are explained in detail on the online class avaliable here (in Spanish):

Understanding FastPitch and the Transformer Architecture

Wed, 23 Aug 2023 00:00:00 +0000

Understanding a modern deep learning model is challenging due to the extensive prior knowledge required and the rapid pace of advancements in the field. In the research project Intercambios Transorgánicos, we are working with TTS, specifically the FastPitch model from Nvidia. I have studied this model to fine-tune it for Spanish and shared my research process in a class to help my colleagues in the research group understand it better.

Understanding Seq2Seq Models

In Intercambios Transorgánicos, we previously used Tacotron2 as our TTS model. While Tacotron2 performs well, it has several issues, primarily during training and inference, due to its auto-regressive nature. In contrast, FastPitch is a non-auto-regressive (NAR) model. To grasp these distinctions, I delved into seq2seq models, exploring their evolution over time and creating a quick overview of the sequence analysis models that have been pivotal:

RNN
LSTM
Transformers

Tacotron2 is based on an LSTM (AR) model, whereas FastPitch utilizes Transformers (NAR). Understanding this technological progression provides crucial background knowledge, particularly about transformers, including positional encoding, a key factor in their non-auto-regressive nature, and the attention mechanism.

The Transformer Architecture

I began by studying the transformer architecture, as it is fundamental to the FastPitch model. I reviewed online resources and the seminal Attention is All You Need paper. Below are some key points I noted during my study:

Self-Attention Mechanism
- Purpose: Dynamically focuses on different parts of the input sequence.
- Mechanism:
  - Query (Q), Key (K), Value (V): Derived from input embeddings.
  - Attention scores are computed as the dot product of Q and K, scaled by the square root of the dimension.
  - Scores are normalized using softmax to create attention weights.
  - A weighted sum of V is computed based on these weights to produce the output.
Multi-Head Attention
- Purpose: Captures diverse relationships between tokens by applying multiple self-attention mechanisms in parallel.
- Mechanism: Outputs from multiple attention heads are concatenated and linearly transformed.
Positional Encoding
- Purpose: Provides information about token order in the sequence, compensating for the Transformer’s lack of built-in sequence order (unlike RNNs).
- Mechanism: A fixed or learnable vector is added to the input embeddings.
Encoder-Decoder Structure
- Encoder: Processes the input sequence into context-rich representations.
  - Components:
    - Multi-head self-attention
    - Feed-forward neural network (FFN)
    - Layer normalization and residual connections
- Decoder: Generates the output sequence by attending to both encoder outputs and previously generated tokens.
  - Components:
    - Masked multi-head self-attention (prevents attending to future tokens)
    - Multi-head attention over encoder outputs
    - FFN, layer normalization, and residual connections
Feed-Forward Network (FFN)
- Purpose: Introduces non-linearity and processes each token independently.
- Mechanism: Two linear layers with a ReLU activation in between.
Layer Normalization and Residual Connections
- Purpose: Stabilizes training and improves gradient flow by normalizing inputs to each layer and adding skip connections.

FastPitch

After covering the theory, I examined each section of the FastPitch architecture in detail. I provided a brief explanation of word embeddings and positional encoding, as these are complex topics, and I wanted to keep the class concise. FastPitch converts text into mel spectrograms, which are then transformed into audio by another model (in our case, HiFi-GAN). The training sequence involves the following steps:

Text to Word Embedding
Word Embedding concatenated with mel spectrogram
Positional encoding and FFT (Feed-Forward Transformer block)
Pitch Prediction
Phoneme Duration Prediction
Another FFT block
Fully connected layer
Output mel spectrogram

For each block, I presented the corresponding equations and provided qualitative explanations for their roles in the model. For instance, phoneme duration prediction is crucial for aligning a phoneme’s duration with the expected duration in the spectrogram.

Online Class

Finally, I summarized the most important points and conducted an online class to share these concepts with my colleagues. You can watch it here (in Spanish):

Comparative analysis of time-frequency transformations

Fri, 11 Nov 2022 00:00:00 +0000

This research is conducted as part of the subject Metodología de la Investigación at UNTREF. The article aims to compare the differences between three types of time-frequency transformations:

Fourier Transform (FT)
Wavelet Transform (WT)
Huang-Hilbert Transform (HHT)

The objective of this work is to understand the differences between these types of transformations and deepen my knowledge of signal processing.

Objective

The general objective of the research is to determine which spectral analysis tool achieves the highest accuracy in pitch detection tasks.

To achieve this objective, the following specific objectives are proposed:

Identify the parameters needed to represent the signal in the spectral domain for each case.
Select an algorithm that identifies the pitch of the signal based on its spectral representation.
Generate the data (audio signals) to be used for the comparison.
Evaluate the generated data with the different analysis methods and apply statistical processes to validate the results.
Establish a measure of accuracy for the pitch detection task.
Compare the results of the different analyses and determine which method achieves the highest accuracy in pitch detection.

The pitch detection task was chosen because it is one of the main applications of these types of transformations.

Algorithms

The theoretical analysis of all the transformations is performed in the continuous domain, but to conduct the experiments and comparisons, the discrete domain is used, enabling all calculations to be performed digitally.

FFT

The FFT is an algorithm that optimizes the DFT (Discrete Fourier Transform). With this algorithm, the spectral representation of the signal is obtained according to Fourier analysis, which decomposes a complex signal into a sum of sines or cosines. The DFT is calculated using the formula:

$$ X_k = \sum_{n=0}^{N-1} e^{-i\frac{2\pi}{N}kn} x_n $$

Where $ N $ is the number of signal samples, and $ k $ are natural numbers from $ 0 $ to $ N – 1 $.

WT

The Wavelet Transform (WT) uses an oscillatory function (wavelet) and applies a convolution between the signal and the chosen wavelet function to determine whether that wave shape is present in the signal. The wavelet is stretched and scaled in frequency and amplitude, allowing a single wavelet function to recreate the entire spectrum of interest.

In this research, the CDWT (Cyclic Discrete Wavelet Transform) will be used, the most common implementation when discretizing the WT. Conceptually, this transform extends Fourier analysis by projecting the signal onto a basis of wavelet functions instead of sines and cosines. It is calculated as follows:

$$ Wf[n, a^j] = \sum_{m=0}^{N-1} f[m] \psi_j[m-n] $$

Where $ N $ is the number of signal samples, $ \psi $ is the wavelet function, and $ j $ represents the deformation of the wavelet according to the selected wavelet bank.

HHT

Lastly, the Huang-Hilbert Transform (HHT) will be used for spectral representation. It employs a method called Empirical Mode Decomposition (EMD) to decompose the signal into subsignals that contain the relevant information of the original function.

Like the previous analysis methods, the key part of the analysis is the decomposition of the signal into simpler signals. However, instead of sine or wavelet functions, EMD finds intrinsic mode functions (IMFs) that form the basis of our decomposition and are unique to each signal.

The relationship between the IMFs and the frequency of the original signal is established with the equation:

$$ z(t) = f(t) + i H\{ f(t) \} $$

Where $ f(t) $ is an IMF of the original signal, and $ H $ is the Hilbert Transform. This allows the IMF to be represented as a complex signal by projecting it onto the imaginary axis using the Hilbert Transform.

Thus, the IMF is represented as a complex signal, and the amplitude and phase of each moment can be extracted to construct the spectral representation. Since a signal generally has multiple IMFs, this process is repeated for all of them, and the results are summed to obtain the complete spectrum.

Procedure

This research analyzes the relationship between types of spectral representation and accuracy in pitch detection.

First, the parameters for the different transformations will be selected. Among the most critical parameters to determine is the number of samples for temporal windowing, as it determines the trade-off between temporal and frequency resolution.

To ensure a fair comparison among the methods, data representing various cases of interest will be generated, including four types of signals:

Monophonic: Signals with a single note corresponding to $ F_0 $.
Polyphonic: Signals with multiple notes where harmony determines $ F_0 $.
Slow Transitions: Signals with gradual changes in $ F_0 $.
Fast Transitions: Signals with abrupt changes in $ F_0 $.

The real value $ V(t) $ will be compared to the result $ P(t) $ from each transform, integrating the difference over time to calculate the accuracy.

Results

At this stage, the task was to complete the research plan detailing the procedure and analysis methods. Dummy data was generated and statistically validated to simulate expected results.

The graph compares the precision achieved by the three transformations for different signal types. Based on the properties of the transforms, the Wavelet Transform (WT) is expected to outperform the Fourier Transform (FT), and the Huang-Hilbert Transform (HHT) is expected to achieve the highest precision overall.

Conclusions

In pitch detection tasks using spectral analysis, the Huang-Hilbert Transform (HHT) generally provides higher precision than the Fast Fourier Transform (FFT) and the Cyclic Discrete Wavelet Transform (CDWT).

The significance of this precision gain depends on the type of signal being analyzed, with fast-transition signals benefiting the least from the transformation change, while polyphonic signals show the most significant improvement when using the HHT.

This project allowed me to deepen my understanding of signal processing and grasp the foundations of why tools like the WT and HHT are used based on the characteristics of the signal being analyzed.

All details of this work are available in the following report.