Work on Matias Di Bernardo

Generative AI, Flow Matching and TTS

Mon, 24 Mar 2025 00:00:00 +0000

At Intercambios Transorgánicos, we continuously work with state-of-the-art Deep Learning models. To achieve this, we stay up to date with the latest developments in the field. In this presentation, the objective was to explain the fundamentals of the F5-TTS model, a cutting-edge speech synthesis system. During the session, the concept of generative artificial intelligence and flow-based models was introduced, as F5-TTS is based on this technique. The purpose is to share this knowledge with the team and with anyone interested in getting started with Text-to-Speech (TTS) models.

Introduction - Generative AI and TTS

The class was structured as follows:

Introduction to TTS models: Description of the different models used, their main characteristics, and the reasons behind changing approaches.
Evolution of TTS models: A graph-based survey illustrating the progression of various TTS technologies.
Paradigm shift in language models: Explanation of how using large datasets has enabled better generalization in language models.
Example of generalization: Illustrative cases, such as generating an image of an astronaut riding a horse, audio examples, and synthesis of specific accents (e.g., an angry Cordoban speaker).
Change in dataset structuring:
- Traditional approach: Use of a single high-quality speaker with a large amount of data (model trained for each speaker).
- Current and future approach: Use of diverse datasets with labels that enable training models capable of understanding speech concepts and generalizing across multiple tasks (VoiceBox examples).
- Computational cost: Discussion of the increasing model size and training difficulty, similar to large-scale language models (LLMs).
Explanation of Generative AI:
- Introduction to generative models and the concept of underlying distribution.
- Explanation of Transformers and their function in classification with context. Clarification that while Transformers are Non-Autoregressive (NAR) during training, they operate Autoregressively (AR) in inference.
- Difference between classification models and generative models based on dimensionality reduction and latent variables.

TTS Models: VoiceBox, E2, and F5

This section details the key advancements in TTS models leading to the development of F5-TTS.

1. VoiceBox TTS - In Context Learning

The initial architecture of VoiceBox is introduced, focusing on generalization through In Context Learning. This approach allows the model to infer information based on prior examples. To achieve this, it is given an audio sample with a missing segment, forcing it to complete it coherently.

“We show that Voicebox’s text-guided speech infilling approach is much more scalable in terms of data while subsuming many common speech generative tasks.”

This method, introduced by Meta in VoiceBox, represents a paradigm shift by demonstrating that TTS models can benefit from techniques previously used in language and image models.

However, the model still relies on classical TTS techniques such as phoneme duration prediction and Forced Alignment to map phonemes to Mel spectrograms, as their lengths do not match those of text tokens.

2. E2 TTS - Filler Tokens

E2 TTS simplifies many of the architectural decisions of VoiceBox, particularly in handling the differing lengths of Mel spectrograms and text. Instead of using forced alignment, it introduces Filler Tokens, which allow text and audio dimensions to be equalized more efficiently. This approach makes it easier for the model to learn temporal associations between the two elements.

Another significant change is that the model directly processes text instead of phonemes. While this might seem disadvantageous (since homographic words can have different pronunciations), context provides enough information to infer the correct pronunciation, similar to how humans process natural language.

3. F5 TTS - Open Source

Although E2 TTS is a major step forward, its training is slow due to the need to infer associations that were previously guided. F5 TTS optimizes the previous model with the following improvements:

Use of ConvNeXt: Enhances text processing.
Architectural change: Replaces the U-Net network with DiT (Diffusion Transformer).
Sway Sampling: Optimizes the inference process.

Additionally, F5-TTS is open-source, facilitating adoption and improvement by the community.

Flow Matching - Theoretical Foundations

To understand how F5-TTS works, it is essential to review the models it is based on, known as Flow Matching Models. These models have evolved from previous techniques in probabilistic distribution modeling.

1. Normalizing Flow

Normalizing Flows are a family of probabilistic models that transform a simple distribution into a more complex one through a series of invertible and differentiable functions. They are used to model high-dimensional data distributions and generate realistic samples with precise control over probability.

2. Residual Flow

Residual Flows extend Normalizing Flows by allowing more flexible transformations without strictly adhering to the invertibility condition. This is achieved through residual networks that approximate transformation functions without structural constraints.

3. Continuous Normalizing Flows (CNF)

Continuous Normalizing Flows reformulate Normalizing Flows using ordinary differential equations (ODEs). Instead of applying discrete transformations in successive steps, these models learn a continuous dynamic in the latent space, allowing for greater expressiveness and computational efficiency.

4. Flow Matching

Flow Matching is a technique that optimizes the transformation of a reference distribution into a target distribution by minimizing a specific cost function. It differs from previous approaches by not requiring explicit density function calculations, making it particularly useful for large-scale generative models such as F5-TTS.

Online Presentation

All of this sections are explained in detail on the online class avaliable here (in Spanish):

Understanding FastPitch and the Transformer Architecture

Wed, 23 Aug 2023 00:00:00 +0000

Understanding a modern deep learning model is challenging due to the extensive prior knowledge required and the rapid pace of advancements in the field. In the research project Intercambios Transorgánicos, we are working with TTS, specifically the FastPitch model from Nvidia. I have studied this model to fine-tune it for Spanish and shared my research process in a class to help my colleagues in the research group understand it better.

Understanding Seq2Seq Models

In Intercambios Transorgánicos, we previously used Tacotron2 as our TTS model. While Tacotron2 performs well, it has several issues, primarily during training and inference, due to its auto-regressive nature. In contrast, FastPitch is a non-auto-regressive (NAR) model. To grasp these distinctions, I delved into seq2seq models, exploring their evolution over time and creating a quick overview of the sequence analysis models that have been pivotal:

RNN
LSTM
Transformers

Tacotron2 is based on an LSTM (AR) model, whereas FastPitch utilizes Transformers (NAR). Understanding this technological progression provides crucial background knowledge, particularly about transformers, including positional encoding, a key factor in their non-auto-regressive nature, and the attention mechanism.

The Transformer Architecture

I began by studying the transformer architecture, as it is fundamental to the FastPitch model. I reviewed online resources and the seminal Attention is All You Need paper. Below are some key points I noted during my study:

Self-Attention Mechanism
- Purpose: Dynamically focuses on different parts of the input sequence.
- Mechanism:
  - Query (Q), Key (K), Value (V): Derived from input embeddings.
  - Attention scores are computed as the dot product of Q and K, scaled by the square root of the dimension.
  - Scores are normalized using softmax to create attention weights.
  - A weighted sum of V is computed based on these weights to produce the output.
Multi-Head Attention
- Purpose: Captures diverse relationships between tokens by applying multiple self-attention mechanisms in parallel.
- Mechanism: Outputs from multiple attention heads are concatenated and linearly transformed.
Positional Encoding
- Purpose: Provides information about token order in the sequence, compensating for the Transformer’s lack of built-in sequence order (unlike RNNs).
- Mechanism: A fixed or learnable vector is added to the input embeddings.
Encoder-Decoder Structure
- Encoder: Processes the input sequence into context-rich representations.
  - Components:
    - Multi-head self-attention
    - Feed-forward neural network (FFN)
    - Layer normalization and residual connections
- Decoder: Generates the output sequence by attending to both encoder outputs and previously generated tokens.
  - Components:
    - Masked multi-head self-attention (prevents attending to future tokens)
    - Multi-head attention over encoder outputs
    - FFN, layer normalization, and residual connections
Feed-Forward Network (FFN)
- Purpose: Introduces non-linearity and processes each token independently.
- Mechanism: Two linear layers with a ReLU activation in between.
Layer Normalization and Residual Connections
- Purpose: Stabilizes training and improves gradient flow by normalizing inputs to each layer and adding skip connections.

FastPitch

After covering the theory, I examined each section of the FastPitch architecture in detail. I provided a brief explanation of word embeddings and positional encoding, as these are complex topics, and I wanted to keep the class concise. FastPitch converts text into mel spectrograms, which are then transformed into audio by another model (in our case, HiFi-GAN). The training sequence involves the following steps:

Text to Word Embedding
Word Embedding concatenated with mel spectrogram
Positional encoding and FFT (Feed-Forward Transformer block)
Pitch Prediction
Phoneme Duration Prediction
Another FFT block
Fully connected layer
Output mel spectrogram

For each block, I presented the corresponding equations and provided qualitative explanations for their roles in the model. For instance, phoneme duration prediction is crucial for aligning a phoneme’s duration with the expected duration in the spectrogram.

Online Class

Finally, I summarized the most important points and conducted an online class to share these concepts with my colleagues. You can watch it here (in Spanish):