Deep Learning on Matias Di Bernardo

Generative AI, Flow Matching and TTS

Mon, 24 Mar 2025 00:00:00 +0000

At Intercambios Transorgánicos, we continuously work with state-of-the-art Deep Learning models. To achieve this, we stay up to date with the latest developments in the field. In this presentation, the objective was to explain the fundamentals of the F5-TTS model, a cutting-edge speech synthesis system. During the session, the concept of generative artificial intelligence and flow-based models was introduced, as F5-TTS is based on this technique. The purpose is to share this knowledge with the team and with anyone interested in getting started with Text-to-Speech (TTS) models.

Introduction - Generative AI and TTS

The class was structured as follows:

Introduction to TTS models: Description of the different models used, their main characteristics, and the reasons behind changing approaches.
Evolution of TTS models: A graph-based survey illustrating the progression of various TTS technologies.
Paradigm shift in language models: Explanation of how using large datasets has enabled better generalization in language models.
Example of generalization: Illustrative cases, such as generating an image of an astronaut riding a horse, audio examples, and synthesis of specific accents (e.g., an angry Cordoban speaker).
Change in dataset structuring:
- Traditional approach: Use of a single high-quality speaker with a large amount of data (model trained for each speaker).
- Current and future approach: Use of diverse datasets with labels that enable training models capable of understanding speech concepts and generalizing across multiple tasks (VoiceBox examples).
- Computational cost: Discussion of the increasing model size and training difficulty, similar to large-scale language models (LLMs).
Explanation of Generative AI:
- Introduction to generative models and the concept of underlying distribution.
- Explanation of Transformers and their function in classification with context. Clarification that while Transformers are Non-Autoregressive (NAR) during training, they operate Autoregressively (AR) in inference.
- Difference between classification models and generative models based on dimensionality reduction and latent variables.

TTS Models: VoiceBox, E2, and F5

This section details the key advancements in TTS models leading to the development of F5-TTS.

1. VoiceBox TTS - In Context Learning

The initial architecture of VoiceBox is introduced, focusing on generalization through In Context Learning. This approach allows the model to infer information based on prior examples. To achieve this, it is given an audio sample with a missing segment, forcing it to complete it coherently.

“We show that Voicebox’s text-guided speech infilling approach is much more scalable in terms of data while subsuming many common speech generative tasks.”

This method, introduced by Meta in VoiceBox, represents a paradigm shift by demonstrating that TTS models can benefit from techniques previously used in language and image models.

However, the model still relies on classical TTS techniques such as phoneme duration prediction and Forced Alignment to map phonemes to Mel spectrograms, as their lengths do not match those of text tokens.

2. E2 TTS - Filler Tokens

E2 TTS simplifies many of the architectural decisions of VoiceBox, particularly in handling the differing lengths of Mel spectrograms and text. Instead of using forced alignment, it introduces Filler Tokens, which allow text and audio dimensions to be equalized more efficiently. This approach makes it easier for the model to learn temporal associations between the two elements.

Another significant change is that the model directly processes text instead of phonemes. While this might seem disadvantageous (since homographic words can have different pronunciations), context provides enough information to infer the correct pronunciation, similar to how humans process natural language.

3. F5 TTS - Open Source

Although E2 TTS is a major step forward, its training is slow due to the need to infer associations that were previously guided. F5 TTS optimizes the previous model with the following improvements:

Use of ConvNeXt: Enhances text processing.
Architectural change: Replaces the U-Net network with DiT (Diffusion Transformer).
Sway Sampling: Optimizes the inference process.

Additionally, F5-TTS is open-source, facilitating adoption and improvement by the community.

Flow Matching - Theoretical Foundations

To understand how F5-TTS works, it is essential to review the models it is based on, known as Flow Matching Models. These models have evolved from previous techniques in probabilistic distribution modeling.

1. Normalizing Flow

Normalizing Flows are a family of probabilistic models that transform a simple distribution into a more complex one through a series of invertible and differentiable functions. They are used to model high-dimensional data distributions and generate realistic samples with precise control over probability.

2. Residual Flow

Residual Flows extend Normalizing Flows by allowing more flexible transformations without strictly adhering to the invertibility condition. This is achieved through residual networks that approximate transformation functions without structural constraints.

3. Continuous Normalizing Flows (CNF)

Continuous Normalizing Flows reformulate Normalizing Flows using ordinary differential equations (ODEs). Instead of applying discrete transformations in successive steps, these models learn a continuous dynamic in the latent space, allowing for greater expressiveness and computational efficiency.

4. Flow Matching

Flow Matching is a technique that optimizes the transformation of a reference distribution into a target distribution by minimizing a specific cost function. It differs from previous approaches by not requiring explicit density function calculations, making it particularly useful for large-scale generative models such as F5-TTS.

Online Presentation

All of this sections are explained in detail on the online class avaliable here (in Spanish):

The effect of denosing on TTS

Fri, 22 Nov 2024 00:00:00 +0000

This study was conducted in the context of the class Laboratorio de Acústica at UNTREF. I chose this topic because it aligns with research I have been pursuing as part of the group Intercambios Transorgánicos. The class assignment involved conducting a subjective study using a survey to explore the relationship between objective and subjective variables.

In my research group, I have been investigating how denoising algorithms affect Text-to-Speech (TTS) systems trained on low-quality recordings. The focus is on Rioplatense Spanish, a regional accent with limited high-quality data. Within this context, it was natural to combine both tasks and perform a subjective test on the impact of denoising algorithms on TTS systems.

Overview

The key points of this investigation are:

Evaluation of three denoising algorithms: Wave U-Net, HiFi-GAN, and DeepFilterNet.
Use of both subjective (CMOS) and objective metrics (PESQ, STOI, MCD).
Insights into resource-efficient TTS model development for underrepresented accents.

Methodology

Algorithms: Wave U-Net, HiFi-GAN, and DeepFilterNet evaluated with the FastPitch TTS model.
Dataset: Subset of the ArchiVoz collection (15 minutes of noisy audio).
Testing: CMOS subjective test and objective metrics (PESQ, STOI, MCD).
Participants: 24 valid responses, including both experts and non-experts.

Key Findings

DeepFilterNet Performance:
- Achieved the highest CMOS score, reflecting the best subjective quality.
- Demonstrated significant improvements in TTS output despite mixed correlations with objective metrics.
Objective Metrics Analysis:
- PESQ and MCD showed limited correlation with subjective preferences.
- STOI scores were consistent across algorithms, indicating preserved intelligibility.
Algorithm Comparisons:
- DeepFilterNet: Superior subjective evaluations, moderate MCD.
- Demucs: Comparable to DeepFilterNet in PESQ but lower subjective scores.
- Wave U-Net: Poor subjective and objective performance.
Subject Expertise:
- No significant differences were observed between expert and non-expert evaluations in subjective testing.

Implications

Efficiency: Advanced denoising methods like DeepFilterNet can enhance TTS systems without requiring high-quality recordings.
Limitations: Objective metrics like PESQ and MCD are insufficient standalone indicators of subjective TTS quality.
Future Work:
- Expand datasets and noise levels for more robust analysis.
- Explore TTS systems trained jointly with denoising algorithms.

Conclusions

This work concludes that preprocessing with DeepFilterNet significantly improves TTS performance, with a 1.1 CMOS score increase. These findings underscore the importance of algorithm selection in optimizing low-resource TTS systems. Additionally, I gained valuable insights into subjective evaluations and the statistical analysis required to draw meaningful conclusions from data.

All the information for this study can be found in the academic report.

Understanding FastPitch and the Transformer Architecture

Wed, 23 Aug 2023 00:00:00 +0000

Understanding a modern deep learning model is challenging due to the extensive prior knowledge required and the rapid pace of advancements in the field. In the research project Intercambios Transorgánicos, we are working with TTS, specifically the FastPitch model from Nvidia. I have studied this model to fine-tune it for Spanish and shared my research process in a class to help my colleagues in the research group understand it better.

Understanding Seq2Seq Models

In Intercambios Transorgánicos, we previously used Tacotron2 as our TTS model. While Tacotron2 performs well, it has several issues, primarily during training and inference, due to its auto-regressive nature. In contrast, FastPitch is a non-auto-regressive (NAR) model. To grasp these distinctions, I delved into seq2seq models, exploring their evolution over time and creating a quick overview of the sequence analysis models that have been pivotal:

RNN
LSTM
Transformers

Tacotron2 is based on an LSTM (AR) model, whereas FastPitch utilizes Transformers (NAR). Understanding this technological progression provides crucial background knowledge, particularly about transformers, including positional encoding, a key factor in their non-auto-regressive nature, and the attention mechanism.

The Transformer Architecture

I began by studying the transformer architecture, as it is fundamental to the FastPitch model. I reviewed online resources and the seminal Attention is All You Need paper. Below are some key points I noted during my study:

Self-Attention Mechanism
- Purpose: Dynamically focuses on different parts of the input sequence.
- Mechanism:
  - Query (Q), Key (K), Value (V): Derived from input embeddings.
  - Attention scores are computed as the dot product of Q and K, scaled by the square root of the dimension.
  - Scores are normalized using softmax to create attention weights.
  - A weighted sum of V is computed based on these weights to produce the output.
Multi-Head Attention
- Purpose: Captures diverse relationships between tokens by applying multiple self-attention mechanisms in parallel.
- Mechanism: Outputs from multiple attention heads are concatenated and linearly transformed.
Positional Encoding
- Purpose: Provides information about token order in the sequence, compensating for the Transformer’s lack of built-in sequence order (unlike RNNs).
- Mechanism: A fixed or learnable vector is added to the input embeddings.
Encoder-Decoder Structure
- Encoder: Processes the input sequence into context-rich representations.
  - Components:
    - Multi-head self-attention
    - Feed-forward neural network (FFN)
    - Layer normalization and residual connections
- Decoder: Generates the output sequence by attending to both encoder outputs and previously generated tokens.
  - Components:
    - Masked multi-head self-attention (prevents attending to future tokens)
    - Multi-head attention over encoder outputs
    - FFN, layer normalization, and residual connections
Feed-Forward Network (FFN)
- Purpose: Introduces non-linearity and processes each token independently.
- Mechanism: Two linear layers with a ReLU activation in between.
Layer Normalization and Residual Connections
- Purpose: Stabilizes training and improves gradient flow by normalizing inputs to each layer and adding skip connections.

FastPitch

After covering the theory, I examined each section of the FastPitch architecture in detail. I provided a brief explanation of word embeddings and positional encoding, as these are complex topics, and I wanted to keep the class concise. FastPitch converts text into mel spectrograms, which are then transformed into audio by another model (in our case, HiFi-GAN). The training sequence involves the following steps:

Text to Word Embedding
Word Embedding concatenated with mel spectrogram
Positional encoding and FFT (Feed-Forward Transformer block)
Pitch Prediction
Phoneme Duration Prediction
Another FFT block
Fully connected layer
Output mel spectrogram

For each block, I presented the corresponding equations and provided qualitative explanations for their roles in the model. For instance, phoneme duration prediction is crucial for aligning a phoneme’s duration with the expected duration in the spectrogram.

Online Class

Finally, I summarized the most important points and conducted an online class to share these concepts with my colleagues. You can watch it here (in Spanish):

AI learns to play 2048 game

Fri, 17 Feb 2023 00:00:00 +0000

I used this project as an introduction to Reinforcement Learning. Having mostly worked with supervised and unsupervised learning, I wanted to start with a small, simple project to quickly grasp the main ideas and create something fun. I followed this YouTube video as a reference, which explains how to use Reinforcement Learning (RL) to train a model capable of playing the snake game. To make it more challenging, I applied the same network to the 2048 game.

Code

The code has three main components:

The Game: Implements the game logic and the graphical user interface (GUI).
Agent: Controls the gameplay.
AI Model: A neural network that learns how to play and guides the agent.

Game State

I modeled the game state as a 4x4 matrix representing the board. The actions in the game are represented by a vector, with the following possibilities:

Left: [1, 0, 0, 0]
Right: [0, 1, 0, 0]
Up: [0, 0, 1, 0]
Down: [0, 0, 0, 1]

Reward

The RL model works with rewards to quantify when the agent performs well or poorly. Initially, I defined the following rewards:

+10 points: When the agent doubles the points. This is crucial because, in 2048, points increase exponentially.
-10 points: When the game is lost. This serves as a straightforward negative reward.

Initially, the model showed slow improvement. To address this, I added an additional reward:

+2 points: When points are increased. This reward incentivizes actions that maximize board clearance and progress.

How the Model Works

The model is a simple linear neural network. It takes the game state as input and predicts the best next action to maximize the reward.

Rewards are managed using a technique called Q-Learning. A Q Value represents the quality of a decision based on the loss function. The loss function is derived from the Bellman Equation:

$$ NewQ(s, a) = Q(s, a) + \alpha [R(s, a) + \lambda \, \text{max}Q'(s', a') - Q(s, a)] $$

Where:

$Q(s, a)$: The Q Value for a specific state and action.
$\alpha$: Learning rate.
$R(s, a)$: Reward for a specific state and action.
$\lambda$: Discount rate.
$\text{max}Q’(s’, a’)$: Maximum expected future reward.

Experiments and Results

Random Test as a Baseline

To establish a baseline, I tested the average score achievable by taking random actions. After 1,000 iterations, the average score was 170, far below the 2048 points needed to win the game.

Initial Results

My initial attempts were discouraging. The model performed worse than random movements. Here are some early results:

Game 1,000 | Score: 208 | Record: 416 | Mean Score: 200 | Reward: 640
Game 1,000 | Score: 116 | Record: 346 | Mean Score: 161 | Reward: 310
Game 1,000 | Score: 112 | Record: 348 | Mean Score: 146 | Reward: 320

In these attempts, the agent developed a suboptimal strategy of filling the board before increasing points.

Improvements

After experimenting with the reward parameters, I focused on the exploration phase. Initially, the exploration games parameter, which randomly picks moves, was set to 25 games. Increasing this parameter allowed the agent to explore more strategies, leading to better results:

Game 1,000 | Score: 478 | Record: 478 | Mean Score: 230
Game 1,000 | Score: 514 | Record: 964 | Mean Score: 382

As the model improved, I extended the training to more games:

Game 3,796 | Score: 770 | Record: 1,366 | Mean Score: 469
Game 4,852 | Score: 631 | Record: 1,462 | Mean Score: 483

Finally, with 200 exploration games and 5,000 training games, the results were as follows:

Game 5,000 | Score: 840 | Record: 1,678 | Mean Score: 512

Although the model didn’t beat the game, I was satisfied with the progress. I believe that with longer training (the final run lasted 4 hours) and additional network layers, it would be possible to win the game using this architecture.

Demo

Here is a demo of the application, showcasing the agent’s learning process:

I implemented keybindings to control the game’s speed, providing three options:

Fast: For rapid training.
Slow: To observe and analyze the agent’s progress and mistakes.
Medium: Rarely used.

The complete code for this project is available in this repository.

Evaluation of different models for Speaker Diarization task

Fri, 04 Nov 2022 00:00:00 +0000

This project started as the final assignment for a seminar class at UNTREF called Seminario en Aplicaciones de Redes Neuronales en la recuperación de información musical. The objective was to use a Siamese Neural Network (SNN) in a different context than the one explored in class (music similarity detection).

For the final project of this course, we developed an SNN model from scratch using the Keras framework and the SincNet architecture to reduce audio dimensionality, achieving good results. Later, to expand this project, I tried another approach by using Wav2Vec for dimensionality reduction and re-implementing the entire project in the PyTorch framework. This attempt, however, yielded suboptimal results, indicating that the dimensionality reduction using Wav2Vec lost critical information required for the speaker diarization task.

Speaker Diarization Task

The goal of a speaker diarization model is to identify different speakers in an audio stream containing multiple speakers. For example, in a podcast with two people (A and B), the model must determine the time steps where speaker A is talking and the time steps where speaker B is speaking (and implicitly identify the periods of silence). These models are incredibly useful for audio editing and analyzing long audio sequences with multiple speakers.

Why Siamese Neural Networks?

In class, we explored the Siamese architecture to compare similarities between pieces of music, developing a tool capable of identifying covers of famous songs.

A Siamese Neural Network consists of two or more identical subnetworks sharing the same weights and parameters. It is designed to compare input pairs and measure their similarity, typically using a distance metric like Euclidean distance. Each subnetwork processes one input, and the outputs are combined to compute a similarity score.

With this similarity comparison in mind, we wanted to apply these networks to the speaker diarization task. The idea was to generate speaker embeddings from audio using a pre-trained model and compare the outputs of different audio segments. Based on the similarity score, we aimed to identify the segments where different speakers are talking.

Experiments

I tested two different methods for audio feature extraction to serve as speaker embeddings.

Keras Implementation with SincNet

In this approach, we used the SincNet architecture to extract speaker-specific features from audio. SincNet applies learnable sinc functions as its filters, which are particularly well-suited for audio processing as they mimic traditional bandpass filters. These features were then fed into the Siamese Neural Network, which compared pairs of audio segments to calculate their similarity scores. The model was trained on labeled audio datasets, and we observed strong performance in clustering audio segments by speaker, achieving clear boundaries between different speakers.

A report with the results can be found in the following Jupyter notebook (in Spanish).

PyTorch Implementation with Wav2Vec

For this method, I utilized Wav2Vec, a powerful pre-trained model for extracting deep audio embeddings. Unlike SincNet, Wav2Vec embeddings are derived from self-supervised learning, capturing high-level representations of audio. These embeddings were used in the Siamese Neural Network for similarity comparisons. However, the results were suboptimal for the diarization task. It appears that Wav2Vec, while excellent for speech recognition tasks, lost some speaker-specific details necessary for distinguishing between speakers in our setup.

Results

The experiments demonstrated that the choice of feature extraction method is crucial for speaker diarization. The Keras implementation with SincNet outperformed the PyTorch implementation with Wav2Vec, showing higher accuracy in identifying speaker transitions. This suggests that task-specific feature extraction, like SincNet, is more effective than general-purpose embeddings like Wav2Vec for speaker diarization.

The code for this project is available in this repository.

Conclusions

This project was one of my first experiences with deep learning models, where I applied my knowledge to a problem without following a specific paper or using a pre-trained model. I explored different solutions and concluded on the importance of feature extraction and model selection.

It also helped me become familiar with the syntax of the most popular deep learning frameworks and solidified my understanding in the process.