Academic on Matias Di Bernardo

The effect of denosing on TTS

Fri, 22 Nov 2024 00:00:00 +0000

This study was conducted in the context of the class Laboratorio de Acústica at UNTREF. I chose this topic because it aligns with research I have been pursuing as part of the group Intercambios Transorgánicos. The class assignment involved conducting a subjective study using a survey to explore the relationship between objective and subjective variables.

In my research group, I have been investigating how denoising algorithms affect Text-to-Speech (TTS) systems trained on low-quality recordings. The focus is on Rioplatense Spanish, a regional accent with limited high-quality data. Within this context, it was natural to combine both tasks and perform a subjective test on the impact of denoising algorithms on TTS systems.

Overview

The key points of this investigation are:

Evaluation of three denoising algorithms: Wave U-Net, HiFi-GAN, and DeepFilterNet.
Use of both subjective (CMOS) and objective metrics (PESQ, STOI, MCD).
Insights into resource-efficient TTS model development for underrepresented accents.

Methodology

Algorithms: Wave U-Net, HiFi-GAN, and DeepFilterNet evaluated with the FastPitch TTS model.
Dataset: Subset of the ArchiVoz collection (15 minutes of noisy audio).
Testing: CMOS subjective test and objective metrics (PESQ, STOI, MCD).
Participants: 24 valid responses, including both experts and non-experts.

Key Findings

DeepFilterNet Performance:
- Achieved the highest CMOS score, reflecting the best subjective quality.
- Demonstrated significant improvements in TTS output despite mixed correlations with objective metrics.
Objective Metrics Analysis:
- PESQ and MCD showed limited correlation with subjective preferences.
- STOI scores were consistent across algorithms, indicating preserved intelligibility.
Algorithm Comparisons:
- DeepFilterNet: Superior subjective evaluations, moderate MCD.
- Demucs: Comparable to DeepFilterNet in PESQ but lower subjective scores.
- Wave U-Net: Poor subjective and objective performance.
Subject Expertise:
- No significant differences were observed between expert and non-expert evaluations in subjective testing.

Implications

Efficiency: Advanced denoising methods like DeepFilterNet can enhance TTS systems without requiring high-quality recordings.
Limitations: Objective metrics like PESQ and MCD are insufficient standalone indicators of subjective TTS quality.
Future Work:
- Expand datasets and noise levels for more robust analysis.
- Explore TTS systems trained jointly with denoising algorithms.

Conclusions

This work concludes that preprocessing with DeepFilterNet significantly improves TTS performance, with a 1.1 CMOS score increase. These findings underscore the importance of algorithm selection in optimizing low-resource TTS systems. Additionally, I gained valuable insights into subjective evaluations and the statistical analysis required to draw meaningful conclusions from data.

All the information for this study can be found in the academic report.

Time Scale Modification Algorithms

Sat, 11 Feb 2023 00:00:00 +0000

Time scale modifications algorithms are used to speed up or slow down the reproduction velocity of an audio. When you change the sample rate of an audio, the velocity changes but it also changes the pitch (when the audio is speed up it sounds highier pitch). There are different algorithms that change the velocity of the audio but mantain the pitch.

The main reference for this study is the following article, where all the different algorithms are describe in detail.

A Review of Time-Scale Modification of Music Signals.
— Jonathan Driedger and Meinard Müller¹

Algorithm comparison

There are two main algorithms, the Overlap-and-add (OLA) and the Phase Vocoder (PV). Both achieve good results under different signals and conditions. For this, a final implementation using Harmonic Percussion Separation (HPS) combines both algorithms and achiving the best results.

OLA

This method works on time domain and overlap sections of the audio (windows) and rearange it to achive a certain desire change on speed. This method works well for percussive signals, but it introduces artifacts when used with harmonic or tonal signals.

PV

This method works on frequency domain, and it combines chunks of audio in the frequency domain to achieve the desire change in time. This uses the phase vocoder principle to propagate the phase between the windows, this grantice the continuity of when applied to harmonic signals. On the contrary, it does not work for percussive signals becouse the phase propagation process eliminate the transients in the signals.

I created visualizations using Manim to enhance my class presentation. The first video demonstrates how the PV algorithm aligns windows to ensure smooth transitions in the generated signal over time. To achieve this, a Gaussian window is applied, which maintains continuity and smoothness, even at the start and end of the sequence.

The second video showcases the effects of applying the PV algorithm to a signal containing transients.

As predicted by theory, the transients vanish because the PV algorithm disrupts the vertical phase alignment. While these examples utilize idealized signals, they effectively demonstrate the key strengths and limitations of the algorithm.

HPS

To use both methods with their ideal signals, the HPS algorithms is used. This algorithm separete the signal into the harmonics and the percussive parts. It works by comparing the continuty of the signal in the STFT representation and using a filter to compare vertical versus horizontal presence on the spectrogram. With a threshold, a binary mask can be define over the spectrogram to separete the percussive parts from the harmoincs sections.

Results

We succesfully implement all the algorithms and compare them, verifying the theortical contents presented on the referenca article. In the process, we develope the toolkit to use this algorithms with the python programming language. All the code is avaliable on this repo

Academic Presentation

The study was preseented with my classmates on the JAAS (Jornadas de Acustica, Audio y Sonido). The main ideas and conclusions were preseneted on the conference. In the following repoert there is all the details and anlysis done for this proyect (in spanish).

ALGORITMOS DE MODIFICACIÓN DE ESCALA TEMPORAL.
— Matías Di Bernardo; Matías Vereertbruhggen; Sebastían Carro ²

A Review of Time-Scale Modification of Music Signal paper. ↩︎
JAAS 2023 - Algoritmos de Modificación de Escala Temporal paper. ↩︎

Comparative analysis of time-frequency transformations

Fri, 11 Nov 2022 00:00:00 +0000

This research is conducted as part of the subject Metodología de la Investigación at UNTREF. The article aims to compare the differences between three types of time-frequency transformations:

Fourier Transform (FT)
Wavelet Transform (WT)
Huang-Hilbert Transform (HHT)

The objective of this work is to understand the differences between these types of transformations and deepen my knowledge of signal processing.

Objective

The general objective of the research is to determine which spectral analysis tool achieves the highest accuracy in pitch detection tasks.

To achieve this objective, the following specific objectives are proposed:

Identify the parameters needed to represent the signal in the spectral domain for each case.
Select an algorithm that identifies the pitch of the signal based on its spectral representation.
Generate the data (audio signals) to be used for the comparison.
Evaluate the generated data with the different analysis methods and apply statistical processes to validate the results.
Establish a measure of accuracy for the pitch detection task.
Compare the results of the different analyses and determine which method achieves the highest accuracy in pitch detection.

The pitch detection task was chosen because it is one of the main applications of these types of transformations.

Algorithms

The theoretical analysis of all the transformations is performed in the continuous domain, but to conduct the experiments and comparisons, the discrete domain is used, enabling all calculations to be performed digitally.

FFT

The FFT is an algorithm that optimizes the DFT (Discrete Fourier Transform). With this algorithm, the spectral representation of the signal is obtained according to Fourier analysis, which decomposes a complex signal into a sum of sines or cosines. The DFT is calculated using the formula:

$$ X_k = \sum_{n=0}^{N-1} e^{-i\frac{2\pi}{N}kn} x_n $$

Where $ N $ is the number of signal samples, and $ k $ are natural numbers from $ 0 $ to $ N – 1 $.

WT

The Wavelet Transform (WT) uses an oscillatory function (wavelet) and applies a convolution between the signal and the chosen wavelet function to determine whether that wave shape is present in the signal. The wavelet is stretched and scaled in frequency and amplitude, allowing a single wavelet function to recreate the entire spectrum of interest.

In this research, the CDWT (Cyclic Discrete Wavelet Transform) will be used, the most common implementation when discretizing the WT. Conceptually, this transform extends Fourier analysis by projecting the signal onto a basis of wavelet functions instead of sines and cosines. It is calculated as follows:

$$ Wf[n, a^j] = \sum_{m=0}^{N-1} f[m] \psi_j[m-n] $$

Where $ N $ is the number of signal samples, $ \psi $ is the wavelet function, and $ j $ represents the deformation of the wavelet according to the selected wavelet bank.

HHT

Lastly, the Huang-Hilbert Transform (HHT) will be used for spectral representation. It employs a method called Empirical Mode Decomposition (EMD) to decompose the signal into subsignals that contain the relevant information of the original function.

Like the previous analysis methods, the key part of the analysis is the decomposition of the signal into simpler signals. However, instead of sine or wavelet functions, EMD finds intrinsic mode functions (IMFs) that form the basis of our decomposition and are unique to each signal.

The relationship between the IMFs and the frequency of the original signal is established with the equation:

$$ z(t) = f(t) + i H\{ f(t) \} $$

Where $ f(t) $ is an IMF of the original signal, and $ H $ is the Hilbert Transform. This allows the IMF to be represented as a complex signal by projecting it onto the imaginary axis using the Hilbert Transform.

Thus, the IMF is represented as a complex signal, and the amplitude and phase of each moment can be extracted to construct the spectral representation. Since a signal generally has multiple IMFs, this process is repeated for all of them, and the results are summed to obtain the complete spectrum.

Procedure

This research analyzes the relationship between types of spectral representation and accuracy in pitch detection.

First, the parameters for the different transformations will be selected. Among the most critical parameters to determine is the number of samples for temporal windowing, as it determines the trade-off between temporal and frequency resolution.

To ensure a fair comparison among the methods, data representing various cases of interest will be generated, including four types of signals:

Monophonic: Signals with a single note corresponding to $ F_0 $.
Polyphonic: Signals with multiple notes where harmony determines $ F_0 $.
Slow Transitions: Signals with gradual changes in $ F_0 $.
Fast Transitions: Signals with abrupt changes in $ F_0 $.

The real value $ V(t) $ will be compared to the result $ P(t) $ from each transform, integrating the difference over time to calculate the accuracy.

Results

At this stage, the task was to complete the research plan detailing the procedure and analysis methods. Dummy data was generated and statistically validated to simulate expected results.

The graph compares the precision achieved by the three transformations for different signal types. Based on the properties of the transforms, the Wavelet Transform (WT) is expected to outperform the Fourier Transform (FT), and the Huang-Hilbert Transform (HHT) is expected to achieve the highest precision overall.

Conclusions

In pitch detection tasks using spectral analysis, the Huang-Hilbert Transform (HHT) generally provides higher precision than the Fast Fourier Transform (FFT) and the Cyclic Discrete Wavelet Transform (CDWT).

The significance of this precision gain depends on the type of signal being analyzed, with fast-transition signals benefiting the least from the transformation change, while polyphonic signals show the most significant improvement when using the HHT.

This project allowed me to deepen my understanding of signal processing and grasp the foundations of why tools like the WT and HHT are used based on the characteristics of the signal being analyzed.

All details of this work are available in the following report.