Automatic Speech Recognition (ASR), also known as Speech to Text (STT), uses Deep Learning technologies to transcribe speech-included audios to texts. In the fields of Deep Learning Artificial Intelligence, Large Language Models (LLMs) mimic human brains in processing words and phrases, and have the ability to understand and generate text data. LLMs usually contain millions of weights and pre-trained with various kinds of datasets. Specifically, an ASR LLM will convert audio inputs to desired input formats by feature extraction and tokenization.

To customize an ASR LLM with ideal performance, fine-tuning procedures of Whisper, an ASR LLM developed by OpenAI, were tested on Google Colaboratory first. Larger models were then deployed in GPU-equipped environments in Windows OS to speed up training and alleviate GPU availability or limit issues on Colab and MacOS. Audio data were investigated on reliability based on information such as audio quality and transcript accuracy. Models were then improved and optimized by data preprocessing and hyper-parameter tuning techniques. In case of failing to resolve GPU memory issues by means of regular fine-tuning, Parameter-Efficient-Fine-Tuning (PEFT) with Low Rank Adaptation (LoRA) was utilized to freeze most parameters to save memory allocation without sacrificing too much in performances. Results were visualized along with loss curves to ensure the fitness and optimization of fine-tuning processes.

Possibility of multi-speaker support in Whisper was explored using Neural Speaker Diarization. Integration with Pyannote was implemented using pipeline and WhisperX, a project containing similar ideas with extra features of word-level timestamps and Voice Activity Detection (VAD). WhisperX was tested on long-form transcription with batching as well as diarization.

Besides Whisper, other models with ASR functionality were installed and compared with Whisper baseline, including Massively Multilingual Speech (MMS) by Meta AI research, PaddleSpeech by PaddlePaddle, SpeechBrain and ESPNet. Chinese datasets were used to compare these models in CER metrics. In addition, Custom Speech in Azure AI, which supports real-time STT features, was introduced to compare performances (mainly Mandarin Chinese). Then a choice can be made between trained Azure models and loadable models like Whisper for deployment.

Google Colaboratory is a hosted Jupyter Notebook service that has limited free GPU & TPU computing resources. In Google Colaboratory, ipynb extension format is used to edit and execute Python scripts.

Log in to Google Colab through Google account, share written scripts with others via “share” on the right top corner of the page, and optionally authorize Colab with a Github account.

How to set up environments on Colab:

Besides Colab, environments can also be prepared on local PCs. Anaconda is a well-known distribution platform for Data Science field, including data analysis and building machine learning models in Python. It contains Conda, an environment and package manager that helps to manage open-source Python packages and libraries.

How to set up environments with Anaconda:

Visual Studio Code, or VS Code, is a powerful source-code editor for Windows, MacOS and Linux with various programming languages available for editing. It supports multiple tasks, including debugging, executing in integrated terminals, enriching functionalities by extensions, and version control by embedded Git.

How to set up environments in VS Code:

An alternative way is to use the ipynb extension (Jupyter Notebook).

Compute Unified Device Architecture (CUDA) is a parallel computing platform and Application Programming Interface (API) developed by NVIDIA. It allows developers to use NVIDIA Graphics Processing Units (GPUs) for multiple computing tasks.

How to use CUDA GPU:

On Windows, GPU information can be visualized and surveilled in Task Manager

Utilization is a useful indication to see if GPU is utilized as expected. If CPU is used heavily instead of GPU, then it means GPU has not been used for training and might indicate some problems within GPU settings. On the other hand, dedicated GPU memory measures GPU memory usage. A high memory usage will probably cause CUDA Out of Memory (OOM) errors.

Hugging Face is a company and an open-source platform that is dedicated to the fields of Natural Language Processing (NLP) and Artificial Intelligence.

It is important to create a Hugging Face account in order to utilize the models published or upload models customized by our own. Personal READ and WRITE tokens can be created on https://huggingface.co/settings/tokens, where READ token is for downloading models or datasets from the platform and WRITE for uploading local models or datasets. To specify tasks and categories, select tags from the left side of the website.

It is convenient to find most popular models and audio datasets based on downloads and likes. One point to notice is that some models are only supported in certain languages or certain tasks. And for the same model architecture or type, there can also be size differences.

On Hugging Face Platform, most well-known organizations or companies will publish their speech-to-text models and collected datasets. Sometimes an agreement is needed for data access. There are also source files on Hub that can be downloaded to the local.

Here are some common ASR LLMs and their relevant information:

Here are some audio datasets with their relevant information:

Open SLR is another useful website that hosts speech and language resources with compressed files. Various audio datasets can be seen along with their brief summaries in the Resources tab. Some datasets that are not existent on Hugging Face might have access or link to websites.

Specifically for Chinese audio datasets, there are resources that are ideal for ASR purposes and not found on Hugging Face:

Whisper is an ASR (Automatic Speech Recognition) system released by OpenAI in September, 2022. It was trained on 680,000 hours of multilingual and multitask supervised data, enabling multiple language transcription and translation. The architecture is an encoder-decoder Transformer.

The audios will be chunked into 30 seconds and converted into a log-Mel spectrogram, which enables frequencies to be changed into the Mel scale. Then it will be passed into an encoder.

The official website for OpenAI Whisper can be seen in Introducing Whisper, and its research papers are Robust Speech Recognition via Large-Scale Weak Supervision.

Training losses, evaluation losses and WER result in each evaluation steps or epochs can be seen in the training process, a fine-tuned model will then complete after the training, and checkpoints can be made and stored in specified output directory.

If CUDA Out of Memory (OOM) Error occurred during the training process, adjustments are needed on the previous steps to reduce the GPU memory usage:

There are many popular Python libraries that are suitable for performing tasks in Machine Learning and Data Processing. In Automatic Speech Recognition field, some Python Libraries are powerful in modifying audios.

Load the dataset using the load_dataset function. In many popular audio datasets, the splits of train, test and validation have already been preprocessed.

# common_voice for data source
common_voice = DatasetDict()

# split datasets
common_voice["train"] = load_dataset("common_voice"
    , "ja", split="train+validated", use_auth_token=True)
common_voice["validation"] = load_dataset("common_voice"
    , "ja", split="validation", use_auth_token=True)
common_voice["test"] = load_dataset("common_voice"
    , "ja", split="test", use_auth_token=True)

# create DatasetDict, choose the sample size for training and evaluation
common_voice = DatasetDict({
    "train": common_voice['train'].select(range(3500)),
    "validation": common_voice['validation'].select(range(500)),
    "test": common_voice['test'].select(range(100)),
})

# remove columns that are not needed for the training
common_voice = common_voice.remove_columns(["age", "client_id", "down_votes", "gender", "path", "up_votes"])

For non-streaming mode, ensure enough space for the download before calling load_dataset function. In order to create randomization, use shuffle(seed=42) to randomly rearrange the column values. select() and filter() are options that reduce the dataset numbers and return rows that match a specified condition.

As described in the Fine-tuning on Colab Section, columns can be manipulated by removing and renaming columns. Use cast_column() to change the feature type of a single column (or more in cast()).

Here the sampling rate should always be transformed to 16k Hz because of the requirements in Whisper architecture. flatten() is called when inside features are needed from nested structures in the dataset.

To combine datasets that come from different sources

For streaming mode, there are many similar call functions compared to non-streaming mode, but some are completely different. To use shuffling as the regular mode, set up a buffer size and randomly sample examples from this buffer.

It is also possible to create a new dataset with the first n examples by using take(n), or with the rest of the examples by skipping the first n examples with skip(n)

There are specific explanations on Hugging Face Documents: Process

The map function in HuggingFace's datasets is used to apply a given function to each example in a dataset. It performs transformation and pre-processing in a convenient and efficient way.

Here is the basic syntax:

Self-defined functions can be applied to every sample in the dataset and decided on whether it is processed in batches or one by one with the boolean parameter batched. num_proc parameter defines the number of processes to use for parallel processing. It helps speed up the data processing, especially when dealing with large datasets. However issues related to multi-processing may occur when increasing the number. Meanwhile,  unnecessary columns should be removed after execution, and if the desired dataset is created within the defined function, the initial train column should be removed.

Sometimes human labeled transcripts are not ideal, having extra symbols or different formats with other data sources. Transcript cleaning is then an essential preprocessing step. It involves the process of refining or correcting the transcriptions of speech data. It directly impacts the performance of the ASR models, so it's crucial to ensure accurate and error-free transcriptions. Sometimes it is also called text normalization.

Audio chunking, also known as audio segmentation or audio slicing, is the process of breaking down a continuous audio stream into smaller, manageable segments or chunks.

Librosa, Pydub and SoundFile libraries are recommended for doing audio chunking. Use librosa.load call to read an audio file. The AudioSegment class in Pydub is helpful to create chunks, like AudioSegment.from_file(). The SoundFile library can read and write sound files in various formats.

In Whisper, Seq2SeqTrainingArguments and Seq2SeqTrainer are utilized to perform training because of its sequence to sequence architecture. There are parameters in the training arguments that can significantly affect the learning ability of LLMs.

Abbreviations

Tables:

According to Graph 1, Whisper-small has poor WER performance (above 60%) on Hindi transcription with small sample size. While combining training data sources and increasing sample size can improve models by some levels, better performances might be restricted because of complexity. Hence increasing model complexity is more effective. In the meantime, possible overfitting or catastrophic forgetting issues might occur when training with a large sample size or with a single source. Such WER results should not be considered as good references. In addition, Whisper model in English transcriptions had a WER rate much smaller than Hindi and Japanese. As Japanese is character-based, a more suitable evaluation metric is Character Error Rate (CER).

Sometimes out of memory errors might occur while full fine-tuning, and they are usually caused by low-resource hardwares and large model weights. In case the full fine-tuning can not be completed after attempts on adjusting hyper-parameters like batch sizes, then Parameter-Efficient Fine-tuning (PEFT) approaches might be a useful alternative. This approach only fine-tunes a small number of model parameters while freezing most parameters of the pre-trained LLMs, thereby greatly decreasing computational and storage costs. This also overcomes the issues of catastrophic forgetting, a behavior observed during the full fine-tuning of LLMs.

LoRA is for Low Rank Adaptation. The goal of Low Rank Adaptation is to improve the efficiency and performance of adapting a pre-trained model to a new task by reducing the dimensionality of the model's parameters. Low rank adaptation decomposes the weights of pre-trained models into low-rank matrices or tensors and significantly reduces the number of parameters that need to be fine-tuned.

Compared to normal full fine-tuning, the difference comes when using pre-trained checkpoints from Conditional Generation. load_in_8bit=True will quantize the model to use one-fourth precision than float32 with minimal loss to performance. The device_map=”auto” argument automatically determines how to load and store model weights. prepare_model_for_int8_training will cast all the non int8 modules to full precision (FP32) for stability, adds a forward hook to the input embedding layer and enables gradient checkpointing for memory-efficient training.

Now configurations on the PEFT model can be set using LoraConfig

model.print_trainable_parameters() call can print the output of the sizes of trainable parameters, full parameters and its percentage. The percentage shows the exact percent usage compared to the full fine-tuning. Usually the percentage is around 1.

Compared to full fine-tuning, it’s necessary to explicitly set remove_unused_columns=False and label_names=["labels"] as the model doesn't inherit the signature of base model. Additionally, predict_with_generate call can not be used as it internally calls generate function without auto-casting. For the same reason, compute_metrics can not be put into the training arguments.

We can choose to write a custom TrainerCallback to save model checkpoints during training. The functions will save the adapter_model weights and remove the base model weights in pytorch_model.bin. Pass the Seq2SeqTrainingArguments, model, datasets, data collator, tokenizer, and callbacks to the Seq2SeqTrainer. Set model.config.use_cache = False to silence warnings. Finally PEFT model is ready for training!

Trained PEFT model can be pushed to or obtained from Hugging Face. PEFT model will have only two files, adapter_model.bin and adapter_config.json. Set model.config.use_cache = True to enable inference. An evaluation loop should be designed to evaluate model performance. As the call to predict_with_generate is disabled as previously stated, the eval loop could be hand rolled with torch.cuda.amp.autocast().

DataLoader loads test dataset in batches. The model generates predictions for the given input features using the specified decoder prompt IDs and a maximum limit of new tokens. The decoded predictions and labels are appended to the respective lists. The predictions and labels are converted from tensors to NumPy arrays and then decoded back into text format using the tokenizer's batch_decode method.

After each batch evaluation, intermediate variables are deleted, and garbage collection is triggered to manage memory. Finally WER and normalized WER are shown in the output.

Abbreviations

From Graph 2, Whisper with PEFT had poor WER performance on Japanese transcription in any model and sample size. However, as Japanese is character-based language, WER might not fully represent the model performance. After switching to Vietnamese, which is a language with English-like composition, the results were within expectations.

Loss curves are often used in machine learning to monitor the performance of a model during training. They show how the loss functions change over epochs or steps. The loss function quantifies how well the model's predictions match the actual target values. Matplotlib is a powerful Python library for data visualization in graphs.

After inputting losses and epochs data manually, a graph will be created

There are several patterns and rules to identify if the loss curves indicate a good fit for Whisper or any other machine learning models. Strategies can be applied based on the information from plotted visualized graphs. This helps us speed up finding an ideal model.

With Graph 4 results on Whisper Baseline Inference, the plots suggests qualities and splits of audio data sources could have some effects on model performances.

In the English category, TED LIUM (long form) had the worst accuracy with lowest 24.049% WER, and Common Voice the second worst, GigaSpeech third and LibriSpeech the best, with 4.746% WER. This is reasonable, as TED LIUM consists of TED talks that contain noises and is not taking ASR tasks as primary purposes. Common Voice, on the other hand, has significant variation in audio qualities since it collects data from vast amounts of volunteers and contributors. GigaSpeech has the longest trainable hours of audios, but it was recorded from Podcast and Youtube, and thus may have some sound quality losses. Librispeech consists of narrated audiobooks collected from the LibriVox project. It had been carefully segmented and cleaned by researchers.

In the Chinese category, Aishell and MagicData resulted in significantly different CERs. As both of them were divided by speakers, there might be considerable fluctuations of performance within the dataset. However, MagicData has better claimed transcript accuracy, longer training hours, and more speakers. A possible reason is audio quality differences. Aishell audios were recorded by high fidelity microphone, and Hi-Fi audios were created and then downsampled to 16k Hz within the pre-processing steps, while MagicData audios were recorded by mobile phones.

Speaker Diarization is a field in speech recognition that involves segmenting a speech audio into distinct segments corresponding to different speakers. The goal is to identify and differentiate individual speakers in an audio stream, making it possible to assign diarized time segments to specific speakers. This is particularly useful in scenarios where there are multiple speakers in a recording, such as conference calls, interviews, podcasts, and recorded meetings.

Pyannote-audio is an open-source toolkit developed by the Pyannote team for various speech and audio processing tasks. It provides a collection of pre-built models, algorithms, and utilities to perform tasks like speaker diarization, voice activity detection, and speech turn segmentation.

How to use Pyannote.audio with Whisper:

WhisperX is a model that integrates Whisper, Phoneme-Based Model (Wav2Vec2) and Pyannote.audio. It is claimed to be 70x faster in real-time speech recognition than Whisper large-v2 model with word-level timestamps and speaker diarization with VAD feature. It uses faster-whisper backend, and consumes GPU memories smaller than 8 GB.

How to use WhisperX:

However, WhisperX is not perfect for such multi-speaker speech recognition with diarization tasks. Sometimes it will be less accurate if the speakers speak from different channels. Speaker diarization might get disordered if it does not identify the correct number of speakers; and it performs poorly when it comes to voice overlapping and speeches that contain interjections.

Of course, we can also simply use the transcribe functionality in WhisperX. As the architecture is built with Voice Activity Detection and Cut & Merge features, long audios can be processed to the model. Then a comparison will be possible to Whisper pipeline in Transformer (for long audio transcriptions).

The results show that there were little accuracy differences in using different model sizes and computation types. WER results were not much different to the original Whisper model, thus indicating WhisperX is a good alternative other than Whisper Pipeline. However, it was the case only for single-channel clean audio datasets in English and a few other languages. As for Chinese datasets, the outcomes became unreliable.

Here are several possible reasons:

Besides Whisper, there exists other competitive LLMs in ASR field that built with different architectures and techniques, including Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Connectionist Temporal Classification (CTC), etc. Some popular models are researched and evaluated with model inference.

The Massively Multilingual Speech (MMS) is a project led by Meta (Facebook research) included in the Fairseq (a sequence-to-sequence toolkit). It expands speech technology from around 100 languages to more than 1,100 languages, more than 10 times as many as before. Language identification models are able to more than 4,000 spoken languages, 40 times more than before.

MMS uses religious audios and texts, such as the Bible, to be the data source because most of them have been translated in many different languages. The MMS is built on pre-trained wav2vec 2.0 models. The researcher claimed that MMS halves the WER of Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.

There are several kinds of model size available: mms-1b-fl102, mms-1b-l1107, mms-1b-all. Among these, mms-1b-all is the largest size. Unlike Whisper, Wav2Vec2FeatureExtractor and Wav2Vec2CTCTokenizer are used.

The target language ids could be found in tokenizer.vocab.keys(). For example, id for Chinese is cmn-script_simplified. Different language adapter weights can be loaded for different languages via load_adapter(). MMS loads weights for English by default, so target_lang=<target-lang> and ignore_mismatched_sizes=True need to be specified when loading with other languages. An alternative way is to use an ASR pipeline from Transformer. Set “model_kwargs” dictionary to have the two settings above.

For inference purposes, switch out the language adapters directly from the default English model using the codes.

The transcribing process of Hugging Face MMS resembles that of Wav2Vec2ForCTC Model.

The model generates logits, which are the raw output scores from the model. After obtaining the logits from the model, it calculates the index of the highest logit value along the last dimension. This effectively finds the predicted token index with the highest probability. The [0] indexing is used to access the only example in the batch. Finally, the predicted token index ids is decoded to original text transcriptions.

PaddleSpeech is a Chinese open-source toolkit on the PaddlePaddle platform for a variety of tasks in speech and audio, with the state-of-art and influential models. It provides production ready streaming ASR and streaming TTS systems. There are a variety of speech models with different architectures and pretrained data source available:

The recommended OS system is Linux, but others are also sufficient for certain tasks.

There is also a detailed introduction of the exact datasets, models, decoding and augmentation techniques used in the feature list in official GitHub Repo https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/source/asr/feature_list.md

There are several model architectures available in PaddleSpeech.

The DeepSpeech2 online model is a modified DeepSpeech2 version. The model is mainly composed of 2D convolution subsampling layers and stacked single-direction RNN layers.

It also has a separate vocabulary for English data and Chinese data. A technology called Cepstral Mean and Variance Normalization (CMVN) is used. A subset of or full of the training set is selected and be used to compute the feature mean and standard deviation. For feature extraction, the released DeepSpeech2 online model uses the linear feature extraction method (Fast Fourier Transform without using filter bank). Encoder and Decoder architectures are shown below.

The DeepSpeech2 offline model (non-streaming) is similar to the online one. The main difference is the offline model uses the stacked bi-directional RNN layers (bi-directions with RNN Cells). For data preparation and decoder architecture they are identical.

The Conformer is a convolution-augmented transformer for speech recognition. It combines Convolution Neural Networks (CNN) and Transformers to improve speech recognition performance in a parameter-efficient way.

Conformer comprises two feed-forward layers with half step residual connections with Multi-Head Self Attention and Convolution modules at the middle. A post layernorm is followed at the end. By combining CNNs for local feature extraction and transformers for global context understanding, the Conformer model achieves state-of-the-art performance on various speech recognition benchmarks.

Unified Streaming and Non-streaming Two-pass End-to-end Model, also known as U2, applies hybrid CTC/attention architecture with dynamic chunk-based attention and streaming CTC decoding. By adjusting the chunk size during inference, the latency of the speech recognition system can be controlled. After the CTC decoder generates n-best hypotheses, the attention decoder is used to rescore these hypotheses and generate the final result. It allows for more efficient and flexible speech recognition, making it suitable for real-time applications and scenarios with variable-length audio input.

The installations are dependent on how much PaddleSpeech will be utilized. It is possible to choose the specific version on PaddlePaddle official website (Baidu mirror / Tsinghua mirror): https://www.paddlepaddle.org.cn/en/install/quick?docurl=/documentation/docs/en/install/conda/windows-conda_en.html 

How to use PaddleSpeech:

Another approach is to compile source code with commands given on the official website

Copy

git clone https://github.com/PaddlePaddle/PaddleSpeech.git
cd PaddleSpeech
pip install pytest-runner
pip install .

💡

Some features are not supportive on certain OS systems

However, it is also possible to customize executor parameters in asr arguments, the available parameters are shown in Figure 15:

The list of available PaddleSpeech models for ASR inference with target language:

SpeechBrain is an open-source conversational AI toolkit developed by the Speech and Audio Processing Group at the University of Montreal. It aims to provide a flexible and comprehensive platform for speech-related research, development, and applications. SpeechBrain is PyTorch-based and supports state-of-the-art methods for end-to-end speech recognition, including models based on CTC, CTC+attention, transducers, transformers, and neural language models relying on recurrent neural networks and transformers.

SpeechBrain offers a wide range of functionalities for various speech and audio processing tasks, including ASR. The official website is here SpeechBrain

How to use SpeechBrain:

ESPnet is an end-to-end speech processing toolkit covering end-to-end speech recognition, text-to-speech, speech translation, speaker diarization, etc. ESPnet uses PyTorch as the engine and also follows Kaldi style data processing, feature extraction/format, and recipes.

ESPnet is now upgrading to version 2, ESPnet2, and has been shifted with most of the developments. It will have On-the-fly feature extraction and reduce preparation complexity. It contains various ASR recipes such as Hybrid CTC/attention based end-to-end and Transducer based end-to-end. The most recommended OS system is Linux Ubuntu.

How to use ESPnet:

The baselines tests are primarily for Chinese transcription accuracy investigation purposes. Whisper will be used as base references for other models.

for English inference results, Meta MMS had similar transcript accuracies compared to Whisper.

On the other hand, for Chinese inference results, PaddleSpeech had a better performance compared to Whisper. While conformer-aishell with attention-rescoring and Whisper-large-v2 baseline have similar CER results, conformer-wenetspeech chooses WeNetSpeech as its source and provided a better performance in MagicData test datasets. For SpeechBrain models, wav2vec2-ctc-aishell appeared to have better performance on unseen data, and other models have similar performance compared to Whisper large-v2 baseline. Meta MMS Chinese transcription results were worse than Whisper and ESPnet models are the least accurate.

Azure AI Speech Services is a collection of cloud-based speech-related services offered by Microsoft Azure. These services enable developers to integrate various speech capabilities into their applications, including speech recognition, text-to-speech, and speech translation. These services leverage advanced machine learning algorithms to provide accurate and natural language processing functionalities.

Speech Studio in Azure AI is a set of UI-based tools for building and integrating features from Azure Speech service. Custom Speech Projects in Speech Studio can be created in different languages. Endpoints are then available for deployment using the Speech SDK, Speech CLI, or REST APIs. There are mainly four sections in a Custom Speech Project: Speech Datasets for uploading datasets, Train Custom Models with uploaded train datasets, Test Models with uploaded test datasets, and Deploy Models, which create endpoints for customized (trained) models.

There are three methods for uploading training and testing datasets for Custom Speech:

Speech Studio (direct upload), REST API and CLI usage.

For direct upload to Speech Studio, prepare data in local directories and follow these steps:

There will be options for choosing dataset location in either local file or remote location like Azure Blob URL. For local files, simply find the address of the directory on the local device. However, if we want to ensure maximum security of dataset files by using a trusted Azure services security mechanism, Azure Blob is a good option.

Azure Blob Storage is a scalable and cost-effective cloud-based object storage service. It is designed to store and manage unstructured data, such as documents, images, audio files, videos, backups, logs, and more. Azure Blob Storage provides a reliable and secure way to store massive amounts of data, making it an essential component for many cloud-based applications and services.

How to use Azure Blob:

Unlike Speech Studio, it is not necessary to choose whether a dataset is for testing or training at the time of upload with STT REST API. The API kind is dependent on formats.

Speech to text REST API v3.1 will do the tricks. There are several features in this version: Datasets, Endpoints, Evaluations, Models, Projects, etc. It contains GET, POST, DELETE and PATCH methods.

How to use REST API:

There are restrictions for data to upload. Usually the data composition determines whether it can be used for training or testing. It is better to include the audios and transcripts at the same time.

For audio files, the format should be WAV, sampling rate to be 8k Hz or 16k Hz, single channeled. The maximum audio lengths are also different for testing and training. The archive should be in zip format, under 2GB and 10k files within.

The transcriptions for all WAV files are contained in a single plain-text file (.txt or .tsv). Each line of the transcription file contains the name of one of the audio files, followed by the corresponding transcription. The file name and transcription are separated by a tab.

For different languages, the texts should also be normalized with a defined recipe.

Take zh-CN (Chinese) for example, human-labeled transcriptions for Mandarin Chinese audio must be UTF-8 encoded with a byte-order marker. Avoid the use of half-width punctuation characters. It is also required to write out abbreviations in spoken form.

Additionally, Editor help us to edit and combine uploaded datasets in the same project. Automatic selection of audios in certain lengths and manual exportation of a subset as the new dataset are enabled. Also, qualities and correctnesses are testable by enabling play buttons to every audio with its corresponding transcript aside. Quantity will show the total audio length of a dataset.

Training process will be much easier than the local one. We do not need to fine-tune on the Azure models, but rather feed in speech data only.

Name the model and choose a baseline model as a starting point. Select one or more datasets from Speech Studio Datasets. Often it takes hours for the training to complete, or even a day.

After training, inspect or compare error rates with the models using a single test dataset, which should also exist among uploaded datasets.

For inspection, select two models, either customized or baseline type. The inspection is for audio-only datasets, and its purpose is to check the audio qualities.

For evaluation with error rates, select two (customized / baseline) models. The evaluation will follow WER calculation logic, and Error rate, Insertion number, Substitution number,  Deletion number of each model can be seen in Speech Studio. Other than these, original labels, lexical transcripts (the original output from models) and normalized transcripts will be shown under the table in Figure 23 in details. The error rate from every single audio is visible beside these texts.

We can deploy models to applications to integrate with other features, or simply redo the evaluation in local scripts. When deploying a model, it will create an endpoint.

How to deploy a model in Azure Speech Studio:

Abbreviations:

The results showed that Minds14 is not ideal for ASR tasks, as its original purpose is for intent detection tasks. While a large amount of test data might decrease the randomization of audio qualities and accuracies, a mixture with multiple test data sources with reasonable portion divisions may be more insightful. The best Azure model by far was trained with AISHELL-1, mozilla-foundation/common_voice_13_0 and google/fleurs, resulting 2.48% error rate.

In this project, audio sources in English and Chinese datasets were investigated. Whisper models were fine-tuned mainly by these 2 languages. While audio data sources in English have always been sufficient for training purposes, Chinese sources that are available whilst maintaining high transcript qualities are much less in quantities. Additionally, Chinese audio data were often classified by speakers, this indicates a mixture with different speakers might resolve the potential issues of speaker biases.

Because of hardware computing resource limits, full fine-tuning models sometimes produce CUDA OOM error when using a single GPU. This is very likely to obtain a better model in large weights if multi-GPU training or more advanced GPU (NVIDIA 40 series) could be utilized in the process.

A research point that could have been stretched further is effects of LoRA configurations on Parameter-Efficient Fine-tuned model performances. Other than that, having different optimizers and making data augmentation are also possible strategies for improvement. If Linux environment issues are overcomed, training other models for a specific language is also promising to enhance performance (e.g. PaddleSpeech models for Chinese, Meta MMS, SpeechBrain for English).

In the Speaker Diarization field, Pyannote.audio with Whisper integration has proven its potential. While the transcript accuracy has been improved to be close to pure Whisper’s, current diarizing ability of relevant models in multi-speaker meeting scenarios is still not sufficient for multi-speaker speech recognition support.

In the contexts of Azure Speech Services, the most important rule is to keep good audio qualities and word-level accuracy in transcripts. While adding audio source variety is important, filtering training audio files that are not in good quality or too short to identify correct meaning as transcripts can also potentially enhance model performances.

[1] Anaconda, Inc. (2017). Command reference - conda 23.7.3.dev30 documentation. conda.io/projects/conda/en/latest/commands

[2] OpenAI (2022, September 21). Introducing Whisper. openai.com/research/whisper.

[3] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, & Ilya Sutskever (2022). Robust Speech Recognition via Large-Scale Weak Supervision.

[4] Wikipedia contributors. (2023). Word error rate. In Wikipedia, The Free Encyclopedia.

[5] Gandhi, Sanchit (2022, November 3). Fine-Tune Whisper for Multilingual ASR with 🤗 Transformers. Hugging Face, Inc. huggingface.co/blog/fine-tune-whisper.

[6] The Linux Foundation (2023). Previous PyTorch Versions | PyTorch. pytorch.org/get-started/previous-versions

[7] Hugging Face, Inc. (2023). Hugging Face - Documentations. huggingface.co/docs

[8] Vaibhav Srivastav (2023). fast-whisper-finetuning, GitHub repository. github.com/Vaibhavs10/fast-whisper-finetuning

[9] Mangrulkar, Sourab, and Sayak Paul (2023, February 10). Parameter-Efficient Fine-Tuning Using 🤗 PEFT. Hugging Face, Inc. huggingface.co/blog/peft

[10] Bredin, H., Yin, R., Coria, J., Gelly, G., Korshunov, P., Lavechin, M., Fustes, D., Titeux, H., Bouaziz, W., & Gill, M.P. (2020). pyannote.audio: neural building blocks for speaker diarization. In ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing.

[11] Bain, M., Huh, J., Han, T., & Zisserman, A. (2023). WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. INTERSPEECH 2023.

[12] Meta AI (2023, May 22). Introducing speech-to-text, text-to-speech, and more for 1,100+ languages. ai.meta.com/blog/multilingual-model-speech-recognition

[13] Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, & Michael Auli (2023). Scaling Speech Technology to 1,000+ Languages. arXiv.

[14] Hui Zhang, L. (2022). PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations. Association for Computational Linguistics.

[15] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, & Yoshua Bengio. (2021). SpeechBrain: A General-Purpose Speech Toolkit.

[16] Gao, D., Shi, J., Chuang, S.P., Garcia, L., Lee, H.y., Watanabe, S., & Khudanpur, S. (2022). EURO: ESPnet Unsupervised ASR Open-source Toolkit. arXiv preprint arXiv:2211.17196.

[17] ESPnet (2021). espnet_model_zoo, GitHub repository. github.com/espnet/espnet_model_zoo

[18] Eric-Urban (2023, August 2). Custom Speech overview - Speech service - Azure AI services. Custom Speech overview - Speech Service - Azure AI Services | Microsoft Learn learn.microsoft.com/en-us/azure/ai-services/speech-service/custom-speech-overview

[19] Microsoft (2023). Speech service documentation - Tutorials, API Reference - Azure AI services - Azure AI services. Speech service documentation - Tutorials, API Reference - Azure AI services - Azure AI services | Microsoft Learn learn.microsoft.com/en-us/azure/ai-services/speech-service/