dan
/
tacotron2
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
111 Commits
1 Branch
2.7 MiB
 
 
Tree: eb2a171690
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'eb2a171690'
${ noResults }
 ZIP  TAR.GZ
rafaelvalle eb2a171690 Merge branch 'master' of https://github.com/NVIDIA/tacotron2 5 years ago
filelists changing structure for better organization 6 years ago
text text/symbols.py: updating symbols 6 years ago
waveglow@4b1001fa33 adding waveglow submodule 6 years ago
.gitmodules adding waveglow submodule 6 years ago
Dockerfile Dockerfile: updating to use latest pytorch and apex 5 years ago
LICENSE Update license such that it appears on repo fron tpage 6 years ago
README.md README.md: adding instructions to install apex 5 years ago
audio_processing.py adding python files 6 years ago
data_utils.py data_utils.py: rewrite 6 years ago
demo.wav adding demo.wav file 6 years ago
distributed.py Fixing concatenation error for fp16 ditributed training 5 years ago
hparams.py hparams.py: adding ignore_layers argument to ignore text embedding layers when warm_starting 5 years ago
inference.ipynb inference.ipynb: adding fp16 inference 5 years ago
layers.py layers.py: rewrite 6 years ago
logger.py logger.py: using new pytorch api 5 years ago
loss_function.py adding python files 6 years ago
loss_scaler.py loss_scaler.py: patching loss scaler for compatibility with current pytorch 6 years ago
model.py model.py: renaming variables, removing dropout from lstm cell state, removing conversions now handled by amp 5 years ago
multiproc.py adding python files 6 years ago
plotting_utils.py adding python files 6 years ago
requirements.txt requirements.txt: removing pytorch 0.4 from requirements. upgrading to 1.0 6 years ago
stft.py stft.py: moving window_sum to cuda if magnitude is cuda 5 years ago
tensorboard.png tensorboard.png: adding tensorboard image 6 years ago
train.py Merge branch 'master' of https://github.com/NVIDIA/tacotron2 5 years ago
utils.py utils.py: updating 6 years ago

README.md

Tacotron 2 (without wavenet)

PyTorch implementation of Natural TTS Synthesis By Conditioning Wavenet On Mel Spectrogram Predictions.

This implementation includes distributed and fp16 support and uses the LJSpeech dataset.

Distributed and FP16 support uses NVIDIA's Apex and AMP.

Visit our website for audio samples using our published Tacotron 2 and WaveGlow models.

Alignment, Predicted Mel Spectrogram, Target Mel Spectrogram

Pre-requisites

  1. NVIDIA GPU + CUDA cuDNN

Setup

  1. Download and extract the LJ Speech dataset
  2. Clone this repo: git clone https://github.com/NVIDIA/tacotron2.git
  3. CD into this repo: cd tacotron2
  4. Initialize submodule: git submodule init; git submodule update
  5. Update .wav paths: sed -i -- 's,DUMMY,ljs_dataset_folder/wavs,g' filelists/*.txt
    • Alternatively, set load_mel_from_disk=True in hparams.py and update mel-spectrogram paths
  6. Install PyTorch 1.0
  7. Install Apex
  8. Install python requirements or build docker image
    • Install python requirements: pip install -r requirements.txt

Training

  1. python train.py --output_directory=outdir --log_directory=logdir
  2. (OPTIONAL) tensorboard --logdir=outdir/logdir

Training using a pre-trained model

Training using a pre-trained model can lead to faster convergence
By default, the dataset dependent text embedding layers are ignored

  1. Download our published Tacotron 2 model
  2. python train.py --output_directory=outdir --log_directory=logdir -c tacotron2_statedict.pt --warm_start

Multi-GPU (distributed) and FP16 Training

  1. python -m multiproc train.py --output_directory=outdir --log_directory=logdir --hparams=distributed_run=True,fp16_run=True

Inference demo

  1. Download our published Tacotron 2 model
  2. Download our published WaveGlow model
  3. jupyter notebook --ip=127.0.0.1 --port=31337
  4. Load inference.ipynb

N.b. When performing Mel-Spectrogram to Audio synthesis, make sure Tacotron 2 and the Mel decoder were trained on the same mel-spectrogram representation.

Related repos

WaveGlow Faster than real time Flow-based Generative Network for Speech Synthesis

nv-wavenet Faster than real time WaveNet.

Acknowledgements

This implementation uses code from the following repos: Keith Ito, Prem Seetharaman as described in our code.

We are inspired by Ryuchi Yamamoto's Tacotron PyTorch implementation.

We are thankful to the Tacotron 2 paper authors, specially Jonathan Shen, Yuxuan Wang and Zongheng Yang.