电影网站免费建设,做论坛app网站有哪些,中国公共招聘网,营销网站案例什么意思目录 序言 查看GPU配置
复制代码库并安装运行环境
选择预训练模型
上传视频链接#xff08;单个不应长于20分钟#xff09;
自动处理所有上传的数据
训练质量相关#xff1a;实验发现目前使用CJ模型勾选ADD_AUXILIARY#xff0c;对于中/日均能训练出最好的效果#x…目录 序言 查看GPU配置
复制代码库并安装运行环境
选择预训练模型
上传视频链接单个不应长于20分钟
自动处理所有上传的数据
训练质量相关实验发现目前使用CJ模型勾选ADD_AUXILIARY对于中/日均能训练出最好的效果第一次训练建议默认使用该组
开始训练
微调完成后在这里尝试效果
下载模型
模型演示
Text-to-Speech
Voice Conversion 序言
语音合成技术是人工智能领域的重要分支近年来取得了长足的进步。涌现出众多优秀的语音合成模型其中VITS和DDSP是两种代表性的模型。
VITS模型采用了变分自编码器和声码器的组合架构在长期训练下可以生成逼真、自然的语音。然而由于其对显存的要求较高不适合在普通的个人电脑上进行本地部署训练。
为了降低VITS模型的入手门槛我选择使用Google Colab来实现本地部署训练。Google Colab是一个免费的云端计算平台可以提供强大的计算能力。在此基础上其简便性进行了进一步强化目前可以仅通过视频链接来一键进行数据集处理可以大大节省用户的时间和精力。
当然VITS模型也存在一定的缺点在短时间和数据集较少、质量较低的情况下其效果会不如DDSP。因此在选择合适的语音合成模型时需要根据实际情况进行考量。 查看GPU配置
# 查看GPU配置
# Check GPU configuration
!nvidia-smi 复制代码库并安装运行环境
#title STEP 1 复制代码库并安装运行环境
#markdown #STEP 1 (6 min)
#markdown ##复制代码库并安装运行环境
#markdown ##Clone repository Build environment!git clone https://github.com/Plachtaa/VITS-fast-fine-tuning.git
!python -m pip install --upgrade --force-reinstall regex
!python -m pip install --force-reinstall soundfile
!python -m pip install --force-reinstall gradio
!python -m pip install imageio2.4.1
!python -m pip install --upgrade youtube-dl
!python -m pip install moviepy
%cd VITS-fast-fine-tuning!python -m pip install --no-build-isolation -r requirements.txt
!python -m pip install --upgrade numpy
!python -m pip install --upgrade --force-reinstall numba
!python -m pip install --upgrade Cython!python -m pip install --upgrade pyzmq
!python -m pip install pydantic1.10.4
!python -m pip install ruamel.yaml# build monotonic align
%cd monotonic_align/
!mkdir monotonic_align
!python setup.py build_ext --inplace
%cd ..
!mkdir pretrained_models
# download data for fine-tuning
!wget https://huggingface.co/datasets/Plachta/sampled_audio4ft/resolve/main/sampled_audio4ft_v2.zip
!unzip sampled_audio4ft_v2.zip
# create necessary directories
!mkdir video_data
!mkdir raw_audio
!mkdir denoised_audio
!mkdir custom_character_voice
!mkdir segmented_character_voice 选择预训练模型
#title STEP 1.5 选择预训练模型
#markdown ###STEP 1.5 选择预训练模型
#markdown ###Choose pretrained model to start
#markdown CJE为中日英三语模型CJ为中日双语模型C为纯中文模型#markdown CJE for Chinese, Japanese English modelCJ for Chinese Japanese model
PRETRAINED_MODEL CJ #param [CJE,CJ,C]
if PRETRAINED_MODEL CJ:!wget https://huggingface.co/spaces/sayashi/vits-uma-genshin-honkai/resolve/main/model/D_0-p.pth -O ./pretrained_models/D_0.pth!wget https://huggingface.co/spaces/sayashi/vits-uma-genshin-honkai/resolve/main/model/G_0-p.pth -O ./pretrained_models/G_0.pth!wget https://huggingface.co/spaces/sayashi/vits-uma-genshin-honkai/resolve/main/model/config.json -O ./configs/finetune_speaker.json
elif PRETRAINED_MODEL CJE:!wget https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer/resolve/main/pretrained_models/D_trilingual.pth -O ./pretrained_models/D_0.pth!wget https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer/resolve/main/pretrained_models/G_trilingual.pth -O ./pretrained_models/G_0.pth!wget https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer/resolve/main/configs/uma_trilingual.json -O ./configs/finetune_speaker.json
elif PRETRAINED_MODEL C:!wget https://huggingface.co/datasets/Plachta/sampled_audio4ft/resolve/main/VITS-Chinese/D_0.pth -O ./pretrained_models/D_0.pth!wget https://huggingface.co/datasets/Plachta/sampled_audio4ft/resolve/main/VITS-Chinese/G_0.pth -O ./pretrained_models/G_0.pth!wget https://huggingface.co/datasets/Plachta/sampled_audio4ft/resolve/main/VITS-Chinese/config.json -O ./configs/finetune_speaker.json 上传视频链接单个不应长于20分钟
#markdown 运行该代码块会出现一个文件上传的入口上传单个.txt文件。若格式正确的话视频会自动下载并将下载后的文件名打印在下方。#markdown Running this code block will prompt you to upload a file.
#markdown Please upload a single .txt file. If you have put the links in the correct format,
#markdown the videos will be automatically downloaded and displayed below.
%run scripts/download_video.py
!ls ./video_data/ 自动处理所有上传的数据
#markdown 运行该单元格会对所有上传的数据进行自动去背景音标注。
#markdown 由于需要调用Whisper和Demucs运行时间可能较长。#markdown Running this codeblock will perform automatic vocal seperation annotation.
#markdown Since this step uses Whisper Demucs, it may take a while to complete.
# 将所有视频无论是上传的还是下载的且必须是.mp4格式抽取音频
%run scripts/video2audio.py
# 将所有音频无论是上传的还是从视频抽取的必须是.wav格式去噪
!python scripts/denoise_audio.py
# 分割并标注长音频
!python scripts/long_audio_transcribe.py --languages {PRETRAINED_MODEL} --whisper_size large
# 标注短音频
!python scripts/short_audio_transcribe.py --languages {PRETRAINED_MODEL} --whisper_size large
# 底模采样率可能与辅助数据不同需要重采样
!python scripts/resample.py 训练质量相关实验发现目前使用CJ模型勾选ADD_AUXILIARY对于中/日均能训练出最好的效果第一次训练建议默认使用该组
#markdown ##STEP 3.5
#markdown 运行该单元格会生成划分好训练/测试集的最终标注以及配置文件#markdown Running this block will generate final annotations for training validation, as well as config file.#markdown 选择是否加入辅助训练数据/ Choose whether to add auxiliary data:
ADD_AUXILIARY False #param {type:boolean}
#markdown 辅助训练数据是从预训练的大数据集抽样得到的作用在于防止模型在标注不准确的数据上形成错误映射。#markdown Auxiliary data is to prevent overfitting when the audio samples are small or with low quality.#markdown 以下情况请勾选#markdown 总样本少于100条/样本质量一般或较差/样本来自爬取的视频#markdown 以下情况可以不勾选#markdown 总样本量很大/样本质量很高/希望加速训练/只有二次元角色# assert(not (ADD_AUXILIARY and PRETRAINED_MODEL ! CJE)), add auxiliary data is available only available for CJE model!
if ADD_AUXILIARY:%run preprocess_v2.py --add_auxiliary_data True --languages {PRETRAINED_MODEL}
else:%run preprocess_v2.py --languages {PRETRAINED_MODEL} 开始训练
#markdown #STEP 4 (20 min)
#markdown 开始微调模型。
#markdown 训练时长取决于你录入/上传的音频总数。#markdown 根据声线和样本质量的不同所需的训练epochs数也不同。#markdown 你也可以在Tensorboard中预览合成效果若效果满意可提前停止。#markdown Model fine-tuning
#markdown Total time cost depends on the number of voices you recorded/uploaded.#markdown Best epoch number varies depending on different uploaded voices / sample quality.#markdown You can also preview synthezied audio in Tensorboard, its OK to shut down training manually if you find the quality is satisfying.
import os
os.environ[TENSORBOARD_BINARY] /usr/local/bin/tensorboardif os.path.exists(/content/drive/MyDrive/):!python scripts/rearrange_speaker.py!cp ./finetune_speaker.json ../drive/MyDrive/finetune_speaker.json!cp ./moegoe_config.json ../drive/MyDrive/moegoe_config.json%reload_ext tensorboard
%tensorboard --logdir ./OUTPUT_MODEL
Maximum_epochs 200 #param {type:string}
#markdown 继续之前的模型训练/Continue training from previous checkpoint
CONTINUE True #param {type:boolean}
if CONTINUE:!python finetune_speaker_v2.py -m ./OUTPUT_MODEL --max_epochs {Maximum_epochs} --drop_speaker_embed False --cont True
else:!python finetune_speaker_v2.py -m ./OUTPUT_MODEL --max_epochs {Maximum_epochs} --drop_speaker_embed True 微调完成后在这里尝试效果
#markdown ### 微调完成后在这里尝试效果。
#markdown ### 运行后会输出一个public URL, 点击进入网页版UI以使用模型
#markdown ### Try out TTS VC quality here after fine-tuning is finished.
!cp ./configs/modified_finetune_speaker.json ./finetune_speaker.json
!python VC_inference.py --model_dir ./OUTPUT_MODEL/G_latest.pth --share True 下载模型
#markdown ### 浏览器自动下载模型和配置文件
#markdown ### Download option 1: Running this codeblock will download model config files by your browser.
!python scripts/rearrange_speaker.py
%run scripts/download_model.py 模型演示
Text-to-Speech text_to_speech Voice Conversion Voice Conversion
