Would really love some more information on how base models were trained.

[codename0og]

@RVC-Boss @yxlllc
Hey guys ~ Me and many more devs been trying our hardest for the last 2 years to improve RVC and much more.
That's essentially how current Applio and my v4 fork came to be.

Since a longer while, re-creating better base models was our primary target, involving many experiments..

• Different datasets ( including but not limited to: vctk, m4, ears, JL corpus, librispeech and much more.. )
• Various vocoder attempts: RefineGan, Avocado, Hiftnet, istftnet, wavehax, ringformer and currently,
  my custom experimental pcph-gan utilizing pcph signal and silu/snake-beta activations
• Various custom losses to aid the training: envelope loss, multi-scale mel loss, multi-res stft loss and more
• Different adversarial objectives ( Hinge and tprls and iirc, wgan-gp )
• Other discriminators ( avocado discs, ms-sb-cqt, mr-stft, wavelet mpd/msd takes )
  and many experiments involving different combinations.

Unfortunately, we still can't quite well and reliably reach what's presented by original rvc's pretrained models
and ss far as I know, it is still unclear what steps exactly were taken during the training.

What's supposedly known for sure is, for base models:

• Dataset was VCTK only.
• Batch size was 16 ( 4 * 4 gpus ).
• Was trained with 4 gpus.
• There's extra 3 periods in MultiPeriodDiscriminator ( 17, 23, 37 ).
• MultiScaleDiscriminator is set to work as 1 " single scale " like disc.
• Learning rate was 1e-4.
• exponential lr decay with gamma of 0.999875 as scheduler.

Now questions are..

• Was there any pitch-shift transformations / any other alterations performed on the vctk?
• Was it truly just VCTK used? or did you guys also employ a bit of M4 or whatever other " singing " set ?
  and if not, how did you achieve relatively decent f0 handling like that on such a bad and poor in pitch ( since it's fairly monotone ) dataset?
• Any actual field-tested reasons to utilize 3 extra periods?
  Or is it simply that you referenced playvoice's nsf-hifigan take? ( or specifically vtuber-plan's mirror(?) )
• Was there any gradient clipping utilized during training or was left as is?
• Were the base models trained using mixed precision or all fp32?
• Any specific mid-training operations or tweaks?
• Was each pretrains set ( per sample rate ) trained independently? or was it iteratively re-trained from lower sr to higher?
  And lastly, for how long did you train each base models? anywhere close to 200k steps or more towards 1mil+ ?

As a side note, some of us intended to contact you for a longer while, sadly there's no reliable contact or email provided anywhere.
We would truly appreciate some insights!

And yes, I realize 'rvc' is kind of a dead project now and boss, you've taken the TTS direction,
but we don't want to give up too easily, essentially want to continue the legacy as there's so much room for improvements in many areas.. and foremostly, there's nothing like RVC ( and on pair with it's quality ) still up to this date.

Here's my email just in case if you feel more comfortable that way: codenamegithub@gmail.com
In any case, appreciate the response!

[RVCfinder]

your discord contact link is not working. how can i get in touch with you?

[codename0og]

@RVCfinder If you refer to rvc's discord link? it works just fine. In case not? Here
but if you mean my server then that one works just fine as well~

[codename0og]

@RVC-Boss @yxlllc
Bumping up : )

@fumiama I think I should also tag you in here too ~ Not quite sure what kind of training / pretrains-area contribution you had tho~

[TheFrenchPepeAI]

@codename0og Hey, thanks a lot for this detailed message. Honestly, it really resonates with me !!

It’s been almost 2 years that I’ve been waiting for real progress on local voice conversion. I use RVC a lot, especially for French dubbing, and one of the biggest issues I keep running into is pronunciation bugs in French, which clearly come from the fact that the base pretrained models are trained on English-only datasets..

I would have loved to build a French-focused pretrained model, specifically aimed at voice acting / dubbing, because I actually have a huge amount of French dubbing data. I’m a content creator on social media, and my workflow is very specific:
I record myself dubbing characters with my real voice, imitating the original voice actor as closely as possible, and then I apply the AI voice model I trained on top of that. The emotional result is really good !

But the problem is exactly what I mentioned above:
some French words are consistently mispronounced, simply because the base model was never trained on French phonetics. And to be honest, I have no idea how to properly train a pretrained/base model from scratch, even though I have the data. That’s honestly frustrating.

So I have a few questions from a user perspective:

Do you think your v4 fork is currently better than original RVC in terms of stability?

Do you think building language-specific pretrained models (for example French, focused on spoken dialogue rather than singing) is realistically achievable with the current architecture?

And do you agree that voice conversion for dubbing is extremely close to perfection, while most projects are unfortunately shifting everything toward TTS, which is a completely different use case?

Like you said, it feels like RVC is considered “dead”, but from a dubbing perspective, it’s honestly the most promising tech we have, and it feels a bit sad to see that direction being abandoned when we’re so close to something incredible.

Anyway, just wanted to say thank you for not giving up on this, and for trying to push things forward. Even from the outside, it’s really appreciated.❤

[codename0og]

@TheFrenchPepeAI Well, being fully honest with you? Both Applio and Fork are currently the most stable solutions when it comes to "RVC" and "RVC" alikes.

The main difference? Applio is your standard / casual framework to use, Fork is for advanced users or those that want cutting-edge or experimental features, new vocoders.. architecture tweaks and so on.

As of yet, until we finally figure it out or get a response from Boss or yxlllc ( wink wink guys.. )
training pretrained models is.. to say the least, quite problematic. There's as many ways to handle it as there is issues arising during the training.

Since you're already into it that deep, I'd just simply recommend visiting my discord server ( you can find it on fork's repo ) and reading into development channel / just searching for any clues and info.
I'm currently too busy sadly to provide any extensive details but I'm pretty sure you'll find all that you need, you can always ask people around too!

[fumiama]

@RVC-Boss @yxlllc Bumping up : )

@fumiama I think I should also tag you in here too ~ Not quite sure what kind of training / pretrains-area contribution you had tho~

I've not participated in this project during pre-training so I do not know quite a lot about it. As far as I can remember, the pre-trained models are trained under tons of high quality voice data using a separated train script that has never been open-sourced. I would like to help you contact with RVC-Boss but since I have quitted this project due to some reasons, you may have to contact him by yourself. Here's the way: this and this are two Bilibili accounts of RVC-Boss. You can register an account in Bilibili and ask him a question. Please note that you only have one chance to send a message before his reply.

[codename0og]

@fumiama Thank you so much for the reply!
Dang, I competely didn't think of bilibili..

Nevertheless, thank you again! This can potentially boost up our chances ~