willing to pay for help

[perlman-izzy]

Hi,

I don't have much coding experience but I really, really want to play around with this program.

If someone here can help me get this thing running (training samples of music I provide) and explain to me how to use the various features, I will gladly pay you your going rate.

I have spent like 3 months hitting my head against a wall, paying people who claim they can do it but fail.

I'm able to implement the docker image, upload the samples, and get the program to start to run, but every time, I get some error or another. I need someone to help me debug.

Please.

Thank you

[johndpope]

well - probably first step is to show error. You need a stack of VRAM to run this. I didn't bother with it for this reason. > 12gb of VRAM. cards that are suitable are k80 - 24gb/32gb of VRAM / RTX 5000 - 16gb RTX 6000 / RTX 8000. Not even the 2080ti is appropriate at 11gb. the just released nvidida 3090 - the latest / 24gb of VRAM. AMD cards will not work. Wait another month - get the 3090 card or two / there's a limit of these cards being released. This card is better than the one below. (don't get the 3080 it can't be joined together with nvlink) - you'll be in a better position to get anywhere. You'll need a 750Watt power supply. I recommend a HP workstation.

"On a V100, it takes about 3 hrs to fully sample 20 seconds of music"
US $8,399.00
https://www.ebay.com.au/itm/nVIDIA-Tesla-V100-GPU-Accelerator-Card-16GB-PCI-e-Machine-Learning-AI-HPC-Volta-/153223723116

(Alternative is to run it with google colab - but this is problematic.)

[michaelklachko]

@perlman-izzy why do you want it so bad?

[btrude]

@perlman-izzy What's your email address? Sounds like you are using my docker image so I am happy to keep helping you. I also completely understand the desire to do so - jukebox has been the most fun with music I've had in a very long time and the results I'm getting in my own domain (electronic music) are incredible 😭 I've spent pretty much every day since jukebox released editing the code to support various things that I need for better co-composition including the ability to finetune to the 5b model, much larger batch sizes for sampling, editing of tensors for composing before they are rendered as audio, etc. I think I can get you to the same place pretty easily without asking for any money, so let me know and we can get you rolling!

[perlman-izzy]

Yes! OMG. You are a prince among men, sir, and I am in your debt. My email is perlman.izzy@gmail.com.

…

On Sat, Sep 5, 2020 at 7:06 AM btrude ***@***.***> wrote: @perlman-izzy <https://github.com/perlman-izzy> What's your email address? Sounds like you are using my docker image so I am happy to keep helping you. I also completely understand the desire to do so - jukebox has been the most fun with music I've had in a very long time and the results I'm getting in my own domain (electronic music) are incredible 😭 I've spent pretty much every day since jukebox released editing the code to support various things that I need for better co-composition including the ability to finetune to the 5b model, much larger batch sizes for sampling, editing of tensors for composing before they are rendered as audio, etc. I think I can get you to the same place pretty easily without asking for any money, so let me know and we can get you rolling! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#142 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APWDYJSJCSGAOVFR6F7IV3LSEJAXZANCNFSM4Q2LJSWA> .

[perlman-izzy]

Actually, I can't know by your username if you are male or female, so suffice to say you are a noble soul. My goal is to train it on 2 songs of my own. Have the instance w dockerfile set up (thank you!), able to upload my songs. Runs for about 2 min before a broken pipe error with this: file "jukebox/sample.py", line 279, in <module> fire.Fire(run) File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 127, in Fire component_trace = _Fire(component, args, context, name) File "/usr/local/lib/python3.7/site-packages/fire/core.py

…

On Sat, Sep 5, 2020 at 4:41 PM The Crazy 88s ***@***.***> wrote: Yes! OMG. You are a prince among men, sir, and I am in your debt. My email is ***@***.*** On Sat, Sep 5, 2020 at 7:06 AM btrude ***@***.***> wrote: > @perlman-izzy <https://github.com/perlman-izzy> What's your email > address? Sounds like you are using my docker image so I am happy to keep > helping you. I also completely understand the desire to do so - jukebox has > been the most fun with music I've had in a very long time and the results > I'm getting in my own domain (electronic music) are incredible 😭 I've > spent pretty much every day since jukebox released editing the code to > support various things that I need for better co-composition including the > ability to finetune to the 5b model, much larger batch sizes for sampling, > editing of tensors for composing before they are rendered as audio, etc. I > think I can get you to the same place pretty easily without asking for any > money, so let me know and we can get you rolling! > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#142 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/APWDYJSJCSGAOVFR6F7IV3LSEJAXZANCNFSM4Q2LJSWA> > . >

[perlman-izzy]

@johndpope I have access to a GPU with 24gb 'per-GPU' ram (which I think is the same as VRAM?). With that amount of power, how long do you reckon it would take to process 6 min? 10 min?

[perlman-izzy]

johndpope last error message was : file "jukebox/sample.py", line 279, in fire.Fire(run) File
"/usr/local/lib/python3.7/site-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name) File
"/usr/local/lib/python3.7/site-packages/fire/core.py

[btrude]

@btrude Any chance you can make those changes available? I'd like to roll that stuff into the google Colab I've been using.

@bsturk Possibly... I'm not at all familiar with the colab setup as I've been running it locally since may. If the colab simply clones the jukebox repo then it will be a bit difficult as all the changes I'm describing are at the sample.py and train.py levels - though we can definitely figure something out if you can link me to the file you're using. Finetuning the 5b requires 48484mb of VRAM without using the distributed model (ie its only possible to do it with one RTX 8000 at the speed and batch size [1] that such a setup implies) so I don't think you will be able to do that much with colab, but maybe they offer RTX 8ks? Either way, finetuning the 1b_lyrics model is possible with 1/3rd the VRAM of the 5b and yields awesome results as well so not too much of a loss and hopefully I didn't get your hopes up mentioning the 5b. The co-compositional changes I've made are actually just realizing features that are described in the paper but weren't actually shipped with this repo's code, so I have to wonder if some of those features are present in the colab perhaps? Specifically though, I've made most input parameters hyperparameters safely accessible in the code via calls to hps.get('some_param') (so you can specify genre/artist/temperature from the command line and different prompts per-level), and then implemented a new mode truncate and an additional parameter pref_codes which will duplicate the specified codes for subsequent continuations/upsampling. All of this allows you to do as the paper describes, creating an initial batch of n_samples at 1x n_ctx, then listening to the samples and choosing your favorites, then setting them as the pref_codes values and creating multiple different continuations of the same codes, and so on (with the added bonus of being able to shave off unwanted parts of the samples using truncate). All of that is to say let me know if these are actual things you need and I will gladly share the code/modify your colab/whatever.

I have access to a GPU with 24gb 'per-GPU' ram (which I think is the same as VRAM?). With that amount of power, how long do you reckon it would take to process 6 min? 10 min?

@perlman-izzy I sent you an email about your other issue, but an rtx titan doing levels 2, 1, and 0 will take ~48 hours using the default hop fractions for 6 minutes of music. It's hard to say how long it takes to do each level individually, but its probably somewhere around 4-6 hours for level 2, 8-12 hours for level 1, and 30 or so for level 0. All of this depends heavily on n_samples and whether or not you are doing any sort of editing/pruning of codes.

[johndpope]

@perlman-izzy - you can type nvidia-smi to show card. (There are spot prices on ec2 - $.027 for k80 gpu - 24gb - you can get on aws which are cheaper - but they can terminate unexpectedly but you pay 10% of regular price. Although they have heaps of V(ideo)Ram - the k80s that are provisioned on lower end of aws are really slow https://technical.city/en/video/GeForce-RTX-2080-Ti-vs-Tesla-K80 - the current card 2080ti is 333233.3% faster. Again - the latest 3090 cards have 10,000 cuda cores / and if you were to get 2 - you'd be 48gb / and could handle fine tuning larger model. a tid bit - I think the line
"On a V100, it takes about 3 hrs to fully sample 20 seconds of music" - doesn't just reflect a single file of 20 seconds - but a whole suite. happy to be corrected here.
also - @perlman-izzy - presumably - you are using ubuntu? docker + windows + nvidia (historically) don't play well together. you'd be better off with a dedicated ssd drive to boot to with ubuntu 18 / (don't go for ubuntu20 just yet).

[btrude]

you'd be 48gb / and could handle fine tuning larger model

@johndpope This is unfortunately not true. Jukebox as released here uses distributed data parallel which requires the model be duplicated across each GPU. I've experimented with this extensively and as far as I can tell the only way to finetune the 5b (5b_lyrics is too large) is with an RTX 8000 removing DDP or, as mentioned in the readme, with some custom implementation of gpipe.

[johndpope]

@btrude - I stand corrected. So, to train on aws /ec2 with (SPOT PRICING / can terminiate at any instant) p3.8xlarge (64GB of VRAM) is $3-4 /hr in N. Virginia ~ $100 USD/ day
https://aws.amazon.com/blogs/aws/new-amazon-ec2-instances-with-up-to-8-nvidia-tesla-v100-gpus-p3/
That gets you 8x v100 cards. The lower model down has less ram - p3.2xlarge | 1 | 16 GiB won't handle fine tuning larger model. @btrude - did you upload any of your creations to soundcloud?

[Cortexelus]

We successfully fine-tuned the 1b model in the cloud on p3.8xlarge, on a custom dataset, over a few days.

[btrude]

@btrude - I stand corrected. So, to train on aws /ec2 with (SPOT PRICING / can terminiate at any instant) p3.8xlarge (64GB of VRAM) is $3-4 /hr in N. Virginia ~ $100 USD/ day
https://aws.amazon.com/blogs/aws/new-amazon-ec2-instances-with-up-to-8-nvidia-tesla-v100-gpus-p3/
That gets you 8x v100 cards. The lower model down has less ram - p3.2xlarge | 1 | 16 GiB won't handle fine tuning larger model. @btrude - did you upload any of your creations to soundcloud?

Not yet, no. Hoping to do so soon.

[btrude]

@btrude I'm using the colab located here

#72

and yeah, I'm not sure I could utilize the 5b changes you've made in Colab, I'd be interested in the 1b changes though.

@bsturk In trying to implement this I'm finding that the colab linked here is just too janky and at odds with sample.py to just do a clean copy and paste so I'm going to have to hold off for now. It's really weird to me that the colab duplicates code in sample.py when its possible to make calls from the command line from within colab. I also don't think the colab as is really elucidates how the larger sampling process works either, so its not like there is any benefit at all to using it like this. I honestly feel bad for everyone using it because it would be 100x simpler if you replaced all the python with a few example shell commands like those in the readme and then made the relevant parameters into fields on hps. That would take some work, but I have a small demo working in colab already. Now it would just be a matter of providing my sample.py as a gist and then using it instead of openai's. Let me know if you would like to test this and I'll go ahead and put it together.