Well, I'm sure you're fed up of me saying that the moment of truth has arrived, but it really has.
The moment of truth has arrived.
I'm extremely excited, as you could probably tell, and I hope you are too.
This is this is a big deal, a big, a big moment for us.
So first of all, we are in the week seven day three Colab, and I must confess that I'm doing something
a bit naughty in that I have picked a super beefy A100 GPU box, which is the most pricey of the Colab
boxes it consumes.
It gobbles up 11.77 uh units compute units per hour, and I think the current going rate, if I'm if
I'm not mistaken, um, it's roughly 100 units cost about $10.
So this is this is spending about a dollar an hour.
Um, right now those prices do change from time to time.
Um, so, yeah, This is this is definitely not as cheap as as things usually are.
And you don't need to do this.
You can definitely use a, um, let me see.
Uh, change runtime type.
You can be using a t4 GPU.
I will tell you which parameters to change for that and it will be just fine.
Um, but if you don't mind spending a few dollars and you want to have a blast and you want to be changing
hyper parameters and training fast, it's beautiful to have an A100 box for a while and experience the
raw power of a box with 40GB of GPU Ram, so I love it.
I don't I I don't mind giving Google a little bit of money for this, because it's a it's such a treat
to have a juicy, powerful box like this.
So anyways, we start with some pip installs.
This includes this library TRL Transformers Reinforcement Learning from Huggingface, which contains
the SFT trainer supervised Fine Tuning Trainer, which is the trainer that we'll be doing the work for
us today.
Um, and I do want to say I should take just a moment to say that I talked, uh, in the last video,
um, a fair bit about training, and I, I sort of did it almost by the by, I talked about about some
of the hyperparameters and in doing so talked about a bit of the training process of forward passes
and backward passes.
And again, for some people, that's old hat that you're very familiar with this.
For some people that might have gone over your head and you might be be be saying, you know, can you
not take more time to better explain the whole training process?
And one thing I do want to say is that hugging face has made this so accessible, so easy.
Um, they've made the barrier to entry for training a model so low that whilst it's very helpful to
have an intuition for what's going on in terms of, of of optimizing, of taking steps in the right
direction, this is helpful.
It's not essential to know the detail of the theory.
Uh, you just have to know enough to be able to tweak the hyperparameters.
Parameters.
Hugging face makes it incredibly easy to do the training.
So if you did feel like some of that went over your head, then rest assured it really doesn't matter.
The code will make it very clear.
And if you know this stuff back to front already and you're very familiar with optimization, then great.
That will make it so much the better.
It'll make it even easier.
All right, so we do some some imports.
And now we have a bunch of parameters to talk about.
So uh, the base model of course llama 3.18 billion.
We know it well the project name.
So this is the project name that will be used in weights and biases.
Uh, when we get to that, to, to show the, the, uh, the results that will compare.
And we'll also use it when we upload to the Hugging Face hub.
And I'm using the name Pricer for this project, as in something that comes up with prices.
You may remember when we were training a GPT, I called it Pricer GPT.
I've kept the project separate because the results will be so different That just because we'll be measuring
different quantities, it will be confusing to have them in the same project.
Um, so, uh, I've called this just pricer.
Uh, hugging face user, you should put your hugging face name in here.
Uh, because you'll be wanting to post push your fine tuned models to the hub for your models because
you will treasure them and use them in the future.
And you can keep them private.
Uh, they're just for your own consumption.
So when it comes to the data, we need to load in the data set.
And here you that you have a choice.
If a few weeks ago when we were doing data curation, you did go all the way through and you uploaded
the data set to your Hugging Face hub, then you can keep this line in here and it will have your hugging
face username and then pricer data, which is what we we called it, and you'll be able to load it in.
Alternatively, if you decided just to watch that one out and watch me doing the data set stuff but
you didn't, uh, upload it to Huggingface then then.
Shame.
But still, I do understand there was quite a lot to it.
You can simply uncomment this line where I've just hardcoded.
We don't need the f there.
Uh, we've just I've just hard coded the the the price of data in my hub, which will be made public.
So you can just download it there for.
Fine.
You don't need to upload your own one.
And then maximum sequence length, you'll remember the data is always crafted so that it's no more than
179 tokens, adding on a few tokens for the start of sentence and anything any gumph at the end, uh,
means that we're going to say maximum sequence length of 182.
And this is very important because every training data point will be fitted into this.
So that and this needs to be a small enough number to fit in our GPU's memory.
And that's why we've cut off our descriptions to fit within this amount.
So then a few, uh, sort of administrative things, um, I've come up with something called the run
name for each of these runs, which is quite simply the current date the year.
Month.
Day and hour.
Minute.
Second.
Right now of this run and you'll see why in just a second.
I'm going to have something called the Project Run name, which is Pricer, and then a hyphen, and
then the date, and then the hub model name where I want to save the model will be the username.
And then this here.
Uh.
And why am I doing this.
So sometimes people like to just have one model.
And as you run this you just upload different versions that you store against that same model repository.
Because remember everything in Huggingface is just a git repo.
So you could just keep um, you could basically just keep pushing new versions of your model that will
just be, uh, just new versions of that, like, like checking in new versions of code or pushing versions
of code.
It could just be different versions of the model with the same model name.
But I quite like to separate out my different runs and have them as different models, because within
them there'll be different versions, potentially as different epochs.
And I like to keep them separate because I'll have trained them with different hyperparameters, and
I want to keep notes of that.
So that's, that's the way I like to do it.
Um, not strictly necessary, but I find it helpful.
Uh, and just to give you a sense for that, if I just take some if I take the run name, let's start
with that.
If I just show you what run name is right now.
Um, the run name you can see is just the current date that I'm doing this the 22nd and the current
time.
That time is in UTC in universal time.
I'm not actually, it's not actually four minutes past midnight.
Uh, so that's the run name.
And then what was the other thing?
Uh, there's project run name and then hub model name.
So project run name is just Pricer followed by that.
And then hub model name, which is what it will be uploaded as.
Is that so?
This will be creating a model with that name after it's run.
And so if when I look in my models directory, I see a bunch of these because I've run this too many
times, more times than I'm willing to admit.
Uh, but it's been great fun.
Uh, I've got lots of these pricer models.
Uh, and they've all been running.
Uh, so just to finish this off, um, I see it just disconnected and reconnected over there.
Uh, so the hyperparameters then that we are using for training, uh, you know, are, well, the dimensions
of the matrix.
Um, and I'm starting with 32.
As I say, you can bring that down to eight.
Uh, particularly if you're running on a lower box.
It'll be just fine.
Uh, Laura, Alpha should be double R.
Um, and so if you bring this down to eight, then make this 16.
Um, the target modules, of course.
Are you know it too well.
I don't need to tell you what they are.
And this is the standard ones for the llama three.
These are the modules that you target.
The Lora dropout was the thing that I gave quite the long explanation of last time.
It's the the the trick to help with regularization, or with making sure that models generalize well
to new data points by taking 10%, in this case 10% of the neurons, and just wiping them out, setting
their activations to zero, effectively just removing them from the training process a different 10%
each time.
And as a result, the model doesn't become overly dependent on any one neuron.
It just learns to generally for the whole model to get better at receiving training points and giving
the right next token so it helps the model generalize.
10% is 10.1 is a very typical starting point.
You should try 5% and 20% and see how they perform and quant for bits.
True.
Is is quantizing down to four bits.
Okay.
The hyperparameters for hyperparameters for training.
I've set this up to run for three epochs.
You could do it just for one.
By all means, you'll have perfectly decent results after one batch size.
I've got to 16 right here.
Uh, I would, would you need it with.
With, um, a juicy A100 box?
I can pack in a batch size of 16 batches.
Given that this is the max sequence length, it can cram all of them in and still have just about squeeze
that into the 40GB of memory.
Um, but for for for you, if you're going to go with a lower end box like a T4, then this should be
probably one you could try.
Try it higher.
If you get more GPU memory that's still free.
Um, there's this convention that people tend to use powers of two for batch size.
So one or 2 or 4 or 8 or 16.
Um, and in theory, there's a there's various loose bits of evidence that suggest that if you have
it as a power of two, the GPU is better able to parallelize it, and the the performance is slightly
better.
Um, but there's always the data on that is, is always, uh, just a little bit vague.
And so generally speaking, I imagine if you can cram another a little bit more batch, uh, onto your
GPU, then you should do so.
So I wouldn't shy away from having a batch size of three.
If you can fit that on your GPU, it's going to run faster than a batch size of two.
Uh, maybe maybe four would would just be that little bit more, be a bit more efficient if it could
fit.
But I think the general advice is just whatever will fit on your GPU is what you should pick and start
with one and see what kind of headroom you have.
Unless you're splashing out like me, in which case go for 16 gradient accumulation steps.
I explained that last time.
I think actually this helps with the improve the memory of the process.
Uh, you could try, but But I'm staying with one.
You can try that at 2 or 4 as well.
Learning rates I think you know understand is super important is how much of a step you take as you
optimize to try and take a step in the right direction.
And learning rate is a super important hyperparameter that you need to experiment with.
Um, and it's one of those things which which, uh, there's no right answer.
Learning rate could be too high or it could be too low.
The what you're trying to achieve this is going to be a bit hand-wavy.
Again, what you're trying to achieve is, if you imagine that the loss, uh, the true the loss of
your model, uh, is something which has a big dip in it like this.
And you're trying to find this valley, you're trying to locate this valley.
Then you want to be taking these steps, uh, along the direction of the valley, dropping so that you
will end up at the bottom of this valley.
If your learning rate is too high, you might jump over that valley and jump back again and never actually
go down into the valley.
If your learning rate is too low.
You might take little tiny steps and be making two slower progress towards that valley.
There's another problem with taking two small steps.
Supposing that you don't just have one big valley, but you have a smaller valley and then the big valley.
If your learning rate is too low, you might take small steps and end up sinking down into that little
valley.
And actually every small step you take just goes up the two walls of the little valley and never gets
out of that little valley.
And so you never realize that there was this much bigger value valley just next door.
And so that is a key problem.
It's a common problem with learning rates being too small.
People call it being stuck in the local minimum.
This thing is called a minimum and it's local to where you are.
And you haven't found the the global minimum, which is which is all the way down here because you're
stuck in the local minimum.
And that is the problem with having a learning rate that's too low.
There is this nice trick.
I picked a learning rate of 0.00011 times ten to the minus four.
There's a nice trick of using something called a learning rate scheduler, which is something that will
vary the learning rate and make it get smaller and smaller and smaller over the course of your three
epochs, until it's pretty much zero by the end of your three epochs.
Um, and you can give it.
If you choose to use one of these, you can give it a different shape.
And cosine is a very nice one, which starts with a learning rate that starts at the slowly decreases,
and then it decreases quite a lot.
And then it tails off at the end.
Uh, you'll see that visually in a moment.
And that's a really good, good technique.
Um, and the final learning rate parameter is called the warm up ratio, which is saying at the very
beginning of your training process, things are quite unstable because your model has a lot to learn
from the first few data points.
And it's quite dangerous to have a big learning rate initially because it would jump all over the place.
So warm up ratio says start with a lower learning rate and then warm it up to the learning rate that
it becomes the peak.
And then start your cosine, uh, trail.
And you'll see that visually.
So it'll make more sense.
But these are these are very sensible settings.
But you can definitely experiment with a higher or lower starting learning rate or different scheduler
types.
And finally the optimizer.
So here I am picking the paged Adam w uh w means with weight decay a paged Adam w 32 bit.
That is a good optimizer, which has good convergence.
It does a good job of finding the the optimal spot, but it comes at a cost of consuming memory.
Um, I've put down here a link to a hugging face writeup on the different optimizers that you could
pick.
Um, it's clear that the most common for Transformers is Adam or Adam.
W um, and that Adam, uh, does does do well because it stores the rolling average of prior gradients,
and it uses that rather than just the most recent gradient.
But of course, by storing that, that's going to take an extra memory footprint.
And so it's a bit greedy for memory.
So if you're running out of memory then you have an option to choose a cheaper, less greedy optimizer
to save some memory.
But the results might be slightly worse than than using the paged Adam w okay.
And then finally some administrative setup.
Uh, this number of steps is how many batch steps to take before it.
It saves progress to weights and biases to show us how things are going.
Um, and this is how many steps before it actually uploads the model to the hub and saves a proper version
of it.
This is whether or not we're logging to weights and biases, so that that is a tour of the parameters.
And in the next time we really get to to the trainer itself.