So here we are now, back in the Colab, in the same one that we kicked off in the previous day.
It's now somewhat misleadingly named week seven, day three because we are of course on week seven,
day four.
But I daren't touch it for fear that I that I, uh, stop it in some way.
You can see that the GPU Ram continues in its nail, bitingly close to the ceiling level of 38.2 out
of the 40GB.
But it's luckily not changing.
Not going up.
And you'll see here this is the printout of each of the different batch steps as they happen.
We're now on batch step 4350.
Um, and if we zoom all the way up, uh, which again, I'm doing everything very delicately so I don't
accidentally stop the batch in some way.
You can see that we're only a little tiny way into this strip that represents all three epochs.
So one about there perhaps is a complete, uh, run through all of the data.
Um, and you can see we're at batch step.
Uh, I can't say the number because it's ticking too fast, but for for ten, uh, there we go.
Uh, out of this very round number, uh, that you see here of 100,000, uh, and you might wonder why
it's quite so round.
Uh, and, uh, it's just something of a coincidence.
Uh, remember, we have 400,000 data points, and but they're in batches of 16 are in any one step.
So if I take my calculator here for this trivial maths, take 400 000 and I divide that by 16.
Uh, and then I multiply that by the number of epochs.
Um, I thought we had three epochs, but it looks like we have four epochs.
I left it at four epochs.
Uh, so, uh, yeah.
If you then multiply that by four for four epochs.
Uh, you do end up with, uh, just to give evidence of that, there's the camera, uh, 100,000, uh,
and it's that that is the 100,000 here.
It's not just a wild random number, randomly round number.
It is, in fact, exactly the number of batch steps involved in 400,000 data points, grouped into batches
of 16, done four times over.
Uh, so off it ticks.
And you can see just from looking at these numbers that we started with quite high training loss and
that that number has come down, uh, quite significantly.
But it's sort of hard to tell from here exactly what's going on, because the numbers bounce around
up and down.
Um, and it's hard to get a sense of the trend.
If only there were a tool that would allow us to visualize the progress of the batch in a way that could
allow us to compare between different runs and the like.
Of course, there is such a tool and it's weights and biases and it's sitting right here.
Um, and this this is the result of our run.
This is it happening right now.
Um, and this is showing us what's going on.
So.
So this training loss here is the real the real thing.
This is what we really want to look at.
Let me just edit this panel so we get to see it a bit more.
Um, now this the the y axis here goes all the way down to zero where zero will be zero loss, which
would mean perfection.
It would mean that the model is always predicting with 100% confidence the next token and that next
token that it predicts with 100% confidence is the right next token.
Um, so that is an unlikely place to get.
In fact, one would be suspicious if you got much below 1.5.
Typically this is a bit of a rule of thumb thing, but but generally speaking, that would be like a
great, uh, loss to have.
And below that might cause you pause for thought about whether overfitting is going on.
Um, let me change these axes and make the minimum of the y axis one so that.
Well, we can make it.
We can do even better.
We can make it like 1.4 or something, um, so that we can really see what's going on here.
Um, and it's a bumpy line, but it is pretty clear even at this early stage, that that is a bumpy
line that has an improving trend.
Unlike when we saw this with, uh, GPT and the fine tuning, when it seemed to be bouncing up and down
but not really showing a trend, I think you can see that this looks like it's improving.
We can also apply the smoothing thing and see how that looks.
I mean, that's clearly isn't it.
That is that is a line that is improving.
If we apply the smoothing factor.
Now, why do these bump up and down?
I mean it depends really.
It's remember each of these points represents, uh, 16 data points, 16 prompts with a thing for it
to predict at the end of it shoved into one and it's a random it's been jumbled up.
So it's a random set of the 16.
And so who knows which products are shoved into the 16.
There could be some really expensive products that it's making a wild guess at and getting completely
wrong.
So there's plenty of noise.
Um, because the there's a different makeup of each of these different batch steps, and it's good to
have a bit of noise because you want to be shaking things up a bit and getting the model to be, um,
being bounced around.
Um, in a sense, again, I'm being very hand-wavy, but because you're trying to not get stuck in a
local minimum, but be trying out different, different possibilities with the idea that the model will
improve and find a big global minimum, a big valley as it tries to, to, um, find gradients and improve
its ability to predict the next token.
Um, it's just moved on a little bit.
And again, it's very clear that there is visually some improvement happening here.
Um, so what else can we see?
So this is the learning rate.
This is the, uh, the that key hyperparameter about how much of a step it should take.
And I told you we'd chosen a cosine learning rate, and you might be a bit surprised to see something
which doesn't look very cosine y at all and doesn't look the way I described it.
And this what you're seeing here is that thing called warm up, which is the fact that it that it doesn't
start up at the, the 0.0001, the highest number, it starts at zero and builds up to that point.
Because you can see the beginning of the batch is quite a dramatic movement from being way off to to
being in a better position.
And you don't want the learning rate to be too high then.
Or it might do, it might overshoot in all sorts of ways.
So that's the theory behind this warm up.
Um, uh, part of the process.
Um, and you can see it really, really coming to, to, to be here.
Um, and you might wonder why this isn't really showing much in the way of a cosine kind of, uh, shape.
And that's because it's so early in the training.
Still, we're still at the very top of this cosine.
It's about to start curving down, and it's doing that very slowly.
Um, and, um, yeah, there's also some fun charts here that show us what's going on on the GPU.
Uh, some of these are more meaningful than others.
There's some here.
The, um.
Yeah, the power usage and the time spent accessing memory is good.
Uh, it's.
I want to see the CPU utilization.
Where is that?
I mean, the GPU, even the CPU utilization.
The GPU utilization would be the perhaps one of the most important to make sure that the time that the
GPU is hard at work, it's not like it's shepherding memory in and out, and that that's what's taking
the time.
You want to know that the GPU is being utilized.
And that's a great sign that we are hammering our are powerful A100 blocks and making the most out of
it.
And if you're using the different box, then you've got some different hyperparameters.
Then come and check out the GPU utilization, make sure it's doing well, make sure it's nice and hot,
uh, and that you're getting good use out of your GPU.
Otherwise, you might want to tweak some of the hyperparameters to see if you can't get more juice out
of it so that it's your training process is more efficient.
Um, okay.
So what I'm going to do now is do a bit of a cheat.
I'm going to do what they do in those cooking classes or like, like cooking videos when they put it
in the oven and they say, and now here's one that I did earlier, they take it out of the oven and
it's the thing that they put in, you know, it's like, oh, that's uh, that's just cheating.
But I have done that and that I did kick this off a while back and it ran with the same hyperparameters.
So the same thing, um, and, uh, and it's this pink one right here.
And this I should have explained.
I'm so sorry.
This here is showing the four runs that have happened under this project.
The project which is called Pricer.
So up at the top here, this navigation, um, it has the Pricer project.
And down here are the four runs.
Um, and this run here was when I ran either 3 or 4 epochs, um, of this model with this same data
set and with the, uh, yeah.
With otherwise with everything else the same.
Uh, and so if I show you that, we'll see what happened as a result of that.
And here we go.
So this this is, uh, this is the meaty one.
So let's let's bring this up and we are going to have to change the scale.
We're going to have to come down.
There we go.
Now you can see everything.
If I leave it at one.
Okay, so a few things to point out.
So first of all you're seeing a purple.
Let me click.
I think if I do this one here it's going to bring it up.
There we go.
So you can see here a blue and a purple line.
The blue is just here and the purple is here.
The blue is the current run that is running right now over on this tab.
It's this this guy that we are running right at the moment.
The purple is the one that I kicked off a while ago.
I of course it has the data in there about ten days ago.
Um, and you can see that the blue is tracking extremely closely to the purple, which is further evidence
that I'm not I'm not cheating here.
Uh, it is the case that that blue will continue to follow the same trajectory as the purple.
The purple has just had its course.
Uh, now, what you'll see is that the trend indeed improves and improves and improves, which is good
Ish.
Um, so first of all, you'll, you'll see that it that it improves and then it takes a little dive
and then it improves, and then it takes a little dive again and improves and it takes an even bigger
dive.
And so you might be wondering what are these dives?
Well, these dives are the end of each of the epochs.
So this is an entire epoch.
That's epoch one, that's epoch two, that's epoch three, and this is epoch four.
And unfortunately it crashed.
Uh, or Google reset the instance uh, halfway through uh, epoch four.
So, uh, we didn't get to see how epoch four ended.
Um, but as you'll see, that that proves to be unimportant.
Um, so why does the why is there this sudden drop at the end of each epoch?
Well, that's a very good question.
It's extremely important.
Uh, it's because what's starting to happen is a little bit of overfitting.
What's happening here is that the model is seeing some of the same data that it already saw in the first
epoch.
Now, sure, it's muddled up differently.
and seeing them in, uh, in batch batches that are different sets of 16 than the ones that got here.
Hugging face takes care of that for you.
The SFT trainer automatically reshuffles the batches each time.
Um, but nonetheless, the models had the benefit of seeing this data before, and it can take advantage
of that.
It's learned something despite the dropout.
Um, despite some some other things we've done to try and regularize, um, it's still has a leg up
on the fact that it's seen this exact data before.
Um, but luckily there's only a small step down there, so we don't need to be too concerned that there's
overfitting happening.
But then it gets worse here and it gets significantly worse here.
Uh, and so this is a big sign of overfitting.
And now, if I had been doing what I told you is a best practice and I should have been doing, which
was running validation runs at the same time, uh, the chances, first of all, they wouldn't have
taken these little jump downs.
But secondly, it would have probably started to go up at this point because we're overfitting.
And indeed I have savings of the batch at this point.
And sure enough, it was saved up to the Hugging face hub every 5000 steps so I could test it.
And sure enough, the results do get worse past the third epoch, so there was no point in going more
than three epochs.
So it didn't matter that Google pulled the plug on my instance at this point, because this data was
actually no longer useful.
The model was already doing poorly.
And again, this is a great example of where you can regularly upload to the hub, and then you can
go back at each of these checkpoints and run your test over them.
And you can pick the model that performs the best.
And that's a very powerful technique for so that you don't have to guess how many epochs to run.
You just run too many and then select the one that has the best results out of your training data.
Out of sample when you're trying something new.
Um, so I think it's a really good illustration of of how this works.
Um, and of the, the, the, the effect of overfitting and the effect of different epochs.
And, but what we know for sure is that during that first epoch, say, because it was always seeing
new data, all of this slope downwards is all representing good improvement.
And I can tell you that the results at the end of here were also distinctly better than here.
So this was also showing some improvement.
And but around here it started to get a little bit more dodgy.
And maybe, maybe coincidentally that is at around the 1.5 level.
And I mentioned before there was that rule of thumb that less than 1.5 maybe is a time to be raising
an eyebrow and looking again at your results.
Uh, so I will pause at this moment, and when we return, we'll talk a little bit more about a couple
of other things.