And now let me make this real for you by showing you some, some diagrams, particularly now looking
at how training works with Chullora, which is how we're actually doing it in practice.
So first of all, let's talk about this forward pass.
So here is a diagram that should be familiar to you because it's the same one I used before.
That shows our llama 3.1 base model that we've quantized all the way down to four bits with its 8 billion
parameters and the white, all of it is frozen.
We're not going to be changing its weights as part of training, because it would be way too much work,
way too much memory, and too slow to try and shift and tweak and optimize these 8 billion parameters.
So what we're seeing here is a bunch of frozen rows of weights, and the ones in yellow are also frozen,
but they're representing our target modules.
Which color is going to be applied to.
And let's bring in our Lora adapters.
There they are.
You may remember that to be technically correct, there are in fact two adapters called A and B for
each of the different, um target modules they have.
Dimensionality given by R, and you may remember that alpha is the scaling factor.
And it just is as simple as the way that these are applied to their target module is that it's alpha
times A times B.
Um simple as that.
So that is our neural network that we know well.
And we are going to be training the weights in this, these Laura adapters, which in our case is about
109MB worth of, of weights.
Uh, okay.
So then what happens?
We have an input prompt over on the left.
It is like price is dollars and then it's the next token.
We want the model to get good at predicting.
The forward pass is when we take that and we in inference mode basically we we put it take it through
the the model to say please predict the next token.
And so that goes through the model and what comes out the other side is a predicted next token price
is.
And then 99.
And again because we've got we're taking advantage of this simplicity of llama 3.1 that in fact 99 is
just one token.
Um, not not.
And that will always be the case for any three digit number.
That's not that that's critical, but it does simplify things a bit for us.
So that's the forward pass.
All right.
Now onto the loss calculation.
So here we are again.
We've predicted the next token.
So now the model is able to look up what or not the model the training process.
The SFT trainer in our case looks up.
What was the actual next token.
Because we've got the training data, we know the actual next token.
And what was it?
Let's say it was 89.
It was lower.
So we were wrong.
Wrong by $10.
Um, or wrong.
By different token, it doesn't know that this represents $10.
It just knows it's a different token.
Um, and so there is some kind of a loss.
And in just a moment I'm going to explain what that loss is and why it's not quite as simple as it just
being a different token.
There's a technicality there that we'll get to, but for now you can just think of it as it predicted
99.
The actual value is 89.
We have a loss.
So that's the loss calculation.
Step three is the backward pass that you hear people calling backprop or backward propagation.
Back propagation.
Um, and in backprop basically we look back through the network, back we go and we say, all right,
so how much if we if we were to tweak these weights by a little bit, how much would that affect the
loss?
How sensitive is the loss to to those those weights.
It gives us what we call the gradients of the weights of the parameters.
So when I say weights are synonymous with parameters.
Uh, so, uh, how, uh, um, yeah.
What are the gradients?
If we were to change those weights, what would it do to the loss?
Because we want to try and improve things a little bit.
Um, and so those, uh, red triangles are meant to represent Delta, uh, for showing like a gradient
calculation that's happened there.
Uh, and then finally we get to the last step, step four optimization.
Here it comes.
So we've got these gradients.
And what we now need to do is we want to take a tiny step in the right direction.
So we want to update the parameters in our Laura matrices a little tiny bit.
So that next time if it gets the same input prompt, the loss will be a bit lower.
It would do a bit better.
So we're taking a step in the right direction, and we use the learning rate to decide how much of a
step to take.
Because as I said before, there are pros and cons of taking smaller or larger steps.
Um, and you may remember that the optimizer we're using, the Adam W optimizer, does something quite
cunning, where it doesn't just use these gradients, it keeps a kind of rolling average of prior gradients,
so that it's being really smart about how to take that step in a way that's most likely to improve things.
And it's also trying to make sure that we don't do things like overfitting and other dangers that we've
talked about before.
So we do that.
And what happens is that the Laura adapters are then improved, and as they are applied in the future
to our base model, it will do slightly better.
The loss will be a bit lower next time, and that is why you see that the training loss is coming down
and our wiggly charts, because it's learning and getting better and better.
And so it's worth noting, of course, that the thing that the weights, the parameters that get changed
are the parameters in the Laura adapters, the green ones, we don't actually change any parameters
in the llama 3.1 base that is just too big, too many parameters.
If you were doing ordinary training, fine tuning, not not Laura based, but ordinary, then it would
be the whole llama 3.1 model that would need to be having gradients calculated and need to be shifted
during optimization.
And that, of course, is what these big companies like meta.
That's how they've trained llama 3.1in the first place, and they've spent significant amounts of money
doing it, which which we don't have easy access to, which is why we're using our Lora adapters instead.
So that's the quick summary of optimization in the sorry of the whole of the training process.
And I imagine that was probably mostly clear to you already, but I'm hoping that these diagrams of
crystallized it for you and mean that everything has fallen into place.
And in the next video, I'm just going to explain one more technicality, a very important technicality
about what we mean by the prediction and the loss calculation.
See you then.