Without further ado, we're going to get stuck into it.
Talking about Laura.
Low rank adaptation.
And this is going to be a section where we will talk a bit of theory for a few slides, but fear not,
we're going to get straight to practice as usual.
The best way to understand these things is by seeing them in action.
So just after a couple of slides, we're going to hit Colab and look at these things for reals.
But first just to set the context then.
So look we're going to be using llama 3.1 for this for this week.
And llama 3.1 comes in three sizes.
It comes in the 8 billion parameter size the 70 billion and then the monstrous 405 billion size.
Um, and of course we're taking the smaller one, the 8 billion.
Um, but even that is going to be way too large for us to be training realistically on, on the sort
of box we want to be able to pay for, like a one GPU box, um, 8 billion weights is already that's
32GB of Ram.
If you add it up, It's, it's and that's just to have the model in memory when you start training it,
which is about running optimization, where you have to be able to get gradients for each of these weights.
Um, that that's something which would consume way too much memory.
We wouldn't have a hope.
Um, and so it would also take a hugely long amount of time because there would be so much to, to be
optimizing across be optimizing these 8 billion weights.
That's that's just a lot of processing.
Um, and it's the kind of thing, of course, that these, uh, made that these frontier labs and places
like meta have spent very large sums of money for the biggest models, more than $100 million it costs
to train one of them.
And so that, you know, probably not the kind of money we're going to be spending.
Uh, so there are some techniques, some tricks, uh, which make it surprisingly low cost to train,
uh, from a base model so that you can make something that's better at achieving your particular task,
Assuming that it's got a lot in common with what the base model was originally trained to do.
Um, so before I explain what Lora is, let me just quickly summarize the llama architecture.
Now, we're not going to get into deep into neural network architecture in this course.
It's something which we'll I'll give you some insight, some intuition behind without going into a lot
of detail.
But the llama 3.1 architecture consists of stacks and stacks of layers of, of neurons.
Um, it's actually got 32 groups of these layers where each group consists.
So each group is called a llama decoder layer.
And it has in it some self-attention layers, some multi-layer perceptron layers and a silo activation
layer and layer norm.
And we'll see this in a second.
You don't you maybe you know what this is already.
If you've got a theoretical background.
If not, it's going to be be more real, be more tangible.
When you see this architecture in Colab in just a second.
Um, And all of these parameters sticking in this, in this big, uh, this, this, this, uh, layered
architecture take up 32 gigs of memory.
So this is now the big idea behind Lora.
The idea is, look, what we can do is we can first freeze all of these weights.
Normally, during optimization, you you do a forward pass through your neural network.
You figure out how, um, you look at the prediction, the next token that your network predicted,
you compare it with what the token should have been, what is the actual true next token.
And then based on that, you figure out how much would you want to shift each of the different weights
a little bit in order to make it so that next time it's a little bit better at predicting the right
next token?
That's the idea of optimization.
A bit hand-wavy, but you get the idea.
Literally.
Hand-wavy.
Uh uh, but uh, the the concept of Lora is, first of all, frees all these weights.
We're not actually going to optimize these 8 billion weights because it's just too much, too many things,
too many knobs to turn to, too many gradients.
Instead, we pick a few of the layers that we think are the key things that we'd want to train.
And these layers, these modules in this, this stacked, uh, layered architecture are known as the
target modules.
So that's where this expression target modules comes from.
That I said, sounds a bit like something out of Star Trek, but it just means the the layers of the
neural network that you will be focusing on for the purposes of training, but the weights will still
be frozen.
Instead, you will create new matrices called adapter matrices with fewer dimensions, so not as many
dimensions as are in the the real guy.
These will be smaller dimensionality or lower rank.
It's called um and and they will be off to one side, and you will have the technique for applying these
matrices into the target modules.
So they will they will adapt the target modules.
There'll be a formula which I will tell you about in a second.
But that formula will mean that that in the future, whatever values are in those blue low rank adapters
will slightly shift.
We'll slightly change what goes on in the target modules.
They adapt them so it's lower rank, it's lower dimensional, fewer weights that will be applied against
these target modules.
And then there's one little technicality because you'll see this in a second.
It's worth mentioning, in fact, because of the way that the dimensions work in these neural networks,
there are in fact two of these low rank matrices, one one is known as a and one is known as B.
And you'll see in the code they're called lora a and lora b.
So there are two matrices.
It's not super important to know that, but I want to make sure that when you see it in the code.
You'll see this and you'll say, okay, there are two matrices.
They get applied to target modules.
This makes sense.
And that at a high level is the story behind Laura Freas.
The main model come up with a bunch of of smaller matrices with fewer dimensions.
These are subject to training.
They will get trained and then they will be applied using some simple formula to the target modules.
And that way you'll be able to make a base model that will get better and better as it learns.
Because of the application of these Laura matrices.
And Laura stands for low rank adaptation because they are lower rank, lower dimensions and they adapt
the target modules.
There we go.
A lot of talking, a lot of words, but hopefully you've got an intuition for how this fits together,
and that intuition will become clearer when you see it in the code.
Um, but in the next session, we'll just talk quickly about one more thing, which is the Q, the quantization.