I want to take a moment to talk about something that's very fundamental to an LLM, which is the number
of parameters that sits inside the LLM parameters, also called weights.
Generally, parameters and weights are synonymous.
There is a detail that they're not exactly the same in some situations.
But but basically think of weights and parameters as the same thing.
Model weights.
These are the levers that are within a model that controls what kinds of outputs it generates when it's
given some inputs.
How does it go about predicting the next word that's going to follow.
And these weights are weights that are set when you train an LLM.
It sees lots and lots of examples, and it uses those examples to shift around its weights until it
gets better and better at predicting the next thing to come out the next token.
We'll talk about tokens in a minute, but it gets better and better at that.
And the way it gets better is by adjusting all of its weights.
And for some of you that work in data science, this is all stuff you know very, very well for people
that are new to it.
During the course of the course, we will be looking at this in different ways, so you'll get a better
and better intuition to what it means to have these parameters, these weights that control the output.
But the first thing that one has to come to, to, to realize, to appreciate is how many weights we're
talking about and what this means in the days of, of the the simpler of traditional data science,
traditional machine learning, one would build a model such as a linear regression model, which is
something which sort of takes like a weighted average, and it would typically have somewhere between
20 and 200 parameters, or weights 20 and 200 is about the range you'd often usually be talking about.
And one of the somewhat bizarre, remarkable thing about these LMS is that we're talking about such
a different number of weights.
GPT one that came out back in 2018, I had 117 million waits.
Now, this was actually something that was personally galling for me, because at the time we had a
deep neural network, of which the LMS at types, we had one in my startup, and I used to go around
showing off that our deep neural network had 200,000 parameters, which I thought was a staggeringly
large number, and I couldn't imagine any possibility of a model that had more than 200,000 parameters.
So when GPT one came out with 117 million parameters, I was stumped.
It really made me appreciate, uh, how how the enormity of GPT one.
But then, you know, as you can probably see, this, this scale that you're seeing here is a logarithmic
scale, which means that every tick doesn't mean one more notch.
It means ten times the number of parameters as the, as the, as the tick before it.
And let's layer on to this diagram.
The subsequent versions of GPT, GPT two with 1.5 billion parameters.
GPT three 175 billion parameters.
I mean, this is just unspeakable.
Number of parameters.
GPT 41.7 6 trillion parameters.
And then the latest frontier models.
They haven't actually announced how many parameters they have.
It is believed that they have around 10 trillion parameters.
It is an almost unthinkable number of these weights that are running.
Um, and now let's layer on top of this some of the open source models.
So Gemma is 2 billion.
It's a lightweight model.
And you may also remember that llama 3.2 that we worked on when we were using llama also had 2 billion
uh parameters.
Uh, then llama 3.1, which is it's it's uh a Bigger Cousin comes in three varieties an 8 billion version,
a 70 billion version, and then llama 3.1 405B, which is the largest of the open source models at this
time and which has similar capabilities really to some of the frontier closed source models.
And then I mentioned mixed rail here, the mixture of experts model.
So this is just here to give you some insight into how enormous these models are.
And it's hard to even comprehend what it means to have 10 trillion different weights, different sort
of levers, different numbers.
You can think of them as little knobs within this enormous model that controls the output given an input.
And again, compare that in your mind to an old fashioned linear regression model that might have between
20 and 200 parameters.
Just to get a sense of the enormity of these large language models.