Just before we go on to some of the more advanced metrics, I want to mention for a second something
called the Chinchilla Scaling Law, which is a wonderfully named law coined by the Google DeepMind team
after one of their models called Chinchilla.
And it's related to how you think about the number of parameters that you need in a model, the number
of weights in the neural network, and what the law says is that the number of parameters, how many
parameters you have, is roughly proportional to the size of your training data to the number of training
tokens.
And what that means, basically, is that supposing you've got a model, let's say it's an 8 billion
parameter model, and you get to the point where you start to see that you're getting diminishing returns.
Adding in more training data isn't significantly affecting the model.
So you have this sense, okay, I've got now the right amount of training data for this size of model.
This is a good a good match up.
We've we've used our training data successfully for the model to learn to its to its most capacity of
learning.
And the question might be all right.
So if I wanted to add more parameters, give the model more flexibility to learn more and to be more,
more powerful and nuanced.
How many more parameters do I need given extra training data?
And the answer is, if you were then to double the amount of training data from that that point of diminishing
returns, you would need double the number of weights you'd need to go from 8 billion to 16 billion
parameters to be able to consume twice the training data and learn from it in an effective way, and
be that much more powerful and nuanced at the end of it.
So it gives you a sense of how many more parameters do you need to absorb more training data effectively.
And it also gives you the sort of the flip side, the opposite, uh, relationship to that.
If you're if you've been working with a model which is an 8 billion model, and then someone says,
we'd like to upgrade to a 16 billion parameter model, let's use that instead.
Uh, and you're thinking, all right, well, obviously, if I'm going to take advantage of all of this
extra flexibility, all of this extra predictive power in this bigger model with more, more dials,
more weights to learn from.
How much more training data am I going to need to be able to to take advantage of that?
And the answer is you would you would roughly need to double the size of your training data set.
So that relationship between the number of training tokens and parameters, uh, was, was suggested,
uh, a few years ago, and it stood the test of time.
It turns out that that for transformers for the transformer architecture, this scaling law appears
to apply.
Well.
And it's a great rule of thumb to keep to hand.
All right.
With that, let's just move on now to benchmarks.
So benchmarks are the common metrics that people talk about uh, which are used to weigh up different
models.
They are a series of tests that are applied and used in various leaderboards, which is where you rank
different, different LMS, uh, to see the different pros and cons of different models.
Now I've got this table of different benchmarks.
I'm going to go through them one at a time and get a sense for each one.
Now, you don't need to remember what each of these benchmarks are because you can always look it up.
It's useful for you to have a sense of it so that it comes back to you quickly.
So definitely focus and take this in and and and do some research.
If you have questions.
We're going to see these numbers in some of the analysis that we'll be doing later as we compare different
models.
So the first one I'm going to mention of the the seven most common benchmarks you see all over the place.
The first one is called Arc, which is a benchmark that measures scientific reasoning.
It's basically a bunch of multiple choice questions.
Drop is a language comprehension test which involves looking at text, distilling it, and then doing
things like adding or sorting or counting from that text.
Hella swag, which stands for harder encodings, long context and low shot activities, is a kind of
common sense reasoning test.
MLU is super famous.
You'll see it all over the place.
It was a really common metric that involves reasoning across 57 subjects.
There's been some there were some questions raised about how well formed the questions were.
And there's some, some doubts on the effectiveness of them.
Lou, it was perhaps overused.
And you'll see later that there's a variation on Lou which is now more popular called MLU Pro.
So this has somewhat been replaced.
Now, truthful QA is about accuracy and robustness, particularly in adversarial conditions when the
model is encouraged to not be truthful.
Winogrand is testing that a model can resolve ambiguity in more confusing contexts, and then GSM eight
K grade school math at the eight K level is both math and also word problems that are in elementary
and middle school level.
So these are seven common benchmarks you come across these a lot.
Uh, keep note of them.
They're in the resources.
And you will as I say, these are these are things you will see a lot.
And hopefully you will now recognize them.