Okay.
It's moment of truth time.
I have just taken our class tester.
You remember this class?
Uh, it's the class that runs the evaluation across the full, uh, 250 data points from our test data
set.
It's slightly different.
Uh, if you if you go through this, you'll notice there's some very subtle differences, because we're
not taking an item object.
We're taking a data set, a data point from our data set.
So there's a couple of small differences.
But otherwise this tester is basically exactly the same, um, in terms of what it does.
And it ends of course with this single line tester test.
The model predict is the function that we just wrote that tries out our model against 250 points, and
we pass in the test data set.
And of course I've already run it and let me scroll through the results.
So you get a sense it's right up at the there we go.
We'll take it to the top.
So you'll see there, of course, that first red item is where it predicted $1,800 for something that
costs 374.
There's some more reds, there's some greens.
Um, and it certainly gives you a sense that it's not like we're getting that.
It's the model understands the task for this one.
For example, it guest $89.99.
And the truth was 101 79.
It's interesting that it's not sticking to the nearest whole dollar.
It's it's still creating things with $0.99.
Um, and you'll see that, uh, yeah, there's some there's some other problematic ones here, but there's
a greens and reds, greens and reds, but quite a few reds, quite a few reds.
So I will put us out of our misery and go straight to the charts.
Here it comes.
Oh my goodness.
It is a horrible, horrible result of 395.
In terms of the error, 395 uh, which you will remember, uh, is considerably worse than taking a
random guess.
I think it's certainly worse than just taking the average number of the training data set.
Not that it knew the training data set.
Uh, and, uh, yeah, it's, uh, it's generally a horrible result.
Um, Perhaps not massively surprising.
It's a tiny model.
It's been hugely quantized as well, and you can see visually what it's doing.
You can see that it has, um, had a few different levels that it's been most comfortable guessing at,
and it's guessed most often at one of these three levels.
And unfortunately one of them is far too high.
We've never told the model not to go above $1,000.
It's not like that's a requirement.
It can guess whatever it wants, as could GPT four.
We've, uh um, not not given it that that, uh, that intelligence to to know that, um, and so it's
gone too high far too much of the time.
Uh, and it's really ruined the results.
So a poor performance from our base model.
Not that surprising given the small number of parameters.
And of course, the challenge on our hands now is going to be, can we take this poor performing model
and use fine tuning, use a training data set to make it stronger?
And can we get close to what a trillion parameter model can achieve.
So this is an 8 billion parameter model and it's been quantized.
Can we get close to the the trillion plus model parameters in a major frontier model.
Because this is open source, it's free.
There's no API cost.
Wouldn't it be amazing if we could perform at that level, um, or at least beat a human?
Right now the human beings are winning over an untrained llama.
At least this human is one final thought to leave you with.
You remember that this is quantized down to four bits.
You might be asking yourself, how would it look if we quantized just to eight bits?
If we kept the eight bit version and ran it through, it would be interesting to see whether the how
much the performance was impacting by going all the way down to the double quantized four bit.
And indeed you can do that, of course.
And this framework gives us a lovely way to to in a very simple, tangible way.
See the difference.
So remember 395 is how far this is wrong.
So in this other tab I have just run it with the only quantized to two eight bits.
So if I go up to the top, you can see that I've got up here four bits set to false and otherwise it's
exactly the same.
Notebook.
Let's scroll all the way down and go straight to the results.
It looks like it's in here.
Hold on.
Build the tension.
And here we go.
So it's also pretty horrible performance.
But it is better 395 became 301.
And that's not surprising at all.
You know it's got a twice the amount of information.
Um so you know again quantizing did have an impact on accuracy, but perhaps we would have expected
the eight bit model to have done even better.
So so it wasn't there wasn't such a great difference between them.
Uh, but it does show you, of course, that the bigger model is able to do a better job.
All right.
With that, that's been pretty interesting.
Pretty revealing.
Let's go back to the slides to wrap up and summarize.