Let's now see our results side by side.
We started our journey with a constant model that was at $1.46 error from the average of all of the
data.
In the training data set, we looked at a traditional machine learning model with just a few features.
It got to 1.39.
It beat the average random forest.
The best of our traditional machine learning models got to 97.
The human got to 127, GPT four got to 76, and our fine tuned customized model has got us to $47.
And this, of course, brings me to the challenge for you.
The challenge for you is to improve on this $47.
Uh, this is something where hyperparameter optimization can go a long way.
And so the task at hand is now to play with the hyperparameters experiment, use weights and biases.
Uh, explore maybe different optimizers, different learning rates, different batch sizes, and see
what you can do to improve on this number.
You can also explore different ways of running the model at inference time to see if that improves it.
And then there is one factor that is actually perhaps the way that you can make the the biggest impact
on the results with the smallest change is to relook one more time at the data set, at the data curation
step, and challenge yourself to see whether you can think of different ways to be prompting or organizing
that information in order to be getting better outcomes.
And then there is one final thing that you could do, which is a bigger step, but very exciting.
And if you don't do it, I'm definitely going to do it, which is to try the other models on this.
Try Jama, try Kwon the powerhouse, try fi three, see how they perform.
There are a couple of places where it might be a bit fiddly, because they might not predict one token
as the right number, just maybe that's only at that inference function.
Otherwise everything else might just be fine.
But you'll need to experiment with that and convince yourself that it's that it's so good.
So try some different models.
You could also try doing the whole thing with a version of llama three that's quantized to eight bits
instead of four bits, depending on your appetite for that.
There are also some larger models.
There is a version of 53, I think that is 14 billion parameters that you could experiment with and
see whether that improves things.
So that is the objective.
I would love to hear from the first person that can get this below 40.
That has to be possible.
I think there is like a hard limit on how low you can get it, given the reality of uncertainty in pricing,
but I think you guys, someone is going to be able to get it below $40.
You'll build a model that can get within $40 across the 251st items in the test set, and I can't wait
to hear about that.
So please do reach out and tell me when you get below 40, and tell me your hyperparameters and your
model so that I can try and recreate it myself.
I would love that.
And with that, let's wrap up for the week.
It's the end now of week seven where you can now do, of course, obviously generating text and code
with frontier APIs and with open source models and hugging face, you can solve problems including dataset
curation, a baseline model, and fine tuning.
And at this point, you can confidently carry out the full process for selecting and training an open
source model that can outperform the frontier.
And that's a big accomplishment.
So next week is the finale of this course, and I promise you, I've kept the best to last.
It's going to be a triumph.
Next week is going to be so much fun.
You've you've got this far.
Hang on in there to the very end to see everything come together.
There's some stuff that's really important we're going to do about now packaging up what we've done
and being able to deploy it as behind an API so we can use it for production purposes, and then really
package everything into an application that can can make a real impact.
And at that point, you'll be in a position to be creating your own end to end solutions to commercial
problems.
Using groundbreaking llms that you'll be able to train yourself.
So there's a lot ahead, and I can't wait for next week.
And as always, I will see you there.