Well, before we do a postmortem on what happened, let's just quickly look at the standing the ranking
orders that we've got here.
So you'll remember that when we did a constant run based on the average, we had an error, an average
prediction difference of 146.
When we did traditional machine learning, it went to 139.
Random forest was 97.
The human, one human in particular, uh, who shall not be named, was 127, uh, GPT four or mini
when we ran it the first time, I'm afraid to say was $80.
GPT four was 76, and what we just came back with was 91.
As I say, there were things about it that actually were improved on the prior run, but for whatever
reason, uh, it is what it is.
I can't I can't fudge the results.
The unfortunately, the business metric that we're most focused on was slightly poorer fine tuning,
uh, has, uh, fine tuned in the wrong direction, it seems.
So let's talk about that.
It was obviously a sobering moment for us, an important learn on our journey.
Uh, so it does, uh, take a moment.
We need to take a moment to think about, um, what is the objective of fine tuning with a frontier
model?
Uh, fine tuning is often used, and we will be using it for taking an open source model that has fewer
parameters and trying to train it on a data set to make it rival with a frontier model.
But when you have a frontier model that has trillions of parameters already and has been trained over
enormous data sets, what is your objective?
Um, and so here, these, these five main objectives for why you fine tune a frontier model.
I actually basically took these from OpenAI's website itself.
These are OpenAI's reasons for why you would want to train to fine tune something like GPT for many.
Um, and it's, it's if you want to craft the like the style or tone of the responses, it gives an
example of adding some sarcasm to some responses.
If you want to improve reliably producing a particular type of format, a construct, you need the format
to be in a particular style or way or structure.
Um, the third one is correcting, uh, where the model is failing to follow a difficult or challenging
prompt.
There's something very complex it's being asked to do, and it's it doesn't get the joke, it's missing
it.
Um, handling edge cases when there are things that are occasional flaws that get exposed in the model
that you need to correct for and then performing something new.
And this is perhaps what we were trying to do, a new task, but one that's hard to articulate in a
prompt.
And that's really what OpenAI stresses on the site that it's about trying to solve for things that you
can't already fix with good prompting, and it really urges you to start by working as much as you can
on the prompt, because much of the time with something like GPT four or mini, you're going to be able
to get to a very high level of performance just through prompting.
Um, and really, for a frontier model that that's the key here.
Uh, the we can already specify the question at hand and the style of output very clearly in a prompt.
And in fact, if you remember back to the prior results, GPT four mini responded accurately in terms
of a proper structure.
In every single case, it never we weren't ever not able to pluck a number out.
And the numbers were always, you know, within within an error close ish to the product.
It was guessing, um, so it wasn't a problem with it understanding the challenge or the output format.
Um, and you have to remember that GPT four and GPT four mini have an absolutely staggering size of
training data with a great world knowledge, and it's unlikely that giving it 500 more training examples
is going to move the needle in terms of its world knowledge.
And there is then this, this slight point, um, that, uh, that I talked about a while back now about
what they call catastrophic forgetfulness, forgetting, which is where sometimes adding in more fine
tuning causes you to erode some of the deeper knowledge that was gained during pre-training.
So it's not always a good thing to be fine tuning.
Um, and I don't know if it was catastrophic forgetting that caused this, this slight dip down or whether
it's just a bad luck, just that there is some some noise in the system and and it just didn't happen
to do so well on the test set.
Um, but we certainly didn't appear to improve things.
That's the bottom line.
And it's because, in my view, and from from the way that I understand it and the way the experiments
show, we were already doing a great job of clearly prompting what was needed.
GPT four, in many, was already understanding that well, and the caliber of results was already very
good.
So having said that, the challenge for you, though, is to keep working on this.
I've done a bit of hyperparameter optimization or trial and error to try and improve things a bit.
Um, but not much.
And I would be shocked if it's not possible to get to a point where this fine tuning is at least doing
a little bit better, a little bit better than what we had before.
So that's the challenge for you.
Do some more, you know, I mean, whilst OpenAI doesn't recommend that one puts in massive data sets,
particularly while it's free to do so, I would certainly be interested in in, in trying bigger data
sets.
Try a training dataset of 1000 or 2000.
Maybe try some more epochs.
I did do that and it didn't make a difference for me, but try something different.
There are other hyperparameters you can explore.
You can look up on OpenAI's website.
There's a couple that you can try changing if you wish.
Just pass them into that same dictionary of hyperparameters.
Um, and uh, yeah, you could also try putting in different training data points, and you can try
playing with the prompt.
I mean, OpenAI's biggest point on the website is that you will get the most mileage from improving
the prompting.
And obviously this is something where we spent a bit of time curating the data and perfecting the prompts,
but there's much, much more that can be done there.
So have a shot at that as well.
The challenge for you is do some hyperparameter optimization, do some playing around with the prompting,
at least do better.
Let's look back at where we were.
Your challenge is to do better than 76.
Um, and I will tell you that that I have been able to do better than 76 at one point with a with a
prior run.
And so I have done I know that it's possible to do better than 76 without without making too many changes.
Not massively better, but better.
And that is the challenge for you.
Uh, do so please, and let me know how you get on.
And if you particularly if you get, uh, optimized prompt or hyper parameters, then then push the
code, do a PR so that I can look at it and share it with others and see where we get to.
Um, and that will be, uh, that will be your your challenge accomplished when you do better than 76.
All right.
That brings us to a conclusion for week six.
You are remarkably now 75%, three quarters of the way to being an LM engineer, proficient LM engineer
who has mastered AI and LM engineering.
And I hope you're as excited about that as I am.
It's just a fantastic progress that you should be super proud of everything that you've learned.
Uh, obviously generating text and code with frontier models assistance and using open source models
with hugging face transformers, library, uh, lang chain rag.
And then most recently, the five step strategy for problem solving curating data.
We did a lot of curating data.
But you know, the life of an LM engineer involves a lot of data curation.
That is a knack that you get into, and it's one of the most important parts.
Certainly in all of the experiments that I did, changing the data structure was the thing that moved
the needle more than anything else.
Uh, and you're already seeing it post a lot of, uh, experiment.
Um, but you I'm sure you can do better.
Um, you've played with traditional machine learning.
Uh, just just to get a good sense of a baseline that that we've beaten comfortably.
Uh, you made a frontier model solution, and now fine tuned frontier models.
So the results were a little disappointing.
Gotta be real.
But nonetheless, this is something that you can use in your own projects.
And there are situations such as if you want to change the style or you're having difficult edge cases
that are causing you problems, then fine tuning is the answer.
And now at least you have a good recipe.
You know how to do it and you've seen a run happening.
You've checked its status, you've watched it in weights and biases.
You know everything that's involved.
All right, next week we turn over a new leaf and we we start a new segment of the voyage as we turn
to open source models.
Fine tuning open source models is a very different proposition to fine tuning a frontier model, fine
tuning, an open source model.
What we're trying to do is start with something that is massively smaller than the large models that
we're dealing with.
I mean, it's still going to have billions of parameters.
It's still a big model in the in the general scheme of things, but it doesn't compare with the trillions
of parameters in GPT four and GPT four mini.
So we're going to be fine tuning open source models, and we're going to be using something called Lora,
which you may have heard of or you will have heard of because I've mentioned it a few times, but you
may have perhaps seen some examples of Lora, and you may have heard of its cousin, Lora, which is
a quantized version of Lora.
We will be working on both, and by the end of it you will know them both back to front.
Um, and in the next, next session, we're going to be selecting the base model, which you already
know what that is.
But but we'll be choosing it for reals.
Um, that is going to be the model that we will have to compete with GPT four.
Our current the current winner on our leaderboard.
That is going to be our challenge for next week.
It's going to be a big challenge, but I can't wait to take it on.
And I hope you can't wait as well.
I will see you then.