Welcome back.
So about ten minutes later, maybe 15 minutes later, the run has completed.
And how do I know this?
A few ways.
One of them is that I just got an email from OpenAI, as you can see right here in my email, uh, to
tell me that my fine tuning job.
Blah blah blah has successfully completed and a new model, blah blah blah has been created.
And you'll notice in the name of that model there is that word processor, because we specifically asked
for the suffix processor to be included.
Uh, just shows you how how it works.
It's the name of a feat for fine tuning, and then the name of the GPT four mini variant that we've
been working with colon personal colon processor, and then a code at the end.
So that's my email.
Here is the JupyterLab.
Uh, this was the thing we were running.
Um, and now we're looking at the final ten messages in the status, and you will see that it completed
step.
Um, the last five steps here then.
Fine tune model created.
Evaluating model against our usage policies before enabling.
That was the thing I mentioned to you that it does.
Um, and then usage policies complete and the job has been successfully completed.
And that is that I will now show you in weights and biases how this looks.
This is the final weights and biases chart.
Um, you can see that the things that really matter are training loss and validation loss uh, that
you can see that the validation loss of course, is not or didn't happen nearly as regularly as the
training loss was calculated.
Um, and as I say, because again, because we only did one epoch that all of the training data was
new.
So the training loss, uh, is, is just as useful for us, um, or the validation loss because it's
always calculating it on the same set.
It's particularly useful for, for trying to spot trends.
Um, and at first blush, looking at the validation loss, there doesn't appear to be much of a trend
there, which again, is concerning.
We can bring this up.
We can edit this panel and zoom in.
We can change the y axis here.
And the minimum should be zero.
And let's make the maximum like three.
So we zoom all the way in.
And you can see that it doesn't particularly look like it's improving.
In fact um, you could if you wished, uh, be uh, almost argue that maybe it's increasing slightly.
Um, but uh, I'm not sure if that we can necessarily say that there is a smoothing function that's
available in this chart that we can go with, and that is the smoothed version of it.
Uh, and, uh, yes, I suppose it's, it's certainly not going up, but it appears that it made some
improvements and then it just kind of stayed flat.
But it does look like there was some improvement up until the 300 point.
Um, so these are all things for you to look at and spend more time on yourself.
But at this point, it's time for us to now go and evaluate this model against our test data.
So I will kick that off and then flip to a video when it completes.
So let's go back to the Jupyter Lab.
So this is our fine tuned model right here.
Um we can get the job ID and we can collect the fine tuned model.
Let me just quickly show you what that's going to be.
If I, um, show you this, you can see when we look in here now that right here there's a new attribute
fine tuned model, and it contains that same name of the fine tuned model that was in the email as well.
So you could equally copy and paste it from the email, but we might as well just pluck it out with
some code.
So here we do.
Uh, so just to show you that that's done when I'm suggesting, obviously if I run this, it's got that
same name.
All right.
So we're going to redo this messages for function again.
Uh, this time, uh, I'm, uh, just using the one that doesn't reveal the answer.
Obviously, we don't want to give it that information.
Uh, let's just convince ourselves that that is actually going to work.
There you go.
So it gives the question.
It does not reveal the price.
And the challenge for our model is going to be to finish this off.
You will remember from last time a utility function that we that we have that will pluck out the price
from what comes back.
Uh, and I remember last time I did this, the price is roughly 99.99 because blah blah, blah.
And if we run that of course get price function just strips out the price from there.
Uh, as I think you're familiar.
So then this is the function, the function that we will be about to test against.
GPT fine tuned response is OpenAI ChatGPT completions create you call it, just as you would call it,
for the normal GPT four.
Oh, same API.
Exactly.
There's only one difference, one tiny difference, minute difference.
That is this we don't pass in GPT four mini.
We pass in the name of our fine tuned model, this name right here, that is what we will send in to
OpenAI.
And it will automatically it will know that that means that we want to use our fine tuned version.
We take back the response, we get the price.
So let's just print one example.
Let's print a test, uh, something from the first thing on our test set, which was that that thing
that cost 200 and something that was, uh, caught me off guard.
And then we will call our GPT fine tuned for the first time.
Let's see what one result looks like.
Okay, so that is the sorry, the the number we were looking at earlier was a training price.
This is the price of the first test item.
Uh, let's see what the first test item actually is.
So.
Let's have a look at it.
It is an AC compressor repair kit for Ford, uh, body parts.
And, uh, so this is one that I had to do myself and, uh, yeah, obviously it's not done a very good
job of that first data point, but who cares about one data point?
What matters is doing it across the lot, at least the 250 that we've been using consistently for all
of our testing.
So without further ado, let's run it off it goes.
So the first couple of results look a bit red, and then it looks a bit better and gets green, but
then a whole bunch of red.
So some mixed results here.
And at this point I'm not going to have you hanging for the 250 of them.
I'm going to pause.
And then the next video we will reveal the outcome.
I will see you there.