Here we are back in Colab, looking at the week seven, day five of the Colab notebooks and I'm on a
T4 box, so it's a low end cheap box.
That's all that's required since we are doing inference today, not training.
Um, or the training is still happening in this other tab, as you can see.
Uh, as, as as we speak, uh, we start with a few installs and then some imports with the usual kind
of stuff.
And let me just tell you about the constants that we've got here.
Um, the base model, of course, llama 3.1 project name is Preiser, the hugging face user.
So you choose here, you can put in.
Well, I hope that you'll be putting in your name and you'll be running this inference, this test against
the model that you have fine tuned and uploaded to the Hugging Face Hub.
It is possible, though, that you have, uh, either, uh, lost patience and or you just want to see
how my one did, in which case you can keep my name in there because this model will be public.
So you will be able to run against this model too.
Um, and I've selected in here the run name.
This is the run name.
You may recognize that that 39 there is the run name in hugging face of the one that I ran for multiple
epochs in the past.
Um, and this revision, this is where I am specifying which of the different checkpoints I am selecting.
I'm selecting the one before it started to badly overfit.
Um, and this was the one where it was still getting good, good results.
Um, and then this becomes the name of my fine tuned model.
It is, of course, the hugging face name.
I'm sorry.
I should make this, uh.
Otherwise, I'm hard coding my name in here.
But what I'll do is I will make two versions of this.
One will be for the hugging face user that you have entered in.
And the other one I will comment out and I will say uncomment this line if you should use my model.
Uh, and either of course, if you're using your model, you'll need to change the run name and the
revision to match whatever you're using.
And you can start by not putting in put revision equals none.
Um, or revision equals none if you're not using a revision at all okay.
And then for the data set uh, either again we load in the data set that you have carefully, lovingly
curated and uploaded to the Huggingface hub.
Uh, or you can just use my one should you prefer.
Um, and now and by the way, if you have gone with the lower cost version of this and you've trained
your model for appliances only for home appliances, then of course you should be filling in the light
data set that you'll have built and your model for for home appliances, and you will get similar kinds
of results.
Quant for bit is true.
We are quantizing and then you may remember these are the nice ways that we can print colorful lines
to the output.
Okay.
Then we log in to hugging face.
You're used to this now.
We don't need to log into weights and biases because we're not training.
And then we load in the data set.
And as well, you know, at this point if I look at the first training data set, we won't be using
it anymore.
But it has the price baked into it.
It looks like this.
We will of course now be using the test data set which looks like this.
The text does not have the price.
The price is only in the answer which is not given to the model.
It's only given this text as well.
You can double triple check in a moment when we get to the part that we'll be actually doing this prediction,
It would be a bit of a gaffe, wouldn't it, if we were accidentally passing in the price itself?
But we're not.
Okay, so then first of all, it's time to it's time to load in our tokenizer and our fine tuned model.
So we first pick the right kind of quantization.
You're familiar with this same as before.
This is also the same as before.
Well with a slight difference just here.
But we load in the tokenizer.
We put in that boilerplate stuff to set up some of its parameters.
We load in the base model as before using the right quant config.
And we have got that one liner again.
And now this is new.
So we are now loading something called a left model.
If you remember stands for parameter efficient fine tuning.
It's the name of the package which which which has coded Laura.
So a left model represents it's a hugging face model that represents a model that has a base.
And then it's got some adapter applied on top of the base.
And so that is what we load.
Now you call that with Frompretrained.
And you can pass in the base model, the fine tuned model name which we set up above.
And then a revision if you wish.
So if revision is not null none then I pass it in.
Otherwise we just don't don't don't bother passing it in.
Um and so that will load in our fine tuned model.
And at the end of that we'll print the memory footprint.
You may remember the memory footprint was what, 5.6GB before and now it is 5.7.
It's 5700MB because there's that extra 100MB or 109MB of our Laura adapters.
Um, our Laura A's and Laura B's in there.
Uh, one more time, we can just print this fine tuned model.
You may remember we did this right back in week.
Uh, in day two, when I mentioned we were, we were taking a look into the future because I was using
this.
This model itself was the one that we looked at.
And this is how it appears.
If you remember this, you can see all the different layers of the neural network, and you can see
that when you get to these, the, the tension layers, that there's a dropout layer in there.
Now, you know all about dropout with 10% probability of dropouts.
And then there's Laura A and Laura B that are in there as well.
Um, and uh, yeah, you can see that Laura A and Laura B are for all of the layers that have been adapted,
our target modules.
And you also worth just noting down at the very end here, the LM head, since I just talked about that,
this is the final, uh, the, the, the final fully connected layer that outputs the logits, the number
for each of the possible, uh, vocab token vocab entries, um, which will then go into a softmax in
order to predict the probability of the next token.
All right.
Are you ready?
So, uh, we're going to go in and run inference.
Uh, the, uh, I want to give you one more time, a quick, uh, memory that GPT four zero got to $76
llama 3.1 base model.
This this untrained model was $396.
Very disappointing.
Uh, this human being here got 127, uh, as my error, uh, and very much hoping to see that llama
can beat a human.
Um, as an open source model, it is important to keep in mind I don't, uh, in case you're expecting
something, uh, crazy here, that prices of things have a lot of volatility, and the model doesn't
know anything about that.
It's not going to know if the price of a product has been slashed, uh, because it's on sale by by
a huge amount.
So there is a natural big variation in these product prices, as I discovered when I was trying to do
it for myself and got wildly out.
This is this is it's a very difficult challenge.
You might think that it sounds like a it's not that hard.
It is very hard.
Try it for yourself and you'll see.
Um, okay.
With that caveat in mind, let's keep going.
So extract price is the function that you, you know, well it takes a string, it looks for price is
dollars.
And then it finds the number that comes at any point after that one more time, let's just satisfy ourselves,
extract price and put in a string.
Price is dollars a fabulous, uh, eight, nine nine, 99 or so, whatever I want to say and out comes
the price.
Uh, I'm sure you get it.
Uh, so that's extract price.
Uh, and then this is the model predict function, the function that we used before, um, that takes
the the inputs.
Um, that takes the attention mask.
Is that thing I told you about that you use to avoid it?
Throwing a warning and to make it very clear that we don't need it to predict the the most of the input
prompt.
Um, and then the outputs we call generate on the, on the fine tuned model, we pass in the inputs.
We pass in this attention mask, we only need up to three new tokens because we're going to get the
the next token is really going to be the one that we care about.
But we'll, we'll put in some more just to make sure if, if it makes some horrible mistake that we
capture that, um, and then we take, we say only one response, please, we take that one response
and we extract the price.
Now, as it happens, we can do a little bit better than this prediction function.
This doesn't make a whole massive amount of difference.
But but since, since, since we've got so much control over this model, we can actually do something
a bit smarter with how we handle this next token.
And so I've written this function.
That's an improved model predict function.
Improved model predict uh which is um, yeah.
It's just um, it's a bit more involved.
So, um, I guess, uh, I'll, uh, I'll just explain it in simple terms, but it's not super important.
What it does is instead of just taking the most likely next token, it takes the most likely three next
tokens, the three with the highest probability.
Uh, and then it says, okay, what probability did you give for these three.
And they represent real numbers.
Like maybe you said it was very likely to be worth $100 and then a little bit less likely to be 99,
but a lot more likely to be 101.
But 100 was the most.
And then it just takes a weighted average between those three numbers.
And that's a way for us to get a little bit more precise about what's it trying to predict.
Um, and it allows it to predict something that's not necessarily always a whole number.
Um, so it's a it's a technique I've used.
It's sort of solving for the fact that we're treating what is really a regression problem as a classification
problem.
It's not super important that you know about this, but but and it doesn't make much difference if you
use the function above.
It just makes a bit of difference.
But it is maybe worth looking through this if you're interested in these last layers of the neural network,
because you can see that what I do is I take the outputs of the fine tuned model passing in the inputs,
and these are considered the the logits that I mentioned this vector across all of the possible vocabulary
entries for a tokenizer.
And then I call softmax in order to convert that into probabilities.
And then I go through the top three.
And I just this is some some gumph that just takes the weighted average between those top three.
It's weighted prices and sums up the weighted prices.
And that's what it returns.
So it's very similar to model predict.
It just gives a slightly more accurate answer that's based on the top three predictions, not just the
top prediction.
Uh, and so then we have our class tester.
This is just exactly the same tester class that we've used before.
Um, and it is worth just pointing out that the thing I mentioned before this, this is obviously this
is the meat of the whole thing.
When we take whatever functions passed in and we call it, and what we pass in is only the text associated
with the data point.
We obviously don't tell it the price.
All it knows is the text, uh, so that it doesn't have any knowledge of the price.
Of course, of course.
Um, and then, uh, we then just call test a test where I'm going to use the improved function, and
we pass in the test and like some kind of a soap opera, I'm now, of course, going to say we will
get the results of this in the next video.
I will see you there.