So you may remember eons ago when we were building our data set.
At the end of that, we uploaded our data to Huggingface.
Since that point, we since we also produced pickle files, we've been loading in the data from pickle
files from that point onwards.
But now that we're in Google Colab, it's easiest for us to collect that data back from the Huggingface
hub again, which is a very typical task in this kind of, uh, process of building your own model.
So here I go.
I load the dataset using, uh, hugging face load data set method, passing in the data set name, and
then I break it up into a train and a test and the data set name I set in the constants at the top.
Once I've done that, we can take a look at the first training data point.
And what it looks like is it has text and a price.
You may remember we set this explicitly ourselves.
The text is our prompt.
How much does this cost to the nearest dollar followed by the description of the product followed by
price is dollars and then the price itself, but rounded to the nearest whole number.
And in the top.
Here I say, how much does this cost?
To the nearest dollar.
And the reason I'm doing that is I want to make the task a bit easier for the llama 3.1 model with its
puny 8 billion parameters.
When we were sending it to a frontier model, we didn't need to specify that because it's easily powerful
enough to make its own decisions about cents.
But in this case, we want to give our model every, every, every simplicity we can.
Um, and since it's going to, since this will always map to one token in llama 3.1, we're making it
quite easy that all it's got to do is be able to predict that one token right there.
That's going to be what it's going to try and learn how to do well.
Um, and in this data set, we also have the real price in here too.
Uh, if I look at the test data and take the first point, the test data is going to look very similar
in structure with one tiny difference.
Do you know what that difference is?
I'm sure you do.
It is, of course, that there is no price provided at this point in the test data.
The text is going to be the text.
How much is this to the nearest dollar?
And then we pass in this text.
And the assignment to our model is to predict the next token.
What is the probability of the next token coming after this.
And we hope it will give a high probability to the token that matches the number that is, uh, 3.74,
uh, matching the actual price.
Uh, and so that is the assignment.
And because this maps to one token, it's really the challenge is just to get good at predicting the
next token, the single next token that represents that cost.
Uh, one other point to mention is that you may remember, we did some futzing around to make sure that
this text always fitted into exactly 179 tokens or less.
Um, and because of that, we're able now to tell, uh, the I've got a constant up here that says maximum
sequence length is 182.
I've added in a few tokens.
There it is, in fact 179.
But I'm adding in a few extra spare tokens, uh, because, uh, the the tokenizer is going to add in
a beginning of sentence token to the start of the sequence, and it may add in an end of sentence or
a pad token or two at the end.
And I want to have no risk at all that we accidentally chop off the price of the most important token,
uh, which is going to come at the end of this.
So given a little bit of extra leeway, in fact, this doesn't become important until we get to training.
But I wanted to point it out now since we're looking at the data.
So there we go.
We've just sorry, gone too far.
We've just looked at this data.
The next thing that we do is we pick the right quantization config.
I set a constant up above, uh, quant four bit.
In this case I set it to true.
Let's just go and check that we will see.
There we go.
Quant four bit is set to true.
And so now when I come back down again, we're going to pick the four bit quantization.
And I show you here what it would look like if we picked eight bit.
But we're going to pick the really minuscule four bit version.
And then we load in the tokenizer and the model.
I'm not going to run this cell because I already ran it.
You can see it's sitting in the memory here.
If I run it a second time I'll run out of memory.
And what we do here is we load in the tokenizer.
There's a bit of stuff here that's very boilerplate that you'll see a lot.
Um, we're telling the tokenizer that when if it ever needs to pad the end of a sequence, it should
just use the end of sentence token and just have that repeatedly.
And it should do that off to the right hand side.
This is standard stuff that will happen when we train.
We won't actually use it right now.
Um, so you don't need to worry about it.
But but you'll see this all over the place.
It's a very standard setup, as is this line here, which you also don't need to worry about right now.
What we're doing is creating a tokenizer and loading in the llama 3.1 base model.
And it's using up the 5.6GB of memory that you're expecting.
There it is, 55.9, it seems because because I've done some, uh, inference down below.
Uh, but yeah, it's, um, it's the, the very slimmed down four bit version of the model.
So now this function is one that should be familiar to you because we used it recently with frontier
models extract price, which is going to take some text, any text and pluck out from it the price that's
being predicted.
And so if I do something like extract price, price is dollar 999, I should have this as a string perhaps.
So that's not going to work very well is it.
Price is dollar 9999 blah blah.
Uh price is 999.
So cheap.
Whatever.
Uh, then hopefully what we'll see.
Yes.
Is that it's going to pluck out 99999.
But the model, we know it's going to be provided with this in the in the prompt.
It's what comes next has got to have nine, nine, nine in it.
Um, and then this here model predict.
So this is the function which we're going to be using in our test harness.
This is the function where we tell it.
We're going to give you a prompt.
And we want to know how much does this cost.
And so this is how we call our model in inference mode similar to what we did several weeks ago.
Uh, we take the prompt and we encode it using tokenizer dot encode.
And this thing here will push it off to the GPU.
Uh, this is just, uh, something that's, uh, not super important.
It stops it from printing a warning.
So, so this doesn't actually affect anything.
Uh, and then, um, yeah, to be precise, this this prevents it from trying to predict anything that's
happening in that input token area, which we don't want it to predict, that we want it to predict
what's coming afterwards.
That's what it would do anyway.
But it would give a warning if we didn't explicitly tell it this.
So then we say for our outputs, we're going to call our base model lemma 3.1.
And we're going to call the generate method on it.
We pass in the inputs.
We're going to say the maximum new tokens we is for.
We could make that a much smaller number.
We only really need one token.
I'm giving it to to generate up to four tokens, just in case it prints another dollar sign or something
like that.
Um, uh, that we pass in the attention mask that I've just set that stops it giving a warning.
And this is just saying we only want back one answer.
We don't want it to come back with multiple answers.
And then for the reply we take that one answer, it sends us back.
And we call tokenizer dot decode to turn that back into a string.
And then we extract that string.
All right.
So that's exciting.
Let's just remind ourselves.
So if we take the zeroth, the the first test item, here it is.
It's the OEM AC compressor.
The actual price is $374.
Who knew.
So let's have our first shot at this.
So we're going to say Model.predict.
Test zero.
And to get the prompt out of that I just call text on that.
So are you ready.
Here we go.
Llama 3.1 base model is going to try and predict the price of an OEM AC compressor with something repair
kit okay.
And it's predicted $1,800 which is rather far off.
So that is potentially a bad omen for how the llama 3.1 base model will work.
But we might have just gotten unlucky with the first example, but that will be revealed in the next
video.