And welcome back to us continuing our journey through the model class in Hugging Face Transformers library.
We were just looking at the architecture of the llama model that you get when you simply write it,
and saying that you should look at it for the others.
One other thing to point out is to always look at the dimensionality.
As I briefly mentioned, look at the number of input dimensions representing the vocab, and see that
that matches the output down here.
And you can follow the dimensions through the architecture and get a sense of what's going on.
All right.
But now that we have done all of this and we've talked through what's going on and we've built our inputs,
it is time for business.
This is the method model dot generate.
It takes our inputs which are sitting on our GPU, ready for this.
Um, and we can say we want up to 80 new tokens.
Um, a reminder, in case you forgot that what we asked for was a joke.
A joke for a room of data scientists.
Our favorite little experiment.
Uh, and then we take the the outputs.
We take the the first in the list of outputs, there will only be one.
Um, and we then call tokenizer dot decode to turn it from tokens back into letters to text again.
And we print the result.
Let's do it.
So it starts to run.
I like to to watch what's going on by looking down here and seeing it do a forward pass.
And there comes our answer to a lighthearted joke.
Why did the regression model break up with the neural network?
Because it was a bad fit and the neural network was overfitting its emotions.
Ah, you know, it's okay.
It's not terrible.
It's, uh.
Yeah, it's it's a perfectly plausible joke.
It's not the funniest that I've heard, but it's, uh.
It's not bad.
Why did the logistic regression model go to therapy?
Because it was struggling to classify its emotions.
I think that's really good, actually.
I think that's great.
It's simpler and it's, uh, spot on for a data science audience.
I think that's a better gag than, uh, than some of the ones at the frontier models came up with.
Uh, so good job.
Llama 3.1.
Uh, the thing to bear in mind again, of course, is that we're dealing here with the 8 billion parameter
version of llama 3.1, the smallest version of it, and we've quantized it down to four bits, and then
we double quantized it.
Uh, so it's a this super slim version of the model, and it just told a perfectly respectable joke.
Uh, so I hope you enjoyed that.
The next thing we do is we do some cleanup to free up some memory.
Otherwise, if we keep running, different models will very quickly run out of GPU.
You may find this happens to you, in which case you can always restart your session by going runtime
restart session and then continue from where you left off after doing the, uh, the imports.
So the next thing I'm going to do is take everything we've just done and package it up into a nice little
function that does all of it.
The function will take the name of a model and messages a usual list of dictionaries.
And let's just go through this line by line as a way of revising what we just did.
We start by using the auto tokenizer class to create a new tokenizer based on the model that we're working
with.
This line is the thing that sets the padding token to be the same as the end of sentence token.
This sort of standard boilerplate thing to do.
And then this.
We know and know it well.
This is where we apply the chat template that's suitable for this tokenizer to the messages the list.
And it will return back a series of tokens.
We then put that onto the GPU and that we assign to inputs.
This is new.
So just as another little little skill to add I'm going to say let's stream back the results.
And the Huggingface library supports that as well.
You do you create this thing called a text streamer.
You need to give it the tokenizer because as it streams back tokens, it's going to need to convert
them back into, into text.
Uh, so it needs to know what tokenizer you're using.
So you provide that and then action.
Uh, we first of all get the model.
This is auto model for causal lm.
This is the equivalent to the auto tokenizer.
But to load the model we say from pre-trained, we tell it the name of the model.
We say device map is auto, meaning user GPU.
If you've got one and we pass in our quant config that we set way up there somewhere, uh, to be four
bit double quantized NF for uh, type of four bit numbers.
And the bfloat16 is the calculation, uh data type.
And it's now time for business model dot generate.
That's the big method.
And we pass in the inputs we'll generate up to 80 new tokens and we'll give it our streamer.
That's this is the piece that means that it will then stream the output and then we'll do our cleanup.
So that is the function which kind of wraps everything that we did before, but also adds in streaming.
And with that, let's quite simply call Phi three with our messages, uh, using the function we just
wrote.
So Phi three will now load in again.
This will take a little bit longer for you because you will, uh, be loading it for the first time.
Uh, I have already loaded it, so it's cached on disk.
Uh, so it doesn't need to redownload the whole thing from hugging faces.
Hub.
Um, there's still a little bit to do to load it in while it's doing that.
We could, uh.
Oh, I was going to say we could look at the resources, but I think it's going to be so quick that
I want you to see it streaming back.
And I think, uh, I should prepare you for the fact that you may be disappointed.
Um, so I found that from using at least the prompt that I've got there, I was not able to get Phi
three to tell a joke.
Rather, it gives some sort of general stuff that a data scientist might be talking about and sort of
rambles away.
Now, I don't know whether I can improve the prompt to be something that's a bit more assertive for
53, or whether it's simply not something that 53 is willing to do.
53 will do a lot of things very admirably indeed, but not this particular task.
So I also leave that as an exercise for you as as well as trying some other models.
Also see whether you can improve the prompting to get 53 to tell a joke or if it's not a jokester,
you can find some of the things it's good at.
It will answer some of the other questions that we've asked llms about things like use of Llms very
well indeed.
Um, so that is the the 53 outcome.
Now let's see how Gemma does.
So the same approach, we can use our utility function for Info Google's Gemma two model, and it's
worth noting that Gemma doesn't support a system prompt, so you have to just pass in the user prompt
like this, which is fine because the system prompt didn't say anything special anyway.
And let's give Gemma a whirl.
It is, of course, a 2 billion model.
It's a very small model.
In addition to being a very small model, we are also quantizing it down to four bits and then quantizing
it again.
So we are really, uh, dealing with a very slim model at this point, which shouldn't use up much memory
and also should load nice and quickly and tell a joke quickly.
And there is its joke.
Why did the data scientists break up with the statistician?
Because they had too many disagreements about the p value.
It's, uh, another nerdy joke about p value.
I don't get it.
But maybe there's something obvious that I'm missing.
Uh, welcome.
Anyone to tell me if it is.
Uh.
But still, I like the way it's nice and friendly.
It's got another.
Let me know if you'd like to hear another joke.
Uh, maybe when you run this, you're going to get a better joke, I don't know.
Uh, but, uh, it's certainly, uh, an enjoyable, uh, tone.
And I think that that, uh, Gemma two has done a laudable job of, uh, certainly it's data science
relevant.
Um, and, uh, particularly when you remember that this is a tiny model that we are further quantizing.
I think it's a fine showing from Gemma two, but certainly I fully expect when you use quantum, which
I have used uh, that you'll see, uh, superior results and, uh, you maybe you'll get something better
from Pi three as well.
And then whatever, whether you pick the mixed trial model or something a bit slimmer that you can also
use, I imagine you'll be able to get some good results.
You could also try asking maths questions, something which they can struggle with.
If you're dealing with difficult maths.
But I tried asking a fairly difficult question to llama 3.1 earlier, and it had no difficulties at
all to see if you can have the same experience.
Regardless, now is a moment for you to explore using these models, trying out different things.
You're working with open source models.
There's no API cost going on.
The only cost you you pay is if you're using up.
If you're not using Free Colab, but you're using up some of your, uh, units from the, uh, the Google
Colab, uh, costs.
Um, the I'm using 1.76 units per hour.
So there's really plenty of time to be, uh, to be working with this and enjoying inference on open
source models using the hugging face Transformers library.