Welcome to Colab.
Welcome to the week seven day two Colab.
And just before we try our base model, we're going to actually see these tokenizers in action.
Uh, start with some Pip installs and some imports as usual.
And now in the constants section, I'm going to set up four different tokenizers that we will just quickly
look at.
After that, we're going to look at our base model llama and do some some work with the data.
And I've also got the constants here for writing to the output in color that you may remember from when
we visualized results in the past.
All right, let's run that.
We log in to hugging face using the usual snippet that you know well at this point.
And now I've written a useful function called investigate tokenizer that takes a model name.
And what that does is it uh, will load in the tokenizer for that model name, and it will then iterate
through these six numbers zero, one, ten, 100, 999, and 1000.
For each of those numbers, it will convert it into a string, and then ask the tokenizer to convert
that string into tokens.
So what are the tokens that represent the string 100 as a piece of text?
Um, and I'm using this add special tokens as false parameter.
What that means is please don't add in things like a start of sentence token and an end of sentence
token or something like that that will interfere with things.
Just simply convert this text into tokens that represents that text.
Um, so that's what it will do.
And then it will print that out.
So to run that and to show it let me call investigate tokenizer.
And we will start by calling it for the llama three one uh model.
And let's see what we get.
So what we get is the string zero.
Text zero converts to the token number 15.
One goes to the token 16.
Ten goes to token number 605.
As it happens.
And you'll see that similarly 109.99 they each mapped to a single token.
1000 though that maps now to two tokens.
In fact, it maps to the token for the number 100.
If you see that followed by the token for the number zero and that.
So just just as you would imagine, it is the tokens for the text 100, followed by the text for a zero.
And what you see here, loud and clear, is that if we're just focusing on three digit text numbers,
on three digit numbers, we have this useful property that whatever price we have is always going to
map to one token.
So the model's task of trying to predict the price of a product will end up being just predict one token
and get that token right, and then you've got the right price of the product.
Um, and so it's not an essential property.
We don't require that these models are perfectly able to generate a sequence of tokens, but it's convenient
that we're going to simplify the problem down to just getting this one token right.
And we can also see how this looks for another model like K-125.
Uh, and now we see something different.
The single digits zero and one mapped to a single token, but ten maps to two tokens.
In fact, the token for one followed by zero 100 is 100999 is presumably 999, and 1000 is 1000, uh,
for tokens being used there.
So you see that different property.
And hopefully that really clarifies, um, why I think llama three one has an edge.
Uh.
Gemma.
Two for Gemma, two.
Sorry.
Uh, Gemma.
Two similar properties to Kwon.
Uh, interestingly, uh, totally different vocabulary.
It's a much bigger number, but that's no surprise.
There's no reason why they should have the same vocabulary.
Um, and, um.
Yeah, it's, uh, clearly, uh, not not a one token for for one three digit number.
Um, and Phi three is, as I say, similar.
Uh, there is actually a different variant of phi three, a smaller variant that has the same nice properties
as Lama three one.
So that is another thing that would be worth potentially trying.
Um, as could any of these be tried?
It's not a disqualifier that that it produces multiple tokens by any means.
All right.
Well, anyway, that gives you a background to the tokenizers and a good sense for why we picked the
model that we did.
Uh, in the next video, we will then load in data and try testing our model.