Welcome back to Google Colab.
Here we are ready to explore the wonderful world of Tokenizers.
So, uh, the first thing I'm going to do is do some imports.
And after I've done that, I want to mention this statement here.
I forgot to mention this in the last video, but I have added it into that colab, so hopefully you
found it anyway and read my explanation.
Uh, you may need to log in to Huggingface in Colab if you've never done that before.
And this is the code that you use to do that.
First of all, if you haven't already created your account with hugging face, you need an account with
hugging face.
It's free.
It's it's terrific and you will never regret it.
So sign up at huggingface and then navigate to settings and create a new API token.
Giving yourself write permission.
We won't need to use the right permission today, but we will in the future, so might as well set it
up right now.
Then when you come back, you go to this key section here in the Colab and you add in a new secret.
The secret should say HF underscore token and the value should be your token.
And then all you have to do is run this code that will get the HF token from your secrets, and it will
then call this login method, which I imported here.
And that login method logs in to hugging face.
Let's run that right away.
And it's done.
And you see it says I have rights permission right there.
Okay, let's talk Tokenizers we are going to start with the fantastic llama 3.1, the iconic model from
meta, which paved the way for open source models.
Now, when you're using llama 3.1, meta does need you first to sign their terms of service.
And the way you do that is you visit their model page on Hugging Face, which is linked here.
And at the top of that page, there are very simple instructions for what you need to do to sign.
Uh, you should you'll need to supply your email address, and it's best if the email address that you
supply matches your hugging face account.
That means they get things done quickly.
In fact, they should approve you in a matter of minutes.
I've done this many times, including once late on a Saturday night.
I got approved very, very quickly.
I don't know whether that's just because they're really on the ball or whether it's all automated,
but it's very quick indeed.
And in case you think there's something evil with this signing terms of service, it's really if you
read the fine print, it's about making sure that you're not going to use lemma 3.1 for anything nefarious
and that you have good intentions, which is very much the case in this class.
So it should be no problems whatsoever signing that.
Once you've done so, you will have access to all the variants of llama 3.1.
It's one one sign and then it applies to the whole family.
If you wanted to use one of the older llama three models, like llama 3 or 2, you would need to go
and sign the terms for that family of models.
If for some reason you don't want to, or you're finding that they're not approving you right away,
you can just skip to later when we start.
Or you can just watch me executing for 3.1, and then you can pick up when we start working with some
of the other tokenizers.
But with that, creating a tokenizer is this single line here.
Hugging face has this class auto tokenizer, which will create whatever subclass of tokenizer is needed
for this particular model.
Don't need to worry too much about that.
Just know that auto tokenizer is the one to do and you call the class method from pre-trained, which
means I've got a pre-trained model and I want you to create the tokenizer that's for that.
And that is the name.
This is the model that we're using.
That's which you can take directly from the Hugging face hub.
It's meta llama's meta llama 3.18 billion.
This trust remote code equals true when as as you bring in this tokenizer, it's possible for there
to be code that is part of of a model.
And we're saying we know who meta is.
We know that this is fine so you can trust it.
If you don't include that, it will still work fine.
It just gives you a warning, an ugly warning.
So if you don't want the ugly warning, then just, uh, put that in there.
Okay.
With that, the next thing I'm doing is I'm using the text.
I'm excited to show Tokenizers in action to my LLM engineers, and we take that text as a string and
we call tokenizer dot encode that text.
And then we will print the tokens that result.
Here they are.
It's something that's super simple.
It's just a list of numbers.
Nothing more than that.
Nothing magical about tokens.
They are just numbers.
And these numbers represent that text.
Let's see how many of them there are.
Well, let's start by saying how many, um, uh, letters were in that text that we gave it.
There are 61 letters in that text.
So now we can count the number of tokens.
And do you remember the rule of thumb for roughly speaking, the how many characters map to a token.
On average, it's four on average.
Roughly.
Rule of thumb about four letters should be one token for normal English or if you have a lot of English.
So we're expecting for 61 letters.
We're expecting around 15 tokens.
Let's see what we get.
15 tokens.
There we go.
Exactly 15 tokens for this text.
Um, and we can in fact do this decode to turn our tokens back into text again.
So we're expecting to recreate the original text.
And what we get is something similar, slightly different.
As you will see what we get back is the text that we were expecting.
But at the front of it is something new, this this funny thing here, this set of text that says in
angled brackets are less than and greater than sign begin of text.
What is this?
So this is something called a special token or all of the, all of what I've highlighted just maps to
one token.
In fact, this token here, this 128,000 token, um, and it is a special token which is indicating
to our model that it is the start of a, uh, of a text of a prompt.
Um, and so it's used for that purpose to be a special indicator to the LM.
Now, again, you might be thinking, uh, okay.
So does that mean that somehow the architecture of the transformer has to be set, set up so that it
expects that kind of token?
Uh, and uh, as you're probably, uh, very comfortable now, the answer is no.
That's not what it means.
Uh, what this means is that in all of the training examples that it saw during training time, it was
set up this way.
The training examples began with this special token begin of text.
So it's got used to through training expecting that.
And in order to ensure the highest quality output, one should recreate that same approach.
Uh, when feeding in new prompts at inference time.
So hope that made sense.
There's another method batch decode.
And if you run that with your tokens what you get back instead of one string, you get back these,
uh, little, um, sets of strings where each string represents one token.
So as I say, this first token here turned into this here.
And then you can follow through to, to to see how that's working.
Um, and there's a few things to note from this.
Uh, as you'll see straight away, one of them is that in most cases a word mapped to a token, because
we've got very simple words here.
So excited, even though it's way more than four characters mapped to one token, because it's such
a common word, it's in the vocab.
Um, another thing to notice is that, uh, as with GPT tokenizer, uh, the fact that something is
the beginning of a word, this space before the word is part of the token.
So and so am as the beginning of the word, and then the letters Am is a different token to just am,
the fragment of characters that could be within something more complicated.
You'll also notice that something like Tokenizers got broken into two tokens, one for the word token
and the other for ISAs.
So that's an interesting, uh, you know, a word ending Isa ISAs.
You could imagine that might be stuck on the end of lots of different things, and that's part of its
tokenization.
One other thing to notice is that it is case sensitive.
So so you can see that, uh, token with a capital T has been been taken there.
Uh, so, uh, the final thing I want to mention here is the tokenizer dot vocab.
If you run tokenizer dot vocab, you get the, uh, it gives you the.
It's the dictionary of the complete mapping between fragments of Words and numbers.
And you can see there's some pretty obscure things here.
There's an awful lot of tokens that are available, and there's some quite odd tokens in here that are
from different languages or used for different purposes.
So very much it does go beyond three letters, four letters, and you'll see a number of different things.
A um, it's printed out quite a lot of them.
Uh, something else that I'll show you from this, uh, as I scroll back through all of our dictionary.
Get back here.
Uh, is, uh, that you can also print, uh, comment that, comment this out.
Uh, just what's called the added vocab, which are the special tokens that I mentioned.
Um, there's a bunch of these reserved special tokens, and sorry, at the top you can see here are
the special tokens that have been reserved in the vocab, uh, to be used to signal to things to the
LM.
Beginning of text.
End of text.
Some reserved, um.
And then a start header, ID and header.
And then some other things here.
And a Python tag.
Uh, something obviously special there.
So for whatever reason, these are the special tokens that have been identified, uh, as, as it being,
uh, useful to include those special tokens in the vocab and provide them during training so that when
you're doing inference, when you're running the model, uh, to, to generate text, you can use these
tokens to indicate things to the model.
All right.
Well, that's a bit of playing around with the llama three model.
Uh, llama 3.1 tokenizer.
When we come back, we're going to look at the, uh, the way that that this applies to chats in particular.
And then we're going to play with some other tokenizers.
So see you then.