Now it's time to talk for a minute about tokens.
Tokens are the individual units which get passed into a model.
In the early days of building neural networks, one of the things that you'd see quite often is neural
networks that were trained character by character.
So you would have a model which would take a series of individual characters, and it would be trained
such that it would predict the most likely next character, given the characters that have come before.
That was a particular technique, and in some ways it had a lot of benefits.
It meant that the number of possible inputs was a limited number, just the number of possible letters
of the alphabet and some symbols.
And so that meant that it had a very manageable vocab size.
And it needed to its weights could didn't didn't need to worry about too many different possibilities
for the inputs.
But the challenge with it was that it meant that there was so much required from the model in terms
of understanding how a series of different characters becomes a word, and all of the intelligence associated
with the meaning behind a word had to be captured within the weights of the model, and that was expecting
too much from the model itself.
And so we then went to almost the other extreme where neural networks, these models were trained of
each individual possible word.
So you would build something called the vocab, which is like like the sort of the dictionary, the
index of all the possible words.
And then each possible word a token could be any one of these possible words.
So that meant that the the model itself could start to understand that each individual word had a different
meaning, rather than having to to appreciate how a sequence of characters would have a meaning.
So that was a good thing.
But the trouble was that it resulted in an enormous vocab.
You needed to have a vocab the size of all of the possible words.
And of course, there are so many possible words because there are also names of places and people.
And so there had to be special tokens for unknown words.
And that that caused some limitations.
Rare words had to be omitted, special places had to be omitted.
And so that caused some some some some oddness.
Um, and then around the time of the of GPT uh, a discovery was made a breakthrough that that there
was a sort of happy medium between these two extremes, rather than trying to train a model based on
individual characters and need it to learn how to combine them to form a word.
And rather than trying to say that each word is a different token, you could take chunks of letters,
chunks that would that would sometimes form a complete word and sometimes part of a word, and call
it a token, and train the model to take a series of tokens and output tokens based on the tokens that
are passed in.
And this had a number of interesting benefits.
One of them is that because you're breaking things down into tokens, you could also handle things like
names of places and proper names.
They would just be more fragments of tokens.
And then there was a second interesting effect, which is that it meant that it was good at handling
word stems, or times when you'd have the same beginning of a word and multiple potential endings that
would be encoded into one token, followed by a few second tokens.
And that meant that the sort of underlying meaning of what you're trying to say could be easily represented
inside the model, because the tokens had the same kind of structure that might have sounded a bit abstract.
Let me make that a bit more real for you.
So GPT, OpenAI actually provides a tool which is a platform openai.com slash tokenizer, and it lets
you put in some text and see visually how that text is turned into tokens.
And so I took a particular sentence, an important sentence for my class of AI engineers.
And you can see that GPT tokenized that.
That's the verb that we use when we're turning from words into tokens.
And it highlights in colors how it turned that into tokens.
And in this case, because these are all common words, every one of these words mapped precisely to
one token.
So this is a clear example.
You can see from the colors.
One other slightly interesting point to make that's that's important.
You see the way that some of these colored boxes, like the word for has a space in front of it.
It's like space.
And then for is what's been tokenized.
That's because the break between words is also meaningful when tokenizing.
That token represents the word for in isolation, like that beginning of word followed by the letters
for that is mapped to one token, the beginning of word for token, and so that that will maybe become
a bit more important in a moment.
But.
But it's worth noting that the gap between words is included as part of a token.
So let's take another example.
Now in this example, I'm coming up with a slightly more interesting sentence, an exquisitely handcrafted
quip for my musterers of LM witchcraft.
Now Musterers is, I believe, an invented word.
As you'll see, the red squiggly underline shows that it's not a true word.
And let's see how the tokenization has happened down here.
So you'll see that that four is still here as one word, uh, with a beginning of token at the beginning
of it.
But and so is an Anne at the start, but exquisitely has been broken up into multiple tokens.
Exquisitely.
And that shows how when you've got a rare word, it doesn't have that word as a single word in its vocab.
And so it had to break it into multiple tokens, but it's still able to pass it in.
And now look at that word handcrafted.
You can see that it also doesn't have that in its vocab as a single token, but it's able to break that
into hand and craft it.
And that does kind of reflect the, the, the, the meaning in a way it does get across that it can.
It's combined from these two hand and crafted.
And you can see as well that the crafted token does not include a beginning of sentence.
So it's a token that represents a word that that has crafted in the middle of it.
That's what that token reflects.
You'll see that quip isn't there at all.
It got broken into and it uh, and then you'll see Masteries.
And this is a good example of what I was saying about word stems.
Masteries has been broken into master, which is after all, the the, the verb that we're going for
here, someone who masters and then errs at the end as an extension to that word.
Um, and so you can see that it's, it's able to reflect the meaning of what we're trying to say by
breaking it into those two tokens, even though it's not a real word.
And you can also see that witchcraft got broken into witch and craft, uh, which is also interesting.
Uh, and so, yeah, handcrafted.
And master, as I say, you can see how the meaning is reflected there by the tokens.
And hopefully this gives you some real insight into what it means to break something into tokens.
So an interesting one here is to now show you this slightly more sophisticated example.
Uh, my favorite number, apparently 6534589793238462643383.
Blah blah blah.
Uh, so, uh, it shows you that when you have something like this, of course, long numbers like pi
are not going to map to one token.
And in fact, what you see is happening here is that every series of three digit numbers is being mapped
to one token.
And that's an interesting property.
It's actually a property of GPT two tokenizer, but many others don't have that many other cases.
You'll see that things map to multiple tokens.
Uh, so the generally speaking, there's a rule of thumb which is helpful to know to bear in mind when
you're looking at tokens.
The rule of thumb generally is that on average, one token typically maps to about four characters.
And that means that a token is on average for normal English writing, it's about three quarters of
a word.
One token maps to about 0.75 words.
And an easier way to think about that.
A better way to put it is that a thousand tokens is about 750 words.
So that's the mapping to have in your mind.
A thousand tokens is 750 words.
And that means that the complete works of Shakespeare, for example, to make this real, to give a
real example, that's about 900,000 words, apparently in the complete works of Shakespeare.
So about 1.2 million tokens, that is the size of the complete works of Shakespeare.
Now, that refers to English.
If you're looking at things like math formulas, scientific terms and also code, then the token count
is much higher because obviously, as we saw here with numbers, things need to be broken into many
more tokens to incorporate Right symbols and stuff like that.
And the other point to make here is that this is showing you GPT tokenizer.
There are no hard and fast rules about how Tokenizers should work.
In fact, we saw a minute ago that in the early days you used to have tokenizer, that every letter
would map to one token, and you'll see that different models have different approaches to tokenization.
And when we look later at open source, we're going to be getting hands on with a bunch of different
tokenizers.
And we're going to explore an interesting property of Llama's tokenizer too.
So different tokenizers work can work differently.
There are pros and cons for having fewer tokens or more tokens.
There's not a single answer that depends on the how many parameters are in the model and how it was
trained, and so on.
But this is a more detailed look at GPT tokenizer, and I hope this has given you some clarity and intuition
on what it means to to go from words and characters into the world of tokens.