Welcome back to our continued exploits with Tokenizers.
What we're now going to look at is what's called the instruct variance of models.
So there are many models that have been fine tuned to be specifically designed for chats, for carrying
out chat conversations with users, as one does with the with, uh, GPT four with with chat GPT um,
typically when you see those models in hugging face, you'll see that they have the same name as their
base models, but with instruct added to the end of it, meaning that they have been fine tuned to be
used in this instruct use case.
Uh, they have been trained to expect prompts in a particular structure with a particular set of special
tokens that identifies the system message, the user message and assistance responses so that it forms
a kind of a chat.
And that is just simply part of the way that it's been trained with enough examples.
So it expects it in this format, and this is hopefully going to bring some things together for you,
because it's now finally going to close the loop on something where I planted a seed some time ago about
the reason for this structure of messages, lists of dicts that we became very familiar with when we
were playing with frontier models.
So I'm going to create my tokenizer this time using the meta lemma 3.18 billion instruct variant.
So this will look familiar to you.
This is one of those lists of dicts that we use so much with, uh, OpenAI and Claude and so on, uh,
where you specify a role and content, a role a system is for the system message and user is for the
user message.
Then the tokenizers that Huggingface provide have a special function apply chat template and it will
take messages in this format in the OpenAI API format, and it will convert it into the right structure
to be used for a this particular model, the type of the prompt that this model is expecting, given
the way it's been trained, if you have tokenized equals true here, um, then what we'll get back is
just a series of numbers and we won't know what's going on.
So I've got tokenized equals false.
So what we'll get back will be the, the text version of it.
And I'm going to print it so you can see what is the uh the what is it that this is converted into that
gets pumped into the model at inference time for this particular conversation.
And here it is, starts with a special token begin of text and then a header.
And then system the word system and then end header.
And then there's some information that's shoved in there about the cutting knowledge date and today's
date.
That's that's special.
And I think that's a llama 3.1 thing.
I don't remember that from previous llama families, but I could be wrong there.
Uh, and then, um, this here is of course, the system message that we provided here.
Uh, then there is another start header for user and header.
And then this is the user message.
Then there's another start header and then the word assistant and then end header because we want the
model to generate the assistance response.
So this is kind of teeing up the model that what should come next right after this should be whatever
the assistance said in response to this uh, prompt following this system instruction.
So I'm hoping this is an aha moment for you that you see now how you can you can have a structure like
this.
And that's how you might think about the conversation with the model.
But at the end of the day, what gets pumped into the model is a prompt that looks like this with special
tokens in the mix.
And because it's been trained with that structure, with those kinds of special tokens, it knows what
needs to come next.
The assistance reply.
So that explains the chat interfaces.
Let's work with a few more models to get some more experience with this.
I'm going to pick three models in particular.
Phi three is a model from Microsoft.
Quinn two is this powerhouse model I keep mentioning from Alibaba Cloud.
Star coder two is a model designed for generating code.
It's built by three companies working together, collaborating ServiceNow and hugging face themselves
hugging face and Nvidia uh, that those uh, three mighty companies have partnered to make this, uh,
group star coder and have built this, uh, this particular model.
Okay.
So, uh, let's give a try for Phi three.
So we use exactly the same approach auto tokenizer from pre-trained and we provide the model.
And now um I'm giving the same text.
I'm excited to show Tokenizers in action to my LLM engineers.
I'm going to reprint the previous the llama 3.1 Tokenizers results to remind you what it's tokens look
like.
Then an empty line, and then I'm going to print Phi three.
And the question is going to be at the end of the day, do they basically produce the same tokens or
is it different.
Let's have a look.
Well you'll see right away they are completely different.
Uh they're different.
Not only is the generated text different, but this first one, which is the start of of message special
token is completely different.
Uh, let's do batch decode so we can see that.
Tokenizer.
Dot Batch decode.
I'll have to say tokens.
Equals.
Tokens.
Let's see what we get here.
Uh, and we do get something completely different.
And actually, interestingly, I was wrong with what I said a second ago.
There isn't a start of sentence special token in the case of 53, so it just goes straight into it.
So that's that's a very different approach.
All right.
Let's use the apply chat template to see how 53 uses chat templates.
Let's start by doing it for llama again.
So we'll see llamas one.
And then we'll print side by side the same the chat template for that same conversation, that same
prompt for 53.
Let's see how they look.
So this is the one we just looked at for for Lama, here is the equivalent for Phi three.
It's obviously much shorter.
It doesn't pass in the the date.
And interestingly, whereas the structure for Lama was about a header and then the word system and end
header and a header the word user and an end header.
In the case of Phi three there's just a special tag for system and a special tag for user and a special
tag for assistant.
So it's this whole sort of different approach.
Um, and that's really interesting to see that these two tokenizers, these two models just have a different
approach for how prompts get sent in.
So obviously, hopefully you're getting the impression if you use the wrong tokenizer for the wrong
model, you'd get garbage, because obviously this with different tokens and different structure is
going to be meaningless to llama three.
And now let's do the same for Quinn two.
We're going to see the original Lama version.
And then we're going to show the Phi three version and then the two version.
Here they come.
Uh, obviously you can see totally different results for the three tokenizers.
Uh, and one more time highlights.
You got to pick the right tokenizer for the right model.
Uh, and, uh, let's just apply the chat template and we'll see again the chat templates for that same
message about telling a joke.
We'll see that for llama.
And then for five three and then for Quinn two all side by side.
Let's see what they look like.
We already saw the one from llama.
We already saw the one from 53.
And here is the one for Quinn two.
And what you'll see is that it's it's sort of somewhere in between.
It does a bit like llama.
It's got the, the Im start im end and system in here.
Um and then user and then assistant.
So it's some somewhere in between the two.
Uh, it doesn't uh, it doesn't have something in between the word.
It doesn't have a header special tag.
It just has, uh, this approach here.
So it's an interesting again a third approach, another variation and with different special tokens.
All right.
And finally let me show you Star Coder two.
This is the code generation module.
We're going to take its tokenizer.
And we're going to put this code in there.
Hello world a def hello world uh taking a person variable.
And it's going to print hello.
And then the person.
And then we just use the same encode to turn it into tokens.
And what I'm then going to do is just print out each token followed by what did that get to uh, get
mapped to what what text did that represent?
And what you'll see here is that there was something at the beginning, and then there's def went into
one token and then hello underscore world and then person.
This here obviously will will reflect the tab and then print hello comma person close brackets.
So it gives you some sense that, um, the star coder two tokenizer is a tokenizer that is designed
around tokenizing code rather than English.
And there's some experiments you can do.
First of all, try out different tokenizers try exploring mapping from text to tokens.
Find out which words.
Try and find the rarest possible word that has a single token in in llamas.
Uh, tokenizer or perhaps the longest word or something like that.
Do some experiments, um, and then satisfy you.
Satisfy yourself that if you take a pretty complicated piece of code, you should find that star coder
tos tokenizer tokenizes it in a more efficient way than one of the tokenizers that's designed for just
English.
And at that point, you will be an expert in the world of open source tokenizers and you'll be ready
to take on the next piece, which is models.
First, let's go back to the slides.