I'm delighted to see you again.
As we get started with day three of week three of our adventure and the, uh, things are going to get
get deeper this time.
We're going to roll our sleeves up as we get into the lower level APIs of hugging Face Transformers
library.
And as always, just a quick reminder you can code against frontier models, you can build AI assistants,
and you can use pipelines.
Pipelines.
What we did last time, such an easy way to use the wide variety of open source inference tasks available
from Hugging Face.
Today, though, we get lower level.
As I mentioned, there are these these these two things, tokenizers and models that are part of the
way we interact with transformers at a lower level than pipelines.
And that's what we're going to be doing today.
We're going to be starting with Tokenizers.
We're going to be learning how to translate between text and tokens for different models, and we're
going to be understanding something called chat templates, which I'm hoping is going to make a few
different things come together.
It's quite an important moment.
Um, so first, to introduce this type of object called a tokenizer in hugging face, it is an object
which translates as you can imagine between text, a string and tokens, a list of numbers.
Um, and there are very simply two functions.
Two things you need to know about encoding and decoding.
Encode takes you from strings to tokens, and decode takes you back again.
And we will see that.
And of course, there's just a little bit of nuance and fiddly stuff, but that's basically all there
is to it.
A tokenizer contains a vocab, which is all of the different fragments of characters of one character,
two, three, four characters shoved together that make up that token.
Um, and it can also include as well as these fragments of characters.
It can include something called a special token.
A few of these special tokens where a special token is again a single token that is going to tell the
the model something that it represents, like start of a sentence or beginning of a chat with the assistant
or something like that.
And as I mentioned before, if you're thinking, okay, but how do we train a neural network architecture,
how do we how do we how do we construct a neural network architecture so that it expects a particular
token to represent something like start of sentence or something like that?
And there's no magic answer.
It just simply comes down to training.
If it's seen enough examples in its training data that has that special token being used for that purpose,
it learns that that is the objective of that special token.
But there's nothing fundamental in the architecture, generally speaking, that expects one particular
type of token over another and also a tokenizer.
In addition to doing this, mapping text to tokens and having a vocab also has something called a chat
template.
At least for a specific type of model, as we'll see.
And that knows how to take a set of messages where you've had system message, user message and so on
and turn that into just a set of tokens.
And that will all make sense when you see a real example.
So every model in hugging face, every open source model has its own tokenizer associated with it.
There's not just one general tokenizer that applies to models because it depends on how the model was
trained.
The tokenizer, um, I mean, obviously multiple models could share the same tokenizer, but but what
matters is which tokenizer was used when the model was trained, because you have to use exactly the
same tokenizer during inference time when you're running it, otherwise you will get back bad results.
Uh, maybe that's an experiment we should try at some point, but I'll you'll see why.
That would be a very unproductive experiment in just a moment.
So for today we're going to look at the tokenizer for llama 3.1 which is the iconic family of models
from Larma that paved.
Sorry.
From.
From Larma.
From Mehta.
That paved the way for open source models.
And we're going to look at a model called Phi three from Microsoft.
And we're going to look at Quinn two again, the powerhouse from Alibaba Cloud, which leads the way
in many of the different metrics.
We're also going to look at something very different, which is a model called Star Coder two, which
is a model for, for for generating code.
We're going to look at its tokenizer to see any differences.
Um, and the reason that these two have similar looking graphics is that Lama 3.1 and Phi three are
extremely similar.
Quantu perhaps it's also very similar, but it's it's got more of a focus on, uh, Chinese as well
as English.
And Star Coder two is of course more about coding.
So with that introduction, we're going to head over to Google Colab and we're going to do some tokenizing.