Welcome.
It's week three.
It's day four.
We are back on the adventure in open source land, back investigating how to run inference over open
source models.
And today it is time to look at the model class in Hugging Face.
We talked originally about pipeline API, the high level API.
Then we started talking about the low level API, beginning with Tokenizers and now onto the model.
So what can you already do?
Of course, in addition to coding with frontier models, building multimodal AI assistants or you can
now do is use hugging faces, pipelines and tokenizers today.
New skills, new classes.
We're going to get into the models part of hugging face, which is when you actually create a transformer
and run it to generate text.
And we'll be comparing results across five different models.
I'm actually going to be doing three of them with you and leaving you to experiment with the other two,
uh, so that you can have an extra exercise, but I'll have all of the code ready for you.
Um, so it should be a lot of fun.
So the models then to introduce them.
We are going to again be working with llama 3.1 from meta, their flagship and groundbreaking model.
We are going to be looking at Phi three, which is Microsoft's open source model, and Gemma from Google.
It's a the small.
The small cousin of Gemini is Google's Gemma.
There are two other models that I'll be leaving you with to experiment with on your own.
One of them is Mistral from Mistral and the other, the other is the powerhouse that is Quinn two.
And I hope that you will enjoy using Quantu.
So we're also going to be covering three aspects of working with open source models in the hugging face
framework.
Um, the first of them is called quantization.
And this is about reducing the precision of the weights in the model so that it is easier to fit into
memory and loads in and also can run faster.
So quantization, a very important technique that allows us to work with, say, a one of the lower
end GPU boxes.
and when we get to training, it's going to be absolutely critical to be able to use quantization,
to be able to train large open source models.
In fact, you've heard me saying, now the Q Laura, that is the name of the technique that we're going
to be using in a couple of weeks time.
And the Q and Q, Laura stands for quantization.
So we will be coming up against quantization a few times on this journey.
Today we're also going to be looking inside a model.
So generally again this is a class that is more practical than theoretical.
But this will be one of those moments.
And we'll just take a peek inside at what what do the PyTorch layers look like that sit behind the hugging
face Transformers library.
And then also, we're so familiar with streaming at this point that it hardly needs to be said that
we want to be able to stream results.
So I will show you how you can work with open source models to stream results as well.
So these are some of the little extra bits that we're going to look into in our voyage into running
inference over the lower level APIs for hugging face.
There's quite enough talk.
Let's get to it.