I know that everybody.
It seems like just the other day that we were embarking on our quest together, and now here we are
at the beginning of week four, rapidly approaching the halfway point.
So what is in store for us this time?
Uh, this week we are getting into an essential period of building core knowledge.
It's going to be about this very challenging question that hits us all, which is we are spoilt for
choice.
There are so many llms out there.
There are so many decisions to be made, closed source or open source.
And and then in each category, so many.
How do you pick the right LLM for a problem at hand.
And that is the main theme of this week.
And in addition we'll be going to be generating code building llms that write code for us in open source
land.
So that will be some fun.
Uh, what we're going to be doing today is getting cracking on this subject of how do you pick the right
model for the task.
We're going to be talking about attributes and benchmarks, and we're going to be using something called
the Openml leaderboard, which is an amazing resource from Hugging Face to help you compare open source
models.
But first, of course, a moment to look at our eight weeks.
We started over on the left.
We're going to finish over on the right.
In week one.
We talked about all things frontier model and we compared a bunch of frontier LMS, six of them.
In week two, we introduced UIs with Gradio agent ization and of course we played with Multi-modality
week three.
Last week we got stuck into open source with Hugging face.
We looked at the hub, we looked at the high level API, the pipelines, and then we looked at Tokenizers
and models and hopefully a lot came together in terms of really understanding how these chat interfaces
work and how these sorts of lists of dictionaries end up being tokens, including special tokens fed
into llms, LMS, resulting in the output the next predicted token.
We get to week four about selecting LMS, and also the problem of generating code which which is going
to be very interesting.
I hope week five is rag.
Week six is fine tuning.
It's when we begin training.
Training on the frontier and then training open source and bringing it all together.
That is our eight week plan.
And so for today we're talking about comparing models, comparing LMS.
And if there's one takeaway from this session, the most important point is that it's not like there's
a simple answer of like this LLM is better than the others.
It isn't a a case of of ranking from best to worst.
It's all about what are you trying to accomplish.
And it's about picking the right LLM for the task at hand.
So understanding weighing up different LMS and comparing them with your requirements is the name of
the game, and there are two different ways that we can pair models.
The first of them is looking just at the basic facts about the model.
Um, obvious stuff like the cost of the model.
Um, and that's going to really cut down your decision space into a smaller number of choices.
And once you've investigated the sort of basic attributes, the basic aspects of the different models,
then then you start looking at the detailed results.
And that involves looking at things like benchmarks, leaderboards and arenas.
And based on all of this, you should end up with a handful of candidate llms that you will then use
for prototyping to allow you to finally select the best LLM for your task at hand.
And so let's talk a bit about the basics.
So when I say we're comparing the basics, I really do mean the most obvious things about different
models that you would have to assess.
And we start with understanding whether or not you're going to be looking at an open source model or
a closed source model.
And of course, there are pros and cons, and it will affect a lot of the other basic attributes.
So as you develop your shortlist, the first thing to note down is is this open or closed source?
When was it released?
What is the release date and and presumably the same the same dates.
But an important thing to note is what is the knowledge cutoff?
What is the date?
The last date of its training data, beyond which typically it won't have any knowledge of current events.
And depending on your use case, that may be important to you or it might not.
Then the number of parameters.
This gives you a sense of the the strength of the model.
It will also give you a strength of costs that will come to, and it will give you a sense of how much
training data is needed.
If you want to fine tune that model, which we will also talk about in just a moment.
So the number of parameters, the size of the model is another of the basic facts that you would note
down The number of tokens that were used during training.
The size of the training dataset is an important thing to note, and it will give you a sense again
of the power of the model, its level, its depth of expertise, and then of course, the context length,
the size of the context window.
The thing we spoke about a lot in the past.
The total amount of tokens that it can keep effectively in its memory while it's predicting the next,
next token, which needs to include the original system, prompt input prompts.
And if you're in a in a instruct in a chat use case, then all of the exchanges between the user and
the assistant all have to fit within the context length.
If you're dealing with a multi shot prompting case where you're you're providing multiple examples at
inference time for the model to learn from, then you need to make sure that you're going to have a
sufficient context length to take all of those examples.
Can you remember the model with the longest context length today?
uh, Gemini 1.5 flash with a million size contacts window as of right now, but we'll see in a moment
where you can go and look up and compare all of the context lengths.
So that's one set of basics.
Let's go on to some more basics that you would you would look at.
So there's a bunch of of costs that you need to be mindful of.
I've divided them into inference costs, training costs and build costs.
So inference costs of course is how much is it going to cost you every time you run this model in production
to generate an output given an input?
That, put simply, is what we're talking about with inference.
Um, and there depending on whether you're dealing with open or closed source or and how you're interacting,
there could be a number of different types of costs.
We know, of course, with frontier models, we're thinking about API costs, which we also know consists
of a count of input tokens and output tokens that would need to go into that API cost.
Uh, if you were talking about using the Pro user interfaces, the chat UIs, you'd be thinking of a
subscription, a monthly subscription cost.
And if you're talking about open source models that you would run yourself, then there would be some
runtime compute cost, which could be like a colab cost, or later we'll be talking in the actually
in probably the last week of this course about ways to deploy to production.
Thinking about platforms like modal, which let you run your model in production on a GPU box, and
then you're paying some, uh, some, some fee to run your compute box in their cloud.
So that runtime compute for open source is another factor.
And typically if you're if you're working with an open source model that you've trained yourself, your
inference costs will be lower because it's your model.
You're not going to be paying that API charge every time.
But there's you know, it's not a clear calculus.
It depends on your use case.
It depends on your choice of model, how many parameters and so on.
And then training cost.
So obviously if you're using out of the box frontier model, then there isn't a training cost.
If you're not further fine tuning it, as we'll do in week seven.
But if you're doing, uh, building an open source model that you want to specialize for your domain
and you're going to be providing it with, with training costs, with, with, with training, then
there will be a cost associated with that.
And you need to factor that into the equation.
Build cost.
So how much work will it be for you to create this solution.
Uh, and that's highly related to the next one which is time to market, which is how long is it going
to take you?
Uh, the, the one of the selling points of using a frontier model.
Is that the time to market?
And the build cost can be very low.
Uh, it can take very little time to be up and running with a powerful solution using frontier models.
Typically, if you're looking to fine tune your own open source model, it's going to take longer and
it's going to be harder.
Uh, so that's that's a major factor to weigh up rate limits using frontier models, you may run into
some limits on how frequently you can call them.
This is typically the case for subscription plans.
Um, and uh, maybe, maybe with rate limits.
I point out reliability as well when using the frontier models through the APIs.
There are times when some of the models I've experienced this with both, uh, GPT four and with Claude
3.5 sonnet that the the, uh, APIs respond with an error that they are overloaded because they are
too busy in production at that time.
So that's sort of related to rate limit, but a sort of stability point, their speed, which is a sort
of throughput, like how quickly can you generate a whole response, how quickly can new tokens be generated?
And very similar.
There's a sort of subtle distinction between speed and latency, which is sort of the request response
time.
When you ask how quickly does it first start responding with each token?
You may remember when we built the AI assistant for our airline, which was multimodal and spoke back
to us.
Latency was a bit of a problem there.
I don't think I mentioned it at the time, but there were some awkward pauses because of course when
there's some text, it's then going out to the model.
It's calling out to a frontier model, generating the audio, coming back and playing the audio.
And it was even more jarring when we were generating images, because those images were taking some
time to come back and we'd sit there waiting for the image.
Obviously, there are ways to to handle that more gracefully than we did in our prototype, but that
is a factor that has to be.
Bear in mind if you're dealing with your own open source model, it's the sort of thing you have more
control over.
And then last of our basics, but definitely not least, is license.
Whether you're dealing with open source or closed source.
You need to be fully aware of any license restrictions in terms of where you are, and are not allowed
to use it.
Um, many of the open source models have very open licensing.
Um, some of them do have fine print.
I think stable diffusion is one that's known that that, uh, um, you are allowed to use it commercially
up to a point.
There's a point at which when your revenues are above a certain level, um, that at that point some,
some kind of an arrangement business arrangement with stable diffusion, uh, is, is needed.
Um, and we experienced ourselves signing the terms of service with, um, with llama 3.1 with meta.
Uh, again, which I think is mostly to make sure that we're using it for good purposes, but still,
it's part of the license that one needs to be aware of.
So that wraps up the basics.
These are all things that you would note down before going in to a more detailed analysis of the performance,
the accuracy of the models for the task at hand.
And we will continue in the next session.