Okay, so welcome to our leaderboard fast as we go through a ton of essential leaderboards for your
collection.
The first of them, the Big Code Models leaderboard.
You can see the URL there, but you can also just search for it in hugging face.
Um, and as with all of them, it's running as a spaces um, app.
And also we'll include links in the class resources.
Um, so what you see here is the, uh, set of models.
Let me start by filtering just on base models.
So it's names that we all recognize.
And you see the scores against the, uh, set of the human eval tests that I mentioned before, which
is Python tests and then tests against Java and JavaScript and C plus plus.
So you can compare the performance of these models in against different programming languages.
And the win rate is a sort of a similar to like an average across them.
And if you go to the about page, you'll get more information about how those are figured out and the,
the way that this this column is calculated, you can see that the top model for coding is a specialized
version of Queen for writing code that is called code Queen.
And there's also a code llama that's not far behind.
Deep Sea Coder is a model that's doing very well.
Uh, a variant of Code Llama.
And then Star Coder two is the model that we used ourselves early on.
And so Star Coder two features here as well.
Um, and then Code Gemma, which is Google's open source code generation model.
If we include all, then we'll include some of the ones that have been, uh, tuned specifically on
more specific data sets.
And you'll see that actually, if you compare the scores, uh, it really Yeah.
There's been a see that the code is far down now that a lot have been fine tuned to do much better.
Um, somewhat surprisingly, Code Kwan 1.5 chat uh, seems to be, uh, outperforming the 1.5, uh,
um, 7 billion down here, but but there may be various reasons for that.
It might be to do with the way the data set that's been used to fine train it for that purpose, um,
along with the kinds of questions that are asked here.
So if the specific problem you're looking to solve involves coding the big code models, leaderboard,
is the leaderboard for you.
Next one we're going to look at is called the LM perf leaderboard, which is about looking at the performance
of different models around things like their speed, their memory consumption and the like.
And if you go to the leaderboard itself, you find the models listed out here with their various variations
and then information about their speed, their consumption of energy and memory and so on.
But I would actually suggest that you don't start with with with that page, but instead you flip to
this, find your best model, you choose the hardware architecture that you're looking at, and then
you pick find your best model and what you see here when you go to find your best model is this very
interesting chart, which is actually a diagram that's displaying, uh, at least three different,
uh, quantities.
But if not, you could argue for, uh, along the x axis.
Here you are seeing something about the speed.
It's the time that this model takes to generate 64 tokens.
So obviously the more to the left is better.
It means faster time.
We're looking for models that come to the left if you care about performance, uh, speed, performance.
If you care about accuracy, that kind of performance, then you could you could use the total.
The open LM score, that that, uh, aggregate score as your measure of model accuracy.
If you care about the cost in terms of the memory footprint and a sense of the of the magnitude of hardware
that you're going to need to run this model, then you need to look at the size of the blob.
A bigger blob represents a greater memory need, and so it gives you a sense of what you'll need there.
So we are looking ideally for models that are small blobs that are over on the left and that are quite
high up.
That would be a nice result for us if we don't need it to be high up.
Particularly you can see a model like this one is doing really, really well and it is a Quan the Quan
1.5 variant.
Um, if you look right up here, if what you care most about is something which perform, which has
very great, very strong accuracy in terms of its benchmark scores and is also quite fast.
Uh, then maybe you come to this one here, which you can see is Llama Llama three model.
And that does bring me to the final point, which is that the other, uh, bit of information that's
being expressed in this chart is the family of models, but that is expressed by the color of blob.
And you can see over here, uh, how to read those colors like yellow means it's a phi model or a Phi
trained model.
And you'll see the phi, uh, yellow model over there.
So when it comes to the trade offs between speed, accuracy and memory footprint, which will affect
your running costs, uh, for open source models, this is a fantastic resource.
The, uh, perf leaderboard, uh, come to this, always turn to the Find your Best Model tab and browse
around to understand what your options are.
And this for example, if we're talking about a T4 hardware, then you would flip to the T4 tab to see
what kind of, uh, options you have here.
And looking that's the 01I hear.
Is Kwan again doing well.
And you can see other other models that might be most appropriate for you based on your use case.
And now I just want to mention that there is a spaces uh, there is a you could you could go to spaces
and search for leaderboards.
All I've done here is done a leaderboard search in spaces, and you will see all of the different leaderboards
that are out there that you could look at to see more details about your benchmarks of your LMS.
And if you're not overwhelmed with the amount of information here, there is great, great utility to
looking at these different leaderboards I mentioned a moment ago.
There is a Portuguese focused leaderboard.
You'll find many languages have their own leaderboards specifically to assess the abilities.
I mentioned the open medical leaderboard.
Let's bring this one up and you can see that there's a bunch of medical specific benchmarks like clinical
knowledge, college biology, medical genetics, uh, and uh, and PubMed QA.
And these are then scored against medical models.
So if you were trying to build a solution that was designed for medical use cases, this is the leaderboard
that you would come to right away.
Um, and so really the and typically the, the, the about page will give you that extra information
about the data sets, how it's used, how they are calculated.
So this should give you a really good sense of how you select the right set of, of uh, open source
models for the problem at hand.
How you find a useful leaderboard and how you, uh, interpret the different metrics and can rank the
different models out there.
Next time we'll look at some leaderboards that combine open source and closed source.