And welcome back to More Leaderboard Fest as we go through some more leaderboards.
But this time we're bringing into the mix open source and closed source models together on some of the
leading leaderboards.
Outside hugging face for for a change.
We are actually not going to be looking at hugging face now.
So the first leaderboard that I want to show you, that is another one that is definitely belonging
on your bookmarks is the vellum leaderboard.
We did touch on this briefly in the past.
Uh, vellum AI company had publishes this essential resource for LM practitioners, which compares different
models at the very top of the page.
You get these comparison charts about some basic benchmarks that are some of the easier benchmarks these
days.
MLU the reasoning one not the Pro, but the basic one that's, uh, you know, pinch of salt on this
metric, but still, it's still quoted a lot.
Um, human eval for Python coding and math.
Um, and what you're seeing here are generally the closed source models that you know and love, like
GPT four and sonnet 3.5, sonnet and GPT turbo and four.
But look at that.
There is an open source model in the mix in the form of llama 3.1 4.05 billion.
That is the largest open source model on the planet.
And you can see that it is competing, competing favorably with some frontier closed source models.
Uh, so it does appear that this is in order of strength with the strongest one first, GPT four zero,
uh, crushing it with MLU.
But you can see that llama 405 B is just fractionally behind.
Um, and they're all neck and neck.
Really?
Uh, so, uh, obviously llama 4 or 5 billion is the open source model is very much a contender.
Then when it comes to coding, you can see that this is the order.
Clause 3.5 sonnet is the leader, then GPT four zero, then llama 405 be in third position, and then
the mini version of GPT four zero.
Not not far off given how much cheaper it is.
And then GPT turbo.
And then here is the ranking for math questions GPT four zero at the at the helm, followed by llama
405 billion right after that, and then followed by the others, with Claude coming in fourth place.
For those top models, here are some super useful charts on performance.
A little bit easier to interpret than the multi-dimensional chart we saw in Hugging face, although
less information, of course.
Uh, so in terms of the speed, the fastest to generate tokens measured in tokens per second is llama
8 billion open source model.
Not surprising because of course with fewer parameters it's doing less, so probably worth understanding.
Uh, yeah.
I see.
So.
So this is all, uh, trying as much as possible to run in a consistent way.
And the information explains a little bit more about that.
Uh, so after llama eight comes llama 70, a bigger model, and then Gemini 1.5 flash.
Uh, and then Claude, three haiku, and then GPT four.
Oh, mini.
Uh, the mini variant.
So obviously the smaller models are the faster ones.
No surprise there.
Uh, latency.
Uh, that's that's measured in the number of seconds until the first token is received.
It's a nice way of capturing it.
That's a good way to explain what I was talking about earlier, when I showed latency on the basic attributes.
And you can see no surprise the smaller models are able to respond very rapidly.
Um, and here GPT four surprisingly has improved latency over GPT four.
zero.
Many, which may be just related to the the hardware setup that it has.
I'm not sure, but they're close anyway.
And then the cheapest models, which is measured in terms of dollars per million tokens.
Uh, llama 8 billion comes in cheapest Gemini 1.5 flash does well, GPT four and mini of course is very
cheap.
And then, uh, the, uh, Claude three haiku, um, and then GPT 3.5 turbo after that.
And this is being shown as two separate bars, one for input cost, one for output cost.
So, uh, there is then a nice little interactive ability to compare two models and see them side by
side against different measures.
This is showing Claude three point uh, sorry, Claude 3.0 Claude three opus against GPT four.
Oh, let's see if we can change this around a bit and pick 3.5 sonnet against GPT four.
Oh, this is the face to face that we like to look at.
So here we go.
I mean, really, it looks like generally it considers them neck and neck.
What are they saying?
88.3% for Claude, 3.5 and 88.7% for GPT four.
Oh.
So giving GPT four the edge there reasoning Claude does better coding.
Claude does better math, Claude does worse tool use.
Uh, of course, what we went through in week two.
Uh, Claude does better and multilingual Claude does better.
So great.
Uh, fascinating to be able to compare the models side by side like this.
Um, then this table has, uh, row by row, the different models.
Um, and so you can come through and look at, uh, closed source models like Claude 3.5 sonnet.
Uh, that in terms of the averages, here is the one at the at the top of this leaderboard.
Um, what you're looking at here is MLU again, which is this metric where everything scores very well.
The one that we talked about in the initial metrics human eval for Python.
This is the be hard benchmark that I mentioned was the benchmark designed to test future capabilities
of models above and beyond what they're capable of.
Um, but would you believe when you look at this cloud 3.5, sonnet is already scoring 93.1% in B hard,
which means that no longer is this a metric that's testing for future capabilities.
It is very much current capabilities.
And cloud 3.5 sonnet is crushing it.
Uh, grade school math and harder math problems.
So here you see the the the the results from these different models.
And something I mentioned early on that that is a bit puzzling is that cloud 3.5.
Sonnet performs better than Claude three.
Opus.
Um, but Claude three opus is still provided as a by anthropic as as an API and costs significantly
more than 3.5 sonnet.
So I'm not sure why anyone would choose Claude.
Three opus over 3.5 sonnet unless there happens to be some specific.
Well, it looks like in the case of, uh, of reasoning, uh, Claude three opus does do better.
So there are some, some, some ways in which it does better, but I'm not sure if it would be worth
that extra price point.
Um, and what you'll also see, of course, is that llama, uh, enters onto this model comparison.
I noticed that llama 405 billion is not shown here, and I can only imagine.
That's because they haven't yet been able to carry out all of these tests for llama 4.5 billion, because
I would, of course, imagine that it far outperforms the 70 billion llama three instruct variant.
Um.
And now coming down to this table, this is the one that I showed you before.
It's one place you can come to that will show you for the different models.
What is their context, window size and what is their cost per input and output tokens.
So it's of course only comparing the um, the, the, the models, uh, where it has that data, but
it's extremely useful.
It's something where, uh, you would either be hunting through many different pages online, or you
can come here and see it all in one place, and that's why you should bookmark it.
Uh, it of course, highlights that Gemini 1.5 flash has the extraordinary a million context window.
That is, of course, 750,000 words or so of common English, uh, almost the complete works of Shakespeare,
a extraordinarily large context window.
The Claude family at 200,000.
The GPT family at 128,000.
Which, as I said before, seems somewhat slim compared to the million in Gemini 1.5 flash.
But that's still a lot of information to be able to digest in a context window and still give a good
response.
Uh.
You'll also see some open source models in the mix here.
You can see mixed trials, context window size, and that the llama three models have an 8000 token
context window.
And that's worth bearing in mind as you compare using open source models to their closed source cousins,
that if you need these massive context windows, then you're probably needing to go to the closed source
route.
Okay, so there is then a coding leaderboard that you can look at to compare human eval and then that,
uh, that concludes the leaderboards on the vellum web page.
There is one more to look at.
Of these, um, business, uh, sites, and it is called the seal Leaderboard, provided by a company
called scale.
Com and scale specializes in generating bespoke data sets.
So if you are working on a particular problem and you need to have a data set, uh, crafted, curated
for your problem, then that is something that scale com is in business for.
Uh, if you aren't able to use the data generator that hopefully you built as part of last week's challenge.
So this leaderboard has a bunch of very specific leaderboards for different tasks.
And there's one on adversarial robustness, which is designed, as it explains very well here on the
Learn More to, uh, test prompts designed to trigger harmful responses from large language models.
And so there's this, specific examples of the kinds of problematic questions that are asked.
And if, for example, you're looking sorry, I didn't mean to do that.
If, for example, you're looking to surface this as a chat as perhaps your airline customer support
chatbot, you will care about the fact that it is robust against being taken off track and doing something
that that could be far off the rails.
So this is a useful benchmark for that purpose.
Coding gives a more detailed benchmark for coding skills, and you can see Claude 3.5 sonnet leads the
way.
Um, and Mistral, of course, this is another set of boards that combines closed and open source.
And Mistral Large two um, is in that top three, uh, as an open source, uh entrant instruction following
uh, here you'll see that, uh, wonderfully, the llama 3.1 405 billion.
They have been able to test this against instruction following.
And it's in second place.
It's ahead of GPT four zero.
It's just behind Claude 3.5 sonnet.
Uh, and so that is an amazing result for the world of open source and for meta coming in second place
there.
Uh, and in math problems.
Llama 3.1 405 B comes in third place, GPT four zero and second Claude 3.5.
Sonnet leading the way for math.
And then there is also a leaderboard for Spanish, uh, which shows some of the results here.
Uh, and Mistral is the open source front runner in fourth place with GPT four zero in pole position
here.
And Qxl.com are adding more of these business specific leaderboards all the time.
So come back to see what else has been added and use this as a great resource for more specific leaderboards
for your business problem.