Welcome to day two of week four, when we get more into leaderboards so that by the end of this, you'll
know more than you ever wanted to know or dreamed of knowing about leaderboards and metrics and benchmarks
and arenas and all of that.
But you'll find it incredibly useful and important as you look to understand how to pick the right LLM
for the task at hand.
So the plan for today, uh, last time we talked about comparing Llms using basic attributes and using
benchmarks.
Today you're going to look beyond the open LLM leaderboard from hugging face at other leaderboards and
arenas that let you compare and evaluate llms.
We're also going to look at some real world use cases of llms solving commercial problems, uh, which
will be hopefully, uh, insightful for you.
We'll just do that quickly, because I think everyone has exposure to so many these days, but it will
give you food for thought and it should, in summary, arm you to be able to choose the right LLM for
your projects.
So we're going to be doing a tour of six essential leaderboards that are available through Huggingface
and beyond.
Huggingface.
For comparing Llms, the first of them is actually one that we've already seen the Huggingface OpenAI
leaderboard comparing.
There's both the old version, but now there's the new version, which has the harder metrics.
It's of course open source models, but it is the go to place to compare open source models.
There is also a leaderboard called Big Code on Huggingface, which is a leaderboard that compares models
specifically that are designed to generate code.
And we'll be doing that looking at those examples and how it assesses models in just a second.
There is one called the LM perf board, which is another Huggingface board that talks, thinks about
performance, uh, about both accuracy and the actual speed and cost of compute.
So this is a super important one that looks at a different dimension.
It's looking at some of the basic attributes, and it's one that again is going to become a go to resource
for you, particularly when you're thinking about inference of open source models compared to their
closed source cousins.
And then there are some other hugging face boards.
There's so much on hugging face.
There are more.
There are specific leader boards designed for different business areas.
You're going to see a medical leaderboard which is designed for, uh, of course, uh, medical use
cases.
You're going to see leaderboards for other languages.
I think there's one Portuguese.
I just saw that.
I'll show you.
There's a bunch of different language specific leaderboards depending on your use case.
That might be the leaderboard that's right for you.
Then we're going to go and look at Valheim's leaderboard.
You may remember we we briefly looked at this uh, early on.
It's a very useful resource which has a number of leaderboards, um, that include open and closed source.
So it's one of the places where you can bring together the full family of models.
It also has that useful table, if you remember, that has cost, API costs and contacts window lengths.
And so it's another one to add to your bookmarks because it's one of the few places which which reliably
has all of that information up to date in one place.
And then the last of the leaderboards that we'll look at is called Seal, which assesses various expert
skills, and they are always adding new leaderboards to to this set of leaderboards.
So it's it is, if I may say, another one to bookmark.
So your bookmarks are going to get crowded because these are great resources and I think you will thank
me when you need to use them in the future.
We are then also going to look at something called the Chatbot Arena, the LMS Chatbot Arena.
It is a fantastic resource.
It is specifically looking at the instruct the chat use case, the ability of models to chat.
But the idea is rather than having some sort of benchmark test, it relies on human judgement to decide
which model is better at the kind of, uh, the, the, um, the instruction following chats.
Um, And it's just a qualitative decision by a human to decide to pick a model A or model B whilst chatting
with both models together.
And it's a blind test in that you don't know which model is which and you have to vote without knowing.
Um, and models are given an Elo rating, which I mentioned before based on their overall, uh, um,
how they rank overall against their, their, their peers, um, from many human tests.
So we'll see this.
We'll get to do some voting ourselves.
And then finally we're going to look at a bunch of commercial use cases, everything from law to talent
to code to healthcare to education, seeing llms in action.
And of course, this this could be ten times more.
The Llms are making an impact in every business vertical you can imagine, but it's always useful to
see a few of them and get a sense of what's out there.
So we will do all of that now.
And, uh, prepare, prepare to do some bookmarking because you're going to see a lot of useful stuff
now.