So I'm aware that there's a big risk that you are getting fed up of leaderboards, because we've done
a lot of talking about leaderboards, and perhaps you are becoming fatigued to looking at benchmarks
and leaderboards.
You've bookmarked them all.
You're thinking, okay, I get it.
There are lots of great resources when it comes to choosing an LMS.
I will look through your leaderboards and I will do it then.
But enough.
Enough with the leaderboards.
But wait, there is one more.
And it's a really fun one.
And it's one you're going to love.
So you're not going to mind this final leaderboard for us to look at.
And it is in fact not a normal leaderboard.
It is an arena and it's called the LMS Chatbot Arena.
And you may have come across this already because it's so popular and fun and lots of people talk about
it.
So if you know it already, then hang on, bear with me because we're going to use it together and have
some fun.
So the idea is that the LMS chatbot arena, which is incredibly popular, is a place where models are
assessed by humans and they're assessed for their chat abilities.
It is, as they say, a crowdsourced open platform.
They've collected over a million human votes, ranking llms and to display results in as as I say,
in what's called an Elo rating type that will be familiar to chess players.
It's a way of assessing an overall kind of leaderboard based on the outcomes of head to head battles.
And you can see here some of the results.
And we will now take a look at what we see in first place is a variant of GPT four called ChatGPT 4.0
that came out very recently.
As of recording this video, it came out in early August 2024, and that previously GPT four was a little
bit further down on this list.
But with this latest trained, fine tuned model, uh ChatGPT four has taken the top position in the
arena with an Elo of 1.316.
Uh.
A useful thing here is that the knowledge cut off is quoted over on the right.
And so it's a if you want one place to go to see all the knowledge cut offs.
Actually, the, uh, open the LMS chatbot arena is a good place to go, so that's worth knowing.
I don't think it's it's intended to be its primary reason, but it's a useful tidbit.
Um, you'll see that Gemini 1.5 Pro actually is is doing really well in this human evaluated assessment.
Um, grok two, which is the latest model from X, uh, formerly known as Twitter is in strong place
here.
Uh, grok one had not done so well.
Grok two is really doing very well.
Um, and it's beating the earlier GPT four zero from, from May.
Um, and uh, it's, uh, yeah.
Uh, and you can see there's not much in that ELO difference, but at this point there is fairly significant
difference from the the leader Claude 3.5 Sonet, which at one point was was the front runner by by
I think by quite a way, if I remember right when it came out has now come down to, uh, I guess equal,
equal sixth or so.
Um, uh, right down here in, in, uh, Elo of 1270 compared to 1316 for the top, uh, GPT chat,
GPT four zero, um, then Gemini 1.5 flash, which of course is a somewhat smaller model.
And here comes llama 3.1 405 billion at 1266.
Uh, so, uh, yeah, very interesting to see this lineup.
You can keep going to find your favorite models.
Claude three opus is way down here.
Uh, and, uh, yeah, this is a great place to get a sense of how humans consider the experience of
chatting with different, uh, instruction tuned uh, chatbots.
Uh, you'll see that, uh, um, command r that we were looking at before.
Uh, I just saw command R that features somewhere here.
Uh, you will have to find that yourself.
I've now lost it.
There it is.
Command R plus found it.
Uh, there it is at 1213.
Uh, it's one of the ones we looked at all the way back at the start in week one.
So this is the famous LM, uh, LMS chatbot arena, and you can vote by clicking on this link, and
it will bring up this voting screen.
And here's how it works.
You get presented with two models model A and model B.
You don't know which one is which and you have to chat with them both.
And so let's do one of our favorite little chat things.
Uh, please tell me a light hearted light hearted joke suitable for a room full of data scientists.
So this question will be asked to both models together and we will see how they answer.
So the model A why do the data scientists break up with a computer?
Because it had too many commitment issues with its null values.
Okay, they did sciency, but not funny, I don't think.
Maybe you found it funny.
And model B, please tell me a joke.
Here is a light hearted joke.
Why does scientists go to the library?
Because they heard it had a lot of volumes.
Oh, yeah, that's pretty good, right?
That's that's different plays on the dual meaning word volumes.
It gives a little explanation.
The joke is simple.
It's kind of pun that data scientists tend to appreciate as it combines technical expertise with a bit
of wordplay.
I think that's solid.
That's solid.
Uh, I think B is better, is what I want to go with.
Hopefully you agree with me.
Uh, if not, you need to go into the arena and do, uh, do a ask a similar question and make your
own vote, but I'm going to vote on your behalf on all of our behalves by pressing the B is better button,
and it will then reveal the models.
Let's see.
Uh, interestingly, the we picked Claude three haiku and we picked that against grok two.
Uh, if I remember right, grok two was quite significantly higher than Claude three haiku.
So we have slightly fractionally shifted that balance through our vote.
So I hope you enjoyed that.
Of course, that was somewhat unscientific, but the idea is it's meant to be a human decision based
on your interaction.
And the ask for you now is go and do this.
It's an important way to add back to the community.
It is data that will be used.
Uh, plus it's a it's a lot of fun.
Uh, and it gives you a bit of sense of the capabilities and scope of the different models and some
hands on experience whilst contributing back to the data science community.
So enjoy spending some time on the chatbot arena, and I will see you next time to talk about commercial
uses.