From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
325 lines
8.6 KiB
325 lines
8.6 KiB
WEBVTT |
|
|
|
00:01.040 --> 00:05.300 |
|
So I'm aware that there's a big risk that you are getting fed up of leaderboards, because we've done |
|
|
|
00:05.300 --> 00:11.000 |
|
a lot of talking about leaderboards, and perhaps you are becoming fatigued to looking at benchmarks |
|
|
|
00:11.000 --> 00:12.020 |
|
and leaderboards. |
|
|
|
00:12.020 --> 00:13.010 |
|
You've bookmarked them all. |
|
|
|
00:13.010 --> 00:14.780 |
|
You're thinking, okay, I get it. |
|
|
|
00:14.780 --> 00:18.110 |
|
There are lots of great resources when it comes to choosing an LMS. |
|
|
|
00:18.110 --> 00:21.590 |
|
I will look through your leaderboards and I will do it then. |
|
|
|
00:21.590 --> 00:22.280 |
|
But enough. |
|
|
|
00:22.280 --> 00:23.390 |
|
Enough with the leaderboards. |
|
|
|
00:23.390 --> 00:25.400 |
|
But wait, there is one more. |
|
|
|
00:25.400 --> 00:26.480 |
|
And it's a really fun one. |
|
|
|
00:26.480 --> 00:27.770 |
|
And it's one you're going to love. |
|
|
|
00:27.800 --> 00:32.390 |
|
So you're not going to mind this final leaderboard for us to look at. |
|
|
|
00:32.390 --> 00:35.240 |
|
And it is in fact not a normal leaderboard. |
|
|
|
00:35.240 --> 00:38.810 |
|
It is an arena and it's called the LMS Chatbot Arena. |
|
|
|
00:38.810 --> 00:43.550 |
|
And you may have come across this already because it's so popular and fun and lots of people talk about |
|
|
|
00:43.550 --> 00:43.820 |
|
it. |
|
|
|
00:43.820 --> 00:48.860 |
|
So if you know it already, then hang on, bear with me because we're going to use it together and have |
|
|
|
00:48.860 --> 00:49.610 |
|
some fun. |
|
|
|
00:49.610 --> 00:57.290 |
|
So the idea is that the LMS chatbot arena, which is incredibly popular, is a place where models are |
|
|
|
00:57.290 --> 01:02.030 |
|
assessed by humans and they're assessed for their chat abilities. |
|
|
|
01:02.090 --> 01:05.830 |
|
It is, as they say, a crowdsourced open platform. |
|
|
|
01:05.830 --> 01:15.760 |
|
They've collected over a million human votes, ranking llms and to display results in as as I say, |
|
|
|
01:15.790 --> 01:20.440 |
|
in what's called an Elo rating type that will be familiar to chess players. |
|
|
|
01:20.440 --> 01:28.360 |
|
It's a way of assessing an overall kind of leaderboard based on the outcomes of head to head battles. |
|
|
|
01:28.720 --> 01:32.410 |
|
And you can see here some of the results. |
|
|
|
01:32.710 --> 01:43.570 |
|
And we will now take a look at what we see in first place is a variant of GPT four called ChatGPT 4.0 |
|
|
|
01:43.600 --> 01:45.640 |
|
that came out very recently. |
|
|
|
01:45.640 --> 01:54.820 |
|
As of recording this video, it came out in early August 2024, and that previously GPT four was a little |
|
|
|
01:54.850 --> 01:56.170 |
|
bit further down on this list. |
|
|
|
01:56.170 --> 02:05.470 |
|
But with this latest trained, fine tuned model, uh ChatGPT four has taken the top position in the |
|
|
|
02:05.470 --> 02:10.490 |
|
arena with an Elo of 1.316. |
|
|
|
02:10.700 --> 02:11.270 |
|
Uh. |
|
|
|
02:11.420 --> 02:16.400 |
|
A useful thing here is that the knowledge cut off is quoted over on the right. |
|
|
|
02:16.400 --> 02:20.210 |
|
And so it's a if you want one place to go to see all the knowledge cut offs. |
|
|
|
02:20.240 --> 02:26.180 |
|
Actually, the, uh, open the LMS chatbot arena is a good place to go, so that's worth knowing. |
|
|
|
02:26.210 --> 02:30.860 |
|
I don't think it's it's intended to be its primary reason, but it's a useful tidbit. |
|
|
|
02:31.490 --> 02:39.770 |
|
Um, you'll see that Gemini 1.5 Pro actually is is doing really well in this human evaluated assessment. |
|
|
|
02:40.040 --> 02:49.730 |
|
Um, grok two, which is the latest model from X, uh, formerly known as Twitter is in strong place |
|
|
|
02:49.730 --> 02:50.600 |
|
here. |
|
|
|
02:50.630 --> 02:52.430 |
|
Uh, grok one had not done so well. |
|
|
|
02:52.460 --> 02:54.860 |
|
Grok two is really doing very well. |
|
|
|
02:55.250 --> 03:01.220 |
|
Um, and it's beating the earlier GPT four zero from, from May. |
|
|
|
03:01.610 --> 03:05.410 |
|
Um, and uh, it's, uh, yeah. |
|
|
|
03:05.410 --> 03:10.310 |
|
Uh, and you can see there's not much in that ELO difference, but at this point there is fairly significant |
|
|
|
03:10.550 --> 03:17.720 |
|
difference from the the leader Claude 3.5 Sonet, which at one point was was the front runner by by |
|
|
|
03:17.750 --> 03:26.240 |
|
I think by quite a way, if I remember right when it came out has now come down to, uh, I guess equal, |
|
|
|
03:26.240 --> 03:27.650 |
|
equal sixth or so. |
|
|
|
03:27.650 --> 03:38.450 |
|
Um, uh, right down here in, in, uh, Elo of 1270 compared to 1316 for the top, uh, GPT chat, |
|
|
|
03:38.450 --> 03:44.930 |
|
GPT four zero, um, then Gemini 1.5 flash, which of course is a somewhat smaller model. |
|
|
|
03:44.930 --> 03:52.610 |
|
And here comes llama 3.1 405 billion at 1266. |
|
|
|
03:52.820 --> 03:57.140 |
|
Uh, so, uh, yeah, very interesting to see this lineup. |
|
|
|
03:57.140 --> 03:59.840 |
|
You can keep going to find your favorite models. |
|
|
|
03:59.840 --> 04:02.570 |
|
Claude three opus is way down here. |
|
|
|
04:02.780 --> 04:11.810 |
|
Uh, and, uh, yeah, this is a great place to get a sense of how humans consider the experience of |
|
|
|
04:11.820 --> 04:17.550 |
|
chatting with different, uh, instruction tuned uh, chatbots. |
|
|
|
04:17.580 --> 04:23.400 |
|
Uh, you'll see that, uh, um, command r that we were looking at before. |
|
|
|
04:23.400 --> 04:27.120 |
|
Uh, I just saw command R that features somewhere here. |
|
|
|
04:27.420 --> 04:30.810 |
|
Uh, you will have to find that yourself. |
|
|
|
04:31.050 --> 04:32.310 |
|
I've now lost it. |
|
|
|
04:32.310 --> 04:32.940 |
|
There it is. |
|
|
|
04:32.940 --> 04:34.830 |
|
Command R plus found it. |
|
|
|
04:34.950 --> 04:37.680 |
|
Uh, there it is at 1213. |
|
|
|
04:37.680 --> 04:41.760 |
|
Uh, it's one of the ones we looked at all the way back at the start in week one. |
|
|
|
04:41.940 --> 04:51.210 |
|
So this is the famous LM, uh, LMS chatbot arena, and you can vote by clicking on this link, and |
|
|
|
04:51.210 --> 04:53.910 |
|
it will bring up this voting screen. |
|
|
|
04:53.910 --> 04:55.290 |
|
And here's how it works. |
|
|
|
04:55.290 --> 04:58.620 |
|
You get presented with two models model A and model B. |
|
|
|
04:58.650 --> 05:02.160 |
|
You don't know which one is which and you have to chat with them both. |
|
|
|
05:02.160 --> 05:05.910 |
|
And so let's do one of our favorite little chat things. |
|
|
|
05:05.940 --> 05:19.930 |
|
Uh, please tell me a light hearted light hearted joke suitable for a room full of data scientists. |
|
|
|
05:19.930 --> 05:26.920 |
|
So this question will be asked to both models together and we will see how they answer. |
|
|
|
05:26.950 --> 05:31.990 |
|
So the model A why do the data scientists break up with a computer? |
|
|
|
05:31.990 --> 05:37.270 |
|
Because it had too many commitment issues with its null values. |
|
|
|
05:37.660 --> 05:43.480 |
|
Okay, they did sciency, but not funny, I don't think. |
|
|
|
05:43.510 --> 05:44.860 |
|
Maybe you found it funny. |
|
|
|
05:45.280 --> 05:47.200 |
|
And model B, please tell me a joke. |
|
|
|
05:47.200 --> 05:48.310 |
|
Here is a light hearted joke. |
|
|
|
05:48.310 --> 05:49.990 |
|
Why does scientists go to the library? |
|
|
|
05:49.990 --> 05:52.720 |
|
Because they heard it had a lot of volumes. |
|
|
|
05:52.840 --> 05:54.670 |
|
Oh, yeah, that's pretty good, right? |
|
|
|
05:54.670 --> 05:58.030 |
|
That's that's different plays on the dual meaning word volumes. |
|
|
|
05:58.030 --> 05:59.050 |
|
It gives a little explanation. |
|
|
|
05:59.080 --> 05:59.770 |
|
The joke is simple. |
|
|
|
05:59.770 --> 06:04.540 |
|
It's kind of pun that data scientists tend to appreciate as it combines technical expertise with a bit |
|
|
|
06:04.540 --> 06:05.410 |
|
of wordplay. |
|
|
|
06:05.440 --> 06:06.790 |
|
I think that's solid. |
|
|
|
06:06.790 --> 06:07.900 |
|
That's solid. |
|
|
|
06:08.140 --> 06:10.630 |
|
Uh, I think B is better, is what I want to go with. |
|
|
|
06:10.660 --> 06:12.310 |
|
Hopefully you agree with me. |
|
|
|
06:12.340 --> 06:17.560 |
|
Uh, if not, you need to go into the arena and do, uh, do a ask a similar question and make your |
|
|
|
06:17.560 --> 06:23.570 |
|
own vote, but I'm going to vote on your behalf on all of our behalves by pressing the B is better button, |
|
|
|
06:23.570 --> 06:25.730 |
|
and it will then reveal the models. |
|
|
|
06:25.730 --> 06:26.600 |
|
Let's see. |
|
|
|
06:26.900 --> 06:34.490 |
|
Uh, interestingly, the we picked Claude three haiku and we picked that against grok two. |
|
|
|
06:34.610 --> 06:39.740 |
|
Uh, if I remember right, grok two was quite significantly higher than Claude three haiku. |
|
|
|
06:39.740 --> 06:45.050 |
|
So we have slightly fractionally shifted that balance through our vote. |
|
|
|
06:45.050 --> 06:47.030 |
|
So I hope you enjoyed that. |
|
|
|
06:47.060 --> 06:52.130 |
|
Of course, that was somewhat unscientific, but the idea is it's meant to be a human decision based |
|
|
|
06:52.130 --> 06:53.690 |
|
on your interaction. |
|
|
|
06:53.900 --> 06:57.260 |
|
And the ask for you now is go and do this. |
|
|
|
06:57.290 --> 06:59.270 |
|
It's an important way to add back to the community. |
|
|
|
06:59.300 --> 07:00.980 |
|
It is data that will be used. |
|
|
|
07:01.010 --> 07:03.020 |
|
Uh, plus it's a it's a lot of fun. |
|
|
|
07:03.050 --> 07:09.620 |
|
Uh, and it gives you a bit of sense of the capabilities and scope of the different models and some |
|
|
|
07:09.620 --> 07:14.630 |
|
hands on experience whilst contributing back to the data science community. |
|
|
|
07:14.630 --> 07:20.930 |
|
So enjoy spending some time on the chatbot arena, and I will see you next time to talk about commercial |
|
|
|
07:20.930 --> 07:21.710 |
|
uses.
|
|
|