From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
349 lines
9.7 KiB
349 lines
9.7 KiB
WEBVTT |
|
|
|
00:00.920 --> 00:06.650 |
|
So to revisit then the solution that we built in the previous day and talk about the metrics. |
|
|
|
00:06.650 --> 00:13.640 |
|
As I say, we've actually got very simple business centric metrics to gauge the performance and evaluate |
|
|
|
00:13.640 --> 00:15.020 |
|
the performance of our results. |
|
|
|
00:15.020 --> 00:17.180 |
|
It's quite simply the performance of the code. |
|
|
|
00:17.180 --> 00:19.100 |
|
How much faster did we make the code? |
|
|
|
00:19.100 --> 00:27.500 |
|
And obviously Claude 3.5 sonnet crushed it and and came through as the front runner, as the leader, |
|
|
|
00:27.740 --> 00:35.720 |
|
followed by GPT four zero, which sometimes failed with the mission, but also sometimes succeeded and |
|
|
|
00:35.750 --> 00:37.670 |
|
with a fairly fast solution. |
|
|
|
00:38.030 --> 00:45.320 |
|
And then, unfortunately, Code Kwan did, as I say, a decent effort, but was not able ever to get |
|
|
|
00:45.320 --> 00:50.000 |
|
the right results despite some attempts to force it through system prompts. |
|
|
|
00:50.030 --> 00:52.610 |
|
Slightly cheating wasn't good enough. |
|
|
|
00:52.760 --> 01:00.230 |
|
Uh, so, you know, again, I do just want to point out that Quinn has 7 billion parameters and it's |
|
|
|
01:00.230 --> 01:04.370 |
|
closed source rivals way more than a trillion parameters. |
|
|
|
01:04.370 --> 01:09.050 |
|
So the very fact that it could do what it could do is incredibly impressive. |
|
|
|
01:09.050 --> 01:15.530 |
|
And remember that aside from that, that cost of running it, um, the $0.80, I was paying an hour |
|
|
|
01:15.530 --> 01:18.920 |
|
for it to be running as an inference end point. |
|
|
|
01:18.920 --> 01:20.540 |
|
Aside from that, it is free. |
|
|
|
01:20.540 --> 01:21.800 |
|
It's open source. |
|
|
|
01:21.920 --> 01:25.070 |
|
Uh, and so we're not paying API costs. |
|
|
|
01:25.070 --> 01:31.190 |
|
And there are ways that you could run it on your local box, um, particularly if you have a GPU, uh, |
|
|
|
01:31.190 --> 01:37.790 |
|
in which case, again, there's no cost other than the cost of running your box, the electricity. |
|
|
|
01:38.030 --> 01:42.110 |
|
Uh, so it's a very impressive, uh, model. |
|
|
|
01:42.110 --> 01:48.350 |
|
It's it's it's frankly extraordinary that an open source model with only 7 billion parameters can do |
|
|
|
01:48.350 --> 01:51.380 |
|
things like converting that code to calculate pi. |
|
|
|
01:51.440 --> 01:56.060 |
|
So at this point, I want to introduce the challenges for you. |
|
|
|
01:56.060 --> 02:01.540 |
|
And this week I've got a number of them and they are quite hard, but they'll be so fulfilling, so |
|
|
|
02:01.540 --> 02:05.770 |
|
satisfying, that I want to encourage you more than ever to give this a shot. |
|
|
|
02:05.890 --> 02:07.390 |
|
I do hope you have time. |
|
|
|
02:07.390 --> 02:12.460 |
|
I've tried to keep this days lectures relatively brief to free up some time so you can do it. |
|
|
|
02:12.460 --> 02:16.060 |
|
This will really solidify a lot of what we learned so far. |
|
|
|
02:16.090 --> 02:18.010 |
|
A couple of smaller things to tweak. |
|
|
|
02:18.010 --> 02:19.690 |
|
The solution we've already got. |
|
|
|
02:19.720 --> 02:28.720 |
|
You can add Gemini into the closed source models so that the Gemini is you've got GPT for Claude and |
|
|
|
02:28.720 --> 02:33.220 |
|
Gemini, and you can also add in more open source models. |
|
|
|
02:33.220 --> 02:40.600 |
|
As I was suggesting, you can add in code Llama and Star Coder, and you could see whether or not you |
|
|
|
02:40.600 --> 02:43.390 |
|
can force Code Gemma to work. |
|
|
|
02:43.420 --> 02:50.140 |
|
Um, maybe it can be convinced to not rewrite the random number generator by a series of aggressive |
|
|
|
02:50.140 --> 02:52.660 |
|
prompts to try and stop that happening. |
|
|
|
02:52.660 --> 02:54.580 |
|
Uh, see if you have more luck than me. |
|
|
|
02:54.580 --> 02:57.600 |
|
If you do, then please, please push the code. |
|
|
|
02:57.660 --> 02:58.680 |
|
Let me see it. |
|
|
|
02:58.710 --> 03:00.840 |
|
Let me include it and I'll share it with others. |
|
|
|
03:00.840 --> 03:03.390 |
|
And then we can all benefit from it. |
|
|
|
03:03.390 --> 03:04.290 |
|
And that would be. |
|
|
|
03:04.320 --> 03:04.710 |
|
Yeah. |
|
|
|
03:04.740 --> 03:05.940 |
|
Really terrific. |
|
|
|
03:05.940 --> 03:09.240 |
|
So give it a try and share your results. |
|
|
|
03:09.240 --> 03:10.470 |
|
But then. |
|
|
|
03:11.190 --> 03:19.050 |
|
But then I have three thorny challenges for you which is going to take your your abilities to generate |
|
|
|
03:19.050 --> 03:19.380 |
|
code. |
|
|
|
03:19.380 --> 03:21.030 |
|
It's going to take it to the next level. |
|
|
|
03:21.270 --> 03:24.630 |
|
So here's some really great ideas. |
|
|
|
03:24.630 --> 03:31.140 |
|
One of them would be to write a coding tool that can automatically add comments or docstrings to your |
|
|
|
03:31.140 --> 03:33.900 |
|
code, something that might be really useful. |
|
|
|
03:34.080 --> 03:38.490 |
|
I know that there's some products out there that do it already, but it'd be great to do it yourself |
|
|
|
03:38.490 --> 03:42.180 |
|
and see it working and see whether code is good enough to do this for you. |
|
|
|
03:42.600 --> 03:47.610 |
|
Uh, another would be a tool that can write unit tests. |
|
|
|
03:47.610 --> 03:53.340 |
|
Uh, unit testing is something which is the bane of some of our lives. |
|
|
|
03:53.400 --> 03:55.620 |
|
Uh, it can be very grueling. |
|
|
|
03:55.870 --> 04:02.170 |
|
and sometimes unit tests can be very poorly written if all they're doing is just sort of replicating |
|
|
|
04:02.170 --> 04:06.190 |
|
inputs and outputs of a function rather than doing something sensible. |
|
|
|
04:06.190 --> 04:15.040 |
|
So see if you can use frontier models and maybe open source models to generate unit test cases for a |
|
|
|
04:15.040 --> 04:17.410 |
|
particular Python module. |
|
|
|
04:17.620 --> 04:21.220 |
|
And then finally, this is a really fun one. |
|
|
|
04:21.220 --> 04:21.940 |
|
And I've done this. |
|
|
|
04:21.940 --> 04:26.980 |
|
So I have this myself just just for for entertainment value alone. |
|
|
|
04:27.160 --> 04:28.330 |
|
And it's great. |
|
|
|
04:28.330 --> 04:29.320 |
|
It's great. |
|
|
|
04:29.440 --> 04:37.870 |
|
And you should try it to write a code generator that writes code to make trading decisions for buying |
|
|
|
04:37.900 --> 04:41.590 |
|
and selling equities in a simulated environment. |
|
|
|
04:41.590 --> 04:43.420 |
|
It's a bigger project, I realize. |
|
|
|
04:43.420 --> 04:47.050 |
|
But you could get you could take it baby steps at a time by just having it generate code. |
|
|
|
04:47.050 --> 04:49.810 |
|
You don't necessarily need the simulated environment. |
|
|
|
04:49.900 --> 04:52.090 |
|
So you provide it with an API. |
|
|
|
04:52.120 --> 04:57.460 |
|
You could you could look for some of the common APIs that are available for looking up equity stock |
|
|
|
04:57.460 --> 04:58.120 |
|
prices. |
|
|
|
04:58.120 --> 05:03.430 |
|
Or you could just invent an API and just provide it in the prompt so that the model knows what methods |
|
|
|
05:03.430 --> 05:10.720 |
|
it can call to get the price of a ticker on a particular date, and then have it generate functions |
|
|
|
05:10.720 --> 05:14.530 |
|
that will carry out, buy or sell trading decisions. |
|
|
|
05:14.530 --> 05:17.710 |
|
And yeah, I had terrific fun with this. |
|
|
|
05:17.710 --> 05:25.780 |
|
I was able to build a ton of different trading signals, both using frontier models and open source |
|
|
|
05:25.780 --> 05:26.380 |
|
models. |
|
|
|
05:26.380 --> 05:31.120 |
|
One thing I will say is please, whatever you do, if you do this, do not actually use it to make trading |
|
|
|
05:31.120 --> 05:31.960 |
|
decisions. |
|
|
|
05:31.960 --> 05:36.970 |
|
I strongly advise you not to use these sorts of models to make trading decisions. |
|
|
|
05:36.970 --> 05:41.770 |
|
In fact, if you try and ask frontier models to generate these methods, they'll do so, but they will |
|
|
|
05:41.770 --> 05:47.440 |
|
warn you that it should not be used for real trading decisions under any circumstances. |
|
|
|
05:47.440 --> 05:49.720 |
|
So consider that a stern warning. |
|
|
|
05:49.930 --> 05:59.050 |
|
Use this purely for your own enjoyment and to to see the capabilities involved, not for actually risking |
|
|
|
05:59.080 --> 06:05.950 |
|
fame and fortune on the whim of an LM, whether it's closed or open source. |
|
|
|
06:06.250 --> 06:08.530 |
|
But this is a great project. |
|
|
|
06:08.530 --> 06:09.820 |
|
I'm sure you'll enjoy it. |
|
|
|
06:09.820 --> 06:14.950 |
|
I might share my code if no one else does, but I you know, first of all, I put the challenge to you |
|
|
|
06:14.980 --> 06:16.150 |
|
have a shot at doing it. |
|
|
|
06:16.180 --> 06:19.420 |
|
If you generate some trading signals, then I challenge you. |
|
|
|
06:19.420 --> 06:25.930 |
|
I'll go head to head with you and see in a simulated environment only, of course, which of our different |
|
|
|
06:25.930 --> 06:32.980 |
|
generated trading functions are able to make more more money in a fake market? |
|
|
|
06:33.070 --> 06:34.030 |
|
There you go. |
|
|
|
06:34.030 --> 06:38.500 |
|
That should be a really fun project for you, and I can't wait to see what people come up with. |
|
|
|
06:38.530 --> 06:41.050 |
|
And with that, would you believe it? |
|
|
|
06:41.080 --> 06:42.490 |
|
Would you believe this? |
|
|
|
06:42.520 --> 06:51.680 |
|
We have just reached the 50% point in this journey to to being a proficient LM Engineer. |
|
|
|
06:51.710 --> 06:53.510 |
|
You are half way there. |
|
|
|
06:53.540 --> 06:57.230 |
|
At least you're halfway there after you've done some of that homework and built some of those projects. |
|
|
|
06:57.230 --> 07:01.100 |
|
Because that's, as I say, the most, the best way to learn is by doing. |
|
|
|
07:01.130 --> 07:07.490 |
|
And so once you've done that, then you can pat yourself on the back and say you've got 50% of the way |
|
|
|
07:07.490 --> 07:08.120 |
|
there. |
|
|
|
07:08.480 --> 07:15.860 |
|
And there is so much more to come next time we're going to be talking about rag Super Hot Topic, and |
|
|
|
07:15.860 --> 07:19.040 |
|
we're talking about the big idea behind Rag. |
|
|
|
07:19.040 --> 07:25.310 |
|
We're going to walk through the high level flow of adding expertise into your queries, and we're going |
|
|
|
07:25.310 --> 07:31.340 |
|
to implement a toy version of Rag, a simple version without introducing these things called vector |
|
|
|
07:31.340 --> 07:35.900 |
|
databases, which is the big, the big thinking behind Rag. |
|
|
|
07:35.900 --> 07:38.480 |
|
So I can't wait to get to all of that. |
|
|
|
07:38.480 --> 07:41.300 |
|
Next week is going to be really, really exciting. |
|
|
|
07:41.390 --> 07:50.060 |
|
And again, one more time, a massive congratulations for being halfway on your journey to mastering |
|
|
|
07:50.090 --> 07:50.990 |
|
Llms.
|
|
|