So to revisit then the solution that we built in the previous day and talk about the metrics.
As I say, we've actually got very simple business centric metrics to gauge the performance and evaluate
the performance of our results.
It's quite simply the performance of the code.
How much faster did we make the code?
And obviously Claude 3.5 sonnet crushed it and and came through as the front runner, as the leader,
followed by GPT four zero, which sometimes failed with the mission, but also sometimes succeeded and
with a fairly fast solution.
And then, unfortunately, Code Kwan did, as I say, a decent effort, but was not able ever to get
the right results despite some attempts to force it through system prompts.
Slightly cheating wasn't good enough.
Uh, so, you know, again, I do just want to point out that Quinn has 7 billion parameters and it's
closed source rivals way more than a trillion parameters.
So the very fact that it could do what it could do is incredibly impressive.
And remember that aside from that, that cost of running it, um, the $0.80, I was paying an hour
for it to be running as an inference end point.
Aside from that, it is free.
It's open source.
Uh, and so we're not paying API costs.
And there are ways that you could run it on your local box, um, particularly if you have a GPU, uh,
in which case, again, there's no cost other than the cost of running your box, the electricity.
Uh, so it's a very impressive, uh, model.
It's it's it's frankly extraordinary that an open source model with only 7 billion parameters can do
things like converting that code to calculate pi.
So at this point, I want to introduce the challenges for you.
And this week I've got a number of them and they are quite hard, but they'll be so fulfilling, so
satisfying, that I want to encourage you more than ever to give this a shot.
I do hope you have time.
I've tried to keep this days lectures relatively brief to free up some time so you can do it.
This will really solidify a lot of what we learned so far.
A couple of smaller things to tweak.
The solution we've already got.
You can add Gemini into the closed source models so that the Gemini is you've got GPT for Claude and
Gemini, and you can also add in more open source models.
As I was suggesting, you can add in code Llama and Star Coder, and you could see whether or not you
can force Code Gemma to work.
Um, maybe it can be convinced to not rewrite the random number generator by a series of aggressive
prompts to try and stop that happening.
Uh, see if you have more luck than me.
If you do, then please, please push the code.
Let me see it.
Let me include it and I'll share it with others.
And then we can all benefit from it.
And that would be.
Yeah.
Really terrific.
So give it a try and share your results.
But then.
But then I have three thorny challenges for you which is going to take your your abilities to generate
code.
It's going to take it to the next level.
So here's some really great ideas.
One of them would be to write a coding tool that can automatically add comments or docstrings to your
code, something that might be really useful.
I know that there's some products out there that do it already, but it'd be great to do it yourself
and see it working and see whether code is good enough to do this for you.
Uh, another would be a tool that can write unit tests.
Uh, unit testing is something which is the bane of some of our lives.
Uh, it can be very grueling.
and sometimes unit tests can be very poorly written if all they're doing is just sort of replicating
inputs and outputs of a function rather than doing something sensible.
So see if you can use frontier models and maybe open source models to generate unit test cases for a
particular Python module.
And then finally, this is a really fun one.
And I've done this.
So I have this myself just just for for entertainment value alone.
And it's great.
It's great.
And you should try it to write a code generator that writes code to make trading decisions for buying
and selling equities in a simulated environment.
It's a bigger project, I realize.
But you could get you could take it baby steps at a time by just having it generate code.
You don't necessarily need the simulated environment.
So you provide it with an API.
You could you could look for some of the common APIs that are available for looking up equity stock
prices.
Or you could just invent an API and just provide it in the prompt so that the model knows what methods
it can call to get the price of a ticker on a particular date, and then have it generate functions
that will carry out, buy or sell trading decisions.
And yeah, I had terrific fun with this.
I was able to build a ton of different trading signals, both using frontier models and open source
models.
One thing I will say is please, whatever you do, if you do this, do not actually use it to make trading
decisions.
I strongly advise you not to use these sorts of models to make trading decisions.
In fact, if you try and ask frontier models to generate these methods, they'll do so, but they will
warn you that it should not be used for real trading decisions under any circumstances.
So consider that a stern warning.
Use this purely for your own enjoyment and to to see the capabilities involved, not for actually risking
fame and fortune on the whim of an LM, whether it's closed or open source.
But this is a great project.
I'm sure you'll enjoy it.
I might share my code if no one else does, but I you know, first of all, I put the challenge to you
have a shot at doing it.
If you generate some trading signals, then I challenge you.
I'll go head to head with you and see in a simulated environment only, of course, which of our different
generated trading functions are able to make more more money in a fake market?
There you go.
That should be a really fun project for you, and I can't wait to see what people come up with.
And with that, would you believe it?
Would you believe this?
We have just reached the 50% point in this journey to to being a proficient LM Engineer.
You are half way there.
At least you're halfway there after you've done some of that homework and built some of those projects.
Because that's, as I say, the most, the best way to learn is by doing.
And so once you've done that, then you can pat yourself on the back and say you've got 50% of the way
there.
And there is so much more to come next time we're going to be talking about rag Super Hot Topic, and
we're talking about the big idea behind Rag.
We're going to walk through the high level flow of adding expertise into your queries, and we're going
to implement a toy version of Rag, a simple version without introducing these things called vector
databases, which is the big, the big thinking behind Rag.
So I can't wait to get to all of that.
Next week is going to be really, really exciting.
And again, one more time, a massive congratulations for being halfway on your journey to mastering
Llms.