And here we are back in JupyterLab.
It's been a minute.
We've been working in Colab for last week, and now we're back to to Jupyter and running locally, where
we will be enjoying ourselves today.
Uh, before we get to the code and looking at what we're going to do, let's remind ourselves the leaderboards,
which looked at coding abilities of frontier models just to see who is at the in the top of the pack
when it comes to coding.
So the vellum leaderboard you might remember from the AI company, vellum uh, has uh, the human eval
uh, metric, which is the simple Python test.
Um, and you can see against this metric that GPT four zero leads the way.
And then if we look down Claude three sonnet not doing so great down here.
Um, now, for various reasons, I feel like this perhaps isn't the most up to date.
I notice elsewhere it doesn't have llama 3.1 on this site, so I'm thinking that perhaps this is a bit
out of date.
I also know that human eval isn't the best of the tests, and I'm more interested in the seal leaderboard
for coding.
And if you come into this and you read their blurb, you'll see that they do a number of different kinds
of coding tests, including human eval, but also including live codebench, a bunch of programming
puzzles, and a few others.
And so I feel like this is super comprehensive.
I'm also pleased to see that llama 3.1 is on their list, and so I get the impression that this is also
more recent.
And top of this leaderboard is Claude 3.5 sonnet, followed by GPT four zero, followed by Mistral large
and open source model Mistral in third place.
So this gives us a sense of what's going on.
Let's see if GPT four are many features.
Not that I can see.
Uh, and that might suggest that we're going to need to use GPT four.
Oh, if we're going to want to, uh, really compare the top models on this front.
But you can feel free to use GPT two for mini, uh, if you'd rather be, uh, bit more frugal.
Saved a bit of money.
Uh, regardless, let's go over to Jupyter Lab and let's go into week four and into day three to see
the code for this week, when we are going to be doing code generation and building an app around it.
And as usual, this is a lot of things going on for what we'll be learning today.
One of them is actually about the problem of generating code.
Um, but we're also going to be using this as a way of exploring, comparing different models, um,
looking at leaderboards as we just have, understanding how to solve a business problem with LM solutions.
And you know what you might notice here?
There's there's this little thing here, we're going to get another opportunity to play with Gradio
and to show what it's like to package things up into a prototype, because that is just such a great
way to collaborate with others on your LM solutions.
So we're going to embark on this by running some imports, and we're then going to set our environment
variables using the usual load env.
And it's a nice time for me to remind you once again to have a EMV file.
This time we'll be using OpenAI and anthropic.
So have that set up, uh, in this, uh, cell here, we initialize the OpenAI and the cloud interfaces
as usual, and we'll use OpenAI and cloud 3.5.
Sorry.
We'll use GPT four and cloud 3.5.
Sonnet uh, which were in the top two positions in that leaderboard.
So now we have, uh, it's time to develop our system message and our user prompt.
And I'm using the same approach that we used back in the day.
It feels like an age ago now where system message we just hard code it.
And the user prompt we have something where we pass in a variable and generate the user prompt for that
variable.
So the system message I've gone with is you're an assistant that re-implements Python code in high performance
C plus plus four and M1 Mac.
Obviously I'm using an M1 Mac right here.
Um, and I suggest that you substitute in here whatever kind of environment you have to make this most
appropriate for you and you may need to do some some tweaking, particularly with the C plus plus setup
to make sure that this works for you.
Respond only with C plus plus code.
Use comments sparingly.
Do not provide any explanation other than occasional comments.
The C plus plus response needs to produce an identical output in the fastest possible time, so this
is a little bit more wordy than the prompt I showed you in the slide a second ago, but this is what
I found worked best with some tweaking around, and you'll see that the user prompt is even more wordy.
Rewrite this Python code to C plus plus fastest possible implementation.
It's a bit repetitive.
Um, and then just here you can see I've cheated a little bit from doing some of my experiments.
I found actually, as you'll discover, uh, maybe this is as suggested by the leaderboards.
Claude didn't need this extra hinting, but GPT four did need this.
Otherwise, the C plus plus code it generated didn't work.
Um, I had to say pay attention to number types to ensure that there are no overflows.
And remember to Hash include all necessary cplusplus patches such as or packages such as.
I even had to actually explicitly name a particular package, which if I didn't, uh, GPT four would
generate the cplusplus code, but not correctly include that package.
So for whatever reason, that's something that I ended up having to do.
Uh, maybe when you try this out, you'll find that doesn't happen.
You'll find a better way to prompt it without needing to be quite so directive.
Uh, it feels a little bit like that's cheating for GPT four, and we should disqualify it.
But there we go.
Anyway, with that in mind, we now, uh, run this function to we've now defined a function to create
this user prompt.
And then this section here will be very familiar to you.
Uh, messages for uh is where we create the list.
We know so well now, uh, with two elements, uh, the role system for the system message and role
user for the user prompt.
Um, so that generates that messages list given Python, uh, and now Little utility function called
writeoutput.
That will take some cplusplus code, and it will just strip out anything in there that, uh, shouldn't
be there.
There's there's, uh, the models tend to respond with this keep at the top and this at the bottom.
And so I just replace that with, I just remove that from the, from the text and then save it to a
cplusplus file called optimized dot cpp.
So when this runs we will see a file appearing in our directory optimized cpp.
And when it's called.
All right.
And then here is uh function optimized GPT that is going to call the GPT API.
We're going to call OpenAI dot chat dot completions dot create.
Why do you know that that call by now, uh, model equals OpenAI model messages is and now we pass in
the messages for Python and we set that to be streaming.
And we do for chunk in stream.
That means that the results come back and we print each little chunk as it comes back.
And then at the end we write this to a file.
Hopefully I don't need to go through this because this is super familiar to you.
Now you've seen this a hundred times and side by side with it, here is the equivalent version for Claude
doing the same thing.
We're going to call Claude.
Messages dot stream for the Claude model.
Uh, we you remember in Claude's case, we have to provide the system message separately to the user
prompt.
So there we go.
This is, again, a construct you're very familiar with.
We have to tell it the maximum number of tokens.
And then this is how we do the streaming back.
Same kind of thing.
Printing writing the output.
All right.
At this point, because we're getting ready to try this out for reals I will execute these two.
And then I'm going to to pause for the next video.
And in the next video you see us, you'll see us giving this a try and seeing how GPT four and Claude
3.5 sonnet perform when faced with this challenge.
See you then.