Welcome to the Jupyter Lab for day four.
It's going to look very familiar because it's actually I've started with a duplicate of day three.
And in this duplicate there's a few cells.
We're going to want to rerun the imports at the beginning connecting our environment.
We're going to want to connect to our frontier models.
Because guess what.
We're going to be able to get the frontier models to work side by side with open source models, which
is going to be terrific.
Uh, we'll get our prompts, our messages function, if you remember that something that can write to
output C plus plus file.
Um, this I don't think we need to run because it's going to be overwritten.
But then our code for our Pi program, we don't need to try it out because you'll remember that.
But we will get our code for the hard program that looks for subarrays of arrays with the largest cumulative
number.
Um, and we do need this stream, GPT function.
And we need this stream, Claude function.
And we need this optimize function, although we're about to overwrite it.
And you'll remember the gorgeous gradient functions will take execute Python and execute C plus plus,
which, uh, I hope you remember that this is something which is not to be shared with others.
You don't want to allow people to have the ability to execute arbitrary Python and C plus plus code
on your box.
That's not a good idea.
So keep the Gradio interface to yourself and don't say share equals true for that.
And then some CSS to make it look fancy.
All right.
And now we get to our new code.
And it is going to be extremely simple.
We start with some imports.
We're importing from Huggingface hub, the login function you've seen before, but also a class called
inference client.
And we're importing the the tokenizer.
We're going to log in to hugging face.
There we go.
And now just some constants.
Code is this is the name of the code model I've also got here code Gemma in case you want to give it
a try and see if you have more success than me.
This is the URL of my endpoint, so it's just taken from my endpoint page.
I can just take this this URL.
That is where my endpoint is running right now and costing me $0.80 an hour.
And this is code Gemma that is sadly paused as it was not particularly stable.
Okay, so let's give this a try.
So remember tokenizer you can create using the auto tokenizer class and say Frompretrained pass in the
name of your model and you will get the right tokenizer for that model.
And then I'm going to create this.
This is the function that we wrote before that's going to create the nice structured, um list of dictionaries
to ask for the simple function for pi to be rewritten in C plus plus.
And then we're going to use the Tokenizers apply chat template to turn that into text.
Let's just take a look at what I'm talking about here.
So if we remember to run that and then run this.
So first of all remember what messages is.
Messages is a list with two elements.
One of them has a system role.
And that's the content.
You were insistent that re-implements Python code blah blah blah.
And then user uh, and then the user method.
Now I should mention here just to in the spirit of being completely open, that I've added in a little
hint here, keep implementations of random number generators identical so that match, uh, results
match exactly.
Uh, so as you can imagine, uh, I've had to add hints here and there just to keep things on track.
And you will see the results of that in a moment.
So this is, of course, the messages that you're familiar with.
And now, if I show you what what text became after we applied the chat template, this if I print it,
it will be spaced out a bit better.
Let's see it printed.
Here we go.
Have a look at this.
So there is a special token I am start and then system and then the system message.
And then the same thing with user.
And then this is the code.
And then it ends with this little prompt.
That means that the LM is just going to keep going and give the assistance response.
So with that now you would imagine that making the call to a hugging face endpoint, making this kind
of message to an open source model and getting back the response might be really difficult.
And it's not.
It's one of those things that's just a couple of lines of code.
Here it is.
It's as simple as this.
Uh, you create an object, the inference client and you tell it the URL of your endpoint and your hugging
face token, and then you just say client dot text generation.
You pass in your text, you say, in this case I want to stream back results.
If you didn't have that, this would be even even shorter.
But it's nice to see the results streaming back.
Um, and then max new tokens, um, is where you tell it that it can keep going for a little while.
Uh, and let's see what happens.
We're getting streaming back from my endpoint, running in hugging face, the result of the, uh, of
of this conversion.
Now, one thing I will point out while it's doing this, hopefully you're recognizing this code and
seeing that it's that it looks very legit, very solid conversion of py, uh, code from Python into
C plus.
Plus, uh, it has given this, this stuff at the beginning about C plus plus code, you know, giving
a bit of an explanation.
And it's also ending with a bunch of text about why it did what it did.
And this is despite the fact that my system message insists that it shouldn't do that.
Um, it says, uh, do not provide any explanation, but I was not able, through the system message
to get it not to provide an explanation.
Although, as you remember, GPT four and Claude do obey the instructions.
So that's something I imagine you could spend more time working on that prompt to really try and make
sure that it doesn't happen.
But failing that, of course, it's very easy to do some post-processing and strip out the everything
that comes up till the, uh, the start of the C plus plus code and up until the, um, so you could
strip out everything up until this symbol here and take what comes between that and here.
So it'd be pretty easy to do that with some post-processing.
All right.
Anyway, we've seen how easy it is to run inference.
Uh, I will see you next time when we will put this together in a UI and give it a whirl.
See you then.