So it's time to continue our journey into the world of open source and understand which models we should
pick to investigate the ability to generate open source code.
Where do we begin?
We begin, of course, with the big code models leaderboard.
This is the leaderboard running as a hugging face space that we looked at before, and it's the one
that compares the performance of open source models in code generation tasks.
You remember we started by just looking at base models, and we can see some of the base models that
are doing really well, including including Code Llama, Code Kwan and Deep Sikh Coder.
Um, but we're going to look at all models because we're interested in models that have been fine tuned
because they really do perform much, much better than some of the base models when they've been trained
specifically for, uh, for tasks such as C plus plus code.
Um, and we're going to be looking at Python and C plus plus test ability because that is, after all,
what we're going to be doing.
And when we look at these, one thing to note is that the ones that say exit here, their benchmark,
some of their scoring has happened externally.
So I'm personally going to tend to lean towards ones that don't have that symbol, that are the ones
that have been benchmarked as part of the hugging face, uh, spaces platform and the the one that is
top of the list here is code Kwan 1.57 billion, right up at the top.
The code variant of Kwan.
It's been trained for chat, which means that you can interact with it rather than just, for example,
giving it some Python code and expecting it to complete.
You can have a chat with it, such as saying, hey, could you convert this Python code to highly optimized
C plus?
Plus?
Let's go in and have a look.
We're now looking at the code Kwan 1.57 billion chat the top model.
And you can see here it describes some of what it can do.
Um and some of how you can interact with it.
Like you can say, write a quicksort algorithm in Python.
Um, and you can see that you can construct messages in much the way that we are used to, and then
use the tokenizer apply chat template to convert that in tokens and run that through the model.
So this is all very consistent with the way that we ran a number of models for other simple activities,
text generation activities in the past.
If we look at Codeclan we can see that it scores really very well indeed in Python coding and in C plus
plus coding.
It is only beaten by this one here, which is the externally run one which only fractionally does better.
And if you look, this model itself is in fact a fine tune of Quinn itself.
So I think frankly, we're doing pretty well by sticking with code.
Quinn.
There's a big drop off after that.
The other one that I was interested in showing you was I wanted to also show you Code Gemma down here,
which I thought would be great to try a different one and see Google.
But as it happens, when I try to use code.
Gemma uh, it was able to to answer very simple tasks, but it crashed whenever it was given harder
tasks.
You may want to try and see as an exercise whether you can get code.
Gemma to work, but I wasn't able to for this particular problem, at least not for any complicated,
uh, translation or reimplementation of Python in C plus plus.
But certainly an exercise for you will be to try out other models, because we have so many to choose
from and see what you find.
So we've picked, uh, Code Quan.
Let's go into it one more time.
Uh, the next thing to do is to figure out, okay, how are we going to run this various ways?
We could do it.
You already know of one of them.
We could go onto Google Colab and write the code to use the Hugging Face Transformers library to directly
access Codex code, load it in and run it in inference time.
And that would work great.
The thing is, I want to execute the compiled code on my box, so it would be a bit inconvenient.
We'd have to run that in Jupyter, in in Google Colab, and then copy and paste the code over to my
box.
And I was hoping to be able to do it all in one nice user interface.
And so basically I want to be able to run this model as it is, but from my box.
Now there are a few ways of doing that, and I'm not going to go into all of them, but there's one
that's super convenient, and it's actually going to show us a way that you can deploy models in production
for production purposes.
And that is to ask hugging face to run this model for you and to give you an endpoint which you can
use to call the model remotely from your code.
So it's a sort of hosting solution for running a model like this model, or any open source model you
want, and give you an endpoint to run it yourself.
And that sounds super advanced.
You may think that there's going to be a lot to it to do this.
It turns out it's not difficult at all.
All you have to do is go to this deploy button right here.
And the first option here is inference endpoints dedicated inference deployments for production.
And it's saying this is a way you can have your own deployment of this model.
And you'll be able to call it yourself from anywhere and use this model on your own.
Now of course you'll have to pay for that.
It doesn't come for free, but it's not massively expensive.
But this is one that is, it costs a little bit more.
It's something certainly to be aware of.
Um, so when you, when you come in to, to use one of these inference endpoints, you choose whether
you want to run on AWS or Azure or GCP, Google Cloud.
And you can also choose whether you want a CPU box or a GPU box.
In the case of this model, we would need to have a GPU box, and we would need to have at least an
Nvidia L4.
At least that is what it is recommending to us, which has 24GB of Ram on the GPU.
Uh, so we'd simply select this and then we would go ahead and, uh, fill in some of these things and
click Create Endpoint.
And it's as easy as that.
We would have our own dedicated endpoint for for your personal use running with hugging face paying
$0.80 an hour.
If you're in the US, you may see different pricing there.
So you know it's not super cheap, but you can turn it off and on whenever you want.
So for doing an experiment, if this is just going to take you an hour or two, uh, it's something
that's that's great fun to do, I recommend it, uh, if your budget allows for these things, but if
not, never fear, I've done it for you.
You can watch along and see what it's like.
I'm going to go to my dedicated endpoints using this menu right here.
And you'll see that I've got two dedicated endpoints that I've set up.
One of them is indeed Codeclan, the one we've just been looking at.
I basically followed through that screen and it's running on that box that was recommended, the L four.
So I am currently spending $0.80 an hour for the pleasure of running this endpoint.
Don't let it be said I don't do anything for you.
And I also, as I say, I tried code Gemma.
I tried it on an alpha box.
I was not able to get this to run.
It would do simple, simple stuff, but anything too complicated it would crash.
But I love this to be an exercise for you to come in and try this.
Try getting code Gemma to work, and then I'd be interested to see how it matches up with code.
But I think you'll find that just as we saw from the leaderboard, there will be very little doubt that
Codeclan is hugely superior and very, very capable of these kinds of tasks.
So at this point, you have hopefully followed this through.
You have to wait five minutes, go and get a coffee for for the codeclan endpoint to come up and running,
and then we'll be ready to get back to JupyterLab running on our box to call this endpoint and to try
out some code generation.
I will see you in a minute.