So what you just saw was an ephemeral app, as it's called, which means just a temporary app that you're
just using for, for, for testing before finally picking an app to deploy.
And we're going to stay with the ephemeral apps for a little bit longer, because we're now going to
take our proprietary model that we built last time and deploy that as an ephemeral app before we we
do it for reals.
So in order to do that, let me show you this other file here called Pricer ephemeral.py, which is
a Python module which is going to have some similarities with what you just saw, but with one key difference,
which is instead of using llama 3.1, we're going to use our own fine tuned model using very similar
code to the code that we used in the codelab when we were running our model in inference, when we were
testing it, and had the wonderful moment when it when it did so well.
So in this, uh, Python module, then we begin as before by importing modal.
And again I will remove volume.
At one point I was going to use that, but I ended up not.
So you set up, you say that you have an app.
It's called Pricer.
That's going to be the name of the app you saw.
Pricer.
You saw my deployed version of Pricer before.
Um, and now we create an image and we install the same packages.
But wait, there's a new package.
You'll remember the parameter efficient fine tuning package that we need in order to be able to load
in our fine tuned weights and our secrets.
Hopefully you found your way to adding them in and you should put them under HF secret.
Or if you put it under something else, then obviously change the name here.
That's the name of the the the category of secrets.
And then within that you have to have the exactly the usual um, hugging HF underscore token for, for
the token itself.
Um, see if that's actually in the code somewhere.
Um, I don't think it is.
Um, anyways, so so if that hopefully that that that makes sense.
Um, so, um, we then um, specify we want a t4 GPU.
We specify the base model of course.
Llama 3.18 billion.
Uh, the project name.
This is because we're going to be pulling it from the Hugging Face hub.
Uh, so this is my hugging face username.
If you want to use my trained version, then keep my name there.
If you want to use your version because you fine tune the version and maybe you made it even better.
Maybe you've done some hyperparameter optimization and you have beaten me, in which case you should
definitely use year one instead.
Um, but this is mine, and if you use mine, then you should use that run name.
And this revision number, which you'll remember, was the epoch where I got the best results before
it started to overfit.
And so this ends up with the fine tuned model name.
And so we have a simple function price.
And it's a function which is almost identical to the function that we had in the Google Colab.
It's just we've got it sitting here on my local box, and we're going to decorate it with this decorator
that says that we want to run it in this image with these secrets and this GPU.
But this is all the same stuff.
We do some imports.
We note the question and the prefix.
The prompt is going to be how much does this cost to the nearest dollar.
Then the description of the product and then price is dollars.
And then it's going to be almost unbearable for the model not to respond with what the price is.
And then here we do the quantization.
We load the model as before.
And this line hopefully you remember this.
This is where we load in the the Laura fine tuned model.
We provide a base model.
We provide the name of the fine tuned model and in this case, the revision number.
You can leave that off if you're just taking the final save.
You don't need to take an interim point.
The way that I am.
This is for reproducibility.
So that it's the random seed is set.
So you get the same numbers each time we tokenize.
We then make the call dot generate to our fine tuned model and we decode.
And then finally hopefully you also recognize this code.
This is a bit of janky code that strips out the number, the floating point number from within the text
that comes back, just in case there's an errant dollar sign or something like that, and makes sure
we get whatever number we can, um, and returns that number, and that's all there is to it.
It's code that should be pretty familiar to you at this point.
So now we go back here.
Um, I did already run it because as before, because it needs to warm up.
It takes a couple of minutes.
So it will be agonizing if we had to wait each time.
But I already ran it.
This is, of course, what I ran.
You say you don't need to actually have this with modal enable output.
But if you if you don't do that there's lots of other stuff prints.
So this is a way to make it so that it's a nice and neat.
Um, but you do need to have with app dot run.
Um, and then I just say result is priced remote.
And then what I'm asking for is to price a particular object, um, that, uh, just, uh, looking
around for something that I could ask it to price.
And the object that I've asked it to price is the very microphone that I am talking into right here.
So I just typed it in there.
I remember how much it cost.
And so why not?
Let's see how it does estimating that.
Uh, so I ran it.
It did take a couple of minutes.
Now it takes a couple of minutes because it has to warm up that box, uh, and, and populate things,
um, because it has to do everything you can imagine you'd have to do if you ran this in Colab.
It has to download from the Huggingface hub and then load it into memory.
Um, it also has to pip install all of those packages.
So there's a lot that has to happen, which is why it takes a bit of time.
Um, and it did all this stuff and at the end it completed and it returned the number one, 33, $133.
And I think that is round about what this costs.
So, uh, yeah, it's, uh, I think that's pretty much, uh, a spot on, uh, so maybe it was it.
Who knows?
It could have been in the training data set, but but, uh, certainly a nice result.
So hopefully you're doing this to you're trying this.
You will have to wait those two, two, three minutes.
You'll be able to go onto modal while you're doing it, and you'll be able to go to ephemeral apps.
It's no longer running, but while it was running, it will have been here, and you'll be able to watch
it and see it doing its thing and then, uh, get a satisfactory result.
Okay.
But now we've done that, it's time to make the transition from these ephemeral apps, which is just
ways of running functions remotely.
Now we're going to have deployed apps, which is when we tell Modell that we've got some functionality
and we want to have it be, um, permanently sitting on.
Modell will still only pay for it when we actually run it, but we do want to give it a sort of a name
and a proper place where we can quickly get back to it if we need to.
Um, and the way you do it is on the command line, you type model.deploy and then the name of the app
that you want to deploy.
Uh, and it's important to note that this this really this is the moment that we are taking some code,
some model that we have built as data scientists, and we are deploying it in a way that can be used
for production purposes.
It's going to have a Python API behind it.
People will be able to call it, um, from Python and be able to to invoke our API.
And this is an example of productionizing, something that we've built in originally in JupyterLab.
Um, it can also modal also allows you to build Rest endpoints very easily so that it doesn't need to
be called by Python.
It could just be called from any any HTTP query.
Um, we're not going to cover that in this class because everything we're doing is in Python.
So we might as well just just call it from Python.
Um, but it's very easy to do and the docs are super clear about it.
So, uh, one thing I glossed over a moment ago, you may know this already.
Uh, Jupiter, uh, has this great feature that if you want to, you can run something in terminal just
by doing an exclamation mark.
And then the the name of the command you want to run.
You can also actually open up a terminal window by pressing the plus button there and, and create a
terminal.
Uh, but this is even easier because you can do it right in the prompt here, which I did above.
And the command again is modal deploy and then the package name.
So it's priceless service.
So let's go and have a look at this package called Price of Service.
Let me quickly tell you what I've got here.
So uh it's, uh, remarkably, uh, similar.
Um, you can see that basically we, we have, uh, the, the constants, we define the function, uh,
and uh, it's basically the same thing, but this is going to be enough to deploy, uh, our code as
a service, as a, as a proper deployed service, rather than the ephemeral version of the process that
we had before.
Um, but yeah, it is the same thing.
It's the identical code just packaged differently.
And again, with this volume that's not required.
Uh, uh, so on running that if we go back to the day one notebook, uh, you'll see that it deployed
it to as Price's service.
Um, it gives me a little URL at the end of it.
Um, and then once it's done that if I go now back here and I go to deployed apps, you'll see that
price is running there.
Um, sitting on a T4 GPU box, just as we specified.
If we come back here, the way that you call a deployed app is different and actually simpler than the
ephemeral app.
You don't need that.
That, um, scaffolding you saw a moment ago.
Instead you just say modal dot function, dot lookup, and then you give the name of the service and
then the name of the of the function.
Simple as that.
Modal dot function.
Dot lookup the service and the function.
Um, and so once you've run that you can then say price dot remote and make the, the remote call.
And we get back the same number 133.
And that's all there is to it.
We've just deployed an app with our proprietary model to the cloud, and we've called it and it's run
on the cloud.
Uh, now there is a nicer way of doing this, a fancier way of doing this.
When we build it this way, every time that it goes to sleep and needs to be warmed back up again,
which, by the way, is quite quickly like after a minute or two it goes to sleep.
And so every time you call it, you're waiting all that extra time, uh, and basically every time it
has to rebuild an image and do the imports and load the model and so on.
Um, sorry.
I jumped to the version where it doesn't have to do that.
Here we go.
This is the bad version where it has to do all of that.
So I've made a second version of it.
Prices service two pi.
This module, uh, and this one is done a bit in a smarter way.
It's done using a class, and it's done in a way that allows you to pre-build some, uh, some aspects
of initialization.
So you don't need to do it every time.
Um, so when you define a class like this, you have the ability to use this decorator model dot build
and write a function that will populate your box the first time that it's built.
And in this case I download from huggingface the model and I put it in the cache.
And I'm not going to walk through this in detail, because I'm going to leave that as an exercise for
you in the interest of time, but it's perfectly self-explanatory.
This code will load the files from Huggingface and put it in Huggingface cache.
And then you can have this decorator modal dot enter, and this is when the function actually gets called.
And this is where it's going to set things up by setting up a tokenizer and setting up a base model
that will be loaded once.
So it can be used repeatedly.
If this is called before it goes to sleep.
Uh, and this is the method itself price.
This is now the same as before, but the the code to load the model into memory has been pulled out
into this one here into modal into setup under the the A decorator modal or enter.
And so as a result of all of this, if I haven't lost you yet with this, when I go back to day one,
the key point, uh, I then do a modal deploy of process service two.
So process service two gets deployed.
And when I've done that I can then use this code slightly different.
You remember this is what we did last time we did modal dot function dot lookup.
Now we do modal dot class which annoyingly is spelt cls.
Um they probably class is probably something that is already used.
So they couldn't use that.
So modal dot dot dot lookup and then price of service and price now with a capital P price.
So it's the class.
So this will will find the class that we've just just deployed.
You can instantiate that class.
So I say price is is an instance of price.
And then I can call Pricer dot price dot remote.
Uh and when I do that, I get back that answer again.
And when I ran this before, it ran super fast because it was already in memory, but now it's probably
swapped out already.
So yeah, I won't do it now to avoid holding us up, because we'll press on to the final example, which
I will run.
Um, I'm going to kick it off now so that I can talk while it's doing its thing.
We'll let it run.
So I have packaged this into a nice, useful class called Specialist Agent, which is our first agent
that we're going to look at.
Um, and it is something which is going to look up our pricer running on modal and call our pricer.
And it's something which just takes a single function, a single method price.
And you can pass in what you want it to price.
Let's go and look.
It's in agents.
And now you get your first preview.
Look at all of the different agents that we're going to be playing with this week.
Uh, and we're going to go in and have a look at this particular agent.
It's called specialist agent because it's using our special model.
So this is our Python module for our specialist agent.
And as I promised you'll see that I've written comments and that there's some type hints going on.
So this class is a very simple class.
It's an agent that runs our fine tuned LLM remotely on modal.
Um, it sets a name, it gives itself a name and a color.
And the reason for that, you'll find out later, is that we're going to have our agents log.
So we'll be able to see what they're all doing.
And the specialist agent is going to take the color red.
So you'll be able to see when it's thinking and doing things and so on.
Uh, so I initialize in the initialization in the constructor method here I log that, that it's getting
started.
And I do this modal dot plus dot lookup.
And I look up the price of service and the price.
And this is the thing that we just deployed a moment ago.
So we look looked that up and we instantiate it.
And then we log that we are ready for business.
And the price call itself does exactly what we just looked at.
We do price dot price dot remote.
It's just like when we did the hello dot remote.
We're taking a function.
We're calling a remote, which means it's going to run on the cloud.
It's almost transparent to us.
It's like we're calling it directly here in Python, but it's actually going to run on the cloud on
the T4 box that we specified.
We pass in the description, we log the fact that it's completed and we return the result.
So hopefully I've jabbered away for long enough that when we turn back to day one, it's going to have
finished and it hasn't finished.
Well then I can also fill in the time by flipping over here and we can take a look at it doing its thing.
Uh, so here now, before we remember, I got lost looking for ephemeral apps.
We are now correctly on deployed apps because we're looking at our Pricer service or Pricer, which
is a deployed app.
We can go into this and have a look at it.
Oh, it's now saying succeeded.
Let's go back and have a look.
It has indeed succeeded.
There we go.
It succeeded and it returned the right number 133.
And if we run it again right away without delay, see how quick it is?
That's how quick it normally is when it's already warm.
That's going out to the cloud.
Let's quickly come up with something different.
Let's go for an iPhone SE second edition.
Let's see how much an iPhone.
There we go.
299 bucks apparently.
Uh, so I'm delighted that that we got to see that and, uh, iPad Pro uh, second generation.
299 again, it seems that must be a popular price.
So you get a good sense once it's warmed up.
Once you've populated the model, you can call it very quickly and you'll get back a quick response.
And again, we're calling our agent.
And what our agent is doing is our agent over here is simply it's done this already.
It's already got the class, it's already instantiated it.
And that class is already loaded in the files from Hugging Face.
And it's warm and ready for business.
And then when we call this method, it doesn't need to load in the files again.
It's got it all ready.
All it needs to do is inference.
And that's why it's so quick.
And that's that's the trick.
That's how you build an API to a model that's running in the cloud, serverless, and how you make sure
that you've cached the model weights in memory so that it's ready for action.
So I know we went through it very fast, but now you have to go back and do this again yourself.
Make sure that you can make it work and that it looks good on modal.
And I will see you in the next video to wrap up.