Well, welcome back to Jupyter Lab for what will be an epic finale to our time in this platform.
And it's going to be such a great conclusion to the last seven weeks.
The first thing you'll notice is that there is quite a lot going on in the week eight folder.
There are a lot of files here, which reflects the fact that we've got a lot to get through, a lot
of work to be done to solve this big problem.
It means I'm going to be moving at a faster pace than usual as we go through this code.
But that's okay, because at this point you are proficient, you're well on your way to mastering LLM
engineering, and you no longer need me to belabor the point about lists of dicts of assistants and
users and things like that.
It's it's now something that is very much old news.
And yeah, it's amazing to think I was explaining what tokens were only, only a matter of weeks ago.
And and now that you're ready for a big, full on agentic AI I.
Solution a quick admin point before we get started.
Uh, the I've made some changes to the packages.
I've added some more packages that are dependencies on this environment in order to, to really make
this a very, very juicy problem.
And that just means you might need if you've if you pulled the code sometime before like late September,
you might need to pull the code again and update your packages to have anything new that I've added.
And that's very easy to do.
Um, all you have to do, of course, is go to the project home directory.
So LM engineering.
Um, and if you're on a PC, then you need to be in an Anaconda prompt.
Uh, and then you do a git pull to get the latest code, and then you run conda env update.
Um, you specify the file as environment.yml and the dash dash prune tells it to remove any packages
that are no longer in the list.
I don't think I have removed anything, but anyways, I think this is always a sensible uh, one liner.
Good to have that to hand to update your environment with a new environment.yml file.
Okay, so here we go.
Let's talk about modal.
So as we get into today's class there only is one import to begin with.
And it is to import modal.
There we go.
Now if you have not set up your tokens which you will not have done, then you need to run a command
line thing called modal setup.
And the way you can run that is you just uncomment this line and you run that statement right there.
Now I'm not going to do it because I've already done it and my environment set up.
But when you do, it's going to pop up with a browser window and have you authorize modal.
And then modal will set your environment variables for you.
And I think that's really great.
All of the challenges that people have had with tokens, particularly with OpenAI, this is a very different
experience.
It's really streamlined.
So good for modal.
They seem to have really figured out the the way to set tokens.
If for any reason that doesn't work for you, then if you go into the modal platform, you can actually
find your tokens in there in much the same way as you can for for things like OpenAI.
And then you can manually set it.
Um, but hopefully this will just work.
It certainly did for me.
And by all accounts, that's that's how it works for for people generally.
All right.
So now, uh, in the next line, I'm going to import some things from a package called hello, which
is just what I've written over here.
And I'm going to take you over to that right now to show you what it looks like.
So this is just a piece of Python.
It's just some, some Python code.
And I'm going to tell you what it does.
So we import modal.
And from modal we import a few things.
App for applications volume and image.
Um and actually now that I look at it I see that I don't actually end up using volume.
So I don't think that is needed.
Keep this simple is better.
Um, and so this is what you do.
You begin with some setup.
You tell modal what kind of infrastructure you need.
And this is an example of modal talks about this a bit on the in their docs.
But this is this is a type of infrastructure as code.
You can use code to describe what kind of box you want.
So if you're thinking when we're in Google Colab, we had to pick various drop downs and choose what
kind of, um, VM, what runtime we wanted from Google Colab, well, here you just get to choose what
you want by by specifying it in code.
So we, we say that we want an image which we want it to be the Debian operating system.
We want to pip install the requests package that very, very common standard package for, for uh,
doing URL web work.
Um, and we specify a GPU, uh, actually, which again is also not used in this example.
Sorry.
That makes it a bit simpler.
We're not going to use a GPU for this example because we're keeping it very simple indeed.
Uh, so uh, the I have got a method called hello.
It's a very simple.
It's a function.
Hello.
That returns a string.
Uh, and what it does is it imports requests, uh, it gets it goes to a website called IP info.io,
which is a useful one to know.
It's just one of these utilities that will return a JSON object which describes where you are the IP
address of, of what just called it.
Um, and so then I take that JSON, I pull out the city, the region and the country, and I say hello
from city, region, country.
Um, and it's just based on your IP address.
So it's just going to say hello from wherever you are, or at least wherever your ISP is serving you
from.
So for me, it's going to be somewhere fairly close to New York, but may not be in New York.
Um, so that's a pretty simple function.
The only potential interesting thing about it is this decorator at the top here, which is a gradient
decorator where we've decorated it with app, which is this thing here, this modal app.
Hello dot function.
And then we're passing in this image.
And that image refers to a Debian image uh which has requests installed.
And just by decorating it with that, we're saying that's the kind of box that we want to be to be able
to run this on should we wish to.
That's all there is to it.
So you you could put that to one side.
You could imagine that this was just part of an existing piece of code.
You had to do something, and now you've just decorated it with something to say that you want to be
able to run it should you wish to, using a Debian operating system with requests installed.
So that is all that's in this hello.py.
Super simple.
We go back to day one dot, the Jupyter notebook.
So I'm now going to import that.
And so now what I can do is I can call my hello I can I can take my hello function that I've, that
I've imported here.
And I can call it by saying hello dot local.
And what that means is I want to run that function that I've just defined, that we just looked at this
function here, and I just want to run it on my local box.
I want to run it in this Jupyter notebook.
Let's see what we get.
So it's running right now.
And it says hello from seaport, New York, US.
That is where my ISP is, I guess.
Let's try that again.
Yeah, that seems pretty consistent.
So that's running locally.
Uh, and, uh, now look at this one here.
It's exactly the same thing.
I've just changed local to remote.
That's the only.
The only thing I've changed.
Let's see what happens.
Let's take a little bit longer and then.
Hello from Ashburn, Virginia, us.
It's running in a completely different state, so I'm sure you you're expecting that.
You get the idea.
Uh, and sometimes it's different.
By the way, I've seen it pop up all over the place.
So simply by calling remote instead of local, you can call this function the same piece of code, the
same piece of Python code.
And it's been deployed to a server and it's running on that server instead.
So I think that's pretty magical.
It's magical because it's so simple and it's just allowed us to deploy some code.
And now we've got a slightly more involved package called llama.
And this is where things like GPUs start to appear.
You can see I paired this back to the hello example, which is why there was some some traces of what
this was.
Um, so I start by, um, uh, calling my app llama.
Um, now I'm going to again have a Debian image, but this time I'm going to install torch transformers,
bits and bytes and accelerate all packages that you know.
Well, at this point, um, I am also getting my hugging face token from my modal secrets, and that
is something I should have shown you before.
I will show you in a second how you set that up in modal, how you get to secrets, and you can set
your hugging face token in there.
And once you've done that, you can read it like this.
And I'm specifying that I want a T4 GPU, which of course is the very cheap basic one.
And I've got a constant here.
And now I have uh, this, this generate, uh, function.
And it takes a prompt which is a string, and it returns a string.
This is an example of the type hints.
If you're not familiar with them.
It's something that I'll be doing this time around in various places.
And it will, uh, I think, become clear after a bit.
So, uh, again, I decorate this function with I pass in the image, I tell it my secrets, and I say
I want a T4 GPU, all configuring my server using code.
And now this is just a function as if I were writing something to run locally.
Now, I couldn't possibly run this locally because my box has nothing like the horsepower to be able
to run a llama model like this locally.
There are ways you can do it with things like llama CP if you're familiar with that, but I wouldn't
be able to do it like it's written here, and if I did, it would be very, very slow.
Um, and so I want to do it on a box which has a T4 GPU and this is how I'll be able to do it.
Um, and so this, this code should all look very familiar to you.
This is the config that you know.
Well that puts specifies the, the four bit quantization that will have for Lama.
Um, we are going to load the tokenizer, this boilerplate stuff, you know.
Well we are going to load our model.
And then this, you should also recognize we encode the the prompt into tokens as our input.
We set this attention mask to avoid getting that warning.
And then we do model dot generate.
We pass in our inputs.
We say we only need five new tokens and we want to just one response.
We will take that one response, we will decode it and we will return it.
It's as simple as that.
Now with that, let's start that and let's run it.
So stuff is happening.
So, um, first of all, what what have I actually run?
So I've called the remote function and what I passed in the prompt that I passed in to a llama model
is life is a mystery.
Everyone must stand alone.
I hear now, uh, I, uh, hopefully most people know what comes next from that.
Otherwise I'm going to feel very old.
Uh, but, uh uh, this, of course, would be straight from the opening to Like a Prayer by Madonna.
And I hear you Call my name would be what comes next.
And it's almost unbearable not to say that.
Uh, and now I'm going to hear that song in my head for for the rest of the day.
Uh, but, um.
Yeah.
And sorry if I've put that now in your in your mind forever too, but, uh, such a catchy song.
Anyway, it's what we're seeing now is what's going on on that box.
Um, and we can also flip over to, to modal itself.
Um, and uh, let's see, go to the let me refresh this screen.
Uh, so it's not a deployed app.
Ephemeral apps.
That's what it is.
Uh, give me a second to find that.
So this is it.
Running right now.
Running as an ephemeral app, uh, generates, um, started two minutes ago.
And we can click into it and we can see that its status is running.
Uh, and, uh, let's see what else we can get.
So the containers, it's live.
We can look at the memory, the CPU cores.
And we can see it's a T4 GPU, just as we specified.
Um, and we can also look at the we can see that Hello.py is also on this box we just ran.
And llama.py uh, is sitting there, the, the Python script that we just wrote.
Uh, so here we go.
While we're here, I'm just going to point out that right at the top here is secrets.
And secrets is, of course, where you go to set the secrets, the hugging face token.
So you will need to do that.
I'm not going to click on that now to avoid revealing my hugging face token secret.
Um, but that's that's where you would go so that you can then use your hugging face token in the code
that's deployed in your ephemeral, uh, service here.
So it's still running.
And I think at this point I will hold to this video.
And when I come back, uh oh, it's already just said it's succeeded.
Uh, and I did just get a message saying that it had finished as well.
Uh, and, uh.
Let's see.
Yes.
Back here.
Let's see what we get.
I hear you call my name.
Uh, so, lama, even the four bit quantized version of Lama also couldn't resist completing the Madonna
song as well.
Uh, and so we have just successfully run this piece of code.
Um, using this ephemeral, uh, service on modal, uh, to run a quantized lama, um, uh, model,
uh, Lama 3.1 model to complete a prompt.
All right.
When we come back, we are going to go through and it's going to be time for us to deploy our model
and actually see how we do that and how we put an API around the proprietary model that we built last
time.
I will see you in a minute.