Welcome back to Jupyter Lab and welcome to Day Five's Lab.
And this is going to be lots of creativity and hopefully lots of entertainment.
So to start with I have copied the day four Jupyter Lab.
And I've duplicated that.
And then I've extended it.
So everything above where I am now is just a repeat of day four.
Creates the AI assistant for our airline called flight flight AI, something like that, and arms it
with a tool to be able to get ticket prices.
All of that is already there and I've executed it ready for our showtime today.
We're going to go multi-modal.
We're going to use Dall-E three, which is the image generation model that sits behind GPT four.
We're going to use it to make some images.
And let's start by putting it into a function called artist.
Before that, there are two, uh, service announcements I should make.
Uh, first of all, I should point out that the price associated with generating an image is not tiny.
Everything that we've done so far, I hope, has had a de minimis price in the fractions of a cent.
Unless you've been generating tons of lengthy brochures, you have not racked up a significant bill
from running this course so far.
But now we are doing something that's slightly more on the radar.
Each image that we generate will cost $0.04.
Now, I put it to you that when you see these images, you will agree that they are well worth $0.04
each.
And they are super creative and high value.
And I love them.
So I think it is money well spent.
But I do want to inform you of that so that you can decide whether you want to spend your $0.04 each
time.
Uh, the other thing to mention is that there is a little bit of, uh, um, uh, there's there's a
there's a point about whether or not one should use the term LM when referring to image generation and
audio generation and the like.
Text to audio.
Because of course, these are not large language models sitting behind the scenes.
Now, what tends to happen these days is that people use LM as a bit of a general term for the models
that sit behind gen AI systems.
So actually, in practice, I think this very much is part of the skill set and toolkit of an LM engineer.
But I should mention that, of course, strictly speaking, these aren't language models.
These are image models and audio models that we'll be playing with right now as we add them to our agent
framework.
Anyways, with that preamble, let's get on with it.
So we start by importing some useful image libraries.
Well, the first one isn't.
First two aren't image libraries, but some, some, uh, utilities.
And then the Python image library is going to be very useful for us, a very handy common library.
Uh, so the next thing we do is we're going to write a function called artist and artist calls OpenAI
dot images dot generate.
So it's a very consistent style that you're used to OpenAI images generate.
We pass in the name of a model.
In this case, the model is Dall-E three.
You could also try Dall-E two, its predecessor.
The images are less awesome.
It's a bit cheaper.
I seem to remember it's about $0.02 rather than $0.04, so it's not massively cheaper and in my opinion
well worth the extra $0.02.
Stick with Dall-E three.
We give it a prompt and this isn't now a clever list of dictionaries.
It's just text.
And in this case, the prompt I'm suggesting here is, we say, an image representing a vacation in
city, showing tourist spots and everything unique about city in a vibrant pop art style.
We give it a size that is the smallest size.
Dall-E three will do, Dall-E two will go much smaller.
Um and Dall-E three also does two larger sizes in a portrait and landscape format.
Just google it if you'd like to know those dimensions.
If you'd like to try those images.
We just want one image back.
We say we want this format.
Back comes something in the, uh, this um, uh, base64 encoded format.
We then decode that into bytes, and then we then create a bytes IO object on those bytes, which we
can then pass in to the image dot open function, and that will return an image for us.
Let's execute that.
And now let's give it a try.
So I'm going to say image equals artist.
And what shall we say New York City.
And then display image is the Jupiter way of then getting that to show.
Let's run that or you're seeing one I ran already there.
Sorry it's not that quick, but look how amazing that is.
Uh, you're already getting.
I'm spoiling you by showing you one right away.
This is what it looks like.
It's generating a second one above you get to see the Statue of Liberty, a few different Empire State
buildings, some planes in the sky, and then a sort of image to Times Square with lots of signs and
with New York, spelled out their taxi.
Look at that.
A yellow New York taxi.
And Coca-Cola.
And a hot dog.
A very New York iconic thing.
Fantastic.
Meanwhile, it's built another image for us here.
And.
Wow, look at this one.
It's different.
It's great.
It's got a big jet over here.
It's got the Empire State Building, of course, multiple Empire State buildings, Statue of Liberty's.
And it's got again the sort of thriving shops and taxi in the foreground like that, an iconic New York
taxi and a hot dog again.
Uh, so the thing to mention is that these images, they're so creative and they're so different, we've
got two now that we can see the one I did a moment ago and this one here, uh, and you can see how
great they look.
All right.
Well, I hope that you were entertained by that.
And by all means, can I suggest spend some $0.04, generate a few images for yourself.
They're great.
All right, let's add one more function.
We're going to make a function that uses OpenAI's speech to generate some audio.
So we're going to use a couple of utility stuff here with a library called Pi Dub.
That's very useful.
We're going to write a function called talker.
And talker is going to call OpenAI dot audio dot speech dot create.
So if we look back up the image generation was OpenAI images generate.
And for audio it's a case of uh OpenAI audio dot speech dot create.
We pass in a model.
Um, and this is the model we're using, TTS one TTS stands for text to speech and is, uh, the, the
this kind of model that we're going for, we supply a voice.
In this case we're going to try the voice.
Onyx.
There's something like eight different voices to try again.
You can you can Google to see what they are.
And we pass in the thing that this function was called.
With what comes back, we again create a bytes IO object to represent those bytes.
And then we use this to this audio segment, uh creating it from a file and the audio stream and get
it to play that audio.
So let's create that function and then let's say talker.
Well hi there.
There we go.
As simple as that.
Uh, let's see how another voice sounds.
Let's see how alloy sounds.
Let's put alloy in there.
Well hi there.
And that was alloy.
I think we'll stick with onyx.
But you can try either.
And you can also put in some more there that you can experiment with and pick your favorite.
All right.
Well that's what we'll go with.
Uh and now let's talk about the agent framework.
I think we will break for the next video.
And that's where we'll take on our full agent framework.
See you then.