Welcome back to the wonderful world of JupyterLab.
And here we are in week two.
Day three.
Uh, bring up this notebook.
So we're talking conversational AI, also known as chat bot, and we're going to get right into it.
We start by doing our usual imports and we do our usual setting of our environment variables.
And we initialize OpenAI.
This time we will use OpenAI.
And you can have it as an exercise to switch in other models if you'd like to do so.
So uh, going to start with the basic system message.
You are a helpful assistant.
All right.
Now I want to talk for a bit about, um, message structure.
So first of all, uh, reminder of the structure of a prompt message to OpenAI.
Uh, we've seen this many times now, and so you're probably thoroughly bored of me explaining it.
There it is.
One more time.
You know it well.
A list of dictionaries that give the system the user, and it can have an assistant responding and then
the user and so on.
And you may remember I mentioned there is something else to come, but for now.
System user assistant.
User assistant.
User assistant user and so on.
Uh, now we are going to write a function called chat.
And that function chat is going to take two inputs message and history.
Message represents the current, uh, message that is being being asked that chat needs to respond to.
And history has the history of all prior messages, all prior exchanges.
And the structure of history is going to look like this.
It's going to be a list, a list that consists of lists.
And these sub lists like this are simply what the user said and what the assistant replied, what the
user said and what the assistant replied and so on.
So why am I asking you to do that?
Why are we going to write a function that looks like that with parameters, that arguments that look
like this?
The answer is because that is a particular type of function that Gradio expects for using with its chat
user interfaces.
And that's why Gradio expects us to write a function called chat that's going to take a message.
It's going to take history in this structure, and it will return the next the response, the response
to this chat.
So that's why we're thinking about that format.
So our job for this function is going to be convert this kind of style of message into this.
So we're going to need to iterate row by row through this structure and build this structure that you
see above.
Hopefully that makes sense.
If not, it will make sense when I show you what that looks like.
So I'm defining a function called chat.
It takes a message that we got to respond to an input message, and it takes history of prior messages.
So we first of all, we we set up our list of messages, which is going to be this guy.
And we populate it with the system prompt at the very start.
Of course, then we are going to iterate through history.
Each element of history again is one of these lists with two values.
So we're going to unpack that into user message assistant message.
And then we append the user's message and the assistant message.
Uh each each time.
So each row from from history turns into two rows in this list.
One for the user, one for the assistant.
Hopefully that makes complete sense.
Now.
If not, you can always.
Oh well, you don't need to.
I was going to say you can always put in some print statements.
Uh, I had the foresight to put in some print statements myself.
So we will see this.
We're going to print the history And then we're going to print the messages.
So we get to see that too.
Um, and then the next line is very familiar to you for this particular chat, uh, method at this point,
this function, sorry, at this point we are then going to take, um, this set of messages and we're
going to call OpenAI with them.
So we do OpenAI chat dot completions, dot create.
We pass in the model, we pass in the messages, and we're going to say please stream results.
We might as well.
And then we go through and we yield response.
Um, so again this isn't actually a function.
It's really a generator because we're going to be yielding the responses piece by piece.
Okay.
So what I want to do now is turn this into the kind of user interface that you saw in the slide a moment
ago, a user interface which has an instant message style interaction.
So obviously there's a bit of work to do there because we're going to have to, to to craft that kind
of, um, canvas with the messages that come one after another and figure out how to do that.
Um, based on the response that's coming back from this chat message.
Uh, I don't know if you've cottoned on, but I am, of course, fibbing.
It's going to be really easy.
It's going to be a single line.
Uh, so Gradio comes with something called chat interface out of the box, and chat interface, uh,
expects a single function which needs to have this structure.
If you've written a function which takes a message and history in this particular format, then for
Gradio it's just a single line of code.
Uh, let's see if it's really that easy.
I do need to remember to execute that so that we have defined our chat generator.
And then we will launch our interface.
And here it is.
Here is our chat interface.
Let's bring it up in a separate window, because I just prefer it that way.
And we'll say, uh.
Hello there.
Hello.
How can I assist you today?
I want to buy a tie.
Great kind of tie are you looking for?
Do you have a specific color, pattern or material?
Uh, so you get the idea.
But let me just say, um, a red one red tie is a classic choice.
Here are a few options to consider.
And there comes the answer.
Now, obviously the reason I said a red one is I wanted to demonstrate what you already know, which
is that it has context of this conversation and it knows what came before.
And one more time, it's a bit of an illusion to feel as if this thing has memory from when we first
spoke to it.
And I said, I want to buy a tie.
All that's happening is that every time we interact, that chat method, function generator, I get
it right eventually.
That chat generator is being called.
And what's being what's being passed in is the whole history so far.
And it's building that set of messages and that's what's being sent to OpenAI.
So for each of our calls, the whole history is being provided.
And that's why it has the context of what came before.
It's not as if the LLM, it's not as if GPT four is remembering that that 30s ago we said that.
It's just that with every call, we pass it all in.
I'm sure it's obvious to you at this point.
So I'm sorry I'm belaboring it, but I think it's important to, to really rub it in.
Um, and yeah, so you remember I have some print statements happening below which are going to be quite
chunky now, but let's just look at the the last one there.
So the last one said history is and then this is what Gradio sent us.
And you'll see it's like uh, what we said, what it said, what we said, what it said.
Uh, and then we converted that into the right format for GPT four zero.
Uh, GPT four mini.
Um, we converted it into a list of, like, role system content.
You're a helpful assistant.
And then user said, hello there.
And the assistant replied, hello, how can I assist you today?
And so on.
So that is what we turned it into.
All right, just before we go on, I'm going to have a quick tangent, but it is an important tangent.
So this isn't just me prattling on.
This is something which I want to sow a seed with you.
Something that we will come back to later and is an important point, um, which maybe, maybe something
that's been on your mind.
Um, or if not, it should be.
Um, so just to mention, you might be thinking, so this structure, this system user assistant user.
Uh, so is this.
Does this somehow get passed into the LLM in some structured way?
Like are we somehow when we when we provide this data to the LLM, is it being given maybe as a as a
like a dictionary, a list of dictionaries in some way?
Um, because you may say, I thought Llms just took tokens.
They just take a list of tokens and they generate the most likely next token.
So how does this whole list of dictionaries and so on, uh, translate to the world of tokens?
And that would be a great thought if you had that thought.
Uh, very good.
Uh, and there's a simple answer, uh, it is just tokens that gets passed to the actual underlying
GPT four, uh, GPT four LLM.
What happens is that OpenAI turns this into a series of tokens, and it has special tokens, special
ways of explaining that this is the beginning of a system prompt.
This is the beginning of a user and an assistance response.
It has some markup to say that, and it tokenizes that whole markup, including some special placeholder
tokens that sort of communicate inform the LLM.
We're now switching to system prompt mode.
Here's some system prompt text.
And now we're out of system prompt mode.
Now we're doing a user message and so on.
So this structure is what we send the OpenAI API.
It converts it into tokens.
And it's those tokens that then get fed to the LLM to predict the next token.
And you might say, okay, I hear you, I get that.
But how does the LLM know that this particular special token means system message and should interpret
that to be its its high level directive?
And how does it know that this token means user and this means assistant and so on.
Like like what gives it that ability is that, like, baked into its architecture in some way?
Uh, and there's a very simple answer to that, which is that.
No, it's just because that's how it's been trained.
It's been trained with lots of data, with that structure, with millions of examples like that.
And it's learned through training that when it's being given a specific directive in a system instruction,
the most likely next token, the most likely response is going to be one that adheres to that system
prompt.
There's I've oversimplified.
There's some, uh, more nuance there to do with with things like the technique that is like chef and
things like that.
For for those that know all this stuff are listening and saying, oh, it's a bit oversimplified, but
it's the general idea.
It's the basic idea.
This structure is this sort of the API structure.
This is how we communicate to OpenAI that that's what we want to do.
And OpenAI takes that structure and turns it into tokens.
So to to sort of take a step to, to the very beginning, Gradio gives us data in this format.
We map that to this format, which is what we send OpenAI and OpenAI converts that to tokens, including
some special tokens.
It's that that goes into the LLM for the whole conversation so far, for everything, every time it
gets the entire conversation, and then it generates the most plausible next sequence of tokens that
are most likely to come after that.
Um, and that is what gets returned to us that we then assume represents the assistance response.
So I realized that was quite a long sidebar.
It's very important, foundational understanding.
And we will come back to that when we, particularly when we look at open source models.
And we're actually going to see these kinds of generated tokens, these special tokens ourselves.
So with that, I'm going to pause it until the next video when we're going to press ahead building this
chatbot out.