This now brings us to an extremely important property of LMS called the context window that I want to
explain clearly.
The context window is telling you the total number of tokens that an LLM can examine at any one point
when it's trying to generate the next token, which is its big job.
Its job is to generate the most likely next token, given a number of tokens that have come before and
the number of tokens that it can examine, that it can look at in order to make that prediction is limited
by the context window, which is limited itself by the size, the number of parameters in the LLM and
the the the way that it's been constructed, the architecture of the LLM.
So what does that mean?
Is that just saying that's how many input tokens that you can have for it to make an output token.
Kind of.
But it's worth clarifying what that really means in practice.
You remember at the beginning we had when we were when we were first working with our first LLM, our
first OpenAI call, we had a system prompt and then a user prompt.
This was all part of our input to the LLM, and it was then predicting the most likely next token in
our input prompt.
We passed in a website and it was then and we said, now summarize.
And then the most likely next token was a summary of that website.
Now when you have a chat with something like ChatGPT, you pass in some input and it then produces some
output, and then you might ask another question.
Now in practice, it appears that ChatGPT seems to have some kind of a memory of what you're talking
about.
It maintains context between your discussion threads, but this is something of an illusion.
It's a bit of a conjuring trick.
What's really happening is that every single time that you talk to ChatGPT the entire conversation so
far, the user prompts the inputs and its responses are passed in again, as are the long prompt.
And then it ends with okay, what is most likely to come next given all of this conversation so far?
So what the context window is telling you is that this is the total amount of tokens.
The total amount of information, including perhaps the original prompt the system prompt.
The user prompt the question you made its response.
Your next follow on question, its response to that, your follow on question then and now.
It's having to generate new contacts, new contacts, new tokens to come at the end of this, this long
chain of backwards and forwards.
So the context window then is the total of all of the conversations so far, the inputs and the subsequent
conversation up until the next token that it's predicting.
So it's it's it's important to have that in mind when you're first starting a conversation.
The context window only needs to fit just the current prompt.
But as the conversation keeps going, it needs to be able to fit more and more of what's been said before,
to be able to keep that context.
Um, and so this is particularly important in things like multi-shot prompting and so on.
And for example, if you wanted to ask a question about the complete works of Shakespeare, you would
need to have in the context window, the 1.2 million tokens all at one time.
That's how it works.
So in a nutshell, that is the context window.