I mentioned before an AI company called vellum.
When we were talking about the different questions, they had come up with that question about scales
on reptiles that we asked GPT four zero and zero one preview vellum publishes a leaderboard, which
we will come back to in future weeks, but it has a lot of super interesting information that ranks
and compares different llms.
And there are a bunch of different leaderboards that we'll look at when we talk about picking the right
LLM for the task at hand.
It's a good one to bookmark, but in particular, there's something useful on the vellum website, which
I don't think I find somewhere else that you can also download should you wish.
According to that pop up, and it's right here, about halfway up on the Valimai LLM leaderboard website,
and it's a table that compares the context windows with the costs of the different frontier models.
And it's really useful to have to hand, particularly when you're talking about costs.
First of all, let's just mention the context windows that we talked about.
So the model with the largest context window is Gemini 1.5 flash, and it has an outrageous context
window.
Outrageous.
1 million tokens.
And you may remember a million tokens means for normal English language about 750,000 words, which
isn't far off the complete works of Shakespeare.
If you remember, I think we thought that was 1.2 million tokens, perhaps.
So you could almost fit the complete works of Shakespeare in one prompt into Gemini to have it to continue
generating Gemini 1.5 flash.
So that is extraordinarily wide context window.
The Claude series of models have a 200,000 token context window, and the GPT set have mostly 128,000
token context window.
And again, remember that that means that the all of the series of prompts in one conversation, the
input in a conversation, the response the next input, the response, and also the system prompt at
the beginning.
They all have to fit within that context window.
And actually also all of the tokens that get generated in the output, because each each token is generated
one at a time and then passed in and it generates the next token given all of the input and output that
came before it.
So that's what has to fit into the context window.
But there's a lot of space for that in 128,000 tokens.
I guess it's what about about 100,000 words.
So that's that's how to look at the context window size.
And now over here we get a sense of the cost.
And at first blush you might think this doesn't exactly look like it's super cheap if you take something
like Claude 3.5 sonnet, which is the one that I frequently find myself using in the most powerful LLM
on the planet.
By most measures, you'll see that its cost is $3 per input token, $15 per output token.
It sounds quite expensive, but no, it's because it's not $3 per input token.
It's $3 per million input tokens, $3 per million input tokens.
So if you passed in in your input prompt, the complete works of Shakespeare, then you'd be paying
a little bit more than $3.
So I just want to get across that whilst these numbers are meaningful and they are real numbers that
one needs to watch for, and you can bring up the dashboard with each of these and monitor carefully
your API costs.
Typically, the kinds of things we're going to be doing on this course are asking short business questions,
summarizing things, trying to draw quick conclusions.
You'll find that you will be costing less than a cent for most of them, less than than a fraction of
a dollar.
The model that we will use most often is GPT four mini, and that will cost us $0.15 per million input
tokens and $0.60 per million output tokens.
And so, as you can imagine, you have to generate a fair amount of content in order to be able to scratch
the surface in terms of a real price, and bearing in mind that these are large models and a lot of
compute is required to run these and be able to deliver the quality of results we get.
These API costs do seem to me to be to be very reasonable in the circumstances.
So I hope this gives you some some, some clarity, some perspective on what these costs are like and
what they mean.
The total cost is then the sum of your cost for your input tokens by by this number, plus the cost
of any output tokens that got generated.
And in the APIs you can specify a a maximum number of output tokens to generate to stop it from going
too far.
Uh, so that gives you a good sense of the costs.
And you should bookmark this page and have it as something that you come back to anytime that you want.
A little bit more clarity on how to think about these costs.
And as I say, the time when it really becomes meaningful is when you're building a system that's going
to maybe make a thousand calls to the model, and then you have to really bear in mind, how are these
costs going to add up?