You must be feeling absolutely exhausted at this point.
And if you are, that is okay.
You have done a phenomenal job.
This week has been a grueling week with tons of new information that you've had to take on and from
the beginning, with the leaderboards and the arenas all the way through to our implementation of code
translators, both in with frontier models and with open source models.
We've gone through a lot, and it's been been quite a journey.
You've acquired a ton of new skills, so I'm here to congratulate you and to tell you that this last
session will be quite quick for today as we prepare for the upcoming week.
So first of all, just to say, of course, again, it was score one for the frontier.
This time we've got to let Claude win the win the show.
Uh, we had great fun writing software that can write code and translate code between Python and C plus.
Plus.
And we saw Claude, uh, rule the roost with.
It's a fantastic job of re-implementing those algorithms.
Um, at the end of today, we're just going to do a little bit more on discussing the performance of
open source and closed source models.
Um, and we're going to talk about commercial use cases for generating code.
And there's of course going to be an assignment for you.
It's a big assignment for you.
So whilst this session will be relatively short, the work for you is relatively great and I can't wait
to see what you come up with.
But first, I want to take a moment to talk about a seriously important problem, a question which is
about how do you decide whether or not your AI solution is actually doing a good job?
Uh, what is what is the what are the techniques for evaluation?
Uh, and it's as I say here, it's perhaps the single most important question, uh, because so much
rides on how you will gauge success.
It's something that needs to be thought up front and it needs to be established and worked on.
Uh, but there are actually two different kinds of performance metrics, and it's important to understand
the differences between them and to use both in the right context.
The first kind is what sometimes known as model centric or technical metrics.
And these are the kinds of metrics that data scientists live and breathe by, because these are metrics
which we can optimize our models with, and they tend to measure in a very immediate way the performance
of the model.
Now I'm going to talk to a couple of these metrics, but not all of them, because some of them are
more related to traditional machine learning.
And you may already have experience, and it doesn't matter if you don't.
But the first one there, loss is just a general term for talking about how poorly an LLM has performed
in its task, and is typically used during optimization.
You look to try and minimize loss, that is, that is the task of optimization.
When you are training a model.
The type of loss that we use most frequently in this field is called cross-entropy loss, and this is
how it works.
So imagine you've got an input set of tokens, a sequence of tokens which is your input text.
And you're trying to predict the next token.
And you have what the next token actually is in your training data.
But as part of training, you're going to try and feed in some amount of this sequence to the model,
say, predict the next token, and then you're going to have the real next token.
And you want to use something about these two results to calculate a loss.
Here's one way of doing it.
What the model actually does is it doesn't just predict the next token in the way I've been saying it
up to this point.
What it really does is gives you a probability distribution of the probabilities of all of the possible
next tokens that could come in the list.
And we may, for example, pick the one that has the highest probability.
The way you calculate cross-entropy loss is you say okay, well now we know what the real the true next
token was.
Let's find out what probability did the model ascribe to that token?
If the actual next thing that was coming was, you know, we started with hello, and the next thing
was the token for the word there.
Then let's look up what kind of probability the model gave to the word there.
And that probability is what we will use as the basis for the cross-entropy loss.
And in fact, to turn it into a loss.
Because if we just took the probability that a higher number would be better.
And loss is a bad thing, we want to we want a higher number to be worse.
So what we do is we actually take the negative log.
We take minus the log of the probability.
And that might sound a bit confusing.
Why do we do that?
Because if you take the if the probability were one, which would be a perfect answer, it would mean
that we said there was a 100% likelihood that the next token was exactly the thing that turned out to
be the next token.
So a probability of one would be a perfect answer.
Well, the negative log of one is zero zero loss.
Perfect answer.
So that works.
And if the probability is a very small number, as it gets smaller and smaller, negative log of that
number as it gets closer and closer to zero becomes a higher and higher positive number.
So again that works.
It becomes a loss.
A bigger loss is bad news.
And so taking the negative log of the probability, the predicted probability of the thing that turned
out to be the actual next token, that is called cross-entropy loss and is one of the fundamental metrics
that are used.
And very commonly with training LMS, we'll be using it ourselves at some point.
Uh, another metric that you hear about quite a lot, which is very much related, is called perplexity,
which is just, uh, e to the power of cross-entropy loss.
It means when it turns out to be, is that a perplexity of one would mean that the model is completely
confident and correct in its results.
It's 100% accurate with 100% certainty.
That would give you a perplexity of one.
A perplexity of two would be like a 50 over 50.
It's right half the time.
Perplexity of four would be a 25%, uh, probability.
So that gives you a sense.
A higher, uh, perplexity gives you a sense of of how many tokens would need to be, uh, if all things
were equal, uh, in order to, to predict the next token.
So that gives you a sense of loss and perplexity.
I won't talk about the others, but you get the sense that these are immediate ways to measure the accuracy
or the inaccuracy of a model that can be used during optimization or for analysis of a model.
So that's the model centric metrics.
What's the other kind of metrics then?
The other kind of metrics are business centric or outcome metrics.
And these these are the ones that are going to resonate the most with your business audience.
And ultimately this is the problem that they are asking you to solve.
So it's KPIs that are tied to the actual outcomes that your business people have asked for.
Maybe it's return on investment.
Maybe it's if this is meant to be optimizing something, then it's improvements in the time.
If you think about what we've just done, then the ultimate metric would be for code.
For the code solution we just built.
How much faster is the C plus plus code than the Python code?
How many times faster if it has the same answer?
So that would be an example of a business centric or outcome metric, because it requires us to to run
the full product and see what comes out at the end.
Uh, another example might be that could be comparisons to benchmarks.
If you're doing something, you're building a, uh, some sort of solution that is then going to surpass
other benchmarks at carrying out a certain business task.
So obviously, the huge benefit of these kinds of metrics is that they are tangible, they are concrete,
and they will speak to your business goals.
If you are able to move the needle on these metrics, then you've delivered impact and you can prove
it.
The problem with them, of course, is that they're not so obviously immediately tied to the model performance.
It's related to all sorts of other things, like the kind of data you've got, the environment, how
it's used, and whether the original idea really works in solving the business problem.
So there's a lot of unknowns that sit between your model's performance and the business metrics.
But the business metrics have the great advantage of actually being meaningful in the real world.
So the the answer is you need to use both these kinds of metrics.
You need to use them in concert.
One allows you to optimize your model to fine tune your model to to demonstrate its its fast performance.
And the other of them is what you use to ultimately prove the business impact behind your solution.
With that, I'm going to pause in the next session.
We're going to go back one more time and look at our coding solutions, and then talk about what you
can do to take it to the next level.