So I hope you enjoyed our first bash at Scrubbing Data, and that you are now looking through the code,
convincing yourself that I've done good things when I have cleansed the data and I haven't thrown out
anything that we should be including for the model.
If you do find something like that, please let me know right away, because I love squeezing a bit
more juice out of the model and making the results a little bit better, and changing the data.
Improving the quality of the data that goes into the model has had by far the most profound impact on
the performance of the model.
You can spend huge amounts of time doing what they call hyperparameter tuning, which we will all be
doing for for a while to try and squeeze a little tiny bit of juice out of your model and make it a
little bit better at predicting outcomes as we try and go head to head against frontier models, which
is going to be hard.
Um, but it turns out that improving the data quality of what goes in is the easiest and most effective
way to move the needle.
So that's why it's worth spending the time doing it.
And in that last Jupyter notebook, we we really worked on numbers one, two and three on this list,
and some of four and five.
We've got more of four and five to go, but most important is that you look through that Isom class
and see how it's doing its thing yourself.
Now just before we we we wrap up, I do want to measure and mention again the topic of performance evaluation,
how we're going to decide whether or not we have actually succeeded in our goals with this model.
Uh, there are, as I mentioned a while back, these two kinds of of metrics that you can use to evaluate
performance.
There are model centric metrics which are more in the sort of data scientist's domain of understanding
how how well your model is performing mathematically.
Uh, and then there are outcome business centric metrics, which are the ones that will resonate with
the, the the user, the sponsor, the the CEO.
They're the ultimate goals of the project.
The model centric ones are much more directly measurable by the model, and can usually be measured
while it's training and optimized.
The business centric ones are the ones that matter the most, but not necessarily as immediately tied
to the model.
Although in our case, we have this wonderful luxury of having business centric metrics which are going
to be tied to the model and are going to be great first on the model centric metrics.
So you'll often hear talk of training loss and validation loss that training losses is is the loss with
the training set.
Validation loss is with the validation set, which is a part of the data that you've held back.
And this is going to be a metric that the model will calculate that we'll talk more about later, and
we'll see it happening.
And during the course of training you will see your training loss hopefully gradually going down and
down and down, which means that the model is getting more and more accurate at doing its task.
Um uh, and there are some more sophisticated data science metrics that uh, we will also be looking
at at various points.
There's one called the root mean squared log error.
RMSE l.
And the benefit of this particular error is that it's something that will both penalize the model if
it's if it's wrong on an, on an absolute basis and on a, on a percentage basis in different ways.
And it's not going to unfairly penalize the model if it's out by a few dollars on a low priced item.
So it's a it's a good balanced metric that weighs up absolute differences and relative percentage differences.
Um, appropriately.
So that's something that we'll be looking at too.
Another one that people often look at is much simpler is just mean squared error, which is the the
square of the difference between the prediction and the actual price.
The challenge with mean squared error is that it blows up for larger prices.
So if we have something that costs $800 and we predict $900, then there's a difference of $100 there.
And if you square that, you get 10,000, which is a big number and which will dwarf all of your other
errors.
So mean squared error can be more troubling for our kind of problem.
So those are the model centric metrics.
But now let's talk about the business metrics.
So the wonderful thing about our problem is that there is this metric which is really it's really belongs
in both camps.
It's just the the average absolute price difference.
Basically how wrong was the model.
It said that the fridge cost 100 bucks and it actually cost 120 bucks.
So it was out by 20.
That average price difference is so simple, so human, understandable.
And that is a great business outcome metric.
It has some flaws.
It's sort of unfair that that more expensive things like something again that costs like $800 if you're
out, if you if you guess that it costs $850, you didn't do badly.
But that's going to count as a $50 error, which is quite significant.
Whereas if something costs $10, $10, then you would want to do much better than be out by $50.
So it's not ideal.
It has some some problems, but it's it's pretty good.
And it's going to give us such a human understandable way of telling whether our model is doing any
good.
In conjunction with that, we could look at the percentage price difference, uh, you know, which
which has the opposite problem.
That's sort of more makes more sense, more reasonable for high priced items, but seems a bit unfair
for cheap items.
Something costs ten bucks and you get 12.
Then saying that that's 20% out seems a bit a bit harsh.
Um, and then another way to do it is to have some criteria that you will use to judge that an estimate
is good quality, and that criteria could combine an absolute difference and a percentage difference.
You might say, look, if something is, is, um, either within $40 or it's within 20%, then I consider
that a hit.
I consider that a good estimate.
Good enough, good enough estimate.
Um, and so then you could just measure what percentage of the model's predictions meet that criteria
and are considered good.
So that's another business metric that we could use and that we will use.
Uh, and so these these are ways that we can assess the performance.
And as I say, the delight with our project is that they're going to be so human.
Understandable.
So we're going to be able to do things like try different model, different hyperparameters.
We're going to try different models.
We're going to try frontier and open source.
And we'll simply be able to see which ones are better able to predict product prices.
And so we are very nearly ready to roll our sleeves up and get coding and get assessing how this performs,
how we can predict prices.
And it's going to be such a lot of fun.
Uh, but there are just a couple of housekeeping things we have to do first.
I want to talk to you a bit about, uh, the sort of business This strategy for how you go about solving
a problem like this.
And I realize that if you're like me, you are itching to get into it, itching to get in, to actually
finally building training our model and seeing how good are these different models out there.
Um, but I just want to give you this, this grounding, this foundation, so that you have this context
that's going to be reinforced when we then do go and solve the business problem.
I'm going to contrast the different techniques for improving performance and explain, like what's the
difference between them?
I really want to to clarify, in what circumstances do you use Rag?
In what circumstances do you use fine tuning?
It's a question I get asked a lot.
Um, and I think it's helpful to clarify that before we get into it.
So you've got that, that, that context, and then we still have some, some homework to do with the
data set.
We've still got to curate and upload our final data set.
It's going to be big.
It's going to be a big data set, because we're going to want to do a lot of training.
So we have some more to do there.
All of that, all of that coming tomorrow.
See you then.