You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

367 lines
11 KiB

WEBVTT
00:00.620 --> 00:07.460
So I hope you enjoyed our first bash at Scrubbing Data, and that you are now looking through the code,
00:07.460 --> 00:13.310
convincing yourself that I've done good things when I have cleansed the data and I haven't thrown out
00:13.310 --> 00:15.380
anything that we should be including for the model.
00:15.470 --> 00:21.200
If you do find something like that, please let me know right away, because I love squeezing a bit
00:21.200 --> 00:26.360
more juice out of the model and making the results a little bit better, and changing the data.
00:26.390 --> 00:32.270
Improving the quality of the data that goes into the model has had by far the most profound impact on
00:32.270 --> 00:33.830
the performance of the model.
00:33.950 --> 00:39.980
You can spend huge amounts of time doing what they call hyperparameter tuning, which we will all be
00:39.980 --> 00:44.690
doing for for a while to try and squeeze a little tiny bit of juice out of your model and make it a
00:44.690 --> 00:49.310
little bit better at predicting outcomes as we try and go head to head against frontier models, which
00:49.310 --> 00:50.450
is going to be hard.
00:50.540 --> 00:57.560
Um, but it turns out that improving the data quality of what goes in is the easiest and most effective
00:57.560 --> 00:58.850
way to move the needle.
00:58.850 --> 01:01.640
So that's why it's worth spending the time doing it.
01:01.640 --> 01:06.950
And in that last Jupyter notebook, we we really worked on numbers one, two and three on this list,
01:06.950 --> 01:08.600
and some of four and five.
01:08.630 --> 01:13.790
We've got more of four and five to go, but most important is that you look through that Isom class
01:13.790 --> 01:16.460
and see how it's doing its thing yourself.
01:17.690 --> 01:26.780
Now just before we we we wrap up, I do want to measure and mention again the topic of performance evaluation,
01:26.810 --> 01:33.710
how we're going to decide whether or not we have actually succeeded in our goals with this model.
01:33.950 --> 01:40.520
Uh, there are, as I mentioned a while back, these two kinds of of metrics that you can use to evaluate
01:40.520 --> 01:41.150
performance.
01:41.150 --> 01:48.020
There are model centric metrics which are more in the sort of data scientist's domain of understanding
01:48.020 --> 01:52.310
how how well your model is performing mathematically.
01:52.430 --> 01:59.030
Uh, and then there are outcome business centric metrics, which are the ones that will resonate with
01:59.030 --> 02:03.650
the, the the user, the sponsor, the the CEO.
02:03.680 --> 02:07.430
They're the ultimate goals of the project.
02:07.460 --> 02:13.310
The model centric ones are much more directly measurable by the model, and can usually be measured
02:13.310 --> 02:15.620
while it's training and optimized.
02:15.650 --> 02:22.520
The business centric ones are the ones that matter the most, but not necessarily as immediately tied
02:22.550 --> 02:23.540
to the model.
02:24.080 --> 02:30.680
Although in our case, we have this wonderful luxury of having business centric metrics which are going
02:30.710 --> 02:34.760
to be tied to the model and are going to be great first on the model centric metrics.
02:34.760 --> 02:41.120
So you'll often hear talk of training loss and validation loss that training losses is is the loss with
02:41.120 --> 02:41.720
the training set.
02:41.750 --> 02:46.430
Validation loss is with the validation set, which is a part of the data that you've held back.
02:46.880 --> 02:53.300
And this is going to be a metric that the model will calculate that we'll talk more about later, and
02:53.300 --> 02:54.620
we'll see it happening.
02:54.620 --> 02:59.540
And during the course of training you will see your training loss hopefully gradually going down and
02:59.540 --> 03:05.120
down and down, which means that the model is getting more and more accurate at doing its task.
03:05.510 --> 03:13.890
Um uh, and there are some more sophisticated data science metrics that uh, we will also be looking
03:13.890 --> 03:15.240
at at various points.
03:15.240 --> 03:19.230
There's one called the root mean squared log error.
03:19.260 --> 03:20.190
RMSE l.
03:20.490 --> 03:27.690
And the benefit of this particular error is that it's something that will both penalize the model if
03:27.690 --> 03:33.390
it's if it's wrong on an, on an absolute basis and on a, on a percentage basis in different ways.
03:33.390 --> 03:38.910
And it's not going to unfairly penalize the model if it's out by a few dollars on a low priced item.
03:38.910 --> 03:47.460
So it's a it's a good balanced metric that weighs up absolute differences and relative percentage differences.
03:47.640 --> 03:48.990
Um, appropriately.
03:48.990 --> 03:51.450
So that's something that we'll be looking at too.
03:51.480 --> 03:56.790
Another one that people often look at is much simpler is just mean squared error, which is the the
03:56.790 --> 04:00.900
square of the difference between the prediction and the actual price.
04:00.990 --> 04:06.270
The challenge with mean squared error is that it blows up for larger prices.
04:06.270 --> 04:14.100
So if we have something that costs $800 and we predict $900, then there's a difference of $100 there.
04:14.100 --> 04:20.610
And if you square that, you get 10,000, which is a big number and which will dwarf all of your other
04:20.610 --> 04:21.330
errors.
04:21.420 --> 04:26.460
So mean squared error can be more troubling for our kind of problem.
04:26.850 --> 04:29.340
So those are the model centric metrics.
04:29.340 --> 04:32.040
But now let's talk about the business metrics.
04:32.040 --> 04:36.960
So the wonderful thing about our problem is that there is this metric which is really it's really belongs
04:36.960 --> 04:37.890
in both camps.
04:37.890 --> 04:42.480
It's just the the average absolute price difference.
04:42.480 --> 04:45.210
Basically how wrong was the model.
04:45.210 --> 04:49.560
It said that the fridge cost 100 bucks and it actually cost 120 bucks.
04:49.560 --> 04:51.060
So it was out by 20.
04:51.090 --> 04:56.220
That average price difference is so simple, so human, understandable.
04:56.460 --> 05:00.240
And that is a great business outcome metric.
05:00.240 --> 05:01.710
It has some flaws.
05:01.710 --> 05:10.050
It's sort of unfair that that more expensive things like something again that costs like $800 if you're
05:10.050 --> 05:14.670
out, if you if you guess that it costs $850, you didn't do badly.
05:14.670 --> 05:18.990
But that's going to count as a $50 error, which is quite significant.
05:19.230 --> 05:24.720
Whereas if something costs $10, $10, then you would want to do much better than be out by $50.
05:24.720 --> 05:26.100
So it's not ideal.
05:26.100 --> 05:29.460
It has some some problems, but it's it's pretty good.
05:29.460 --> 05:34.110
And it's going to give us such a human understandable way of telling whether our model is doing any
05:34.110 --> 05:34.710
good.
05:34.890 --> 05:39.990
In conjunction with that, we could look at the percentage price difference, uh, you know, which
05:39.990 --> 05:41.970
which has the opposite problem.
05:41.970 --> 05:47.610
That's sort of more makes more sense, more reasonable for high priced items, but seems a bit unfair
05:47.610 --> 05:48.990
for cheap items.
05:49.200 --> 05:51.540
Something costs ten bucks and you get 12.
05:51.570 --> 05:56.370
Then saying that that's 20% out seems a bit a bit harsh.
05:57.120 --> 06:03.690
Um, and then another way to do it is to have some criteria that you will use to judge that an estimate
06:03.690 --> 06:09.000
is good quality, and that criteria could combine an absolute difference and a percentage difference.
06:09.000 --> 06:18.120
You might say, look, if something is, is, um, either within $40 or it's within 20%, then I consider
06:18.120 --> 06:18.780
that a hit.
06:18.780 --> 06:20.370
I consider that a good estimate.
06:20.400 --> 06:22.530
Good enough, good enough estimate.
06:22.680 --> 06:28.830
Um, and so then you could just measure what percentage of the model's predictions meet that criteria
06:28.830 --> 06:30.870
and are considered good.
06:30.900 --> 06:36.030
So that's another business metric that we could use and that we will use.
06:36.180 --> 06:40.860
Uh, and so these these are ways that we can assess the performance.
06:40.860 --> 06:46.110
And as I say, the delight with our project is that they're going to be so human.
06:46.110 --> 06:47.070
Understandable.
06:47.070 --> 06:53.340
So we're going to be able to do things like try different model, different hyperparameters.
06:53.370 --> 06:54.510
We're going to try different models.
06:54.510 --> 06:56.790
We're going to try frontier and open source.
06:56.790 --> 07:03.360
And we'll simply be able to see which ones are better able to predict product prices.
07:04.920 --> 07:13.620
And so we are very nearly ready to roll our sleeves up and get coding and get assessing how this performs,
07:13.620 --> 07:14.820
how we can predict prices.
07:14.820 --> 07:16.740
And it's going to be such a lot of fun.
07:16.830 --> 07:20.970
Uh, but there are just a couple of housekeeping things we have to do first.
07:21.000 --> 07:29.230
I want to talk to you a bit about, uh, the sort of business This strategy for how you go about solving
07:29.230 --> 07:30.340
a problem like this.
07:30.370 --> 07:36.010
And I realize that if you're like me, you are itching to get into it, itching to get in, to actually
07:36.040 --> 07:40.810
finally building training our model and seeing how good are these different models out there.
07:40.900 --> 07:48.010
Um, but I just want to give you this, this grounding, this foundation, so that you have this context
07:48.100 --> 07:52.060
that's going to be reinforced when we then do go and solve the business problem.
07:52.690 --> 07:59.410
I'm going to contrast the different techniques for improving performance and explain, like what's the
07:59.410 --> 08:00.130
difference between them?
08:00.130 --> 08:04.990
I really want to to clarify, in what circumstances do you use Rag?
08:04.990 --> 08:07.570
In what circumstances do you use fine tuning?
08:07.570 --> 08:09.340
It's a question I get asked a lot.
08:09.370 --> 08:12.250
Um, and I think it's helpful to clarify that before we get into it.
08:12.250 --> 08:17.410
So you've got that, that, that context, and then we still have some, some homework to do with the
08:17.410 --> 08:18.040
data set.
08:18.070 --> 08:21.640
We've still got to curate and upload our final data set.
08:21.640 --> 08:22.570
It's going to be big.
08:22.570 --> 08:25.690
It's going to be a big data set, because we're going to want to do a lot of training.
08:25.810 --> 08:27.430
So we have some more to do there.
08:27.460 --> 08:30.580
All of that, all of that coming tomorrow.
08:30.610 --> 08:31.420
See you then.