You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

148 lines
4.1 KiB

WEBVTT
00:00.920 --> 00:02.300
The moment has arrived.
00:02.300 --> 00:03.230
Here we go.
00:03.260 --> 00:04.790
We're in fine tuning.
00:04.790 --> 00:05.810
We do fine tuning.
00:06.020 --> 00:06.740
Train.
00:06.770 --> 00:10.130
There is also a line here to push it to the hub after it's finished.
00:10.160 --> 00:11.750
Let's kick that off.
00:13.130 --> 00:16.910
So while it's kicking off, we'll take a quick look.
00:16.940 --> 00:22.340
The GPU is starting at six, which is what we expect.
00:22.370 --> 00:27.560
It's about as much as it needs for the the model itself.
00:28.550 --> 00:37.490
And it's going to be warming itself up and getting ready for this training while it's doing that.
00:37.640 --> 00:42.170
Since we're waiting for it just for a second, I might as well just talk to some of these other training
00:42.170 --> 00:46.130
parameters that we that we didn't mention before.
00:46.610 --> 00:48.230
Um, eval strategy.
00:48.230 --> 00:55.460
So it is typical when you're doing this kind of training that you would repeatedly have a held out data
00:55.490 --> 01:01.790
set that you would use for evaluation, that you would use as a constant set that's not involved in
01:01.790 --> 01:07.670
training, that you would use to get the model to, uh, validate that it's making progress.
01:07.700 --> 01:09.800
Now, I've not done that here.
01:09.890 --> 01:15.790
Um, partly because we've got so much training data that I think it's got plenty to be training on.
01:15.850 --> 01:21.850
Um, just in one epoch alone and and partly because I just performance, the speed of training was so
01:21.850 --> 01:22.300
important.
01:22.300 --> 01:26.590
I didn't want it to be stopping to do validation repeatedly through this.
01:26.590 --> 01:29.710
But it is a best practice to have a validation data set.
01:29.740 --> 01:31.600
We've got plenty of data for it.
01:31.630 --> 01:39.970
We we held out 2000 test data points so you can use a bunch of them, um, as part of your validation
01:39.970 --> 01:42.490
step uh test test set.
01:42.490 --> 01:50.260
So one certain improvement to this, that would be more of a best practice would be to use an eval strategy
01:50.260 --> 01:53.620
and pass in validation data as well as training data.
01:53.770 --> 01:58.840
Um, but but it wasn't a super necessary in this case, so I didn't do it.
01:59.650 --> 02:01.600
Um, okay.
02:01.600 --> 02:07.690
Well, you may notice if you are glancing over this way that the GPU memory has shot up and out of the
02:07.690 --> 02:15.280
40GB of GPU memory that I've got, 38.2 is currently being used.
02:15.280 --> 02:23.710
So I really did squeeze this so that, uh, with 16 batch steps, uh, a batch size of 16, I would
02:23.710 --> 02:31.270
really use up almost the entire GPU, very close to running out there and you can see that stuff is
02:31.270 --> 02:32.890
happening down there.
02:32.890 --> 02:36.520
So let's just, uh, scroll down and see what we're seeing here.
02:36.880 --> 02:42.850
Um, so it's off and running and we can see that that things are happening.
02:42.940 --> 02:51.100
Uh, and now to run the entire three epochs is going to take, uh, according to this estimate here,
02:51.160 --> 02:53.140
uh, just over 24 hours.
02:53.170 --> 02:55.390
26 hours or so.
02:55.450 --> 03:04.120
Uh, so, you know, it is something like eight hours per epoch, um, on this, uh, even on this beefy
03:04.150 --> 03:10.240
machine, because the training data is so enormous, but it's not strictly necessary to go through all
03:10.240 --> 03:13.210
400,000, uh, training data.
03:13.210 --> 03:18.310
And as I say, you can run this on a much lower end box and let it run for for a while, and that is
03:18.310 --> 03:19.750
all completely fine.
03:20.080 --> 03:26.710
Um, but what you're seeing here is it's it's it's off every 50 steps, just as we configured.
03:26.740 --> 03:31.450
We're getting an output that shows us our training loss so far.
03:31.660 --> 03:36.940
Um, and this is now running and outputting to weights and biases.
03:37.060 --> 03:41.350
And in the next video, I will give you a peek into how that's looking.