You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

217 lines
5.8 KiB

WEBVTT
00:00.410 --> 00:01.670
Welcome to Colab.
00:01.670 --> 00:04.910
Welcome to the week seven day two Colab.
00:04.910 --> 00:10.760
And just before we try our base model, we're going to actually see these tokenizers in action.
00:10.820 --> 00:15.410
Uh, start with some Pip installs and some imports as usual.
00:15.560 --> 00:22.430
And now in the constants section, I'm going to set up four different tokenizers that we will just quickly
00:22.430 --> 00:23.120
look at.
00:23.120 --> 00:27.830
After that, we're going to look at our base model llama and do some some work with the data.
00:27.830 --> 00:33.800
And I've also got the constants here for writing to the output in color that you may remember from when
00:33.800 --> 00:35.930
we visualized results in the past.
00:36.440 --> 00:37.850
All right, let's run that.
00:37.880 --> 00:43.070
We log in to hugging face using the usual snippet that you know well at this point.
00:43.070 --> 00:49.100
And now I've written a useful function called investigate tokenizer that takes a model name.
00:49.460 --> 00:58.790
And what that does is it uh, will load in the tokenizer for that model name, and it will then iterate
00:58.790 --> 01:06.030
through these six numbers zero, one, ten, 100, 999, and 1000.
01:06.090 --> 01:12.180
For each of those numbers, it will convert it into a string, and then ask the tokenizer to convert
01:12.180 --> 01:14.160
that string into tokens.
01:14.160 --> 01:20.130
So what are the tokens that represent the string 100 as a piece of text?
01:20.520 --> 01:24.660
Um, and I'm using this add special tokens as false parameter.
01:24.660 --> 01:29.940
What that means is please don't add in things like a start of sentence token and an end of sentence
01:29.940 --> 01:33.510
token or something like that that will interfere with things.
01:33.600 --> 01:38.760
Just simply convert this text into tokens that represents that text.
01:38.970 --> 01:40.260
Um, so that's what it will do.
01:40.260 --> 01:41.730
And then it will print that out.
01:41.730 --> 01:50.520
So to run that and to show it let me call investigate tokenizer.
01:51.450 --> 01:59.460
And we will start by calling it for the llama three one uh model.
01:59.460 --> 02:00.950
And let's see what we get.
02:02.390 --> 02:06.200
So what we get is the string zero.
02:06.200 --> 02:09.710
Text zero converts to the token number 15.
02:09.740 --> 02:12.230
One goes to the token 16.
02:12.260 --> 02:15.380
Ten goes to token number 605.
02:15.410 --> 02:16.190
As it happens.
02:16.190 --> 02:21.380
And you'll see that similarly 109.99 they each mapped to a single token.
02:21.890 --> 02:25.310
1000 though that maps now to two tokens.
02:25.340 --> 02:28.580
In fact, it maps to the token for the number 100.
02:28.610 --> 02:33.230
If you see that followed by the token for the number zero and that.
02:33.230 --> 02:41.030
So just just as you would imagine, it is the tokens for the text 100, followed by the text for a zero.
02:41.090 --> 02:47.480
And what you see here, loud and clear, is that if we're just focusing on three digit text numbers,
02:47.480 --> 02:53.690
on three digit numbers, we have this useful property that whatever price we have is always going to
02:53.690 --> 02:55.130
map to one token.
02:55.160 --> 03:02.330
So the model's task of trying to predict the price of a product will end up being just predict one token
03:02.330 --> 03:05.930
and get that token right, and then you've got the right price of the product.
03:06.110 --> 03:08.690
Um, and so it's not an essential property.
03:08.720 --> 03:15.470
We don't require that these models are perfectly able to generate a sequence of tokens, but it's convenient
03:15.470 --> 03:20.270
that we're going to simplify the problem down to just getting this one token right.
03:20.720 --> 03:26.270
And we can also see how this looks for another model like K-125.
03:28.280 --> 03:30.350
Uh, and now we see something different.
03:30.380 --> 03:38.450
The single digits zero and one mapped to a single token, but ten maps to two tokens.
03:38.450 --> 03:50.990
In fact, the token for one followed by zero 100 is 100999 is presumably 999, and 1000 is 1000, uh,
03:50.990 --> 03:53.210
for tokens being used there.
03:53.210 --> 03:54.830
So you see that different property.
03:54.830 --> 03:59.930
And hopefully that really clarifies, um, why I think llama three one has an edge.
04:00.290 --> 04:00.590
Uh.
04:00.630 --> 04:01.230
Gemma.
04:01.230 --> 04:02.670
Two for Gemma, two.
04:02.700 --> 04:03.450
Sorry.
04:03.630 --> 04:04.290
Uh, Gemma.
04:04.290 --> 04:08.280
Two similar properties to Kwon.
04:08.460 --> 04:11.880
Uh, interestingly, uh, totally different vocabulary.
04:11.880 --> 04:14.670
It's a much bigger number, but that's no surprise.
04:14.670 --> 04:17.610
There's no reason why they should have the same vocabulary.
04:17.940 --> 04:19.860
Um, and, um.
04:19.890 --> 04:26.640
Yeah, it's, uh, clearly, uh, not not a one token for for one three digit number.
04:26.910 --> 04:30.660
Um, and Phi three is, as I say, similar.
04:30.990 --> 04:37.230
Uh, there is actually a different variant of phi three, a smaller variant that has the same nice properties
04:37.230 --> 04:38.220
as Lama three one.
04:38.220 --> 04:42.420
So that is another thing that would be worth potentially trying.
04:42.480 --> 04:44.040
Um, as could any of these be tried?
04:44.040 --> 04:48.270
It's not a disqualifier that that it produces multiple tokens by any means.
04:48.450 --> 04:49.020
All right.
04:49.020 --> 04:54.420
Well, anyway, that gives you a background to the tokenizers and a good sense for why we picked the
04:54.420 --> 04:55.680
model that we did.
04:55.740 --> 05:01.410
Uh, in the next video, we will then load in data and try testing our model.