Welcome back.
If you are following along with me in JupyterLab, as I hope you are, then you will need it to have
gone off for a coffee break because it will have taken about 20 minutes or so to have downloaded all
of our datasets, but they will now be downloaded and lovingly crafted.
Here they are.
The automotive one is the largest with north of 900,000 data points and office.
Uh, sorry, electronics has more than 400,000.
So in total.
Let's have a look at what we've got.
We have a grand total of just over 2.8 million data points.
That's a lot of data points.
It's too many data points.
We don't need anything like that number for the sorts of training we're going to be doing.
Um, and that means that there's an opportunity for us to hone this data set and select the data points
that are going to be most valuable for us and give us the most signal.
So first of all, let's take another look at the distribution of how many tokens we have.
This is the same chart we did last time, and it shows you that we don't ever have more than 180 tokens
in any of our training prompts, which is something that we specifically set out to achieve in order
to be able to fine tune well with our open source llama model next time, but also to keep costs low
when we're dealing with frontier models.
Let's look at the prices again.
This is the complete price distribution across the 2 million or so, and you'll see that it is constrained
to be, uh, no more than $999.
So it's between 1 and 999, because that's the constraint we've put in to make sure that we've got a
manageable data set without crazy outliers that will distort all of our, uh, training.
Um, but you'll see that we still have the same problem, that the data set is very skewed to the smaller
numbers.
And there's a very thin trail.
Uh, and this only goes up to 300.
So if we go all the way up to 1000 to the, uh, to the end of our, uh, data set.
There you go.
Look at that.
This is the.
We do have, uh, data points in there which reach up to 909.49, but you can barely see them.
They barely touch the, uh, the the axis.
Um, because the data set is so dominated by the 800,000 or so that are coming in at lower cost points.
Uh, one other thing to do is just to have a quick look at the categories.
This nice little bar chart is showing us how many we have in each of the different categories of data
of product.
So again, automotive dominating here with 900,000.
And you can see it's followed by tools and home improvement followed by electronics with 400,000.
So one of the things we want to do now is do some massaging of our data so that we have a more balanced
data set, because we don't want the model to be skewed, distorted towards learning more about one
particular price of one particular category.
Um, we don't mind if it's if it's somewhat, uh, favors some some, like, like cheaper prices because
that is the reality in the world.
But we don't want to go so far that it distorts or impedes our training progress.
Um, so, uh, what I'm going to do now is, is, is do some selection from this data set sample from
our data set to get a smaller data set that is going to have a better representation of prices and categories.
And the sort of data set size I'm going for is about 400,000 data points.
Um, and even that's a large data set for fine tuning purposes, really doesn't need to be that big.
But I wanted to have a big juicy data set.
So 400,000 is what I've gone for.
Um, and we'll talk about how I do that.
So first of all, I've created a dictionary called slots.
And let me tell you what this is.
And then you'll understand exactly why I've done it.
Slots is a dictionary where the key of the dictionary is every whole dollar price of a product.
So it's from $1 to $9.99 one, two, three all the way through to 999.
So there are 999, uh, keys to this dictionary.
And the value is going to be a list of all of the products, all of the items which have that price.
So in the slots dictionary in slot number two will be a list of all of the items which cost $2.
And so it's organizing everything into these slots.
It's bucketing our data set basically.
Um hopefully that makes total sense.
If not of course bring up this code.
Step through it.
I'm using Defaultdict is a nice little thing to know about, which is basically a dictionary which will
if something is missing from the dictionary, it will automatically initialize it to be of whatever
type you pass in.
It avoids you having to put a sort of if test in your code.
So it makes a nice, nice, elegant code.
All right, so here's a bit of a meaty function here, but I explain what's going on.
Amity.
Jupyter notebook.
Cell.
Um, I am going to go through each of these slots.
Each of the 999 slots.
And I'm going to sample from those slots a subset of the data, which I think will be a nice representative
sample to use for training.
Now, some of this I've tweaked around with arbitrarily until I've gotten comfortable with the histograms
that will follow this.
So it's not like there's any particular special reason.
It's more of a case of trial and error and getting to a point where you feel good about the balanced
data set you're producing.
And of course, I've then run it through training and satisfy myself that I'm getting higher quality
results, uh, by doing this.
Um, and so what I do is I go through each of the slots in turn, and I've decided that for anything
that's worth more than $240, I simply take that whole slot.
I take all of those points and add them to my sample.
Um.
For something less than that.
I basically have some code here that samples 1200 items from that slot.
So it takes that slot, and that slot might have in it several thousand.
I just pick 1200 from that slot, and I use a numpy method called choice, which lets you pick a certain
number from the slot.
And one of the nice things about choice is that you can pass in something called the weights, which
is telling it to give more importance to some of your items over others.
And uh, hopefully this comes together no surprise for the weights.
What I'm saying is let's give anything that's an automotive, a weight of one, and everything else
gets a weight of five.
And again, this was I just played around with different numbers until I got comfortable with what it
was coming up with.
And I didn't want to take it too far because we want to stay roughly true to to the the kind of data
we have in the real world.
But we wanted to correct for some imbalances in the data set.
So I'm not going to go line by line through explaining this.
I've given you the construct, and I'm hoping you'll now look through this and satisfy yourself that
it's doing what I say and that you like the outcome.
And of course, if you prefer to craft the data set a bit differently, this is your chance.
Uh, it's also perfectly possible that you will be able to beat my results in terms of my model performance,
and you may think that it would be better to to perhaps have a different weighting of the categories
or to choose differently from the slots.
So you should absolutely experiment, um, and see what you come up with.
But I've run this now.
It has now created a sample list, and there are 408,000 data points in that sample.
So that's about the size that we were aiming for.
Um, and now let's see the distribution of prices.
And that looks a lot more reasonable in terms of the distribution of prices.
We've got a lot that are cheaper still, but but it's a consistent number for every price point in the
cheaper end.
And as we get to more expensive prices, there's a perfectly decent set of of data points with higher
price.
You'll notice this interesting effect that there are various points.
Uh, predictably enough, it's things that are priced $399, $499 that have a little spike in terms
of how many data points there are.
And that's great because that reflects the real world.
So it's good that we're going to have that in our data set.
I wouldn't want to to to squash that out.
Um, so when we compare this histogram of prices with our earlier histogram of prices here, hopefully
you immediately see the improvement we have made to the distribution of prices in our data.
This is clearly a more it's still skewed and the real world is skewed.
Um, but there's a better representation of higher priced products.
And it's going to mean that we're going to be able to learn in a high quality way and validate our sample
more.
If you're not satisfied by that, by all means create a couple of data sets.
And when we get to training, you can try them both and see the impact it makes to have a well-balanced
data set.
Let's also look at the categories again.
Um, this is the categories.
So actually it hasn't made a ton of difference.
It's slightly shifted.
Um, we've got a bit of a better balance.
Um, I didn't want to further correct it because I feel that this is, after all, somewhat reflective
of the real world.
And so we don't want to overly distort.
There are a healthy number of automotive products on sale, more so than others.
And so this this seem good enough, but it's slightly corrected some of the imbalance there.
Perhaps another way of looking at this is looking at a pie chart.
Generally speaking, often pie charts are unpopular with data scientists because bar charts are better
for seeing quantities side by side and seeing them in a very quantitative way.
But pie charts sometimes are useful visuals, and let's have a look at it.
Here is a pie chart by category, and I should obviously do a bit of work to separate out some of these
words, but you get the idea.
Um, and it's showing you here that automotive does have the biggest the lion's share, but it's not
like it's massively dominating.
And obviously a couple of these together are more than automotive.
So it's perfectly reasonable.
And the little guy here is appliances.
The one that we started with way back yesterday has 1% the smallest, the smallest piece of the pie.
Uh, quite literally in this case.
So that is our data set curated.
Uh, it was um, a bit of work, I agree.
And I did gloss over some of the, uh, thornier, uh, pieces in there, like the sampling.
And I urge you to come back and look through that and evaluate it yourself and potentially craft a better
data set.
Uh, we're finally going to do some last analysis on it before we upload it to the hub.
And I will see you for that in the next video.