And this is the first time that we'll be coding against our big project of the course.
Welcome to Jupyter Lab.
Welcome to the week six folder as we embark upon our big project.
So again, our project is to build a model that can estimate how much something costs based on the description
of the product.
And today we're going to be doing the first step in data curation.
And we'll start by looking at a subset of the data for home appliances, washing machines and the like.
So first let me just show you the data set itself.
The data set is is at this link right here.
This is the data set on the hugging face datasets section of the Hugging Face hub.
Um, and it is a series of um, of scraped Amazon reviews that that goes back in time.
But this latest scrape that was from 20 was from late 2023.
Um, it contains a huge number of reviews, but it also contains almost 50 million items in there.
So there's a lot of different products and they're divided into these different categories.
We are not going to be working with all of these.
We're going to pluck out a subset of this that are the kinds of categories that interest us the most
for this exercise.
Otherwise, everything would take an awfully long time to train and that wouldn't be any fun.
So this gives you a good sense of the kind of data that we're working with.
And if I go into the folder, the hugging face folder that contains the data for what, what the dataset
calls meta data, which is the data of the products and descriptions and prices themselves, which is
what we really care about.
Here it is.
And you can get a good sense if you look at the data set for things like electronics, you can see that
it's just over five gigabytes in size.
So these are big data sets and they're going to have a ton of useful information.
And it was uploaded seven months ago.
So this is all quite recent.
So let's get going.
We begin with some imports.
Nothing particularly complicated there.
Not as yet.
There will be more to come.
Uh, we're going to set up our environment.
Not that we're going to be using any of this today.
We're just going to be using hugging face.
Log in to hugging face.
Um, and, uh, this this makes sure that, uh, matplotlib can show us charts in the Jupyter notebook.
So the first thing to do is to load in our data set.
And what we're going to do is we specify the name of the data set, Amazon reviews.
And we're going to just choose to start with the appliances category.
Appliances home appliances like like uh fridges and, and uh, washing machines and the like, um,
are going to be the first things that we're going to load in using hugging faces load data set, uh,
function.
Um, and the first time you run this, it will actually download it from the Huggingface hub, since
I've already done that, that won't be required for me.
It will just bring in and this has already completed and we'll see how many appliances we have.
We have 94,000 home appliances in there.
So let's have a look at one of these guys.
Let's have a look at uh let's say data point equals data set.
Let's take the first one.
Why not.
Let's have a look at it.
So this is what it looks like.
It's got tons of information.
But in particular you can see it has something called features.
It has a title and it has a few other things that are probably going to be useful for us.
And in particular it has a title, a description, features, details and price.
Let's just print each of them out so we can have a quick look.
So this is the title.
This is an ice maker machine countertop.
This is its description which is empty.
This is its details.
So features lots of features, details here and price which we immediately see a problem.
Price is none in this case.
So clearly not all of the items have a price.
And you'll notice that description appears to come in the form of a list.
So A features whereas details comes in the form of a dictionary.
Although that is deceiving, it's actually not a dictionary.
It is a string that contains JSON.
So this is text that if we want to read into that, we would need to load that in and use A and convert
it into a dictionary using JSON load string load s.
Um, okay.
Let's look at a different data point just to see the next one.
Looks like an egg holder for a refrigerator, uh, and holds up to ten eggs also none.
No price for that one either.
And this third one doesn't have a price either.
It's a brand new dryer drum slide.
So at this point we might have our first moment of being concerned.
We've got 94,000 appliances.
The first three that we've looked at don't have a price.
So let's see how many do have a price.
So a simple way to do that is we will iterate through all of the data points in our data set.
And we will get the price.
And we will put that in a try block.
Because if it doesn't have one, it will fail and we will just skip that data point.
So we'll also ignore anything that is priced at zero.
So we're just going to be looking at things that have a price that is a number.
And that that price is non-zero is more than zero.
I don't think there are any negative prices in there, but if there are, they're not going to get counted.
So this is now going to be going through and trying to figure that out.
And there we go.
So it tells us that there are 726, which is almost 50%.
So it's not terrible.
That's fine, that's fine.
It's, uh, for a moment might be worried that there would be a, that it would be slim pickings, but
no, at least for the appliances.
Um, uh, data set, half of them have prices.
It's a tiny side point.
I don't know if you've spotted when I've been printing numbers.
Generally, they've had a comma to separate the thousands, which I always find so useful when when
being able to read these kinds of things.
The way that you do that is, if you're using Python's f strings, you say colon comma like this.
Um, to you use that for your formatting and then you'll get numbers in this style.
Just a little a hot tip.
Uh, you may have known that already, but if not, it's a useful one to be aware of.
Okay.
So what we're going to do now is we're going to take all of the ones with prices.
Um, and we're going to figure out how many characters it has in its title, description, features
and details.
We're going to add up the total number of characters and put that into a list of lengths.
So what we now have is a list of prices and a list of lengths, so we can get a sense of how how many
characters of detail we have and see if it's uniform or if it's something that's that's in some way
skewed.
So now we're going to use matplotlib, which we'll be using a lot, uh, to make a plot of the lengths
in the form of a histogram.
And hopefully you remember from statistics classes of some time ago, a histogram is basically going
to take everything and bucket it into into bins and show how many we have in each bin.
It's easier to show you what that looks like.
This is what it looks like.
So along the x axis we have the lengths of our different, um, uh, appliances, our different washing
machines or whatever, uh, in terms of how many characters they have in that description.
And this is the count of how many appliances do we have with that many characters.
And you can see that there's a nice kind of peak around here, but there is this long tail of more characters
coming in now.
This is going to be a challenge for us when we're training, because ultimately we're going to want
to use our own, uh, open source models and train them.
And one of the constraints that's very important for us to understand is the maximum number of characters
that we might pass in, or actually the maximum number of tokens that we might pass in to the model
at each point.
And the more tokens that we might need to pass in for each of our training points, the more memory
that we need for training and the harder it is to achieve.
Another point is that even when we're using frontier models, whilst they don't have that problem,
they have a different problem, which is that it's more expensive if we're passing in more tokens than
they are going to of course cost us more, which doesn't really mean very much for a for a few of these.
But if we want to do this in anger for a large number of products, then the numbers will start to add
up.
So ideally we would pick a cutoff and we would constrain our data at that point.
Um, and so that's something that we'll be thinking about later.
Another thing for us to look at is the distribution of the prices.
So how much do things cost?
You may have gotten the hint from our earlier analysis that whilst we thought appliances was going to
be full of fridges and washing machines and the like, the things that we looked at were rather smaller.
They were egg holders and ice makers, and it shouldn't be that much of a surprise when you think about
it, that the data is probably going to have a very large number of cheaper things that might sort of,
um, squash out some of the higher priced items.
So let's see that.
Let's see how this looks.
Well, that does appear to be the case.
So the average price in our data set is $6.
The highest price is $21,000.
There is a home appliance for $21,000 in this list, but you can see that there's a very large number
that have smaller prices.
And for those that remember the difference between mean, median and mode.
Again, from school statistics, this is a nice illustration of where the mean can be pulled up by expensive
items and is clearly going to be bigger than, well, certainly than the mode and and probably the median
too.
Uh, so, um, yes, you can you can certainly see we have skewed distribution where there is a very
large number of cheap products.
And that might be challenging during training because the training data is going to be really crowded
out by these low cost items.
Let's just have a quick look for this super expensive thing and see what it is.
This this $21,000, uh, item.
We will go through our data set and pluck out whatever it is that costs more than $21,000.
It is, it seems, a turbochef bullet.
Rapid cook electric microwave convection oven.
Now, if someone had told me that description, I would never have thought that that was going to cost
$21,000.
I did find something not identical, but something that I think is probably the latest version in Amazon
right now.
And if we go over to have a look at this, you can see here this is also made by Turbochef.
It's a bargain price of only $18,000, not $21,000.
But I don't know about you.
I had no idea that microwaves could cost this much, but it's clearly a very professional microwave,
a very high end microwave, and going, as I say, for that bargain price.
That is the $21,000 version of that is over here somewhere in our way off the scale in our data.
So it's now time for us to curate our data, and we'll do that in the next video.