And welcome back to the week six folder.
We're now at day two, which is the second and final stage of data curation.
We're going to be extending our data set to be much bigger.
And we're going to craft it into something that is exceptional, uh, just right.
For our training purposes, uh, we start with some imports and setting up our environment and logging
into hugging face, and we get ourselves ready.
So last time I talked about a Python module called items that I wrote that allows us to parse our data
points into a nice item.
You remember it well, and hopefully you have spent some time digging through this yourself and confirming
the way that it works.
There's another Python module that I've made that is shorter and simpler called loaders, uh, which
has a single class item loader and is just a nice way to load in one of the data sets from the hugging
face repository.
And what it's going to do is it's going to use some fancy, uh, stuff from the concurrent futures package
in Python to do this with multiple workers, because otherwise this takes a long time.
Now, you don't need to do look through this in too much detail.
There's a few things that I want to tell you about, and then you should just convince yourself that
it's doing what it says on the tin again.
Uh, so the main, uh, method that gets called is called load.
And you pass in a number of workers and by default it assumes eight workers.
I'm working on a MacBook Pro here that has eight cores and so it can handle it.
It does really hammer my machine.
While this is going.
You might want to pass in a smaller number of workers, depending on how much you're willing to give
up your machine's CPU if you're not doing other things at the same time.
Uh, so then, um, this load in parallel method is the one that uses this process.
Pool executor.
If you're familiar, if you have used this before, this basically spawns a number of other.
Um, I've just seen a mistake there that that should obviously say workers so that, uh, it does actually
use the eight workers.
Otherwise it will always be using six.
Uh, so this, um, this will spawn up, uh, the number of workers that are specified, uh, and it
will then load in each of our data points, but it will do so in chunks.
And you can see that that really what I've done is I've just created a generator familiar with generators
now because of us using them earlier with when we're streaming back results.
Um, but we're using a generator to, to chop up our data set into chunks of.
As it happens, I set a chunk size here of 1000.
So a thousand data points at a time.
So it's chunked up into sets of 1000 data points at a time.
And each chunk is then passed in and ultimately made to create a new data point using the same item
class that we worked on last time.
So no real magic here.
This is just some fancy packaging to load in items in an efficient way.
1000 at a time and spread out across eight different workers.
But I will say it's worth reviewing this code because this is something that's the kind of thing, again,
that one tends to do a lot with these kinds of projects.
You need to make good use of your box because it's going to be hefty, and the the kinds of tasks involved
often are very suitable for being divided into multiple workers.
And this code should be quite reusable.
I've hopefully written it in a way that's easy to read and and and reuse so much encourage you to do
so.
Um, and uh, there is one other little trick that I want to point out, not a another trick.
Sorry.
One other, um, uh, decision that has been made that is going to affect our results.
Uh, I've decided that we will select only products which cost anywhere between $0.50 and $999 and and
$0.49.
So we're going to limit it to things that price in that range.
And there's various reasons I did that.
Um, one is that if you take things that, uh, that go, uh, much more than that number than then
the results are distorted.
There's a very small number of things with a huge price and that can completely mess up things like
our test performance.
If we happen to pick something that costs an enormous amount, then our errors can be wild if we're
off by a bit.
And because we do want to be using absolute error, just the difference between the recommended price
and the actual price.
It's nice if we can keep our prices within a reasonable range so that we have a pretty decent sense
of how the model's performing, and we don't have any wild things out there.
So essentially, I'm saying for the scope of this project, for what we're doing, we are going to be
talking about things that cost under $1,000.
That's going to be our scope, and it allows us to focus in our data set.
You could also experiment with doing this with different boundaries and try try bigger ranges and see
how it goes.
But I found this to be easiest to work with and give good, good results across the board.
So that's what is going on here.
Um, so let me save this and go back to our day.
And I'm going to reload this because I made that little bug fix, restart kernel and run this again.
Run our imports, log into Huggingface again.
And now we're going to load in the appliances data set like we did before, and it's now going to be
hammering my computer.
I can see it doing its thing, and I'll tell you that before it took about a minute to load in all of
the appliances data set last time, and this time it takes 0.2 minutes.
So it's rather faster when you have it broken into.
Eight workers hammering my computer.
And so I recommend you do the same.
But obviously pass in here workers equals four or less if you if you have fewer processes.
Um so that is the appliance data set.
It has the 28,625 data points that are priced within that range that we have restricted it to.
Um, and they have all been loaded in, uh, let's have a look at the first one.
Do you remember what the first one was?
I did that again.
I did that last time.
There we go.
Try this.
And here we go.
It is the rack, roller and stud assembly kit.
Let's, uh, print its prompt.
And we'll hopefully also again convince ourselves that those, uh, clunky part numbers are filtered
out.
Yes they are.
And this is, again, the dishwasher top wrack reels and stud assembly kit.
And this is the door pivot block.
Always needed one of those.
Uh, so there we go.
Uh, here are our data points loaded in for us.
Um, now it's going to be time for us to scale up and embark upon a much bigger problem.
We are going to be bringing in all of these data sets from the the Amazon.
Uh, Product Prices dataset repository.
I've selected these to be fairly similar kinds of things that are sort of stuff you might find at a
large home retail store.
So it's really almost anything on Amazon, not including stuff like clothes, beauty products, books
and software and things like that.
So it's everything that felt like it was sort of similar kind of stuff.
Um, that would make a really good comprehensive data set.
So here are the data sets.
And of course, um, if you're doing this as you follow along, if you're doing this when you do this,
as you do this, you can play with this and you can choose a different set of data.
If you wish.
You could do this for clothes and see how it performs.
Um, if you are concerned about the size of data and you want to be doing things quicker, you can just
limit it.
The appliances data set is one of the smallest ones there.
Um, electronics is a nice and one in the middle.
So you could just focus on electronics.
Um, and you'll still have great fun, and you'll still get much the same kinds of performance results
as we will get with the entire data set.
So those are all of the data set names.
And now I'm going to read all of them in using the item loader to get after them.
And it's going to read them in one at a time.
The biggest one is automotive which is the first one on the list.
I've generally ordered it, I think with the biggest ones at the beginning.
Um, it might take a bit longer for you because the first time you run this, it has to download the
data from Huggingface to your computer because I've already run it once that step has happened, and
so it just reuses it from a cache on my computer.
Um, but it may it may take a bit, um, a bit longer for you because of that.
And then off it will go.
Now the total time for for on my computer is about 20 minutes.
Uh, my can see my, my CPU is flat out, so I'm going to take a pause and see you again right after
the break when the data has all loaded.