Welcome back.
So we've been doing the thoroughly distasteful, unsavory work of feature engineering.
Very grotty, uh, work.
But I still find it a bit fun, I have to confess.
But it's quite, quite, uh, hacky and involves getting very deep into the data.
We went through a bunch of stuff to figure out the weights of items in our data set and stuff in an
average weight if we can't find the weights.
Um, we're now going to look at the best sellers rank for each of our items.
And so we're going to try and collect best sellers rank from its features.
Uh, and then uh, what comes back is in fact itself a dictionary, because a product on Amazon can
actually be ranked against multiple different bestsellers lists.
Um, and so we're going to do something again, very rough and ready.
And if it's features in multiple bestsellers lists, we're just going to take the average if it ranks
first in one list and 10,000in another, we're just going to take the midpoint.
We're which is we're just going to take the the one around the 5000 mark.
Uh, so this this is, it's it's, uh, it's all a little bit of guesswork.
Um, it's a bit of trial and error.
And often this kind of traditional data science is a bit like this, actually.
So as we'll discover, is the modern data science as well.
There's plenty of trial and error.
Typically, what you do with this, this kind of technique is that you try lots of features.
You might try taking the average, or you might try taking the best and a few other things.
And you shove all of these features in there and you see which one wins out.
Now, in this case, we're just going to pick the average.
But if you've got the stomach for it and you're enjoying this, this, as I say, slightly distasteful
work of digging around in features, then try some more features.
Try adding in the minimum rank, the maximum rank, uh, whatever you wish, um, to see what gives
the most signal.
So in our case, we picked the average rank.
We just do it.
And we're then going to to do the same trick we did with weights.
We're going to find out what is the average of our average ranks, which turns out to be that slightly
curious number of 380,000 or so.
And then we're going to give ourselves a get rank with default function, which tries to get a rank.
And if something doesn't have a rank, it gives you the average, the average rank from the training
data set.
Okay.
And then one more feature I'm going to add into the mix that I didn't mention before.
Uh, you may have guessed, I don't know is I'm going to say how long is the test prompt with all of
the detail that it's got in there?
I don't know if you remember, there was that scatter diagram that we did, uh, a couple of days ago,
or maybe just one day ago, uh, with lots of red dots on it that was trying to see, is there any correlation
between the price and the amount of text?
And when we looked at that visually, it appeared that there was a slight correlation.
I probably got that up.
We can just take a quick peek at that to see, oh no, it's not there anymore.
I've cleared it out.
You'll have to look back yourself if you ran it.
I hope you did go back and look at that red diagram again and you'll see what I mean.
There is a slight correlation there.
So let's add that in.
Let's get get text length and we'll use that as well.
And then the final one we're going to look at the brands.
Let's first look at the most common 40 brands.
So we're going to count them all up using the same approach as before.
Brands most common 40.
Let's look at the most common 40 brands.
Here they are.
And what you'll notice here is that there's a few, um, automobile car related brands, which I'm not
very knowledgeable about.
You may be more knowledgeable than me.
You may.
You may think I'm missing a trick.
You may say, oh, there's a beautiful feature there that we could engineer of looking at top auto brands,
in which case you should create that feature, add it in and see how you do.
Uh, I sadly don't have that domain expertise.
Um, and so what I've plucked out is a little category called top electronics brands, where I have
shoved in things like HP, Dell, Lenovo, Samsung, Asus, Sony, canon, Apple, Intel which I've
just plucked out of here into this category.
And then that gives me a feature is Top Electronics brand.
And this is one where again, I've done one feature.
You could come up with a bunch of features.
You could you could pick out different kinds of brands.
You could pick out some auto brands.
You can create as many features as you want.
There's no harm in having more features, because the regression model is going to decide which of the
features actually gives you some signal.
And so a fun competition for you is to be generating features and see how well you can do with handcrafted
features.
I'll make one more important observation I mentioned a moment ago.
I don't have the car expertise, which means I can't pluck out auto brands.
And that leads to an interesting point, which is in this kind of traditional data science, it was
important that data scientists had some strong knowledge of the domain they were working in.
If you were working in products you needed to understand about different products.
You needed to understand about different car manufacturers, because you needed to know which features
to engineer to have the most likely chance of success.
One of the curious and remarkable surprises of deep neural networks and modern machine learning and
modern modern deep learning is that the model figures out for itself which features matter.
And so there's no longer this requirement for data scientists like you and me to have deep domain expertise
in the field that we were building models around, because we just have to have expertise in how to
build llms and models and both of any kind of deep neural network.
And they have billions of parameters, and they are able to use the the understanding power of all of
their parameters to learn about the business area.
But back in the day, in feature engineering, one had to understand it oneself and make things like
top electronics, brands features which we have done.
And all of this brings us to this function here.
Get features.
It takes an item and it creates this nice little dictionary here with a weight, a rank, a text length,
and an is top electronics brand, which is either a one or a zero.
Um, and that, that is uh, our features group for this first model.
The first real model that we're building.
Uh, and please, I urge you to, to spend some time turning this into your features of your dreams.
Uh, see how well you could do by engineering features.
And you can probably do quite well.
Um, but I don't think you'll be much of a match for what's to come.
But give it a try.
Now.
Give it your best shot.
But after this, uh, this coming up in this next video, we will actually run.
This run our traditional machine learning model and see how it fares.
I will see you then.