So last time we looked at a humble linear regression model with feature engineering, and now we say
goodbye to features and we start working with natural language processing.
And we're going to begin with a rather simplistic type of NLP, which is called Bag of words, which
I briefly mentioned before.
So we're going to start by putting making these two very useful things.
One is called prices and one is called documents and prices is just the list of all of the prices in
our training data set and documents is all of the test prompts from our training data set.
Just a bit of text, and what we're going to try and do now is build models which use the text in our
documents, rather than using features that we have engineered ourselves.
Just use the text.
Um, and yeah, one one little trick you might notice here is that I use the test prompt, not the training
prompt.
If I use the training prompt, it would include the price in the training prompt.
And that wouldn't work because then of course the model would just learn to spot the price itself is
actually in the prompt.
And then when it got to test time, it would fail rather miserably.
So that's a little trick to watch out for.
Um, a trap I'm more likely to watch out for.
So we prepare that.
We're now going to use something called a countvectorizer, which sounds super fancy.
At any time that you think you're going to be working today with a count of Vectorizer, sounds like
you're going to be building something that's highly sophisticated, but it is, alas, not sophisticated.
What it's going to do is simply look for the counting, the number of words, and then build a vector
where each location in the vector represents a particular word, and then how many times that word appears
in your document.
So each document will be a vector, and every row in that vector represents a word in your vocabulary.
And the counter counts how many times it's there.
And that whole way of thinking about things is known as a bag of words model, because it's like you
have a bag of words that you are counting up, and it doesn't matter what order the words appear in,
it's just the fact that they are there a certain number of times.
Uh, and so we're doing that we're going to count up to the 1000 most common or most important of the
words.
And we're going to use this parameter here to make sure that it only, uh, it, it removes, uh, common
stop words, which is what people call things like and and the and in and it that are going to be not
useful for the model and only a distraction.
And so they will get plucked out leaving us with juicy words behind.
And um, what we're then going to do is, uh, and now you can see we do things much quicker.
We're going to, uh, we're going to create our data set based on these documents.
We're going to create a linear regression model again just as before.
And we're going to fit our bag of words, um, to our prices.
In other words, we're instead of using features this time we're replacing our features with this bag
of words, this vector counting the number of words, um, for each word, each of our 1000 words in
our most common dictionary vocab.
So that's what we're going to do.
We're going to run that linear regression.
It's happening right now.
It's counting up the uh, the, the 1000, uh, vector points across all of our data set.
Um, and then once it's done that, we're going to put that into a bag of words, linear regression,
pricer.
and we are then going to test it, so we will test it again.
It's just completed running.
Um, so this is our function, our simple function that will try it out and we will test it with tester
dot test.
Bag of words.
Linear regression Pricer.
Let's see how it does.
Lots of greens there I see.
But also lots of reds.
You can see.
Let's just pluck one out.
It guest $74.
And it was in fact $46.
So you can see that it's got things that are right.
It's also got things that are way off.
How do you think the graph is going to look.
Let's see.
Okay okay.
Well it's looking more like there's some something good happening.
We're starting to see things converging around the line.
We're seeing a lot more green dots.
The average is $113.
So distinctly better than the linear regression with features and distinctly better than guessing.
Uh, so, uh, progress happening.
There's still some weird outliers here, some some problems.
Um, but, uh, that this was the true value and it should have been right up there somewhere.
Um, but there you can see that there is a progress being made.
So next, the the last of the of this set before we go into more advanced models is going to be using
the amazing Gensim library to, uh, introduce this word to vec model.
That was uh, it was one of the first times that I really encountered, uh, neural networks with NLP,
uh, using a vectorization model, um, with, with something that is a more sophisticated, uh, vector
embedding model.
Um, and we are going to use this word two vec function class from Gensim, and we are going to build
vectors with 400 dimensions.
And it's I've set it to use eight workers, which means it really hammers my box.
This still took several minutes to run, and I ran it in advance so we wouldn't have to wait for for
all of this.
Uh, and I also then ran this, and that should mean that hopefully we are ready just to run this and
to immediately see the results tester dot test.
We should be able to pass in the word two vec linear regression pricer and see how this fancy vectorization
model performs with linear regression.
Well, it looks good so far from the first two lines, but oh no.
Then there's some red, some green, some red, some greens, lots of reds that scoot down past the
250 test data points to the chart.
So here we have it.
It looks quite decent again.
Interestingly, the bad news is it's actually a hair worse than linear regression based on the simple
bag of words model.
So unveiling the lovely word two vec vector hasn't yet particularly helped us.
We're still much the same territory as before, and that may be because the linear regression model
isn't powerful enough to take advantage of all of the extra information that we have in these word two
vec vectors.
So in the next time we're going to explore some slightly more sophisticated models, and then we'll
be done with our with our traditional machine learning.
So just hang on in there for a bit longer, because I want to dig a little bit more juice out of this.
We want to get a bit better with our baseline models, because we don't want our LMS to have an easy
run at this at all.
We want to we want to put up a fight.
So see you next time.