So as the very final step on this part four of day two of week eight, we are now going to build an
ensemble model that brings all of this together.
And first, I just wanted to show you that if I take a product like my microphone right here, the Quadcast
HyperX condenser mic, we've got these three objects now, Specialist Frontier and Random Forest, and
we can ask each of them to price this product.
And you'll see we get these three numbers.
In this case, I think the frontier model is closest to the truth.
I seem to remember when we and maybe I must have had slightly different text when we called the specialist
model last time.
I think we got even better.
We got like 129, I think, which is even closer.
Uh, but yes.
Anyways, you can see that the random forest didn't do so great, but the other two were, uh, were
reasonable.
So what we do right now is quite simple.
I take, uh, the first, I take a selected 250 test data points.
Uh, I actually picked the ones from 1000 to 150 to keep it separate from the ones we've been using
for actually testing.
Um, and basically I take each of those items, I find its description and I add in, what price do
we get from the specialist model, from the frontier model, and from the random forest model?
And then I also have a list of prices where I put the actual true price of that item.
So we will end up with these four lists a list of specialist results from our proprietary LLM frontier
Rag based results that come from GPT four, with our extra context and the random forest results, and
then the ground truth, the real numbers.
And so we build all of that.
I'm going to do a trick now which which is fairly common.
It's the kind of thing you can really play with.
I'm going to add two more, uh, into this.
One of them is called mins and is the minimum of those three, and the other is called Max's and it's
the maximum of those three.
It's just it's another Um, fact that might might have some signal in there.
It might be useful to also look at what is the lowest estimate that the three models had, and what
is the highest estimate that they had for any one product.
So now at this point we now have five results for each each of the 250 products, the specialist one,
the frontier one, the random forest one, the minimum of those three, and the maximum of those three.
They are sitting in five collections and I make a pandas dataframe out of those five specialist frontier,
random forest, min and max.
And I take the prices, the ground truth, and I convert that into a series.
And I call this x and I call this y, which will be familiar to anyone from a traditional machine learning
background.
And I can then do exactly what we also did during week six, which is I can say, uh, let's train a
linear regression model, a simple linear regression that says what weighted average of these different
series gives you the best fit, the best result for this data.
And so we do that.
Um, and this is what we get.
These are the coefficients.
So both the min's and the Max's get pretty high weighting.
Uh, so generally speaking it's taking its most, uh, been looking at some combination of the minimum
and the maximum as what it has latched onto.
Um, then it's taken a healthy share of the specialist proprietary LM and a much smaller share of the
frontier model.
And somewhat bizarrely, it's actually said the frontier.
There is there is some signal in Random Forest, but it's going to subtract out that, uh, you'll see
it's given a pretty large intercept and subtracted out a portion of the random forest numbers.
So that's a curious result, which indicates that maybe the random forest numbers weren't weren't that
good.
But it it does think it's useful to incorporate that in the overall puzzle.
Now you're probably you're going to thinking in your mind.
You point out to me that random forest is already baked into these two, so I can't.
You can't read too much into the fact that it's got a negative number there, because it's already factored
into the min and the max numbers.
So you can run this again, taking out min and Max to probably get a better assessment of how it weighs
up those three models.
Um, so that that's all it takes to build an ensemble model, because now we can use this model to take
in these different factors and predict a price, taking the best linear combination of the models that
we feed it.
So first I save that to ensemble model so that we've got that captured for the future.
We don't have to run it every time.
And I have made a new agent called Ensemble Agent.
Let's go and take a look at Ensemble Agent right now.
Here it is.
This is the code for ensemble agent.
And it's very simple.
Uh, it looks like I need to add some comments in here, which I will do.
So before you get to see this yourself, uh, it needs comments.
Bad meat.
Uh, so sorry about that.
In the init, we set it up by creating the three agents that it will be using for the different, uh,
parts of its pricing.
Uh, and we also load in its model weights the weighted combination when it comes to running the ensemble
agent to do a price, uh, we calculate the price of the specialist by calling price.
We call price for the frontier.
We call price for the random forest.
We build a data frame for X, including the min and the max.
And finally we call Model.predict to predict why that should really be Y hat if we're using data science
speak and we return that the prediction.
Uh, so it's hopefully crystal clear for you.
It's simply a way of packaging up the call to our linear regression model that gives a linear combination
of the different models that we've built before.
And so with that of course, the next thing that you can imagine I tried out pricing the same, uh,
the the the microphone I've got right here.
And it came up with a number that's somewhere in the middle, which is exactly what we were expecting.
I package it into a function ensemble processor, and then of course, I call the tester dot test with
the ensemble.
Now, this takes a while to run because it's calling all these different models and modal takes a while.
So I've run it in advance.
And if you're watching this, remember it will take a few minutes for the first one while modal warms
up, and then it's a few seconds for each of these.
Out they come and as you will see, there's a few reds in there.
I will tell you.
Somewhat disappointingly, I was really hoping this would move the needle and beat, uh, the, the,
the amazing proprietary model that we've got, somewhat disappointingly, using this, uh, approach
of Ensembling multiple models seems to have moved us a hair poorer, a hair worse for this test data
set, uh, than than we were at before.
But you've got to imagine that that's that's more an artifact to the fact that it's fairly noisy.
It's very, very close.
It has to be an improvement that we're carrying out this ensemble of different models.
Um, but there's clearly some more work that needs to be done here.
Uh, the chart looks looks very nice, but there's some intercept problem there that might be that that
intercept number was too high.
Uh, on on what it did.
Uh, and rather than spending a lot of time iterating over this, I think this is the time to say it's
over to you.
Now, I've spent a fair amount of time on this, but not so much on the ensembling technique and on
some of these others.
And it's wonderful to experiment with this because it's so easy to add on more terms, more serieses,
and pass that into the linear regression as you build the ensemble.
And this is a data scientist's dream.
You've got data.
You've got a clear, measurable, a clear way of determining success.
And lots to experiment on, lots of hyperparameters and quite quick gratification.
You can make the change and see the response very quickly.
So you can do better than me.
This is very much a challenge.
You're now armed with lots of good tools.
You may have already built a proprietary model that beats me.
And even if not, you can, I'm sure use this ensembling technique to get ahead.
So with that, that concludes the lab work for this part before we return to the slides.
The just to say remember that whilst this the key objective for this was not necessarily to get super
deep on how you price products, it was to solidify your understanding of things like vector embeddings,
rag the running different models and go from a stage of being fairly confident with this kind of material
to being advanced and super confident with it.
And I hope you've got there now.
And if you haven't, go back through these notebooks and go through each cell by cell and inspect the
outcomes and convince yourself until you are very, very confident.
And I will see you back in the slides in the next video.