Hi everyone.
So the reason I'm so fired up about week seven is that this is the time when we actually start to build
our own proprietary model.
It's when we start to build our own AI, and we are looking to build something that we're going to try
and make to compete with frontier models or yes, even we're going to try and see if we can outperform
at least some frontier models.
Now, the reason that that's even possible is that we are trying to solve one particular specific problem
where we have a lot of data, and so we can take potentially a model that is significantly smaller than
a frontier model.
We can take something like an 8 billion parameter model.
And by making it very good at one focused task, we can compete against the, the, the enormous the
Giants, the Giants like GPT four zero.
Because even though GPT four has trillions of parameters, it's designed to do so many different things,
including writing witty poems about presidential Candidates.
We are not going to be able to do any of those things, but we are going to try and get really, really
good at predicting prices of products.
That's going to be our thing.
That's our business problem.
That's what we're setting out to solve.
The first step in this is to decide which model we're going to use as our base model for fine tuning,
and that's something that's worth taking time on, because it's going to be so obviously pivotal to
what we end up building.
You can think of it in itself as like a one massive great hyper parameter.
We can try a different base model and see how that performs.
And as part of that, we're going to have to decide whether we take the original pre-trained version
of that model, the very base model.
Sometimes people use the word foundation model to to describe that, as opposed to the one that's been
fine tuned for chat purposes that sometimes called the instruct variant, which is the one that's expecting
a dictionary or a list of dictionaries with a user prompt system, prompt user assistant, and so on.
That style of input, which you remember, gets turned into just a series of tokens separating out different
sections of the prompt.
So we've got to make a decision about that.
And there are of course, pros and cons.
Um, and then once we've done that, we should then take the base model as it is off the shelf and see
how that does against our challenge, because, you know, we see whether we're already in a position
where we are competing with the frontier models.
And even if we're not competing with frontier models, remember that this is going to be free of charge,
or at least it will be the cost of us.
Running computes so we won't have to pay API costs when we're running our own open source version of
the model.
So even if we're in the right territory, there's still many reasons why we should use open source.
Even before we talk about making it into a proprietary model specialized for this particular business
problem.
So anyways, with that in mind, it's time for us to talk about which model to use.
Uh, first of all, we need to decide how many parameters to use.
And, you know, it is probably a case that more is more.
More parameters is going to give us a better shot at, uh, solving the problem.
Well, particularly in a world as we have it, where we have a lot of training data, we have 400,000
examples.
So we've got tons of training data.
It's not like we're limited in that regard.
And we could always generate more if we wanted to.
So really our constraint is going to be, uh, it's going to be the amount of memory capacity that we
have.
We're going to want to fit it in one smaller box.
And so we pretty much already know at this point that we can we can get in about an 8 billion parameter
model.
7 to 8 billion parameter.
We won't be able to go much more than that.
And whilst there are some small parameter models, very small uh, like like Gemma goes down to, I
think a 2 or 3 billion parameter version.
Uh, we'll take a look in a second.
Um, the, uh, we're probably going to want to, to take eight as the, uh, the largest possible model
that will fit into the boxes that we can afford.
So that's going to be our approach when it comes to the base or the instruct variants.
There are pros and cons and it's something to experiment with.
Generally speaking, if you're fine tuning specifically for one problem where you have a particular
prompt that you'll be using and expecting a response in a particular way, then you might as well start
with a base model, not an instruct variant, because you don't need to apply things like system prompts
and user prompts because you're just going to be training it for one purpose anyway.
So you might as well just take a base one and fine tune it, so that all it's really good at is that
one task.
So that's the kind of default answer.
Now there are some benefits to starting with the instruct variant.
One of them is that it's very good.
It's already been trained to recognize things like a system prompt and the user assistant interaction.
And you can use that to your advantage by using the system prompt to sort of tee it up in a certain
way to to frame it, to be working, uh, to be, uh, taking a persona, uh, rather than having to
rely on it, learning that persona through all of your training data.
So there are some situations where the instruct variant is a better starting point.
Uh, now, for us, my instinct is that base is the better starting point, because we've got such a
set structure that we're going to be using for all of our training data and for our objective.
Um, and I've in fact tried both and base did do slightly better than the instruct variant, but they
were very close.
Uh, you could also try both and see if you get the same results as me.
Uh, but uh, I would think it would be more normal in our kind of situation, when you have one particular
task and you don't need to be applying system prompts and the like that you would start with the base
variant.
But with that introduction, let's head over to the Hugging face, uh, open leaderboard and take a
look at some models.