Welcome to week six.
Day two a day.
When we get back into the data, we look back in anger at our data sets, and we build a massive data
set that is going to allow us to move the needle when we get to training.
But first, the majority of today is actually going to be spent talking.
It's going to be a talky, uh, it's going to be a session when we're speaking about strategy, perhaps
not the most gripping stuff, not why you signed up, but it is very important.
This is good foundational information that is going to ensure that we're approaching what's to come
in the right way.
In particular, I want to talk to you about a strategy for how you go from facing a business problem
all the way through to an LM solution in production and the steps it takes along that path.
And I want to tell you that now, because we're about to do it, we're going to go through that exercise
for our real commercial problem.
And it's important that you're able to relate to the journey that we go through, because you'll be
doing the same thing with your business problems after this.
I also want to take a moment to compare the three types of technique that we'll be talking about, or
that we've talked about for optimizing models.
I'm talking about whether we're prompting using Rag or using fine tuning.
And it's there's a lot of confusion about in what situations do you pick one of those different approaches.
And I want to demystify that and just give you some concrete examples of how you go about deciding what
technique to use.
So first then let me talk about the five step strategy to applying a model to a commercial problem.
And the first step is understanding.
This is about really getting deep into the business requirements and understanding what problem are
you solving?
How will you judge success?
What are the Non-functionals we'll talk about in a second and make sure that that's all carefully documented
preparation is then about things like testing baseline models and curating your data set and and generally
preparing yourself for what is to come.
And what is to come initially is selecting the models, the models that are going to be the either the
the model you're going to be using, or the handful of models you'll be using as part of the rest of
the project.
And this is where we'll Thai will draw on a lot of the content from prior weeks.
When we looked at leaderboards and analyzed the pros and cons of different models.
Customize is where we use one of the big techniques, like Rag or fine tuning to get more juice out
of the model, and then productionize something we've not talked about at all, but is hugely important,
which is then once we've done, we've built and trained our model and it's performing great.
What comes next?
Because it's not exactly like the Jupyter notebook that we've been hacking away at is going to end up
in production.
Something more has to be done, and that's what we will talk about in a sec.
Let's start with step one though.
So understanding and this is all common sense.
But you know this stuff can't be said enough.
So just very briefly of course you need to gather the business requirements.
You need to evaluate.
You need to understand up front how will success be measured.
Super important.
And we're not just talking about the data science metrics that we know well, but also how will your
your users and your your business sponsors decide whether the project has achieved its goals?
What are the ultimate business metrics that you may not have as immediate influence over, but they
need to be understood.
You need to dig into the data as we've been doing the quantity of it.
How much?
What's the DQ, what's the data quality situation like?
And the format, is it structured, unstructured or a bit of both?
really make sure that that is deeply understood, because that will affect the model you choose and
how you go about approaching this.
Determining the non-functional requirements.
Non-functional are stuff like your budget, the how, how much it will need to scale to latency is is
you know how long you can you can wait for each response back from the model if it needs to be a split
second response.
Um, and also understanding time to market.
Is there a requirement that this is built in a very short time frame, or is there time to to be working
on this?
And of course, if it's something that's needed in a very short time frame, it will lend itself to
a frontier model using an API.
And you know, when it comes to the user interface, something like Gradio is of course allows you to
be up and running in a matter of minutes.
So this will steer some of your later decisions.
When it comes to preparation, there are really three activities involved.
First of all, you need to research what is already out there, what kind of existing solutions.
Solve this problem today and get a very good handle for how well they perform and what they do already.
As part of that, you should look at at solutions that don't involve data science at all.
Maybe there's solutions that just have a few if statements in them, and then look at some traditional
data science solutions, perhaps like linear regression kind of model.
If this is something which is trying to predict product prices, say.
Then that would be a place that you would initially go to.
And even if you might say to me, look, I absolutely know, I have no question that an LLM is going
to massively outperform what's already out there or these existing models.
I don't care how they are today.
The answer would be it's still worth doing this because it gives you a baseline.
It gives you a starting point on which you will improve, and you'll be able to demonstrate the improvement
in a quantified way based on the investment that's made in the new model.
So even as just a baseline, this is a valuable exercise to do.
But more than that, you need to know what is already out there.
Then comparing the relevant LMS.
First of all, of course you remember we divided this into the basics stuff like the price, the context
length, the licensing constraints and then the benchmarks.
Looking on on leaderboards, looking at arenas and understanding if there are specialist scores for
what you're trying to do for this particular task.
Using things like the seal specialist leaderboards from scale.
Com that we that we mentioned last time, and of course curating the data, scrubbing it, pre-processing
it.
And then something that we haven't talked about particularly yet is splitting your data set.
Typically you take all of your data and you split it into your training data, and then you reserve
a chunk for what's called validation when you that you'll be using to evaluate your model, and then
you reserve a final chunk for test.
And that's something that you hold all the way out so that you can use the validation to be tweaking
your hyperparameters and getting everything right.
And at the very, very end, you will use the test to gauge the ultimate success of your model.
So, uh, cleaning your data, pre-processing it, uh, which is parsing, which is what we've been
doing, and then ultimately splitting it up is part of preparation.
And then select this is something that we've done already, uh, choosing LMS, uh, based on the criteria,
experimenting with them and then training and validating with your curated data.
We haven't done that yet.
That's something that that we are excited to do.
So I will now pause and we'll continue in the next session with the all important step four to optimize.