And continuing on our strategy to solve commercial problems with LMS, we get to step four, which is
about optimizing our model to solve problems really, really well.
Taking it beyond the pre-trained model that we might have, um, either or an existing frontier model
and getting more juice out of it.
So there are these three different approaches that two of them we've used, and one of them talked about
there is prompting that we've used a ton now, which is things like multi-shot prompting, chaining
when we have multiple prompts and using tools.
These are all ways to get better outcomes.
There is Rag, of course, that you're now super familiar with.
I do hope you've built that extra project to be doing a knowledge work on on your own life.
Uh, so there is rag and then the new thing, Fine tuning, which is about training the model to be
even better.
So these these are the three techniques.
And there's certainly a lot of confusion out there.
There's a lot of questions I get asked about.
How do you decide which technique to use in which situation.
And of course, it's possible to use all the techniques together, but one does typically, uh, focus
on one, uh, at least initially.
And it's worth pointing out that the first two techniques there are inference time techniques.
These are about taking a trained model and at inference time, figuring out how to get more juice out
of it.
And the third one is a training time, uh, technique.
It's about saying, all right, let's take a pre-trained model and figure out how to supply more data,
to tweak the weights to make it even better at solving its problem.
So to talk about the benefits of each of those techniques just very quickly in prompting, obviously
it's super fast to do this.
We've done this so quickly, so easily having different prompting strategies, uh, with maybe the exception
of tools, was a little bit more involved, but still, you get the idea.
You can just replicate that.
You can quite easily get to a point where you are continually improving, uh, the prompt messages to
an LLM and getting better and better results.
And you typically see very quick direct improvement from it.
You add in some multi-shot, uh, prompt, uh, background, some context into your prompts, and you
immediately get the improvement.
And it's a low cost too.
So lots of benefits of using prompting.
So rag, uh, has the benefit of bringing about this, this strong accuracy because you can pluck out,
uh, this, uh, this very specific fact of information to arm the LLM with.
It's, it's very scalable in that you can have huge quantities of data that can pour in, and your Rag
pipeline can pluck out the relevant context, so you don't have to spend all the extra money pumping
bigger and bigger prompts to your model.
And that ties to the third point, which is that it's efficient because you can you can do that.
Um, so fine tuning.
So what are the benefits?
So it allows you to build deep expertise, specialist skill sets into your model.
You can build a model that is really great at doing something in a way that is very nuanced.
So it's not just being given an extra fact.
Um, about the CEO.
What was our CEO's name?
Avery Lancaster.
It's not just being given a specific fact about Avery and what she used to do.
Uh, it's something which over time is learning.
I don't know the careers of CEOs, or it's learning about the insurer Elm Company and more about its
culture and about its communications.
So it gets this deeper insight, which allows it to show a kind of almost human like ability to reason
about the data that it's being showed.
So it's it's much more of a, um, it's, it's a much deeper way to change the abilities and capabilities
of the model than the inference time techniques.
It allows a model to learn a different style and tone.
Of course, you can achieve some of that by just prompting, as we saw early on when we just added a
system prompt and asked for for snarky comedic style, or when we had llms battling and we had a GPT
four zero being the adversary.
So you can do that with with system prompts.
But if you want a very subtle tone, like you want a model that's going to emulate the the style of
your customer service specialists who've been trained over many years, then they need it will need
to see a lot of data.
A lot of examples to learn from.
Um, and then the fourth point is that whilst this is something which requires a big investment in training,
once you've trained it, you can then run it at inference time.
And you don't need to do things like in Rag.
You have to then go and look up the context and provide that in the context that's no longer needed,
because you've already baked that into the model's weights.
So it's faster.
So what about the cons?
Well, many of these cons follow from the pros of the others, as you will see.
Uh, in the case of prompting, one con is that it's limited by the total context window.
Of course, you can only shove so much in the prompt, and even if you do use up, even if you've got
mega context windows like Gemini one five flash, uh, the million tokens, if you remember that, um,
you still find that if you pump lots and lots into that context, then you get somewhat diminishing
Returns from how much it learns from that at inference times, and obviously.
Inference itself becomes slower and more expensive.
The more context you are pumping in.
And if you're doing something like a prompt chaining when you're making multiple inference calls to
solve a bigger problem, then of course that slows everything down.
So rag some of the cons it's more of a lift to build it.
Um, you need the vector database.
You need to populate that vector database.
Um, it needs the, uh, sort of the, it needs the knowledge base to be supplied and kept up to date,
presumably if it's giving accurate data, if Avery Lancaster steps down as CEO, will need to make sure
that the rag, uh, effect reflects that.
Um, and it lacks nuance.
Um, it doesn't have the same, um, ability to to learn the deeper meaning behind the data.
It's just taking facts and the negatives of fine tuning.
Of course it is.
It's hard.
It's harder to to build it.
It's going to be we're going to have a lot of fun with it, but it's going to be that we're going to
be sweating.
It's going to be difficult.
Um, you need a ton of data.
You need a lot of examples.
Uh, it's, uh, depends on on how specialized you want to be and your objectives.
But generally speaking, we'll see that there's going to be a high data need, and there's going to
be a training cost.
There's one more con that's that's often talked about, which is known as catastrophic forgetting,
which sounds very serious.
Uh, catastrophic forgetting, if you hear that, is saying that, um, if you take a pre-trained model
like llama 3.1 and you fine tune it with a large amount of data, it will get better and better at solving
your particular problem, but over time it will.
Over over training time it will start to forget some of the base information in the base model, and
as a result, some of its quality might degrade if it's taken outside the specific kinds of questions
you're training it for.
Um, and so that's a that's a behavior that's been noticed and that has, has some, some concerning
ramifications.
So if you need to make sure that you don't lose any of the information in the base model, if that will
affect your performance, then you need to be careful about this.
All right.
So just to wrap up these then let me finish by saying that, uh, the the times when you typically use
them with prompting, it's often used as the starting point for a project.
Often your first version of your model will be perhaps a frontier model, and you will use prompting
as a way to, to add, uh, performance Rag is in the specific case where you want, you need the accuracy.
You don't want to spend the extra money on training and you have an existing knowledge base of data.
Then.
Then you're in perfectly suited for a Rag kind of workflow, and fine tuning is you have a specialized
task, you have a very high volume of data, and you need top performance.
Um, and, and you want nuance as well.
And that, of course, is a situation that we are in with our product price predictor.
We have tons of data.
We have a specialized task.
We want top performance, and we do want a nuanced understanding of products so much that it can differentiate
between a great variety in product prices.
Okay, I will pause here for one moment, and we will come back to wrap up the strategy section before
we then turn back to our data and get to curation.