And welcome to day five.
For reals.
We're actually in the proper Jupyter notebook.
This time we're in day five, in week five, ready for action.
And it's the same as before.
It's a duplicate of day four, not day four and a half.
We're using the Chroma Datastore here, and I'm going to really quickly go through this because you
know all of this already.
And we're going to get back to our Gradio interface.
Just like that.
It caught up with us.
It's drawn.
It's 2D and 3D diagrams behind the scenes.
But no time for that.
Now.
We need to press on.
Uh, first of all, I might as well show you that that aviary test we did before still works in chroma
as it did in vice.
So let's just quickly try that ourselves.
Um, what did aviary spelt wrong do before ensure?
Um, and I imagine.
We'll see.
Yes.
That chroma has no problems whatsoever with that either.
No surprises there.
Okay, but let me now show you something which isn't going to go so well.
First of all, I want to take a peek at an employee HR document.
If we go into our knowledge base and we go to employees, we're going to look at the employee record
for a certain Maxine Thompson.
Let's open this with markdown.
So we see it in its full markdown glory.
Here is HR record for Maxine Thompson, um, a data engineer in Austin, Texas.
And the thing I wanted to draw your attention to for one second is that if you look down here, you'll
notice that Maxine was recognized as the Ensure Elm Innovator of the year in 2023.
She received the prestigious I o T Award in Elm Innovator of the year award in 2023.
Now, I have to confess, I added this sentence in myself.
It wasn't as if this was, uh, invented as part of the synthetic data, uh, by, uh, GPT four or
Claude.
This is all my doing.
And it's awful.
So blame me.
Uh, so what we're going to do now is we're going to go back to our day five, and we are going to ask
the question, who won?
Oh, we say, who received the prestigious, uh and Shriram Innovator of the year award in 2023.
And let's see what it says.
It says I don't know.
And quite a blunt way.
Quite curt.
Uh, so that's interesting.
Uh, it has failed.
That was information that it was provided.
It was there in the documents.
And that is a bit disappointing.
And so the thing to do now is to try and diagnose this problem.
And in doing so, we're going to learn a little bit about how Lang Chain works under the hood.
And it's not going to be very surprising.
Uh, so here we get to see what's going on.
Uh, there is this very useful thing we can do, which is create something called the standard out callback
handler, which, much as it sounds, is going to be something which will let us print to the standard
out what is going on behind the scenes.
So this is the same familiar code that you're very used to.
We create the alarm.
We create the memory.
We create the retriever and we create our conversation chain in this beautiful one liner passing in
the LM, the retriever, the memory.
And now you can see I'm passing in one more thing, which is a list of callbacks.
And I'm only creating one callback in here, which is this standard out callback handler.
And that as you can probably, uh, expect, is going to be printing, uh, repeatedly to standard out
as this conversation chain runs.
So here is the question again who won?
I put it differently.
Let's do it the same way.
Who received the prestigious Iet Award in 2023?
There we go.
We'll ask that question.
We'll get back the answer.
We'll see what it says.
So we get this kind of trace as we look through it, which gives us a bit of insight into how Lang Chain
works.
It has these different objects.
These are called the chain that are sort of hooked together as it goes through the steps of building
the conversation, the Rag query.
And you can actually use different callbacks to be printing lots more detail about what's happening
at each stage, should you wish.
But what we really care about is the prompt that ends up going to GPT four.
And here it is.
System.
Use the following piece of context to answer the user's question.
If you don't know the answer, just say that you don't know.
Don't try to make up an answer.
I think this is really interesting, because this is the prompt that specialists at Lang Chain, like
experts, have crafted as an ideal prompt to send to different llms.
And so this is a great one for for you to steal and use in your own projects.
It's very carefully written.
It's clearly very effective because it stopped, uh, GPT four from hallucinating.
Um, and so it's nice well worded prompting.
But here's the problem.
This is the context that was then provided to the to the LM coming up right here.
And you'll see that it is, in fact a few chunks taken from different chunks that we've got.
It's 2 or 3 chunks and they appear to be taken from HR records.
But they're not right.
Because they don't mention the I o t award.
So it's wrong chunks that have been identified unfortunately in this case.
Um oh, and that this at the end here is is the question.
It says human who received the prestigious I o t award.
I'm the human.
Uh, and clearly there wasn't good context to answer that question in what comes above.
And that's why the response was, I don't know.
So what can we do about this?
Well, it's a it's a very common problem with Rag when you find that you're not providing the right
context.
And there's a few different things that you can do.
Uh, one of them is to go back and look at your chunking strategy.
How are you dividing documents into chunks?
And are you doing that?
Right.
And there's a few things that we could try right off the bat.
One of them is instead of chunking, we could send entire documents in as the context.
So we we just put full documents in Cromer and then we look for the document that's closest.
We could also go the other way and chunk more, have more fine grained chunks, smaller chunks.
We can also investigate that overlap between chunks to see if we increase or decrease the overlap,
presumably increase.
In this case, we are more likely to provide a useful chunk.
So those are all things to investigate to get your chunking strategy working well so the right context
is being provided.
There is another thing.
And it's very simple.
And it's what we're going to do in this case.
And that is to control the number of chunks, the amount of context that actually does get sent in.
Um, so in our case, we're just sending a I think it's actually three chunks that are getting sent
in here, and you can actually control the number of chunks that get sent in, and you can do that in
this way.
When we create the retriever vector store as retriever, we can actually say how many chunks we want
returned and passed in.
And in this case, I have specified now that I want 25 chunks to be created and passed in.
As a general rule of thumb, it's a good idea to send a lot of context to the LLM.
Llms are very good at, at uh, only focusing on relevant contexts and ignoring irrelevant context.
So it's good practice to send plenty of chunks.
There are a few occasional situations where it's better not to do that.
One of them, for example, is in one of the very latest models that OpenAI is offering, a model which
looks in much more detail at the prompt and does some more analysis behind the scenes to really understand
it.
Sort of chain of thought processing on it.
Um, and the recommendation there is that you don't provide lots of extra irrelevant context because
that will slow down and distract the model.
But with those occasional examples to one side, general rule of thumb is that more context is generally
a good thing.
And so in this case, there's not much harm in providing the 25 nearest chunks rather than 2 or 3 nearest
chunks.
We've got a total of what, 123 chunks.
So this is still about a fifth of our total data.
So we're not shipping our entire data set.
We're picking the most relevant 25 chunks, the most relevant fifth of our content to send the LLM.
So let's see if this works.
So we will run this.
And then as as before we will bring up our usual Gradio interface.
And right off the bat we'll ask the question who won the uh sorry who received to use the.
So I keep it consistent.
Who received the prestigious I t y Award in 2023.
And let's see.
Drum roll please.
Maxine.
Maxine received the prestigious I o t 2023 award.
So indeed, providing more chunks to the LM did solve the problem.
So with that, the exercise for you is to now go back, experiment with this.
Try some hard questions.
You could always insert a few things in the documents yourself and see what happens.
Um, and then experiment with different chunking strategies.
Try full documents, try smaller chunks, maybe 100 characters with more and less overlap, and get
a good feel for how that affects the quality of results and and how you can either give too little context.
Maybe you can see some effects of providing too much context.
Maybe that causes the responses to be less accurate.
So experiment, get a good sense for the the the good and the bad, and a good knack for how to do this
in a way that is most effective.
And I will see you for the next video to wrap up.