Welcome back to JupyterLab and welcome to your first experiment with the world of Long Chain.
Let me just remind you one more time how important it is that you are going through these Jupyter notebooks,
as I do.
Ideally, at the same time as I'm talking to you, you also are able to have Jupyter Lab up and you
have it turned to week five and to this day to day two, and you're able to step through this as I step
through it.
If that's not possible, then immediately afterwards as soon as you can, then then give this a shot.
It's so important that you experience this for yourself, particularly as we start talking about concepts
like text chunks and then later vectors.
It's going to be super important that you validate what I'm saying, that you experiment with it, that
you see it for yourself, and try out the code and print out various things to get very comfortable
with the way it's working behind the scenes.
So we are, of course, in week five, in the folder we're on day two, and we're looking now at what
is a bit of a copy of the previous day, but with more to come.
We do some imports and now we've got some new imports.
Our first time importing some code from Lang Chain, we're going to import some things called document
loaders, which are utility classes which help us load in files.
There's one called directory loader that loads in an entire folder and text loader for loading in an
individual text file.
I'm also importing something called character text splitter, which is something that's able to take
in, uh, and divide a document into chunks of characters, as you will see.
Let's run those imports.
Uh, I set some constants that we're not actually going to use today, but we will do next time.
Uh, and let's get to business.
So you remember last time we did a hokey thing where we read in documents and put them in a dictionary?
The key was the name of the document.
The value was the contents of the document.
Well, this time we're going to do something a bit smarter.
Using lang chains help.
So we first get a list of the different folders in our knowledge base.
You'll remember those folders are these company contracts employees and products.
So we put that into folders.
And now for each of those folders we are going to first get the type of document company contracts employees
or products.
And we are then going to load in this directory using the directory loader where you pass in the the
the handle to it, the directory path.
And we also pass in something called the text loader loader class, which tells it that we should use
that to bring in each of these files because they are text files, and it's as simple as that.
We then just call loader dot load and it will bring in all of those documents.
Going to iterate through each of those documents.
I'm going to say can we set the metadata on each document.
We want to add in something called doc type and set that to be the Doctype, whether it's company contracts,
employees or products.
And then add that to a list called documents.
Hope that made sense.
I'm going to run it and just show you that as a result of doing that, we now have 31 objects of type
document and I can show you what one of them looks like.
Here it is.
It is in the knowledge base directory.
The metadata has something called source which tells you where it is.
And then this is doctype is what we added in products.
So it's a product.
It's called realm MD and that is the full contents of the file.
Let's go and satisfy ourselves that if we go into products there is indeed something called realm.md.
And if I double click on that, we'll see presumably that it is the same thing that got loaded in.
We can also look at the first document here.
And it's also in products.
And it's called Mark calm.
And there you can see where that comes from.
And let's just pick some random number 24.
The 24th one of our documents is an employee HR record for Maxine Thompson.
And there is Maxine.
She's a doc type employee and there is her contents.
So nothing very complicated.
We've loaded in the documents and we've given them a doc type, and they are 31 of them sitting in documents.
Next thing we're going to do is use this text splitter thing, which is going to take the the documents,
and it's going to divide each document into chunks of characters.
And you specify two things to lang chain when you do this.
One is the chunk size, which is roughly how many characters do you want to fit into each chunk?
And I say roughly, because we're going to give lang chains some discretion to make sure that it that
it tries to split these chunks in sensible boundaries where there's a space and an empty line or something,
or a section between different parts of the document.
So it's not cutting in the middle of a paragraph or in the middle of a word or something.
That would make no sense.
And that would result potentially in poor contacts that we'd end up providing to the LM.
Chunk overlap says that we don't want these chunks of characters to be, uh, to, to be completely
separate from each other.
We want to have some level of overlap between them.
So there's some content of the document that's in common across two chunks.
Uh, again, so that it's more likely if you put in a query that will will pluck out a bunch of chunks
that will be relevant to that query.
We don't want to to risk because of some association, because of some critical word.
And it only gets included in one chunk that we don't include another chunk that's really close to it.
That's equally important.
So the chunk overlap gives us this way of having potentially multiple chunks that contain some of the
same keywords.
So that is a text splitter.
And we just say split documents and we pass in the documents.
And if I run that it's going to to run.
And it warns me that one of the chunks that it created, uh, has a size of 1088, which is bigger than
what we'd asked for.
And again, that's because it's trying to be smart about how it respects boundaries.
Um, and so it's this, this is the decision that it has made.
So if we look at how many chunks we've ended up with, we've ended up with 123, uh, chunks, 123 chunks
from our 31 documents.
And what we can now do is we can pick a chunk, which we pick, pick chunk number five and have a look
at that chunk.
Uh, the chunk itself has metadata, just like a document had metadata.
Um, it knows the source where it came from.
And it has doc type, the doc type that we that we had set.
up.
So we know that this particular chunk has been plucked from the products.
Um, and it's in fact the product summary about Markham.
And you can see how it just starts with a new section and it ends at the end of that section.
So it's been careful to respect boundaries.
It's got a reasonable chunk that's about a thousand characters.
Um, and there'll be some overlap perhaps with the chunk right before it.
Uh, see if it chose to.
Not in this case, the chunk before it is a very small chunk.
Uh, so anyway, you can you can play around and see if you can find examples of where it's possible
options where, uh, there are overlaps between chunks.
Um, and so have a, have that as a quick to do go in and experiment with some of these chunks.
Find out if you can get an example of where two chunks contain the same information.
So what we're going to do now is we're going to just inspect the Doctype metadata across these chunks
and just convince ourselves that we have all the right doc types.
So let's just see what what we have across all of our chunks.
We have four doc types employees, contracts, company and products.
Which is good because that of course, uh, exactly matches the four directories that we read in.
So that all sounds good.
Uh, let's now let me just, uh, let's, let's have a look to see if we were to look through each of
these different chunks.
Which chunks have the word Lancaster in it?
Lancaster.
Hopefully it's familiar to you, because that is the fictitious name of our fictitious CEO of our fictitious
company.
So Lancaster is her last name.
And let's see which chunks have her last name in it.
So here we go.
Um, so the about insurance, uh, document Has her name in it, of course, as she was the founder
of the of the company.
Um, and then her HR record has Lancaster in it and the bottom of our HR record.
Also, there's some other HR notes that mentions her, uh, by last name as well.
So, uh, you probably realize this, but from the cheap and cheerful version of rag that we did last
time, one of the big problems is that it just looks for the word Lancaster.
So if there's any information about the CEO that isn't reflected in the worst Lancaster, then that
would be missed in our in our toy version from last time.
And you can see that if we do a search for Avery, her first name, you'll see that we get way more
chunks because she's mentioned by first name all over the place.
Um, and so, uh, yeah.
And presumably also if we do CEO, we find that there's a bunch of chunks with with CEO in them as well
that would potentially be missed if we were just, uh, looking at purely the, uh, those with the
word Lancaster.
So it gives you a sense that doing some kind of text based search through the documents is not a great
way of doing it, and will miss important bits of context that we need to find.
So what we're looking for in our vector search is something that can be smarter about the way it finds
chunks that are relevant using not just text, but using some understanding of the meaning behind what
you're looking for.
And that, of course, is the big idea behind Rag.
So at this point, hopefully you are proficient with reading in documents using the text loader and
directory loader and dividing them into chunks.
And if you do this exercise, you'll come in and play with the different chunks and convince yourself
that they're being split in sensible places and that there's some overlap.
And then we'll be ready to vectorize and put in vector databases.