Here we are for the day.
2.1 notebook.
And don't let it be said that I don't ever do anything for you.
As you will see, I have gone out on a limb with this one for your pleasure.
Uh, so, uh, again, we are just going to be visualizing our data store for a moment now.
Um, and to do that, we do some imports.
And there is then a cell here where we select the maximum number of data points that we want to show
in a visualization of our vectors.
And my recommendation is that you stick with 10,000 which is a safe number.
You get a nice image and your machine will not be ground to a halt.
But that would be no fun for you.
And I wanted to show you what it looks like if you get all 400,000 data points to show, but it's precarious
and it puts my machine in a very unsafe position that it might crash at any moment.
And indeed, in preparing for this, I have had my machine crash a couple of times and had to even start
again with this Jupyter notebook, so I do not recommend you do this unless you have a very powerful
machine.
Um, so, uh, in the code, we, we, uh, we connect to the vector database.
We have some code which is essentially a duplicate of what we already did in the rag week.
Uh, when we, um, got some did some pre work to collect from the vector data, store the objects themselves,
the documents, their categories that are in the metadata, and then color pick out the right color
that would be able to allow us to identify the different points.
And remember when I show you this, the thing that's super important to keep in mind is that the vectorization
process, the process of deciding what vector to use for each of the documents, was based purely on
the description of the documents themselves.
It was only based on the language in in each product description that we pulled all 400,000 product
descriptions.
The fact that we happen to know which which ones are appliances, which ones are automotive, which
ones are electronics.
The model is not told that the model that builds the vector is just given the text.
So it's helpful to then color it in so we can see.
All right.
This is the landscape of all of the vectors.
Are there trends.
Can we see that the model was able, just through the language, to separate out some of the different
kinds of thing that's there?
But this this kind of thing, this category was not part of the text that it vectorized.
Okay.
So anyway, with that, uh, this is now doing the t-SNE, uh, dimension reduction process, and this
took about an hour to run on my machine.
Uh, for the 400,000 it would take, it should be five minutes or something for 10,000.
Uh, and then we can create a scatter plot, much as we did before.
Uh, and then we can plot this scatter plot.
And now I will show you what it looks like.
And it's super slow to to do this on my machine.
But this rather beautiful thing here is the result of looking at all of the vectors.
Uh, let me try and and and shrink this a little bit.
The machine is running very slowly.
Uh, but you get a sense of the vector space from all 400,000 vectors.
Here we go.
It's just coming into view now.
And the important thing to see is that.
Yes, indeed.
Uh, there it goes.
Agonizingly slow.
Yes, indeed.
Different products have ended up most of the time in different territories, in vector space, with
some clusters that appear to be, uh, near each other.
And when you have a smaller number in here, you can go in and investigate the different ones and satisfy
yourself that the reason that they are potentially in another territory is because they are perhaps
products that straddle both being appliances and electronics or something like that.
So this is really just for for an opportunity to look at the data and investigate it and understand
it and just give a little bit more intuition about what does it mean to create vectors associated with
documents, um, and to, to store them.
Uh, and so it gives you that hands on tangible sense.
And I hope that you enjoy this image, and I hope it was worth almost breaking my box.
And hopefully you're doing it with a smaller number.
And I will see you next time to see some of this in 3D instead.