So we now turn to the parts of the problem, which is perhaps, let's say, not as glamorous as some
of the rest of it, but it is perhaps the most essential, which is finding and crafting your data.
And there are a bunch of places that people can go to look for data to hunt data.
But the first, the first place you go.
First and foremost, of course, is your own proprietary data that your company has.
That is something which hopefully is pertains directly to the problem you're solving, and is going
to be crucial for your fine tuning.
In the case of the pretend fictitious Rag project, we did.
We we pretended we had the shared drive of the company that we used then to build our knowledge base.
That's an example of where we went to find proprietary data.
In the case of of my business, Nebula, we have information about talent and jobs, careers that we
can use to train proprietary Models.
So finding your own company, proprietary data sets that are specific for your problem, that's the
first place to start.
Of course, then there is Kaggle, a wonderful resource for data scientists.
You probably heard of it and used it.
If not, go and take a look at it.
It's got, uh, so much, uh, data.
Data, um, that goes over a long period of time that people have contributed to Kaggle.
And then there is, of course, hugging face.
Uh, that is just such a fabulous resource for us.
And we will be using hugging Face in just a moment.
There's also synthetic data, as it happens.
For our Rag project, we didn't use a real company's shared drive.
We used an LLM to generate synthetic data and that is an option.
There are pros and cons, of course.
If you're if you're trying to use the frontier model to actually, uh, learn from the data, then it
may not make sense to have the frontier model generate the data and then learn from it, but if you're
trying to build your own model, or you're trying to build a cheaper model, that's that's going to
be seeded by a frontier model, then you could use the front end model to generate some some data and
then use that data to train your smaller, cheaper, more lightweight model.
So the various circumstances where synthetic data makes sense.
And then I'll mention that there are specialist companies whose task it is to go out there and curate
a data set for you.
We actually encountered this company earlier when we were looking at one of the leaderboards, the leaderboard
called seal, which were business specific leaderboards for Llms that is put together by a company called
scale.
And scale specializes in building crafted data sets for your problem.
So that is another place to go.
In our case though, we are going to Hugging Face, which is a treasure trove of data and it contains
much data contributed by the community, including this particular data set.
Scrape over many years of Amazon reviews.
That is, uh, enormous.
It is an absolutely enormous data set.
And in addition to the reviews that it takes, it also has metadata just associated with the products,
including the description of the products and their prices.
And that, of course, is exactly what we're after, product descriptions and prices.
And this data set has them in large quantities.
And so it is perfect for us.
This is where we will be heading.
So how do you go about digging into the data.
What are the steps that you take?
We're going to be doing some of this work today, and we're going to be refining some of it tomorrow.
Um, but there are these six different stages, perhaps, to getting deep into the data.
First of all, there's a time when you're Investigating, just understanding the data, what fields
to have?
How well populated is the data?
What kind of data quality issues do you have?
Then typically at least the way I like to approach this is then parsing in that data into a structure
which is going to be easier to handle.
Um, that, that's, that's uh, so that you're no longer working with raw data sets.
You're working typically with, with objects at that point.
It's great then to do some visualizations, it's important to see things like the, the how wide, how
spread out.
Some of your values are when we're thinking of things like prices of products, what's the range in
prices?
Does it turn out that there are a lot of that the distribution is skewed in some way.
So visualize it so you can get a really good sense of it.
Now a deeper assessment of data quality.
Understand what kind of limitations you have with your data.
Um, that will help you decide how to take action on it and curate is when you then go and decide how
you're going to craft this data set.
For example, if it turns out that you discover that a quarter of your data is in has poor data quality
in some way, you may decide just to exclude that data altogether.
You may think that you have a big enough sample that you can focus on the three quarters.
If you find that your data set is very imbalanced in some way, and you're concerned that that as part
of training, the model will only learn a sort of a particular balance of the data, then this would
be a time to potentially fix address that balance.
So curating is about crafting the data set that is most suitable for your training and then saving it.
In our case, we'll be uploading it to the Huggingface hub.
That's the final step before you'd be ready for training.
So with that, we're going to head to JupyterLab for the first time with our new project, and we're
going to get about curating some data.
See you there.