And welcome once more to our favorite place to be Jupyter Lab, the Paradise for a data scientist experimenter.
So as we launch into the day three notebook in week eight, you'll see that there are precious few comments
this time around.
I'm commenting the code that is our production quality code in the notebooks.
At this point.
You guys are pros.
You don't need a whole ton of waffle from me, so we just dive straight in.
Uh, so I'm going to begin with some imports.
As we get into today's puzzle, uh, I'm going to set up some environment things, and now we get to
the first piece of substance.
Uh, I, uh, sneakily imported something called Scraped deal, uh, here without mentioning it, but
I now have this scraped deal, and I'm going to call fetch.
Um, with setting show progress to true, and I'm going to run that without telling you what it does,
because it takes a couple of minutes, so we might as well let it run.
And then I will talk more about it.
So kick it off.
Off it goes.
So what is scraped?
Deal.
It is sitting in the agents folder in the package.
And here is it is in part of deals dot pie.
So let me show you deals dot pie.
So uh, this is a Python module which starts by defining a series of feeds.
And these are URLs to RSS feeds, which are useful ones that happen to have good deals being announced
in a various categories.
And these categories happen to be close matches for the categories that we know a thing or two about
because our model was trained for them electronics, computers, automotive, and then mostly smart
home and home garden.
So here are some feeds.
There are a bunch of others, and if you have the stomach to wait a bit longer, you can slap in a bunch
more feeds in here and have a bigger data set to work with.
So then there are some utilities here that you can look through in your own time.
A method called extract, a function called extract, which cleans up some HTML and returns useful text.
And then there's a class scraped deal.
And this represents a deal that we have retrieved from an RSS feed.
So it's not exactly scraping, it's retrieving it from RSS.
But we will do a bit of a of a lookup as well.
So so there's there's some truth to it.
Um, what we do in the init we take something called an entry, which is just a dictionary of values.
And we're going to pass in.
And this is something we get straight from the RSS feed.
And what we do is we pick the title, the summary, and we take the links.
And then if you look at here, what we do here is we take its URL and we actually go and fetch that.
So we do are doing some scraping here.
Sorry.
We do some some fetching of that URL and we put the results in stuff and then we parse stuff with beautiful
soup.
So this is similar to what we did in week one, day one when we were parsing URLs that we retrieved
from Requests.get.
Okay.
And then there's some more stuff here that takes the contents and scrubs it.
And then we potentially build some features if we have the features.
So this is all a bit of, uh, scraping code to be able to take something that comes in an RSS feed,
clean it up and turn it into a record.
Um, and then there's something that prints what it is.
And this class method fetch, is exactly the one that we just kicked off a moment ago that you can see
has finished running.
Uh, and what it does is it iterates through all of the feeds, um, and it calls something called feed
parser.parse.
And feedparser is a package that I've imported, which is a useful package that allows you to pull RSS
feeds and it will give them to you as a dictionary.
So we are using Feedparser as our Python package for doing this, and we're just taking the top ten
that comes back from each of these different feeds.
We take the top ten deals that we get from each one.
And of course this is something where you can choose to bring back more data if you wish.
For the moment, I'm just constraining it to that number.
And then for each of them we create this is this is ourselves.
We create an instance of us, uh, for that entry.
That entry, of course, is the dictionary that we looked at a moment ago.
Um, and there's a little time.sleep in here, if you're wondering about that.
Uh, that's because I figured that since we're then going and doing a get to to retrieve that web page
from this deals website, it was antisocial.
If we hammer that website with tons of requests, one after another with a split second between them.
So it's considered good scraping practices to put in a sleep so that you're not overly, uh, beating
up a web server and being too needy.
So this is a way of us being better citizens when we retrieve these deals from the websites, and then
it returns those deals.
So that's what I just did.
And those deals should be sitting waiting for us in this notebook.
But I will first just mention a couple of other things in this useful module.
Um, there are these three classes here that are going to be important in a minute because this is how
we define structured outputs.
When we ask GPT four to respond, um, we are defining here a class deal, a class deal selection and
then a class opportunity.
And you can see that these are subclasses of base model.
Base model is uh, comes from the Pydantic package, which does a number of different things.
One of the things it does is it very easily allows you to switch between JSON versions of of a class
and its structure and the class itself, and it also is able to enforce that a class adheres to a schema.
So there's a lot about it that probably many of you are very familiar with.
Pedantic.
Um, but but all you need to do to use it is simply create a new class that is a subclass of base model.
So our first class that we define is just called a deal.
And it is something which just has a product description, a price and a URL.
That's it description price URL.
And then we have another thing called deal selection.
And this is what we're going to ask GPT four to respond with.
We're going to tell it we want a deal selection.
So this is the important one.
And it's very simple.
It's just something that has a list of these deals in a single attribute called deals.
That's that's all there is to it.
So deal selection means I want a list of deal objects.
So if you think of this in your mind in JSON speak, what?
What this will look like in JSON terms is it's going to be like this deal selection is a single object
which only has one attribute deals.
And that is a list.
And it's a list of things which are each objects.
So when that goes into JSON it will look like this.
It has a product description.
And it has a price which is a float.
And it has a URL which is a some kind of a.
URL like so uh, and that makes a deal.
And there is a whole bunch of them potentially in a list of deals.
And that makes up a deal selection.
So if you look at the JSON that I just typed there and compare it to these class definitions, I hope
it becomes clearer in your mind how they are analogous to each other.
This is just the JSON representation of this structure here.
And indeed, when we say to GPT four, we want the structured output to be in this format, what we're
kind of doing is saying we want this to be the kind of JSON that you respond with.
That's all that's going on.
So that hopefully gives you a sense of of how this works.
And I'll delete that.
Now that's not necessary.
The final thing to mention is that there's also a class called opportunity that we define here, which
basically is something which has a deal.
One of these guys, and also an estimate, which is something later we're going to use when we are estimating
the value of these deals.
And then the discount is simply going to be the difference between the deal's price and the estimate
it's at.
How much of a discount are we finding that this is being offered?
So that is the the setup.
Uh, and with that, let's go back over here.
Um, so, um, let's fix something there.
Uh, so the, uh, the we've just run, scrape, deal.
Dot, fetch.
Uh, we can now look at how many do we have?
We have 50 deals sitting in deals.
The reason we have 50 deals is because we have, uh, we had five feeds, and we asked for ten deals
from each feed.
And so that comes to 50, obviously.
Uh, so that's hopefully what you're expecting to hear.
Uh, if we look at, uh, um, deal number 44, um, it prints out nicely because you might have seen
I had a, I used one of the Python magic functions to make sure that it was going to print nicely the
repro function.
Um, and so that is what deal number 44 is.
And if we do the full Describe.
This is the full bit of information we have about it.
It's loads daily deal garage storage bla bla bla bla bla bla bla.
Choose install to dodge the shipping fee.
Uh.
So here's the thing.
If you look at that, you'll notice a couple of things about it.
One of them is that the price doesn't come separately in the RSS feed.
We don't get the price point, we just get the description of it.
And the other thing you'll see about it is that this one doesn't even have a price.
It's telling you how much off it is.
It's telling you some things about the free shipping and things like that, but it's not actually giving
you a price associated with this product.
And that's a bore.
And so that means that we're not going to be able to always use these properly.
And we're going to have to do some parsing to figure out what is the actual price that's being offered
against each of these items.
What you'll find if you look at these items is also some of them combine Multiple, uh, things in one,
uh, blurb.
There's maybe different models of the Apple Watch that are all being offered 20% off.
And so trying to digest that and pull out what we want is going to be challenging, very challenging,
very hard to code that in a way that would be robust.
And that's why we need to use a frontier model.
We are going to use GPT four zero to take each of our RSS feeds, scraped deals, and turn that scraped
deal into something, which is a good, useful deal for us.
We're actually going to send it all 50 and say, look, we want you to find the best five deals which
are most clearly explained from this big set, pluck them out and summarize it back to us.
And we want that in structured output.
We're going to tell you what format we want, and we're going to ask you to respond with exactly that
format.
And so now you now you've, uh, it's been teed up.
You understand what we're trying to accomplish.
And we're going to do it in the next video.