From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
700 lines
18 KiB
700 lines
18 KiB
WEBVTT |
|
|
|
00:00.950 --> 00:04.760 |
|
Well, the first thing you're going to notice is that I don't have a notebook open for you. |
|
|
|
00:04.760 --> 00:08.090 |
|
And that's because this time we're just going to be looking at code. |
|
|
|
00:08.090 --> 00:11.150 |
|
Uh, no notebooks for this recording. |
|
|
|
00:11.150 --> 00:18.710 |
|
For this video, we are going to look at the actual finally, the actual agent framework itself that |
|
|
|
00:18.710 --> 00:22.880 |
|
we're writing that we've, that we've built, that we're going to run it in. |
|
|
|
00:22.880 --> 00:27.530 |
|
And it sounds like something so fancy, but in fact, it's going to be incredibly simple. |
|
|
|
00:27.530 --> 00:32.240 |
|
And it's the reason why I didn't want to use an off the shelf kind of package, because I just wanted |
|
|
|
00:32.240 --> 00:37.880 |
|
to show you that there's nothing more than connecting together, different llms and bits of Python code |
|
|
|
00:37.880 --> 00:40.340 |
|
to carry out a more advanced function. |
|
|
|
00:40.640 --> 00:45.440 |
|
Now, typically when you talk about these sorts of agent frameworks, what you're looking for is you're |
|
|
|
00:45.440 --> 00:50.630 |
|
looking at something that can handle the database, uh, database connectivity, in our case to our |
|
|
|
00:50.630 --> 00:53.990 |
|
Chrome data store memory. |
|
|
|
00:54.080 --> 00:57.590 |
|
So we're going to want some sense of persistent memory. |
|
|
|
00:57.770 --> 01:03.270 |
|
Um, that can that can survive between different chats or of different instances, so we'll understand |
|
|
|
01:03.270 --> 01:05.040 |
|
how memory is going to work. |
|
|
|
01:05.700 --> 01:12.180 |
|
And also stuff like logging I mentioned is sort of good, good practice to be able to understand what's |
|
|
|
01:12.180 --> 01:13.620 |
|
happened in your environment. |
|
|
|
01:13.620 --> 01:17.610 |
|
And then there's also going to be some user interface concerns that we're going to want to build a Gradio |
|
|
|
01:17.640 --> 01:20.250 |
|
UI that's going to be able to use this framework. |
|
|
|
01:20.250 --> 01:21.960 |
|
So it's going to have to to handle that. |
|
|
|
01:21.960 --> 01:26.850 |
|
But that last one we're going to get to in the next set of series. |
|
|
|
01:27.030 --> 01:31.500 |
|
Uh, so for now we're going to look at our, at our framework. |
|
|
|
01:31.500 --> 01:34.080 |
|
Just before that, a couple of things to point out. |
|
|
|
01:34.080 --> 01:38.580 |
|
First of them, when we looked in our agents folder, we've looked at each of these different classes. |
|
|
|
01:38.580 --> 01:40.920 |
|
That represents a different unit of work. |
|
|
|
01:40.920 --> 01:45.450 |
|
And you'll notice that each of them were subclasses of agent. |
|
|
|
01:45.570 --> 01:47.100 |
|
Uh, and you may have wondered, what is that? |
|
|
|
01:47.130 --> 01:52.260 |
|
What is this agent that he's subclassing is there must be all sorts of functionality in there. |
|
|
|
01:52.260 --> 01:55.920 |
|
Well, actually, no, there's very little functionality in this superclass. |
|
|
|
01:55.920 --> 02:00.210 |
|
Uh, it is actually just an abstract class. |
|
|
|
02:00.270 --> 02:03.820 |
|
Uh, which is able to log a message. |
|
|
|
02:03.910 --> 02:07.090 |
|
And the way that it logs a message is it does logging.info. |
|
|
|
02:07.120 --> 02:08.530 |
|
I imagine you're familiar. |
|
|
|
02:08.560 --> 02:10.360 |
|
You've used Python logging, probably. |
|
|
|
02:10.360 --> 02:11.200 |
|
And so you're pretty. |
|
|
|
02:11.230 --> 02:17.590 |
|
Familiar that logging.info logs, uh, to to to wherever you set up a logger for, um. |
|
|
|
02:17.620 --> 02:18.550 |
|
And. |
|
|
|
02:18.700 --> 02:23.560 |
|
The one thing that I'm doing that's a bit different here, um, is that I'm going to. |
|
|
|
02:23.590 --> 02:27.190 |
|
Add in the name of the subclass of the agent. |
|
|
|
02:27.190 --> 02:28.840 |
|
That's that's, uh, that's. |
|
|
|
02:28.870 --> 02:34.120 |
|
Actually, uh, sending this log message and we're going to give each one a color. |
|
|
|
02:34.210 --> 02:39.400 |
|
So that we can see, uh, what which agent is doing what? |
|
|
|
02:39.400 --> 02:45.220 |
|
And for example, if I go back to, uh, or if I go to the specialist agent, you can see that it's |
|
|
|
02:45.220 --> 02:46.300 |
|
given itself a name. |
|
|
|
02:46.330 --> 02:47.560 |
|
Of specialist agent. |
|
|
|
02:47.560 --> 02:51.310 |
|
And its color is red, so that when it logs, it's going to. |
|
|
|
02:51.340 --> 02:52.270 |
|
Appear in red. |
|
|
|
02:52.270 --> 02:54.610 |
|
And you can see that you might have noticed that I didn't do. |
|
|
|
02:54.640 --> 02:56.980 |
|
Logging.info I did self dot log. |
|
|
|
02:56.980 --> 03:00.070 |
|
And that's so that I can call the log message in the superclass. |
|
|
|
03:00.070 --> 03:04.820 |
|
So you know, there's really nothing fancy to this agent superclass. |
|
|
|
03:04.820 --> 03:06.920 |
|
It just meant that we could log consistently. |
|
|
|
03:06.920 --> 03:11.090 |
|
You could equally well have just included that function in the different classes. |
|
|
|
03:11.090 --> 03:14.480 |
|
Nothing particularly special about the agent. |
|
|
|
03:14.930 --> 03:15.860 |
|
All right. |
|
|
|
03:15.860 --> 03:22.250 |
|
One other bit of groundwork to do is to show you this JSON file called memory JSON. |
|
|
|
03:22.250 --> 03:24.710 |
|
And it is a simple piece of JSON. |
|
|
|
03:24.710 --> 03:29.150 |
|
And if it didn't exist then then you'll see it will it will create it manually. |
|
|
|
03:29.150 --> 03:32.870 |
|
Let's open it with the editor, not with the fancy one, but with the normal editor. |
|
|
|
03:32.870 --> 03:34.700 |
|
And this is what it looks like. |
|
|
|
03:34.700 --> 03:36.440 |
|
It is a list. |
|
|
|
03:36.680 --> 03:43.250 |
|
You can see it's got a list on the outside of it, a list of JSON blobs. |
|
|
|
03:43.250 --> 03:48.320 |
|
And each JSON blob is something which may look familiar to you. |
|
|
|
03:48.350 --> 03:49.700 |
|
Do you recognize what it is? |
|
|
|
03:50.120 --> 03:55.070 |
|
Yes, it's of course, an opportunity object turned into JSON. |
|
|
|
03:55.460 --> 04:02.240 |
|
It's got a deal which has a product description, price and URL, and then it has the estimate that's |
|
|
|
04:02.240 --> 04:08.370 |
|
come from the model and it has a discount, which is quite simply this number minus this number. |
|
|
|
04:08.400 --> 04:08.880 |
|
Oops. |
|
|
|
04:08.910 --> 04:09.630 |
|
That number. |
|
|
|
04:09.660 --> 04:10.020 |
|
Ha. |
|
|
|
04:10.380 --> 04:15.300 |
|
Uh, it's how much is this being offered at a discount to what we think it's really worth? |
|
|
|
04:15.300 --> 04:20.790 |
|
So this memory has three deals that have been surfaced. |
|
|
|
04:21.060 --> 04:26.730 |
|
Uh, now, storing it in a JSON file like this is perhaps not the most industrial strength. |
|
|
|
04:26.730 --> 04:31.830 |
|
If that went into a database, then it's something that could handle multiple of these agent frameworks |
|
|
|
04:31.830 --> 04:32.160 |
|
running. |
|
|
|
04:32.160 --> 04:37.740 |
|
But we only have one agent framework that will only write to this memory, uh, once when it does its |
|
|
|
04:37.740 --> 04:38.220 |
|
work. |
|
|
|
04:38.220 --> 04:42.150 |
|
So that wasn't strictly necessary, but of course it could go in Chrome easily. |
|
|
|
04:42.150 --> 04:46.890 |
|
We've got a database hanging around already, so you could certainly take that as an exercise if you |
|
|
|
04:46.890 --> 04:47.550 |
|
wish. |
|
|
|
04:47.880 --> 04:52.620 |
|
Um, that would certainly be a good, good production improvement to make. |
|
|
|
04:53.250 --> 05:01.620 |
|
But anyway, the time has come to look at the actual, uh, the, the framework itself, the agent framework. |
|
|
|
05:01.710 --> 05:04.620 |
|
Uh, so let's do that right away. |
|
|
|
05:06.220 --> 05:11.710 |
|
Um, and I want to make sure I bring up the right class so that I don't give too much away of what is |
|
|
|
05:11.710 --> 05:12.460 |
|
still to come. |
|
|
|
05:12.460 --> 05:12.970 |
|
Here it is. |
|
|
|
05:12.970 --> 05:14.770 |
|
Deal agent framework. |
|
|
|
05:15.130 --> 05:16.540 |
|
Uh, okay. |
|
|
|
05:16.540 --> 05:24.790 |
|
So the, uh, agent framework, um, is something which, first of all, it sets up logging. |
|
|
|
05:24.790 --> 05:31.870 |
|
This is something which is standard Python stuff that will make sure that when someone does logging.info, |
|
|
|
05:32.020 --> 05:36.790 |
|
um, it will get sent to standard out and it has some, some structure to it. |
|
|
|
05:36.790 --> 05:43.300 |
|
So this is a sort of standard, uh, um, boilerplate Python stuff to make sure we log properly. |
|
|
|
05:43.300 --> 05:46.330 |
|
And then this is the deal agent framework. |
|
|
|
05:46.330 --> 05:51.640 |
|
It's got coded in it, the database, the thing that I accidentally left off a couple of times, uh, |
|
|
|
05:51.640 --> 05:55.480 |
|
so that's coded in here along with the name of the memory file. |
|
|
|
05:55.600 --> 05:58.750 |
|
When it starts up, it will begin logging. |
|
|
|
05:58.780 --> 06:00.730 |
|
It will make a log message itself. |
|
|
|
06:00.730 --> 06:02.170 |
|
It will load the env. |
|
|
|
06:02.200 --> 06:05.770 |
|
This is something that we, of course, always do in all of our Jupyter notebooks. |
|
|
|
06:05.770 --> 06:07.520 |
|
So this needs to do it now. |
|
|
|
06:07.760 --> 06:12.800 |
|
It will, it will create or it will access the database. |
|
|
|
06:13.250 --> 06:17.090 |
|
Um, it will read in memory from that JSON file. |
|
|
|
06:17.390 --> 06:25.010 |
|
Uh, it will get the products collection and it will then set the planning agent, uh, initializing |
|
|
|
06:25.010 --> 06:28.070 |
|
it with the collection that it needs to look at. |
|
|
|
06:28.100 --> 06:30.110 |
|
And then it will say, I'm ready. |
|
|
|
06:30.620 --> 06:35.930 |
|
This this method here read memory does exactly what it says on the tin. |
|
|
|
06:35.930 --> 06:39.140 |
|
It, uh, you can see I haven't hard coded the name of the file. |
|
|
|
06:39.140 --> 06:40.160 |
|
Well done, well done. |
|
|
|
06:40.190 --> 06:40.700 |
|
Me? |
|
|
|
06:40.820 --> 06:43.880 |
|
Uh, so it loads this in from JSON. |
|
|
|
06:43.880 --> 06:50.030 |
|
If it doesn't already exist, then it just returns an empty one, which is completely fine, because |
|
|
|
06:50.240 --> 06:52.130 |
|
there's also a function. |
|
|
|
06:52.160 --> 07:00.890 |
|
A method here writes memory, which writes out these opportunities to, uh, the the memory file. |
|
|
|
07:01.940 --> 07:08.710 |
|
There's a log, uh, a method here to send a log message with its own agent framework tag. |
|
|
|
07:09.490 --> 07:14.590 |
|
Then there's a run method that actually kicks off the planning agent. |
|
|
|
07:14.800 --> 07:20.860 |
|
Uh, and uh, once it gets back a result, it says it calls self dot planner. |
|
|
|
07:21.070 --> 07:22.120 |
|
Plan that. |
|
|
|
07:22.120 --> 07:23.500 |
|
Of course, that's the big moment. |
|
|
|
07:23.500 --> 07:26.200 |
|
That's when we we actually set this thing going. |
|
|
|
07:26.200 --> 07:28.750 |
|
And look it passes in its memory. |
|
|
|
07:28.780 --> 07:32.350 |
|
It passes in the memory of opportunities that it knows about. |
|
|
|
07:32.350 --> 07:38.050 |
|
So in that goes uh, and then it logs the results. |
|
|
|
07:38.050 --> 07:43.600 |
|
And if the result wasn't none, remember that the planner might return none if it doesn't find anything |
|
|
|
07:43.600 --> 07:43.990 |
|
new. |
|
|
|
07:43.990 --> 07:50.020 |
|
But if it was a real result, then it adds it to the memory and writes the memory and returns the result. |
|
|
|
07:50.020 --> 07:51.220 |
|
And that is it. |
|
|
|
07:51.220 --> 07:57.040 |
|
This last piece here is coming up in the UI to the next time, but for now, that is the end of the |
|
|
|
07:57.040 --> 07:57.970 |
|
framework. |
|
|
|
07:58.420 --> 07:59.980 |
|
So how do we run this framework? |
|
|
|
07:59.980 --> 08:01.690 |
|
Well, we run it from the command line. |
|
|
|
08:01.690 --> 08:03.100 |
|
This is now for reals. |
|
|
|
08:03.100 --> 08:05.500 |
|
This is no longer this is all Python code. |
|
|
|
08:05.500 --> 08:09.080 |
|
We're outside the Jupyter world and we're using this interface for convenience. |
|
|
|
08:09.080 --> 08:09.710 |
|
Really. |
|
|
|
08:09.980 --> 08:12.440 |
|
But you could be using VSCode or something like that. |
|
|
|
08:12.470 --> 08:16.310 |
|
The way to run a terminal is we press the plus button here. |
|
|
|
08:16.310 --> 08:20.330 |
|
We can also do it in JupyterLab and we can open a new terminal like this. |
|
|
|
08:20.330 --> 08:25.010 |
|
This is a you can also, of course be doing this however you like to do it. |
|
|
|
08:25.130 --> 08:29.540 |
|
Um, it could be an anaconda prompt or it could be you could be using. |
|
|
|
08:29.540 --> 08:32.240 |
|
If you're using a mac, then you can just bring up a new terminal window. |
|
|
|
08:32.330 --> 08:36.380 |
|
Um, but I am first going to have to activate my conda environment. |
|
|
|
08:36.560 --> 08:47.330 |
|
Um, so I do conda activate LMS, uh, which will mean that as you're now very familiar, uh, my environment |
|
|
|
08:47.330 --> 08:54.350 |
|
changes from base to LMS because I'm now in that Anaconda environment, and I really think that's it. |
|
|
|
08:54.350 --> 09:01.730 |
|
I should now be able to say Python deal agent framework. |
|
|
|
09:02.510 --> 09:04.010 |
|
Spell it right. |
|
|
|
09:06.410 --> 09:07.490 |
|
Dot py. |
|
|
|
09:07.640 --> 09:08.990 |
|
And that should really be it. |
|
|
|
09:09.000 --> 09:11.670 |
|
There is one more thing I need to show you before I do that though. |
|
|
|
09:11.820 --> 09:14.190 |
|
I almost forgot just as well. |
|
|
|
09:14.190 --> 09:16.320 |
|
It's just this very last few lines here. |
|
|
|
09:16.320 --> 09:20.070 |
|
You might have been wondering what's going to happen if I do that, if I don't show you this. |
|
|
|
09:20.460 --> 09:26.700 |
|
So the very end, we of course, make it clear that if this is being run from the command line so that |
|
|
|
09:26.700 --> 09:32.550 |
|
this file is main, then we instantiate, we create a new instance of deal agent framework. |
|
|
|
09:32.550 --> 09:35.760 |
|
And we call this run method that we just looked at. |
|
|
|
09:35.790 --> 09:39.780 |
|
That's the little missing link there just to explain okay okay. |
|
|
|
09:39.810 --> 09:40.320 |
|
With that. |
|
|
|
09:40.320 --> 09:41.640 |
|
Now I should be able to run this. |
|
|
|
09:41.670 --> 09:42.540 |
|
Take a deep breath. |
|
|
|
09:42.540 --> 09:43.410 |
|
Let's do it. |
|
|
|
09:46.950 --> 09:49.020 |
|
A long pause while it thinks, huh? |
|
|
|
09:51.060 --> 09:51.930 |
|
All right. |
|
|
|
09:51.930 --> 09:53.310 |
|
Let's see what's going on. |
|
|
|
09:53.310 --> 09:55.800 |
|
The agent framework says it's initializing. |
|
|
|
09:55.800 --> 09:58.350 |
|
The planning agent is initializing scanner. |
|
|
|
09:58.380 --> 09:59.850 |
|
Agent becomes ready. |
|
|
|
09:59.850 --> 10:02.550 |
|
The ensemble agent gets ready. |
|
|
|
10:02.550 --> 10:04.170 |
|
So then it's a bit hard to see this. |
|
|
|
10:04.170 --> 10:09.600 |
|
Read the specialist agent, which is connecting to modal, gets ready. |
|
|
|
10:10.300 --> 10:12.910 |
|
And then the frontier model gets ready. |
|
|
|
10:12.940 --> 10:14.920 |
|
It has to load in that sentence transformer. |
|
|
|
10:14.920 --> 10:16.360 |
|
You can see it's happening there. |
|
|
|
10:16.360 --> 10:19.810 |
|
And then the random forest agent gets ready as well. |
|
|
|
10:19.810 --> 10:21.340 |
|
And that's done. |
|
|
|
10:21.430 --> 10:24.340 |
|
The ensemble is ready, the messenger is ready. |
|
|
|
10:24.340 --> 10:25.810 |
|
And everything in blue. |
|
|
|
10:25.840 --> 10:28.360 |
|
Here it says Agent Framework is ready. |
|
|
|
10:28.360 --> 10:33.130 |
|
So it went through instantiating and creating all of our agents and setting them all up. |
|
|
|
10:33.130 --> 10:39.190 |
|
They've loaded their model weights that they've the single one the specialist model has connected to |
|
|
|
10:39.220 --> 10:39.970 |
|
modal. |
|
|
|
10:40.390 --> 10:46.180 |
|
And they have now, um, all prepared themselves for action. |
|
|
|
10:46.210 --> 10:51.010 |
|
Model weights have been read in and it's kicked off a run. |
|
|
|
10:51.040 --> 10:56.110 |
|
The scanner agent you can see here, uh, received there's a there's an error there. |
|
|
|
10:56.110 --> 10:57.160 |
|
I'll improve that. |
|
|
|
10:57.160 --> 10:59.020 |
|
It's not printing the actual number of results. |
|
|
|
10:59.020 --> 11:02.290 |
|
It's printing needs an F before it. |
|
|
|
11:02.620 --> 11:05.080 |
|
Uh, I will have fixed that before you see it. |
|
|
|
11:05.170 --> 11:07.660 |
|
Uh, so the scanner agent is making that call. |
|
|
|
11:07.780 --> 11:16.400 |
|
Um, and now the planning agent is pricing up the deal, and it's now sitting with modal while my machine |
|
|
|
11:16.400 --> 11:17.960 |
|
warms up. |
|
|
|
11:18.050 --> 11:26.420 |
|
Uh, but as they say on those cooking shows, uh, if you know what I mean, uh, you have those cooking |
|
|
|
11:26.420 --> 11:32.270 |
|
shows when they put the thing in the oven and they say, and here's the one that I put in earlier, |
|
|
|
11:32.270 --> 11:33.980 |
|
and they pull that out of the oven. |
|
|
|
11:33.980 --> 11:37.100 |
|
I always think that's a devious trick when they do that. |
|
|
|
11:37.190 --> 11:39.800 |
|
But you're going to have to trust me that I did just do this. |
|
|
|
11:39.800 --> 11:45.440 |
|
I just kicked it off a few minutes ago and it ran over here, and that's where I left it, and it finished |
|
|
|
11:45.440 --> 11:46.070 |
|
up its thing. |
|
|
|
11:46.070 --> 11:52.700 |
|
And you can see that basically it for each of the five deals that returned, it made a series of calls |
|
|
|
11:52.700 --> 11:54.140 |
|
to all of the different models. |
|
|
|
11:54.140 --> 11:55.670 |
|
You can read through each of this. |
|
|
|
11:55.700 --> 11:58.490 |
|
The ensemble model returns a number. |
|
|
|
11:58.700 --> 12:01.160 |
|
Um, so we should we should work back. |
|
|
|
12:01.190 --> 12:06.350 |
|
First of all, the specialist model calculates a number and then the frontier model calculates the number |
|
|
|
12:06.350 --> 12:08.990 |
|
with the Rag lookup and then the random forest model. |
|
|
|
12:08.990 --> 12:10.430 |
|
And then the ensemble model. |
|
|
|
12:10.430 --> 12:12.870 |
|
And that goes back to the planning agent. |
|
|
|
12:13.200 --> 12:15.300 |
|
And so you can read through all of this. |
|
|
|
12:15.300 --> 12:20.100 |
|
And at the end of this it completed and it sent a push notification. |
|
|
|
12:20.340 --> 12:29.700 |
|
And I did indeed get a push notification about this particular deal that it found, which was a certified |
|
|
|
12:29.700 --> 12:39.090 |
|
refurbished laptop boasting a 13th generation i5, which the estimate was $925 for that, which was |
|
|
|
12:39.090 --> 12:45.030 |
|
a very healthy discount to what it was going for at $560. |
|
|
|
12:45.030 --> 12:46.950 |
|
So there we have it. |
|
|
|
12:46.950 --> 12:50.970 |
|
That was the run of our agent framework running. |
|
|
|
12:51.210 --> 12:55.980 |
|
And yeah, it's incredibly satisfying to see that happening. |
|
|
|
12:55.980 --> 13:00.870 |
|
I love absolutely love watching it and could watch it all day. |
|
|
|
13:00.870 --> 13:04.200 |
|
I have to say it's it's hugely enjoyable. |
|
|
|
13:04.230 --> 13:07.920 |
|
Look, I didn't even need to put that to bring that one out of the oven, because the one that we put |
|
|
|
13:07.920 --> 13:09.780 |
|
in the oven is now cooking up. |
|
|
|
13:09.780 --> 13:14.960 |
|
You can see once modal is warmed up, it's quite fast in how it turns around the different things. |
|
|
|
13:15.170 --> 13:16.910 |
|
So it's going through there. |
|
|
|
13:16.910 --> 13:18.020 |
|
It's come up with something. |
|
|
|
13:18.020 --> 13:22.790 |
|
And of course, what it's proposing is different to what it proposed a moment ago. |
|
|
|
13:23.000 --> 13:26.870 |
|
And I just got the notification on my phone and no doubt on my watch. |
|
|
|
13:27.350 --> 13:34.880 |
|
So we just watched we just saw the full framework running and me getting the notification. |
|
|
|
13:34.880 --> 13:42.980 |
|
And you can read through this trace and see how each of the agents are collaborating to solve this problem. |
|
|
|
13:43.280 --> 13:51.470 |
|
Um, and so all that remains for this is to make it really autonomous so that it's running all the time. |
|
|
|
13:51.470 --> 13:56.810 |
|
And while we're doing that, we might as well slap on a great UI because we know how to use gradio and |
|
|
|
13:56.810 --> 13:57.680 |
|
we know how good it is. |
|
|
|
13:57.710 --> 14:01.190 |
|
And all of that is what we're going to do tomorrow. |
|
|
|
14:01.250 --> 14:04.460 |
|
But for now, I'm going to let you, of course, enjoy this. |
|
|
|
14:04.460 --> 14:07.670 |
|
I do hope that you're running this a few times because it's so satisfying. |
|
|
|
14:07.700 --> 14:08.900 |
|
Honestly, I love it. |
|
|
|
14:08.930 --> 14:14.060 |
|
Hope you're finding just as much enjoyment as me, and I will see you for the slides to wrap up the |
|
|
|
14:14.060 --> 14:14.540 |
|
day.
|
|
|