Welcome back to Colab and welcome back to our business project.
So again our assignment, we are due to create meeting minutes based on an audio file.
Uh so I found this very useful data set.
It's typical hugging face to have a perfect data set for us.
Um, it's a data set called meeting Bank, which is apparently a pretty well known, uh, data set,
a benchmark created from the City council's six major US cities.
Um, so I use this.
I downloaded a particular Denver City Council meeting, and I actually just took a ten minute segment
of it, uh, to be used for our experiment here.
Uh, either ten or maybe it was a 20 minute segment of it.
Um, but anyway, took that audio cut and I saved it on my Google Drive because my idea here is I'd
like this product to be able to take anything that is in one's Google Drive, or if you're building
this for a company in the company's drive and be able to use that to generate meeting minutes.
So as part of this project, a little sidebar is we're also going to see how you can get a colab to
read from your Google Drive.
So we begin, as usual with some imports these sorry with Pip installs, the one little extra here is.
You'll notice that we're also going to install OpenAI on this colab as well.
We're not just using hugging face, we're using a bunch of hugging face, uh, packages.
And also OpenAI library.
We do some imports and including an OpenAI import as well as everything else.
Um, and then we're going to set some constants.
We're going to use an audio model called whisper, one that may be used yourself.
When I'd set you this assignment, uh, previously.
Um, and then this is the llama 3.18 billion instruct model that we'll be using as well.
So here this is the new capability that that that you'll be learning.
Today as well a little extra.
This is how you connect Colab to your Google Drive.
Super simple.
And it's just a drive dot mount.
And you tell it where locally you would like to.
Mount the drive.
That's basically it.
And then I've set a constant for myself of within my drive.
Uh, it's uh, in the I've got a folder called LMS and within that I have Denver Extract dot mp3, which
is the MP3 recording of, uh, this, uh, segment that somewhere between 10 and 20 minutes from the
Denver City Council.
So if I run this, it pops up with a, um, a it's connected to Google the first.
I'm running this for the second time, and the first time I ran this, it of course popped up with an
authentication, uh, selection for me to confirm that I'm signed in with my Google account and I grant
access.
This time it's telling me it's already mounted there.
Um, if I were to go to the folder here, would be able to go through my Google drive and see all my
all my files under slash content slash drive.
So we then sign in to the Huggingface hub.
Here we go.
Login successful.
And we also sign in to OpenAI.
So this is very similar.
We get our OpenAI key which is also I've set the OpenAI key uh in the secrets of this colab.
Um, and so we retrieve that key and then we call the usual OpenAI uh, constructor to, to establish
the, the interface connection.
But this time I am passing in that OpenAI, OpenAI API key.
Uh, you remember in the past I've not had to specify this because I've relied on the fact that the
environment variable is set.
This time I'm passing it in explicitly.
So there we go.
That is now established the OpenAI connection.
And then what am I going to do?
I'm going to take this audio file, which is sitting on my Google Drive that's now mapped to this colab.
And then I'm going to call OpenAI dot audio.
Dot transcriptions dot create, which is very similar to other OpenAI API methods we've used.
It's particularly similar to the one when we actually made, made it to speak, made it generate audio.
I passed in the name of the model, the whisper one model, the file, and that I want the response
in text.
And then I will print what comes back from OpenAI's whisper model.
So it's been provided with a bunch of audio, or it's being provided as we speak with a bunch of audio
that is now calling the frontier model, and we are currently waiting to get back a transcription of
that meeting.
Well, that's happening.
I'm going to keep going so that we can get ahead on on the other things.
I have to run.
We're then going to set up the prompt for llama three, and there's going to be a system prompt, a
system message and a user prompt system messages.
You're an assistant that produces meetings of minutes from transcripts with a summary, key discussion
points, takeaways and action items with owners in markdown, uh, and then a user prompt that says
below is the transcript of a an extract transcript.
That's fine.
I thought my English was bad, but it's okay.
Other Denver Council meeting.
Please write minutes in markdown, including a summary with attendees, location and date, discussion
points, takeaways, and action items with owners.
And then I shove in the transcript of the meeting right after that user prompt.
Here is the transcript.
It just got printed out And it's a long old transcript.
The Denver City Council meeting.
Talked for quite a while, and a lot of it was about Indigenous Peoples Day, which was the upcoming
federal holiday.
And there was some debate about the right way for the council to recognize Indigenous Peoples Day.
If you read through all of this text or if you listen to the, the, the audio.
So this is all now in text in this transcription variable.
So we started with audio.
We now have text thanks to OpenAI's whisper one model.
We now create our system and user prompt.
Now this will look familiar to you.
This is our quant config.
We're going to be quantizing again.
Why not.
It was very effective with lambda 3.1 before it reduced the memory significantly down to 55. five gigabytes.
But it did not.
At least His performance seemed to be perfectly good to us.
Maybe you tried it without quantizing to see how much better the joke was.
Um.
I wouldn't be surprised if it didn't make much difference at all.
Quantization is very effective.
Okay, it's time for action.
This should all be quite familiar to you because this is what we did last time.
We are going to create a tokenizer for Lama using the auto tokenizer Frompretrained method.
We're going to do this business of setting the pad token as before.
Then we're going to call the apply chat template function method.
Passing in the messages the this this right here.
We're passing that in.
And this of course includes the whole transcript.
It includes the text of the whole meeting and the user prompt.
And we're going to put that massive amount of text on our GPU.
We're going to stream again.
So use this text stream object.
And then here we go.
This is.
This is when we create our model.
We create the auto model for causal Elm.
We pass in the llama model name.
We say, please use a GPU if we've got one, which we do.
We're using the T4 box, the small GPU box for this.
And we pass in our quantization config.
I'm going to start this running now because it will take take a while.
I should have started running before I was talking.
That would have been smarter.
Uh uh.
And so we're going to then create the model and then we're going to do the action.
Action is to call generate on model.
And when you call generate you have to pass in the inputs, which of course is now this entire tokenized,
uh prompts and transcript.
This is a bit bigger than you're used to before.
We used to say in the maximum new tokens was 80.
Now we're saying maximum new tokens is 2000 because there could be quite a hefty response.
Um, so, uh, uh, we that should be enough space to get back our meeting minutes.
And then we're also providing the streamer, which is telling it that it can stream results back into
our colab.
While it's going to be thinking for a little bit, I'll tell you what's going to happen next is going
to stream the meeting minutes back in here.
Um, afterwards, what we can also do is we can also just get that text by taking the outputs, taking
the first of the outputs, and there only will be one.
And then decoding that using tokenizer dot decode.
Uh, and that's something we will then put into a variable called response.
Well here come the minutes.
Um.
It's about to come.
It's so far.
Just put the, the uh.
The transcript in their.
Minutes of the Denver City Council meeting Monday, October the 9th and location attendees who are the
attendees.
Summary.
They met on Monday, October the 9th to discuss and adopt a proclamation for Indigenous Peoples Day.
Councilman Lopez presented the proclamation.
Key discussion points.
Takeaways.
It was adopted recognizing the importance of the day.
They emphasized the importance of inclusivity and respecting all cultures.
Some actions with owners and actions.
Councilman Lopez and clerk.
Clerk is to attest and affix the seal of the City and Council of Denver to the proclamation.
And then, uh, Councilman Lopez to transmit a copy of the proclamation to the Denver American Indian
Commission and some other areas, and then some next steps at the end.
So I've got to hand it to llama 3.1.
This seems to be a very comprehensive, very clear, very thorough set of minutes with attendees with
date with with, uh, all of the right format and the right sections.
Now, you'll notice, of course, that it's come in markdown format.
And you're familiar from, uh, when we were working with frontier models before in Jupyter Notebook
locally, that we can use this display markdown response as our way to see that in markdown in the Colab.
And here we have, uh, the minutes of the Denver City Council meeting, um, and, uh, organized into
those various sections of the summary, the takeaways, the action items, and the next steps.
So I give you an application that uses a frontier model and an open source model to take audio and convert
it to a transcript, and convert that transcript to a meeting summary with actions and next steps.
Well, the obvious exercise for you.
I hope you've already guessed what it's going to be.
It's easy peasy to now put that into a nice user interface.
You can use Gradio very similar to what we've had in the previous week, and you can bring up this into
a nice little Gradio interface.
Perhaps you could type out the name of a file on your Google Drive and press Generate Minutes.
It will read in that audio, convert it to text, and then convert it to meeting minutes actions, takeaways,
next steps.
So that's the task for you.
Please go away and do that.
And I can't wait to see some terrific user interfaces.
Please do push the code when you've got it.
I would love to see them and I will see you for the next lecture in a moment.