Welcome back to the radio day in the lab.
More to do.
Let's keep going.
Where we left off is we had just built a simple user interface that was calling an LLM and telling a
very, uh, good joke.
Uh, let's keep going with this.
What we're going to do next is ask for the, uh, assistant to respond in markdown, uh, as a way of,
uh, um, getting better looking user interfaces.
Um, and wouldn't it be nice if we wanted to show results in gradio with good formatting written in
markdown?
If we could just have that instead of text box, just replace it with the word markdown, and then the
output would be in perfectly formatted markdown.
That would be great wouldn't it?
Wouldn't it be nice?
You're probably getting the idea here.
Things just really are.
This good.
Uh, So let's say your message.
Let's say, um.
Um, how do I get from Times Square to Times Square like that to Grand Central?
I've got a question for New York navigation.
Let's see how it does.
It's thinking about that.
And there we go.
Here's a response.
And you can see to get from Times Square to Grand Central Terminal in New York City, it figured out
that's what I was talking about.
Follow these steps and you can see it's good headings.
And it's got a nice sub bullets and numbers and all the rest of it, as described in the markdown that
came back from GPT four.
Oh very easy, very nice.
Let's have a look at what else we can do.
Uh, streaming.
Streaming is something we got used to last time.
So can we stream results back to Gradio user interfaces, just as we did when it was coming back into
a Jupyter output cell.
So here we go.
We change our function.
It used to be the message GPT.
Now we're making it stream GPT.
Different function.
And the key thing is that this isn't actually a function.
It's a generator in that it's going to end by yielding a result.
And Gradio is going to detect that we're giving it a generator, not a function.
And because of that, Gradio is automatically going to be iterative and decide to fill in, um, piece
by piece as it comes back from this generator.
So usual story.
I create the messages and then you'll remember this time it's the same API call, but we pass in stream
equals true.
You remember how Claude does it?
Hopefully you do.
With Claude, you don't have an attribute, but instead you call dot stream instead of dot create.
But otherwise it's very similar.
So one thing that's worth noting here.
Just a tiny subtlety with Gradio.
When you are streaming back the results to Gradio, you don't stream back chunk by chunk of the results.
You have to stream back the entire cumulative result so far and stream back a longer and longer cumulative
result.
So you can see what I'm doing is I'm I'm sort of starting with an empty string.
And then for each chunk I'm adding that in and then yielding the total cumulative result so far.
Um, and if you don't do that, what you'll see is each individual chunk will appear in the output cell
and then disappear and will be replaced by something else.
So you have to do it this way.
If you don't see what I mean, try doing yield chunk instead of yield result.
Sorry.
Yield chunk .0. delta dot content.
Uh.
And you'll see exactly what I mean.
It's not not going to look good.
Uh, anyway, that is our stream GPT uh, and wouldn't it be nice if all we needed to do was replace
the the function that it used to be message GPT with stream, GPT and Gradio just figured out the rest.
It figured out that okay, this is a generator, not a function.
Therefore, they're going to want to stream back results.
Therefore, I need to have a sort of typewriter animation style effect.
Let's see if it really can be that simple.
Can it be that simple?
Here we go.
Uh, how do I get from Times Square to Grand Central?
And there we go.
Of course, it's that simple.
Of course it is.
Streams the results.
They look great.
Markdown looks fantastic.
Uh, so, uh, of course it wouldn't be doing my job if I didn't show you how easy it is with Claude
as well.
I mentioned it before, and now you can see there is Claude's API call.
It's very similar.
You call dot stream, you don't pass in the parameter.
You remember that you do have to specify max tokens and the system message goes in separately.
Otherwise very similar.
The streaming back is just for results as stream.
So with result as stream a context manager.
And then we yield the full response just as before.
And at this point it's going to be boringly simple.
You get the joke.
You simply pass in that function instead.
And now we are talking to Claude instead.
We'll ask it the same question though.
How do I get from Times Square to Grand Central?
And here comes Claude's response, a bit shorter, just two options, but nicely structured.
Nicely formatted.
Very good indeed.
Um, so we can take this one step forwards by having the ability to choose either GPT or Claude.
But I'm going to get to that in the very next session.
So hang on in there.