1 changed files with 395 additions and 0 deletions
@ -0,0 +1,395 @@
|
||||
{ |
||||
"cells": [ |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "7df20212-bfa8-4b2b-aec5-7b562dd15ee8", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"### Issue with Gradio when using Claude for Conversational AI (chatbots)\n", |
||||
"\n", |
||||
"As explained in Day 3 (notebook), Gradio has been upgraded to pass in history in a format OpenAI accepts.\n", |
||||
"\n", |
||||
"This update simplifies the development work as Gradio manages the history in the background and provides a history structure that matches OpenAi. Fortunately, this works with other models that leverage the client libraries for OpenAI, such as Llama and Gemini. \n", |
||||
"\n", |
||||
"However, leveraging Gradio's ChatInterface while using Claude models generates a BadRequest error. \n", |
||||
"\n", |
||||
"In analyzing the history list from Gradio, it has the following format:\n", |
||||
"\n", |
||||
"`{'role': 'assistant', 'metadata': None, 'content': '[assistant message here]', 'options': None}`\n", |
||||
"\n", |
||||
"OpenAi accepts this format without issues, as do other models such as Llama and Gemini - at least while leveraging the client libraries for OpenAI. They accept both formats. \n", |
||||
"\n", |
||||
"However, Claude's API requires the following format:\n", |
||||
"\n", |
||||
"`{'role': 'user', 'content': '[user message here]'}`\n", |
||||
"\n", |
||||
"Claude rejects anything different from this format.\n", |
||||
"\n", |
||||
"Run the code below to get the details! " |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "b5398289-f4d0-4317-b9c2-fff06c4bbfec", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# imports\n", |
||||
"\n", |
||||
"import os\n", |
||||
"from dotenv import load_dotenv\n", |
||||
"from openai import OpenAI\n", |
||||
"import google.generativeai\n", |
||||
"import anthropic\n", |
||||
"import gradio as gr\n", |
||||
"from pprint import pprint # for a nicely formatted printout of a list" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "76df1fb5-76f8-4b52-afbb-12b892208b50", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# set environment variables for OpenAi\n", |
||||
"load_dotenv(override=True)\n", |
||||
"openai_api_key = os.getenv('OPENAI_API_KEY')\n", |
||||
"anthropic_api_key = os.getenv('ANTHROPIC_API_KEY')\n", |
||||
"google_api_key = os.getenv('GOOGLE_API_KEY')\n", |
||||
"\n", |
||||
"# validate API Key\n", |
||||
"if not openai_api_key:\n", |
||||
" raise ValueError(\"No OpenAI API key was found! Please check the .env file.\")\n", |
||||
"else:\n", |
||||
" print(f\"OpenAI API Key exists and begins {openai_api_key[:8]}\")\n", |
||||
"\n", |
||||
"if not anthropic_api_key:\n", |
||||
" raise ValueError(\"No Anthropic API key was found! Please check the .env file.\")\n", |
||||
"else:\n", |
||||
" print(f\"Anthropic API Key exists and begins {anthropic_api_key[:7]}\")\n", |
||||
"\n", |
||||
"if not google_api_key:\n", |
||||
" raise ValueError(\"No Gemini API key was found! Please check the .env file.\")\n", |
||||
"else:\n", |
||||
" print(f\"Gemini API Key exists and begins {google_api_key[:8]}\")" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "f2429366-a9ab-4e72-a651-56f17b779cf4", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# constants\n", |
||||
"MODELS = { 'GPT': 'gpt-4o-mini', \n", |
||||
" 'LLAMA': 'llama3.2', \n", |
||||
" # 'DEEPSEEK': 'deepseek-r1:1.5b',\n", |
||||
" 'CLAUDE': 'claude-3-haiku-20240307',\n", |
||||
" 'GEMINI': 'gemini-2.0-flash-exp'\n", |
||||
" }\n", |
||||
"\n", |
||||
"CLIENTS = { 'GPT': OpenAI(), \n", |
||||
" 'LLAMA': OpenAI(base_url='http://localhost:11434/v1', api_key='ollama'),\n", |
||||
" # 'DEEPSEEK': OpenAI(base_url='http://localhost:11434/v1', api_key='ollama'),\n", |
||||
" 'CLAUDE': anthropic.Anthropic(),\n", |
||||
" 'GEMINI': OpenAI(base_url=\"https://generativelanguage.googleapis.com/v1beta/openai/\", api_key=google_api_key)\n", |
||||
" }\n", |
||||
"\n", |
||||
"# system prompt\n", |
||||
"system_message = \"You are a nice assistant that like to chat with users\"\n", |
||||
"\n", |
||||
"# to save/print the history structure\n", |
||||
"console = []" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "bb34616a-2304-4a56-af3a-8d353629722b", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"#### Testing GPT\n", |
||||
"\n", |
||||
"This runs without issues. You may change the model to Llama or Gemini for further testing." |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "9e69eb7a-50cf-478c-83f6-4ce923a9c53a", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"def gpt_chat(message, history):\n", |
||||
" # change to Llama or Gemini for test\n", |
||||
" model = 'GPT'\n", |
||||
"\n", |
||||
" # add system message to history\n", |
||||
" if not history:\n", |
||||
" history.append({\"role\": \"system\", \"content\": system_message})\n", |
||||
"\n", |
||||
" # conversation including new user message\n", |
||||
" messages = history + [{\"role\": \"user\", \"content\": message}]\n", |
||||
"\n", |
||||
" # send request to OpenAi\n", |
||||
" response = CLIENTS[model].chat.completions.create(\n", |
||||
" model = MODELS[model],\n", |
||||
" max_tokens = 200,\n", |
||||
" messages = messages,\n", |
||||
" )\n", |
||||
"\n", |
||||
" # save history structure\n", |
||||
" global console\n", |
||||
" console = history[:]\n", |
||||
" \n", |
||||
" # post in Gradio's chat interface\n", |
||||
" return response.choices[0].message.content" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "20444651-440c-4b6d-996d-87a21c28f28e", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"##### Have a conversation of several messages (3 or more)" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "90dac8e6-e1ef-455a-94e9-06735e6c17fb", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# Gradio with GPT\n", |
||||
"gr.ChatInterface(fn=gpt_chat, type=\"messages\", examples=[\"How are you today?\", \"Please, tell me a joke.\"]).launch()" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "b4048069-b712-491a-96d5-f8fd9f0fe533", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"##### Notice how the history structure includes both formats, and this is okay with OpenAi" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "2f8acbde-f5c3-4890-af5b-69a24ded9b45", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# audit console\n", |
||||
"pprint(console)\n", |
||||
"\n", |
||||
"# empty list\n", |
||||
"console.clear()" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "deaeb8a0-266a-4b5d-8536-e5c59391baa5", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"#### Testing with Claude\n", |
||||
"\n", |
||||
"This first test will generate a BadRequest error on the second message from the user. The first message sent follows the required format preventing errors." |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "268e662d-84f5-4b09-8472-c3545a04f2a4", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# Chat using Gradio's history - generates error after the second message from user\n", |
||||
"def claude_chat(message, history): # Gradio requires the history parameter, but it goes unused. \n", |
||||
" \n", |
||||
" # save history structure\n", |
||||
" global console\n", |
||||
" console = history[:]\n", |
||||
" \n", |
||||
" model = 'CLAUDE'\n", |
||||
"\n", |
||||
" # conversation including new user message - this is why the first message does not generate an error\n", |
||||
" messages = history + [{\"role\": \"user\", \"content\": message}]\n", |
||||
"\n", |
||||
" # send the request to Claude\n", |
||||
" response = CLIENTS[model].messages.create(\n", |
||||
" model = MODELS[model],\n", |
||||
" max_tokens = 200,\n", |
||||
" system = system_message,\n", |
||||
" messages = messages,\n", |
||||
" )\n", |
||||
" \n", |
||||
" # post in Gradio's chat interface\n", |
||||
" return response.content[0].text" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "5fd5e221-ab81-4567-a14d-5638d9919a40", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"##### Have a conversation of several messages (3 or more)" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "fbd1a4b2-afcf-4345-bf1a-283c16076e07", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# Gradio with Claude\n", |
||||
"gr.ChatInterface(fn=claude_chat, type=\"messages\", examples=[\"How are you today?\", \"Please, tell me a joke.\"]).launch()" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "834bf737-b3d3-4225-be12-f493f3490f16", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"##### Notice how the history structure changes for the second message. This cause a BadRequest error with Claude." |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "c6cfa123-83d5-46d8-ac33-74fd9cae0ab3", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# audit console\n", |
||||
"pprint(console)\n", |
||||
"\n", |
||||
"# empty list\n", |
||||
"console.clear()" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "c0b5c4e0-7544-4279-8006-fde4be9c656f", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"##### Have a new conversation of several messages (3 or more), but this time leveraging a local history repository instead of Gradio's. \n", |
||||
"\n", |
||||
"The code leverages the list `console` as the local repository. Note that Gradio still requires the second parameter (history) is required even though it is not used. " |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "8c50326e-8496-499a-81d9-28b883ac9b6b", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"def claude_chat(message, history): # Gradio requires the history parameter, but it goes unused. \n", |
||||
" # local history repository instead of Gradio's \n", |
||||
" global console\n", |
||||
" model = 'CLAUDE'\n", |
||||
"\n", |
||||
" # append new user message to history - using Claude's required format\n", |
||||
" console.append({\"role\": \"user\", \"content\": message})\n", |
||||
"\n", |
||||
" # send the request to Claude\n", |
||||
" response = CLIENTS[model].messages.create(\n", |
||||
" model = MODELS[model],\n", |
||||
" max_tokens = 200,\n", |
||||
" system = system_message,\n", |
||||
" messages = console, # use local history repository\n", |
||||
" )\n", |
||||
"\n", |
||||
" # append the assistant response to history - using Claude's required format\n", |
||||
" console.append({\"role\": \"assistant\", \"content\": response.content[0].text})\n", |
||||
"\n", |
||||
" # post in Gradio's chat interface\n", |
||||
" return response.content[0].text" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "f19621f5-8dd6-4a88-9a21-9aab04a9dd90", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# Gradio with Claude\n", |
||||
"gr.ChatInterface(fn=claude_chat, type=\"messages\", examples=[\"How are you today?\", \"Please, tell me a joke.\"]).launch()" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "d6c33236-0d2f-4341-b9ec-e3d7b3eeaec6", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"##### Notice that the history structure follows the required format." |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "bd5dd244-abc7-4654-97d2-c140b2a934d2", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [ |
||||
"# audit console\n", |
||||
"pprint(console)\n", |
||||
"\n", |
||||
"# empty list\n", |
||||
"console.clear()" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "markdown", |
||||
"id": "48fc0b5f-09de-413f-a809-7bbba5a1ae3e", |
||||
"metadata": {}, |
||||
"source": [ |
||||
"### Conclusion\n", |
||||
"\n", |
||||
"1. OpenAi (and models that leverage the client libraries for OpenAI) supports both formats for the conversation history.\n", |
||||
"\n", |
||||
" (Gradio's) `{'role': 'assistant', 'metadata': None, 'content': '[assistant message here]', 'options': None}`\n", |
||||
"\n", |
||||
" (Claude's) `{'role': 'user', 'content': '[user message here]'}`\n", |
||||
" \n", |
||||
"2. Claude only supports the following format for the conversation history:\n", |
||||
"\n", |
||||
" (Claude's) `{'role': 'user', 'content': '[user message here]'}`" |
||||
] |
||||
}, |
||||
{ |
||||
"cell_type": "code", |
||||
"execution_count": null, |
||||
"id": "913b171a-8b3e-43e4-ac38-cedf2c4cdedf", |
||||
"metadata": {}, |
||||
"outputs": [], |
||||
"source": [] |
||||
} |
||||
], |
||||
"metadata": { |
||||
"kernelspec": { |
||||
"display_name": "Python 3 (ipykernel)", |
||||
"language": "python", |
||||
"name": "python3" |
||||
}, |
||||
"language_info": { |
||||
"codemirror_mode": { |
||||
"name": "ipython", |
||||
"version": 3 |
||||
}, |
||||
"file_extension": ".py", |
||||
"mimetype": "text/x-python", |
||||
"name": "python", |
||||
"nbconvert_exporter": "python", |
||||
"pygments_lexer": "ipython3", |
||||
"version": "3.11.11" |
||||
} |
||||
}, |
||||
"nbformat": 4, |
||||
"nbformat_minor": 5 |
||||
} |
Loading…
Reference in new issue