# LLM API Integration Guide This guide explains how to use both OpenAI and Llama (via Ollama) APIs in Python applications, specifically for the Website Summary Tool. ## 1. Overview of Available LLM Clients The application supports multiple LLM providers through a unified interface: ```python from llm.llm_factory import LLMFactory # Create an OpenAI client openai_client = LLMFactory.create_client("openai") # Create a Llama client (via Ollama) llama_client = LLMFactory.create_client("llama") ``` Each client implements the same interface, making it easy to switch between providers. ## 2. OpenAI Integration ### 2.1 Loading the OpenAI API Key The first step in using the OpenAI API is to load your API key: ```python import os from dotenv import load_dotenv def load_api_key(): """Load environment variables from .env file and return the API key.""" load_dotenv(override=True) return os.getenv('OPENAI_API_KEY') ``` This function: - Uses `dotenv` to load environment variables from a `.env` file - Returns the API key from the environment variables - The `override=True` parameter ensures that environment variables in the `.env` file take precedence ### 2.2 Initializing the OpenAI Client Initialize the OpenAI client to make API calls: ```python from openai import OpenAI def initialize_openai_client(): """Initialize the OpenAI client.""" load_dotenv(override=True) # Load environment variables including OPENAI_API_KEY return OpenAI() # The client automatically uses OPENAI_API_KEY from environment ``` **Important Note**: The newer versions of the OpenAI Python library automatically load the API key from the environment variable `OPENAI_API_KEY`. You don't need to explicitly pass the API key when creating the client or making requests. When you call `load_dotenv(override=True)`, it loads the API key into the environment, and the OpenAI client uses it automatically. If you want to explicitly set the API key instead of relying on environment variables, you can do: ```python from openai import OpenAI client = OpenAI(api_key="your-api-key-here") ``` ### 2.3 Formatting Messages for OpenAI The `OpenAIClient` implements the `format_messages` method from the `BaseLLMClient` abstract class: ```python def format_messages(self, messages): """ Format messages for OpenAI API. Args: messages: List of message dictionaries with role and content Returns: list: The messages formatted for OpenAI """ # OpenAI already uses the format we're using, so we can return as-is return messages ``` Since our internal message format already matches what OpenAI expects, this implementation simply returns the messages unchanged. ### 2.4 Making OpenAI API Requests Make a request to the OpenAI API: ```python def generate_content(self, messages, model=None, **kwargs): """Generate content from OpenAI.""" # Format messages appropriately for OpenAI formatted_messages = self.format_messages(messages) response = self.client.chat.completions.create( model=model, messages=formatted_messages, **kwargs ) return response.choices[0].message.content ``` The API key is automatically used from the environment variables - you don't need to pass it in each request. ## 3. Llama Integration (via Ollama) ### 3.1 Loading Llama Configuration Configure the connection to a local Ollama server: ```python def _load_config(self): """Load Llama configuration from .env file.""" load_dotenv(override=True) self.api_base = os.getenv('LLAMA_API_URL', 'http://localhost:11434') ``` The default URL for Ollama is `http://localhost:11434`, but you can customize it in your `.env` file. ### 3.2 Initializing the Llama Client Initialize the Llama client to connect to Ollama: ```python def initialize(self): """Initialize the Llama client by loading config.""" self._load_config() return self ``` ### 3.3 Formatting Messages for Llama The `LlamaClient` implements the `format_messages` method to convert the standard message format to what Ollama expects: ```python def format_messages(self, messages): """ Format messages for Ollama API. Args: messages: List of message dictionaries with role and content Returns: str: The messages formatted as a prompt string for Ollama """ return self._convert_messages_to_prompt(messages) ``` The actual conversion is done by the `_convert_messages_to_prompt` method: ```python def _convert_messages_to_prompt(self, messages): """Convert standard messages to Ollama prompt format.""" prompt = "" for msg in messages: role = msg.get("role", "").lower() content = msg.get("content", "") if role == "system": prompt += f"\n{content}\n\n\n" elif role == "user": prompt += f"User: {content}\n\n" elif role == "assistant": prompt += f"Assistant: {content}\n\n" else: prompt += f"{content}\n\n" # Add final prompt for assistant response prompt += "Assistant: " return prompt ``` ### 3.4 Making Llama API Requests Make a request to the Llama API via Ollama: ```python def generate_content(self, messages, model=None, **kwargs): """Generate content from Llama.""" # Convert messages to Ollama format prompt = self.format_messages(messages) payload = { "model": model or self.default_model, "prompt": prompt, "stream": False } try: response = requests.post( f"{self.api_base}/api/generate", headers={"Content-Type": "application/json"}, json=payload, timeout=60 ) if response.status_code == 200: return response.json().get("response", "") ``` ## 4. Creating Message Structure To interact with either API, you need to structure your messages in a specific format: ```python def create_user_prompt(self, website): return self.user_prompt_template.format(title=website.title, text=website.text) def create_messages(self, website): return [ {"role": "system", "content": self.system_prompt}, {"role": "user", "content": self.create_user_prompt(website)} ] ``` System and user prompt examples: ```python DEFAULT_SYSTEM_PROMPT = ("You are an assistant that analyzes the contents of a website " "and provides a short summary, ignoring text that might be navigation related. " "Respond in markdown.") DEFAULT_USER_PROMPT_TEMPLATE = """ You are looking at a website titled {title} The contents of this website is as follows; please provide a short summary of this website in markdown. If it includes news or announcements, then summarize these too. {text} """ ``` This format includes: - A system message that sets the behavior of the AI assistant - A user message containing the actual content to process - The website object is used to insert relevant content into the user prompt template ## 5. Complete Integration Flow Here's the complete flow for integrating with either LLM API: ```python # Create the appropriate client client = LLMFactory.create_client("openai") # or "llama" # Validate credentials is_valid, message = client.validate_credentials() if not is_valid: print(message) exit(1) # Optional: Test connectivity test_response = client.test_connection("Hello, this is a test message.") print("Test API response:", test_response) # Create a prompt manager (or use default) prompt_manager = PromptManager() # Customize if needed # Prepare website content website = fetch_website_content(url) # Generate summary with the LLM API summary = client.generate_content( prompt_manager.create_messages(website), model=None, # Uses default model for the client temperature=0.7 # Optional parameter ) ``` ## A Key Note On The Abstract Interface The system now uses an abstract base class (`BaseLLMClient`) that defines the common interface for all LLM clients. Each provider-specific client implements this interface, including the format_messages method that handles converting the standard message format to the provider's expected format. This approach eliminates the need to first create messages in OpenAI format and then translate them. Instead, each client knows how to format messages appropriately for its specific provider. ## 6. Additional API Parameters ### OpenAI Parameters ```python response = client.generate_content( messages=messages, model="gpt-4o-mini", temperature=0.7, # Controls randomness (0-1) max_tokens=1500, # Maximum length of response frequency_penalty=0.0, # Reduces repetition of token sequences presence_penalty=0.0, # Reduces talking about the same topics stop=None # Sequences where the API will stop generating ) ``` ### Llama/Ollama Parameters ```python response = client.generate_content( messages=messages, model="llama3.2:latest", temperature=0.7, # Controls randomness (0-1) # Other parameters supported by Ollama ) ``` ## 7. Example Usage Here's an example using both providers: ```python # Example with OpenAI openai_client = LLMFactory.create_client("openai") is_valid, message = openai_client.validate_credentials() if is_valid: print("OpenAI credentials validated successfully") url_to_summarize = "https://example.com" print(f"Fetching and summarizing content from {url_to_summarize}") summary = summarize_url(openai_client, url_to_summarize) print("Summary from OpenAI:", summary) # Example with Llama llama_client = LLMFactory.create_client("llama") is_valid, message = llama_client.validate_credentials() if is_valid: print("Llama credentials validated successfully") url_to_summarize = "https://example.com" print(f"Fetching and summarizing content from {url_to_summarize}") summary = summarize_url(llama_client, url_to_summarize) print("Summary from Llama:", summary) ``` ## 8. Environment Setup Create a `.env` file in your project root with: ``` # OpenAI Configuration OPENAI_API_KEY=sk-your-openai-api-key # Llama Configuration (optional, defaults to http://localhost:11434) LLAMA_API_URL=http://localhost:11434 ``` Make sure to install Ollama locally if you want to use Llama models: [Ollama Installation Guide](https://github.com/ollama/ollama) # Annex: OpenAI vs Llama Side-by-Side Comparison This annex provides a clear comparison between OpenAI and Llama (via Ollama) implementations for each critical step in the integration process. ## 1. Import Statements **OpenAI:** ```python import os from dotenv import load_dotenv from openai import OpenAI ``` **Llama (Ollama):** ```python import os import requests from dotenv import load_dotenv ``` ## 2. Client Initialization **OpenAI:** ```python # Load environment variables load_dotenv(override=True) # Initialize client (automatically uses OPENAI_API_KEY from environment) client = OpenAI() # Alternative with explicit API key api_key = os.getenv('OPENAI_API_KEY') client = OpenAI(api_key=api_key) ``` **Llama (Ollama):** ```python # Load environment variables load_dotenv(override=True) # Get base URL (defaults to localhost if not specified) api_base = os.getenv('LLAMA_API_URL', 'http://localhost:11434') # No client object is created - direct API calls are made via requests ``` ## 3. Message Formatting **OpenAI:** ```python # The BaseLLMClient abstract method implemented for OpenAI def format_messages(self, messages): """ Format messages for OpenAI API. Args: messages: List of message dictionaries with role and content Returns: list: The messages formatted for OpenAI """ # OpenAI already uses the format we're using, so we can return as-is return messages ``` Examples: ```python # OpenAI uses a structured format with role-based messages messages = [ {"role": "system", "content": "You are an assistant that analyzes websites."}, {"role": "user", "content": f"Summarize this website: {website_content}"} ] # Simple concrete example: messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user","content": "What is machine learning?"}, {"role": "assistant", "content": "Machine learning is a branch of artificial intelligence that focuses on building systems that learn from data."}, {"role": "user", "content": "Can you give me a simple example?"} ] ``` **Llama (Ollama):** ```python # The BaseLLMClient abstract method implemented for Llama def format_messages(self, messages): """ Format messages for Ollama API. Args: messages: List of message dictionaries with role and content Returns: str: The messages formatted as a prompt string for Ollama """ return self._convert_messages_to_prompt(messages) def _convert_messages_to_prompt(self, messages): """Convert standard messages to Ollama prompt format.""" prompt = "" for msg in messages: role = msg.get("role", "").lower() content = msg.get("content", "") if role == "system": prompt += f"\n{content}\n\n\n" elif role == "user": prompt += f"User: {content}\n\n" elif role == "assistant": prompt += f"Assistant: {content}\n\n" else: prompt += f"{content}\n\n" # Add final prompt for assistant response prompt += "Assistant: " return prompt ``` Examples: ```python # Simple concrete example of converted messages: # Starting with the same OpenAI-style messages messages = [ {"role":"system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is machine learning?"}, {"role":"assistant", "content": "Machine learning is a branch of artificial intelligence that focuses on building systems that learn from data."}, {"role": "user", "content": "Can you give me a simple example?"} ] # Convert OpenAI-style messages to Ollama format prompt = convert_messages_to_prompt(messages) # After conversion, the prompt would look like: converted_prompt = """ You are a helpful assistant. User: What is machine learning? # Convert OpenAI-style messages to Ollama format prompt = convert_messages_to_prompt(messages) ``` ## 4. Making Requests **OpenAI:** ```python def generate_content(self, messages, model=None, **kwargs): """Generate content from OpenAI.""" # Format messages appropriately for OpenAI formatted_messages = self.format_messages(messages) response = self.client.chat.completions.create( model=model or self.default_model, messages=formatted_messages, **kwargs ) return response.choices[0].message.content ``` **Llama (Ollama):** ```python def generate_content(self, messages, model=None, **kwargs): """Generate content from Llama.""" # Format messages appropriately for Llama/Ollama prompt = self.format_messages(messages) payload = { "model": model or self.default_model, "prompt": prompt, "stream": False, **kwargs } response = requests.post( f"{self.api_base}/api/generate", headers={"Content-Type": "application/json"}, json=payload, timeout=60 ) if response.status_code == 200: return response.json().get("response", "") ``` ## 5. Processing Response **OpenAI:** ```python # Response structure """ { "id": "chatcmpl-123abc", "object": "chat.completion", "created": 1677858242, "model": "gpt-4o-mini", "choices": [ { "message": { "role": "assistant", "content": "This website is about..." }, "index": 0, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 13, "completion_tokens": 7, "total_tokens": 20 } } """ # Extracting content content = response.choices[0].message.content ``` **Llama (Ollama):** ```python # Response structure (if response.status_code == 200) """ { "model": "llama3", "response": "This website is about...", "done": true } """ # Extracting content if response.status_code == 200: content = response.json().get("response", "") else: content = f"Error: {response.status_code}, {response.text}" ``` ## 6. Complete Side-by-Side Example With New Architecture **OpenAI:** ```python import os from dotenv import load_dotenv from openai import OpenAI from llm.base_client import BaseLLMClient class OpenAIClient(BaseLLMClient): def __init__(self): self.client = None self.default_model = "gpt-4o-mini" def initialize(self): load_dotenv(override=True) self.client = OpenAI() # Uses OPENAI_API_KEY from environment return self def format_messages(self, messages): # OpenAI already uses our format, so return as-is return messages def generate_content(self, messages, model=None, **kwargs): formatted_messages = self.format_messages(messages) response = self.client.chat.completions.create( model=model or self.default_model, messages=formatted_messages, **kwargs ) return response.choices[0].message.content ``` **Llama (Ollama):** ```python import os import requests from dotenv import load_dotenv from llm.base_client import BaseLLMClient class LlamaClient(BaseLLMClient): def __init__(self): self.api_base = None self.default_model = "llama3" def initialize(self): load_dotenv(override=True) self.api_base = os.getenv('LLAMA_API_URL', 'http://localhost:11434') return self def format_messages(self, messages): # Convert standard message format to Ollama prompt return self._convert_messages_to_prompt(messages) def _convert_messages_to_prompt(self, messages): prompt = "" for msg in messages: role = msg.get("role", "").lower() content = msg.get("content", "") if role == "system": prompt += f"\n{content}\n\n\n" elif role == "user": prompt += f"User: {content}\n\n" elif role == "assistant": prompt += f"Assistant: {content}\n\n" else: prompt += f"{content}\n\n" prompt += "Assistant: " return prompt def generate_content(self, messages, model=None, **kwargs): prompt = self.format_messages(messages) payload = { "model": model or self.default_model, "prompt": prompt, "stream": False } response = requests.post( f"{self.api_base}/api/generate", headers={"Content-Type": "application/json"}, json=payload, timeout=60 ) if response.status_code == 200: return response.json().get("response", "") else: return f"Error: {response.status_code}, {response.text}" ``` ## 7. Key Differences Summary | Aspect | OpenAI | Llama (Ollama) | |--------|--------|----------------| | **Authentication** | API key in environment or explicitly passed | No authentication, just URL to local server | | **Client Library** | Official Python SDK | Standard HTTP requests | | **Message Format Implementation** | Returns standard messages as-is | Converts to text-based prompt format | | **Format Method Return Type** | List of dictionaries | String | | **Request Format** | Client method calls | Direct HTTP POST requests | | **Response Format** | Structured object with choices | Simple JSON with response field | | **Streaming** | Supported via stream parameter | Supported via stream parameter | | **Error Handling** | SDK throws exceptions | Need to check HTTP status codes | ## 8. The Abstract Base Class ```python from abc import ABC, abstractmethod class BaseLLMClient(ABC): """Abstract base class for LLM clients.""" @abstractmethod def initialize(self): """Initialize the LLM client.""" pass @abstractmethod def validate_credentials(self): """ Validate API credentials. Returns: tuple: (is_valid, message) """ pass @abstractmethod def format_messages(self, messages): """ Format messages according to the provider's requirements. Args: messages: List of message dictionaries with role and content Returns: The properly formatted messages for this specific provider """ pass @abstractmethod def generate_content(self, messages, model=None, **kwargs): """ Generate content from the LLM. Args: messages: The messages to send model: The model to use for generation **kwargs: Additional provider-specific parameters Returns: str: The generated content """ pass ``` This abstract base class ensures that all LLM clients implement the same interface, making it easy to switch between providers. The `format_messages` method is a key part of this architecture, as it allows each client to format messages appropriately for its specific provider.