You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

3.4 KiB

AI Web Page Summarizer

This project is a simple AI-powered web page summarizer that leverages OpenAI's GPT models and local inference with Ollama to generate concise summaries of given text. The goal is to create a "Reader's Digest of the Internet" by summarizing web content efficiently.

Features

  • Summarize text using OpenAI's GPT models or local Ollama models.
  • Flexible summarization engine selection (OpenAI API, Ollama API, or Ollama library).
  • Simple and modular code structure.
  • Error handling for better reliability.

Project Structure

ai-summarizer/
│-- summarizer/
│   │-- __init__.py
│   │-- fetcher.py       # Web content fetching logic
│   │-- summarizer.py    # Main summarization logic
│-- utils/
│   │-- __init__.py
│   │-- logger.py        # Logging configuration
│   │-- config.py        # env configuration
│-- main.py              # Entry point of the app
│-- .env                 # Environment variables
│-- requirements.txt     # Python dependencies
│-- README.md            # Project documentation

Prerequisites

  • Python 3.8 or higher
  • OpenAI API Key (You can obtain it from OpenAI)
  • Ollama installed locally (Installation Guide)
  • conda for managing environments (optional)

Installation

  1. Clone the repository:

    git clone https://github.com/your-username/ai-summarizer.git
    cd ai-summarizer
    
  2. Create a virtual environment (optional but recommended):

    conda create --name summarizer-env python=3.9
    conda activate summarizer-env
    
  3. Install dependencies:

    pip install -r requirements.txt
    
  4. Set up environment variables:

    Create a .env file in the project root and add your OpenAI API key (if using OpenAI):

    OPENAI_API_KEY=your-api-key-here
    

Usage

  1. Run the summarizer:

    python main.py
    
  2. Sample Output:

    Enter a URL to summarize: https://example.com
    Summary of the page:
    AI refers to machines demonstrating intelligence similar to humans and animals.
    
  3. Engine Selection:

    The summarizer supports multiple engines. Modify main.py to select your preferred model:

    summary = summarize_text(content, 'gpt-4o-mini', engine="openai")
    summary = summarize_text(content, 'deepseek-r1:1.5B', engine="ollama-api")
    summary = summarize_text(content, 'deepseek-r1:1.5B', engine="ollama-lib")
    

Configuration

You can modify the model, max tokens, and temperature in summarizer/summarizer.py:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    max_tokens=300,
    temperature=0.7
)

Error Handling

If any issues occur, the script will print an error message, for example:

Error during summarization: Invalid API key or Ollama not running.

Dependencies

The required dependencies are listed in requirements.txt:

openai
python-dotenv
requests
ollama-api

Install them using:

pip install -r requirements.txt

Contributing

Contributions are welcome! Feel free to fork the repository and submit pull requests.

License

This project is licensed under the MIT License. See the LICENSE file for more details.

Contact

For any inquiries, please reach out to: