|
3 months ago | |
---|---|---|
.. | ||
summarizer | 3 months ago | |
utils | 3 months ago | |
.gitignore | 3 months ago | |
README.md | 3 months ago | |
main.py | 3 months ago | |
requirements.txt | 3 months ago |
README.md
AI Web Page Summarizer
This project is a simple AI-powered web page summarizer that leverages OpenAI's GPT models and local inference with Ollama to generate concise summaries of given text. The goal is to create a "Reader's Digest of the Internet" by summarizing web content efficiently.
Features
- Summarize text using OpenAI's GPT models or local Ollama models.
- Flexible summarization engine selection (OpenAI API, Ollama API, or Ollama library).
- Simple and modular code structure.
- Error handling for better reliability.
Project Structure
ai-summarizer/
│-- summarizer/
│ │-- __init__.py
│ │-- fetcher.py # Web content fetching logic
│ │-- summarizer.py # Main summarization logic
│-- utils/
│ │-- __init__.py
│ │-- logger.py # Logging configuration
│-- main.py # Entry point of the app
│-- .env # Environment variables
│-- requirements.txt # Python dependencies
│-- README.md # Project documentation
Prerequisites
- Python 3.8 or higher
- OpenAI API Key (You can obtain it from OpenAI)
- Ollama installed locally (Installation Guide)
conda
for managing environments (optional)
Installation
-
Clone the repository:
git clone https://github.com/your-username/ai-summarizer.git cd ai-summarizer
-
Create a virtual environment (optional but recommended):
conda create --name summarizer-env python=3.9 conda activate summarizer-env
-
Install dependencies:
pip install -r requirements.txt
-
Set up environment variables:
Create a
.env
file in the project root and add your OpenAI API key (if using OpenAI):OPENAI_API_KEY=your-api-key-here
Usage
-
Run the summarizer:
python main.py
-
Sample Output:
Enter a URL to summarize: https://example.com Summary of the page: AI refers to machines demonstrating intelligence similar to humans and animals.
-
Engine Selection:
The summarizer supports multiple engines. Modify
main.py
to select your preferred model:summary = summarize_text(content, 'gpt-4o-mini', engine="openai") summary = summarize_text(content, 'deepseek-r1:1.5B', engine="ollama-api") summary = summarize_text(content, 'deepseek-r1:1.5B', engine="ollama-lib")
Configuration
You can modify the model, max tokens, and temperature in summarizer/summarizer.py
:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[...],
max_tokens=300,
temperature=0.7
)
Error Handling
If any issues occur, the script will print an error message, for example:
Error during summarization: Invalid API key or Ollama not running.
Dependencies
The required dependencies are listed in requirements.txt
:
openai
python-dotenv
requests
ollama-api
Install them using:
pip install -r requirements.txt
Contributing
Contributions are welcome! Feel free to fork the repository and submit pull requests.
License
This project is licensed under the MIT License. See the LICENSE
file for more details.
Contact
For any inquiries, please reach out to:
- Linkedin: https://www.linkedin.com/in/khanarafat/
- GitHub: https://github.com/raoarafat