Browse Source

Added detailed README for Playwright-based scraper contribution

pull/340/head
lakovicb 2 weeks ago
parent
commit
1a7f4e86b0
  1. 79
      community-contributions/playwright-bojan/README.md

79
community-contributions/playwright-bojan/README.md

@ -1,56 +1,67 @@
# 🧠 Playwright-Based Web Scraper for openai.com
### 📚 Community Contribution for Ed Donner's "LLM Engineering: Master AI" Course
# 🧠 Community Contribution: Async Playwright-based OpenAI Scraper
> _“An extra exercise for those who enjoy web scraping...
> In the community-contributions folder, you'll find an example Selenium solution from a student.”_
This contribution presents a fully asynchronous, headless-browser-based scraper for [https://openai.com](https://openai.com) using **Playwright** — an alternative to Selenium.
Developed by: [lakovicb](https://github.com/lakovicb)
IDE used: WingIDE Pro (Jupyter compatibility via `nest_asyncio`)
---
## 🔍 About This Project
## 📦 Features
This is a response to Ed Donner’s bonus exercise to scrape `https://openai.com`, which uses dynamic JavaScript rendering.
A fellow student contributed a Selenium-based solution — this one goes a step further with **Playwright**.
- 🧭 Simulates human-like interactions (mouse movement, scrolling)
- 🧠 GPT-based analysis using OpenAI's API
- 🧪 Works inside **JupyterLab** using `nest_asyncio`
- 📊 Prometheus metrics for scraping observability
- ⚡ Smart content caching via `diskcache`
---
## 🆚 Why Playwright Over Selenium?
## 🚀 How to Run
| Feature | Selenium | Playwright 🏆 |
|----------------------|------------------------------|-----------------------------|
| **Installation** | More complex setup | Minimal + faster setup |
| **Speed** | Slower due to architecture | Faster execution (async) |
| **Multi-browser** | Requires config | Built-in Chrome, Firefox, WebKit support |
| **Headless mode** | Less stable | Super stable |
| **Async-friendly** | Not built-in | Native support via asyncio |
| **Interaction APIs** | Limited | Richer simulation (mouse, scroll, etc.) |
### 1. Install dependencies
---
```bash
pip install -r requirements.txt
```
> Ensure [Playwright is installed & browsers are downloaded](https://playwright.dev/python/docs/intro)
```bash
playwright install
```
## ⚙ Features
### 2. Set environment variables in `.env`
- ✅ **Full JavaScript rendering** using Chromium
- ✅ **Human-like behavior simulation** (mouse movement, scrolling, typing)
- ✅ **Caching** with `diskcache`
- ✅ **Prometheus metrics**
- ✅ **Asynchronous scraping logic**
- ✅ **Content summarization via OpenAI GPT API**
```env
OPENAI_API_KEY=your_openai_key
BROWSER_PATH=/usr/bin/chromium-browser
```
You can also define optional proxy/login params if needed.
---
## 🧠 Why not in JupyterLab?
## 📘 Notebooks Included
| Notebook | Description |
|----------|-------------|
| `Playwright_Solution_JupyterAsync.ipynb` | Executes async scraper directly inside Jupyter |
| `Playwright_Solution_Showcase_Formatted.ipynb` | Nicely formatted output for human reading |
---
Due to the async nature of Playwright and the use of `asyncio.run()`, running this inside Jupyter causes `RuntimeError` conflicts.
## 🔁 Output Example
This solution was developed and tested in:
- GPT-generated summary
- Timeline of updates
- Entities and projects mentioned
- Structured topics & themes
- 💻 WingIDE 10 Pro
- 🐧 Ubuntu via WSL
- 🐍 Conda environment with Anaconda Python 3.12
✅ *Can be extended with PDF export, LangChain pipeline, or vector store ingestion.*
---
## 🚀 How to Run
## 🙏 Thanks
1. Install dependencies:
```bash
pip install -r requirements.txt
Huge thanks to Ed Donner for the amazing course and challenge inspiration!

Loading…
Cancel
Save