From 1a7f4e86b0d3a55b36d2754d1df8cb77dd505208 Mon Sep 17 00:00:00 2001 From: lakovicb <> Date: Wed, 23 Apr 2025 16:53:26 +0200 Subject: [PATCH] Added detailed README for Playwright-based scraper contribution --- .../playwright-bojan/README.md | 79 +++++++++++-------- 1 file changed, 45 insertions(+), 34 deletions(-) diff --git a/community-contributions/playwright-bojan/README.md b/community-contributions/playwright-bojan/README.md index f24b91c..314b468 100644 --- a/community-contributions/playwright-bojan/README.md +++ b/community-contributions/playwright-bojan/README.md @@ -1,56 +1,67 @@ -# 🧠 Playwright-Based Web Scraper for openai.com -### πŸ“š Community Contribution for Ed Donner's "LLM Engineering: Master AI" Course +# 🧠 Community Contribution: Async Playwright-based OpenAI Scraper -> _β€œAn extra exercise for those who enjoy web scraping... -> In the community-contributions folder, you'll find an example Selenium solution from a student.”_ +This contribution presents a fully asynchronous, headless-browser-based scraper for [https://openai.com](https://openai.com) using **Playwright** β€” an alternative to Selenium. + +Developed by: [lakovicb](https://github.com/lakovicb) +IDE used: WingIDE Pro (Jupyter compatibility via `nest_asyncio`) --- -## πŸ” About This Project +## πŸ“¦ Features -This is a response to Ed Donner’s bonus exercise to scrape `https://openai.com`, which uses dynamic JavaScript rendering. -A fellow student contributed a Selenium-based solution β€” this one goes a step further with **Playwright**. +- 🧭 Simulates human-like interactions (mouse movement, scrolling) +- 🧠 GPT-based analysis using OpenAI's API +- πŸ§ͺ Works inside **JupyterLab** using `nest_asyncio` +- πŸ“Š Prometheus metrics for scraping observability +- ⚑ Smart content caching via `diskcache` --- -## πŸ†š Why Playwright Over Selenium? +## πŸš€ How to Run -| Feature | Selenium | Playwright πŸ† | -|----------------------|------------------------------|-----------------------------| -| **Installation** | More complex setup | Minimal + faster setup | -| **Speed** | Slower due to architecture | Faster execution (async) | -| **Multi-browser** | Requires config | Built-in Chrome, Firefox, WebKit support | -| **Headless mode** | Less stable | Super stable | -| **Async-friendly** | Not built-in | Native support via asyncio | -| **Interaction APIs** | Limited | Richer simulation (mouse, scroll, etc.) | +### 1. Install dependencies ---- +```bash +pip install -r requirements.txt +``` + +> Ensure [Playwright is installed & browsers are downloaded](https://playwright.dev/python/docs/intro) + +```bash +playwright install +``` -## βš™οΈ Features +### 2. Set environment variables in `.env` -- βœ… **Full JavaScript rendering** using Chromium -- βœ… **Human-like behavior simulation** (mouse movement, scrolling, typing) -- βœ… **Caching** with `diskcache` -- βœ… **Prometheus metrics** -- βœ… **Asynchronous scraping logic** -- βœ… **Content summarization via OpenAI GPT API** +```env +OPENAI_API_KEY=your_openai_key +BROWSER_PATH=/usr/bin/chromium-browser +``` + +You can also define optional proxy/login params if needed. --- -## 🧠 Why not in JupyterLab? +## πŸ“˜ Notebooks Included + +| Notebook | Description | +|----------|-------------| +| `Playwright_Solution_JupyterAsync.ipynb` | Executes async scraper directly inside Jupyter | +| `Playwright_Solution_Showcase_Formatted.ipynb` | Nicely formatted output for human reading | + +--- -Due to the async nature of Playwright and the use of `asyncio.run()`, running this inside Jupyter causes `RuntimeError` conflicts. +## πŸ” Output Example -This solution was developed and tested in: +- GPT-generated summary +- Timeline of updates +- Entities and projects mentioned +- Structured topics & themes -- πŸ’» WingIDE 10 Pro -- 🐧 Ubuntu via WSL -- 🐍 Conda environment with Anaconda Python 3.12 +βœ… *Can be extended with PDF export, LangChain pipeline, or vector store ingestion.* --- -## πŸš€ How to Run +## πŸ™ Thanks -1. Install dependencies: -```bash -pip install -r requirements.txt +Huge thanks to Ed Donner for the amazing course and challenge inspiration!