You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
lakovicb 1a7f4e86b0 Added detailed README for Playwright-based scraper contribution 2 weeks ago
..
Playwright_Solution_Showcase_Formatted.ipynb Added formatted markdown-only notebook for Playwright scraper output 2 weeks ago
README.md Added detailed README for Playwright-based scraper contribution 2 weeks ago
openai_scraper_playwright.py Added Playwright-based scraper solution to community-contributions 2 weeks ago
requirements.txt Added requirements.txt for scraper dependencies 2 weeks ago

README.md

🧠 Community Contribution: Async Playwright-based OpenAI Scraper

This contribution presents a fully asynchronous, headless-browser-based scraper for https://openai.com using Playwright — an alternative to Selenium.

Developed by: lakovicb
IDE used: WingIDE Pro (Jupyter compatibility via nest_asyncio)


📦 Features

  • 🧭 Simulates human-like interactions (mouse movement, scrolling)
  • 🧠 GPT-based analysis using OpenAI's API
  • 🧪 Works inside JupyterLab using nest_asyncio
  • 📊 Prometheus metrics for scraping observability
  • Smart content caching via diskcache

🚀 How to Run

1. Install dependencies

pip install -r requirements.txt

Ensure Playwright is installed & browsers are downloaded

playwright install

2. Set environment variables in .env

OPENAI_API_KEY=your_openai_key
BROWSER_PATH=/usr/bin/chromium-browser

You can also define optional proxy/login params if needed.


📘 Notebooks Included

Notebook Description
Playwright_Solution_JupyterAsync.ipynb Executes async scraper directly inside Jupyter
Playwright_Solution_Showcase_Formatted.ipynb Nicely formatted output for human reading

🔁 Output Example

  • GPT-generated summary
  • Timeline of updates
  • Entities and projects mentioned
  • Structured topics & themes

Can be extended with PDF export, LangChain pipeline, or vector store ingestion.


🙏 Thanks

Huge thanks to Ed Donner for the amazing course and challenge inspiration!