You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

271 lines
14 KiB

{
"cells": [
{
"cell_type": "markdown",
"id": "36ed9f14-b349-40e9-a42c-b367e77f8bda",
"metadata": {},
"source": [
"## An extra exercise for those who enjoy web scraping\n",
"\n",
"You may notice that if you try `display_summary(\"https://openai.com\")` - it doesn't work! That's because OpenAI has a fancy website that uses Javascript. There are many ways around this that some of you might be familiar with. For example, Selenium is a hugely popular framework that runs a browser behind the scenes, renders the page, and allows you to query it. If you have experience with Selenium, Playwright or similar, then feel free to improve the Website class to use them. Please push your code afterwards so I can share it with other students!"
]
},
{
"cell_type": "markdown",
"id": "c97ad592-c8be-4583-a19c-ac813e56f410",
"metadata": {},
"source": [
"## Mac Users\n",
"\n",
"I find some challenges while setting up this in MAC silicon M1 chip. Execute below commands in MAC terminal.\n",
"\n",
"1. Download chromedriver.\n",
"2. Unzip and add it to the path.\n",
"3. Set Extended attributes."
]
},
{
"cell_type": "markdown",
"id": "b635b345-b000-48cc-8a7f-7df279a489a3",
"metadata": {},
"source": [
"cd ~/Downloads\n",
"wget https://storage.googleapis.com/chrome-for-testing-public/133.0.6943.126/mac-arm64/chromedriver-mac-arm64.zip\n",
"unzip chromedriver-mac-arm64.zip\n",
"sudo mv chromedriver-mac-arm64/chromedriver /usr/local/bin/\n",
"chmod +x /usr/local/bin/chromedriver\n",
"cd /usr/local/bin/\n",
"xattr -d com.apple.quarantine chromedriver\n",
"cd \n",
"chromedriver --version"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "17c7c79a-8ae0-4f5d-a7c8-c54aa7ba90fd",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: selenium in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (4.29.0)\n",
"Requirement already satisfied: urllib3<3,>=1.26 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from urllib3[socks]<3,>=1.26->selenium) (2.3.0)\n",
"Requirement already satisfied: trio~=0.17 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium) (0.29.0)\n",
"Requirement already satisfied: trio-websocket~=0.9 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium) (0.12.1)\n",
"Requirement already satisfied: certifi>=2021.10.8 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium) (2025.1.31)\n",
"Requirement already satisfied: typing_extensions~=4.9 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium) (4.12.2)\n",
"Requirement already satisfied: websocket-client~=1.8 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium) (1.8.0)\n",
"Requirement already satisfied: attrs>=23.2.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio~=0.17->selenium) (25.1.0)\n",
"Requirement already satisfied: sortedcontainers in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio~=0.17->selenium) (2.4.0)\n",
"Requirement already satisfied: idna in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio~=0.17->selenium) (3.10)\n",
"Requirement already satisfied: outcome in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio~=0.17->selenium) (1.3.0.post0)\n",
"Requirement already satisfied: sniffio>=1.3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio~=0.17->selenium) (1.3.1)\n",
"Requirement already satisfied: wsproto>=0.14 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio-websocket~=0.9->selenium) (1.2.0)\n",
"Requirement already satisfied: pysocks!=1.5.7,<2.0,>=1.5.6 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from urllib3[socks]<3,>=1.26->selenium) (1.7.1)\n",
"Requirement already satisfied: h11<1,>=0.9.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium) (0.14.0)\n",
"Requirement already satisfied: undetected-chromedriver in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (3.5.5)\n",
"Requirement already satisfied: selenium>=4.9.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from undetected-chromedriver) (4.29.0)\n",
"Requirement already satisfied: requests in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from undetected-chromedriver) (2.32.3)\n",
"Requirement already satisfied: websockets in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from undetected-chromedriver) (14.2)\n",
"Requirement already satisfied: urllib3<3,>=1.26 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from urllib3[socks]<3,>=1.26->selenium>=4.9.0->undetected-chromedriver) (2.3.0)\n",
"Requirement already satisfied: trio~=0.17 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium>=4.9.0->undetected-chromedriver) (0.29.0)\n",
"Requirement already satisfied: trio-websocket~=0.9 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium>=4.9.0->undetected-chromedriver) (0.12.1)\n",
"Requirement already satisfied: certifi>=2021.10.8 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium>=4.9.0->undetected-chromedriver) (2025.1.31)\n",
"Requirement already satisfied: typing_extensions~=4.9 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium>=4.9.0->undetected-chromedriver) (4.12.2)\n",
"Requirement already satisfied: websocket-client~=1.8 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from selenium>=4.9.0->undetected-chromedriver) (1.8.0)\n",
"Requirement already satisfied: charset_normalizer<4,>=2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from requests->undetected-chromedriver) (3.4.1)\n",
"Requirement already satisfied: idna<4,>=2.5 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from requests->undetected-chromedriver) (3.10)\n",
"Requirement already satisfied: attrs>=23.2.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio~=0.17->selenium>=4.9.0->undetected-chromedriver) (25.1.0)\n",
"Requirement already satisfied: sortedcontainers in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio~=0.17->selenium>=4.9.0->undetected-chromedriver) (2.4.0)\n",
"Requirement already satisfied: outcome in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio~=0.17->selenium>=4.9.0->undetected-chromedriver) (1.3.0.post0)\n",
"Requirement already satisfied: sniffio>=1.3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio~=0.17->selenium>=4.9.0->undetected-chromedriver) (1.3.1)\n",
"Requirement already satisfied: wsproto>=0.14 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from trio-websocket~=0.9->selenium>=4.9.0->undetected-chromedriver) (1.2.0)\n",
"Requirement already satisfied: pysocks!=1.5.7,<2.0,>=1.5.6 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from urllib3[socks]<3,>=1.26->selenium>=4.9.0->undetected-chromedriver) (1.7.1)\n",
"Requirement already satisfied: h11<1,>=0.9.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium>=4.9.0->undetected-chromedriver) (0.14.0)\n",
"Requirement already satisfied: beautifulsoup4 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (4.13.3)\n",
"Requirement already satisfied: soupsieve>1.2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from beautifulsoup4) (2.5)\n",
"Requirement already satisfied: typing-extensions>=4.0.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from beautifulsoup4) (4.12.2)\n"
]
}
],
"source": [
"!pip install selenium\n",
"!pip install undetected-chromedriver\n",
"!pip install beautifulsoup4"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c10bd630-2dfd-4572-8c21-2dc4c6a372ab",
"metadata": {},
"outputs": [],
"source": [
"from selenium import webdriver\n",
"from selenium.webdriver.chrome.service import Service\n",
"from selenium.webdriver.common.by import By\n",
"from selenium.webdriver.chrome.options import Options\n",
"from openai import OpenAI\n",
"import os\n",
"import requests\n",
"from dotenv import load_dotenv\n",
"from bs4 import BeautifulSoup\n",
"from IPython.display import Markdown, display\n",
"from openai import OpenAI"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "6fb3641d-e9f8-4f5b-bb9d-ee0e971cccdb",
"metadata": {},
"outputs": [],
"source": [
"OLLAMA_API = \"http://localhost:11434/api/chat\"\n",
"HEADERS = {\"Content-Type\": \"application/json\"}\n",
"MODEL = \"llama3.2\"\n",
"PATH_TO_CHROME_DRIVER = '/usr/local/bin/chromedriver'\n",
"system_prompt = \"You are an assistant that analyzes the contents of a website \\\n",
"and provides a short summary, ignoring text that might be navigation related. \\\n",
"Respond in markdown. Highlight all the products this website offered and also find when website is created.\"\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "5d57e958",
"metadata": {},
"outputs": [],
"source": [
"class Website:\n",
" url: str\n",
" title: str\n",
" text: str\n",
"\n",
" def __init__(self, url):\n",
" self.url = url\n",
"\n",
" options = Options()\n",
"\n",
" options.add_argument(\"--no-sandbox\")\n",
" options.add_argument(\"--disable-dev-shm-usage\")\n",
"\n",
" service = Service(PATH_TO_CHROME_DRIVER)\n",
" driver = webdriver.Chrome(service=service, options=options)\n",
" driver.get(url)\n",
"\n",
" # input(\"Please complete the verification in the browser and press Enter to continue...\")\n",
" page_source = driver.page_source\n",
" driver.quit()\n",
"\n",
" soup = BeautifulSoup(page_source, 'html.parser')\n",
" self.title = soup.title.string if soup.title else \"No title found\"\n",
" for irrelevant in soup([\"script\", \"style\", \"img\", \"input\"]):\n",
" irrelevant.decompose()\n",
" self.text = soup.get_text(separator=\"\\n\", strip=True)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "56df8cd2-2707-43f6-a066-3367846929b3",
"metadata": {},
"outputs": [],
"source": [
"def user_prompt_for(website):\n",
" user_prompt = f\"You are looking at a website titled {website.title}\"\n",
" user_prompt += \"\\nThe contents of this website is as follows; \\\n",
"please provide a short summary of this website in markdown. \\\n",
"If it includes news or announcements, then summarize these too.\\n\\n\"\n",
" user_prompt += website.text\n",
" return user_prompt\n",
"\n",
"\n",
"\n",
"def messages_for(website):\n",
" return [\n",
" {\"role\": \"system\", \"content\": system_prompt},\n",
" {\"role\": \"user\", \"content\": user_prompt_for(website)}\n",
" ]\n",
"\n",
"\n",
"def summarize(url):\n",
" website = Website(url)\n",
" ollama_via_openai = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')\n",
" response = ollama_via_openai.chat.completions.create(\n",
" model=MODEL,\n",
" messages = messages_for(website)\n",
" )\n",
" return response.choices[0].message.content\n",
"\n",
"\n",
"def display_summary(url):\n",
" summary = summarize(url)\n",
" display(Markdown(summary))"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "f2eb9599",
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"It appears that you have provided a sample website or travel booking platform, specifically for flights and hotels in the Middle East region. The content includes:\n",
"\n",
"1. **Flights**: A search engine to find flights across various airlines.\n",
"2. **Hotels**: A selection of chain hotels available for booking.\n",
"3. **Travel**: A general page with FAQs and information about traveling within Saudi Arabia, Kuwait, and other nearby countries.\n",
"4. **Almosafer App**: An advertisement for the Almosafer app, which offers features like secure payment channels, easy booking processes, and user-friendly designs.\n",
"\n",
"The platform also displays a list of trending searches, airlines, and countries to facilitate searching and planning trips.\n",
"\n",
"Please let me know if you have any specific questions or need further assistance with this website sample."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display_summary(\"https://ae.almosafer.com\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "31b66c0f-6b45-4986-b77c-758625945a91",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
}
},
"nbformat": 4,
"nbformat_minor": 5
}