From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
226 lines
6.8 KiB
226 lines
6.8 KiB
{ |
|
"cells": [ |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 1, |
|
"metadata": { |
|
"vscode": { |
|
"languageId": "plaintext" |
|
} |
|
}, |
|
"outputs": [], |
|
"source": [ |
|
"import os\n", |
|
"import requests\n", |
|
"from dotenv import load_dotenv\n", |
|
"from bs4 import BeautifulSoup\n", |
|
"from IPython.display import Markdown, display\n", |
|
"from openai import OpenAI\n" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 2, |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"load_dotenv(override=True)\n", |
|
"api_key = os.getenv('OPENAI_API_KEY')" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 3, |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"openai = OpenAI()" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 4, |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"from selenium import webdriver\n", |
|
"from selenium.webdriver.chrome.service import Service\n", |
|
"from selenium.webdriver.common.by import By\n", |
|
"from selenium.webdriver.chrome.options import Options\n", |
|
"\n", |
|
"PATH_TO_CHROME_DRIVER = 'B:\\\\Users\\\\ekfon\\\\chromeDriver\\\\chromedriver.exe'\n", |
|
"\n", |
|
"class Website:\n", |
|
" url: str\n", |
|
" title: str\n", |
|
" text: str\n", |
|
"\n", |
|
" def __init__(self, url):\n", |
|
" self.url = url\n", |
|
"\n", |
|
" options = Options()\n", |
|
"\n", |
|
" options.add_argument(\"--no-sandbox\")\n", |
|
" options.add_argument(\"--disable-dev-shm-usage\")\n", |
|
"\n", |
|
" service = Service(PATH_TO_CHROME_DRIVER)\n", |
|
" driver = webdriver.Chrome(service=service, options=options)\n", |
|
" driver.get(url)\n", |
|
"\n", |
|
" input(\"Please complete the verification in the browser and press Enter to continue...\")\n", |
|
" page_source = driver.page_source\n", |
|
" driver.quit()\n", |
|
"\n", |
|
" soup = BeautifulSoup(page_source, 'html.parser')\n", |
|
" self.title = soup.title.string if soup.title else \"No title found\"\n", |
|
" for irrelevant in soup([\"script\", \"style\", \"img\", \"input\"]):\n", |
|
" irrelevant.decompose()\n", |
|
" self.text = soup.get_text(separator=\"\\n\", strip=True)" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 5, |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"system_prompt = \"You are a state-of-the-art website analyzing assistant. \\\n", |
|
"You provide a short summary, and you ignore any navigation-related content. \\\n", |
|
"Your response is presented in nice markdown.\"" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 6, |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"def user_prompt_for(website):\n", |
|
" user_prompt = f\"The website you are looking at is titled {website.title}\"\n", |
|
" user_prompt += \"\\nThe content of the website is as follows: \\\n", |
|
"Provide a short summary of this website. If the website contains news or announcements, \\\n", |
|
"summarize those, too.\\n\\n\"\n", |
|
" user_prompt += website.text\n", |
|
" return user_prompt" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 7, |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"def messages_for(website):\n", |
|
" return [\n", |
|
" {\"role\": \"system\", \"content\": system_prompt},\n", |
|
" {\"role\": \"user\", \"content\": user_prompt_for(website)}\n", |
|
" ]" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 8, |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"def summarize(url):\n", |
|
" website = Website(url)\n", |
|
" response = openai.chat.completions.create(\n", |
|
" model = \"gpt-4o-mini\",\n", |
|
" messages = messages_for(website)\n", |
|
" )\n", |
|
" return response.choices[0].message.content" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 9, |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"def display_summary(url):\n", |
|
" summary = summarize(url)\n", |
|
" display(Markdown(summary))" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": 10, |
|
"metadata": {}, |
|
"outputs": [ |
|
{ |
|
"name": "stdin", |
|
"output_type": "stream", |
|
"text": [ |
|
"Please complete the verification in the browser and press Enter to continue... \n" |
|
] |
|
}, |
|
{ |
|
"data": { |
|
"text/markdown": [ |
|
"# OpenAI Website Summary\n", |
|
"\n", |
|
"OpenAI focuses on the development and exploration of artificial intelligence (AI) technologies that are safe and beneficial for humanity. The website emphasizes several key aspects:\n", |
|
"\n", |
|
"## Mission and Purpose\n", |
|
"- OpenAI aims to create safe AGI (Artificial General Intelligence) that will be advantageous to all people.\n", |
|
"\n", |
|
"## Featured Products and Tools\n", |
|
"- The site introduces multiple AI solutions including:\n", |
|
" - **ChatGPT**: An AI for conversational interactions and various tasks.\n", |
|
" - **Sora**: A new platform enhancing creativity by generating text, images, and video from user input.\n", |
|
" - **ChatGPT Pro and Enterprise**: Advanced versions tailored for business and educational needs with enhanced functionality.\n", |
|
" - **API Platform**: Allows integration of OpenAI models into different applications.\n", |
|
"\n", |
|
"## Recent Announcements (News)\n", |
|
"- **Partnerships and Collaborations**: OpenAI announced a partnership with Apple to enhance their AI capabilities (June 2024).\n", |
|
"- **New Model Releases**: Introduction of the OpenAI o1 series that emphasizes improved reasoning in responses and tools for developers (Dec 2024).\n", |
|
"- **Content Extensions**: Collaboration with Le Monde and Prisa Media to include more diverse news content in ChatGPT (March 2024).\n", |
|
"\n", |
|
"## Research Contributions\n", |
|
"- OpenAI is actively engaged in research to improve AI safety, governance, and functionality.\n", |
|
"\n", |
|
"This summary captures the essence of OpenAI's objectives, product offerings, and recent developments in the field of AI." |
|
], |
|
"text/plain": [ |
|
"<IPython.core.display.Markdown object>" |
|
] |
|
}, |
|
"metadata": {}, |
|
"output_type": "display_data" |
|
} |
|
], |
|
"source": [ |
|
"display_summary(\"https://openai.com/\")" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [] |
|
} |
|
], |
|
"metadata": { |
|
"kernelspec": { |
|
"display_name": "Python 3 (ipykernel)", |
|
"language": "python", |
|
"name": "python3" |
|
}, |
|
"language_info": { |
|
"codemirror_mode": { |
|
"name": "ipython", |
|
"version": 3 |
|
}, |
|
"file_extension": ".py", |
|
"mimetype": "text/x-python", |
|
"name": "python", |
|
"nbconvert_exporter": "python", |
|
"pygments_lexer": "ipython3", |
|
"version": "3.11.11" |
|
} |
|
}, |
|
"nbformat": 4, |
|
"nbformat_minor": 4 |
|
}
|
|
|