Added updated Website class to parse webpages which is designed using JavaScript heavely at the end of week1 day1 notebook and tested it using openai, cnn, edwarddonner webpage
"An arena that pits LLMs against each other in a battle of diplomacy and deviousness\n",
"About\n",
"Posts\n",
"Well, hi there.\n",
"I’m Ed. I like writing code and experimenting with LLMs, and hopefully you’re here because you do too. I also enjoy DJing (but I’m badly out of practice), amateur electronic music production (\n",
"very\n",
"amateur) and losing myself in\n",
"Hacker News\n",
", nodding my head sagely to things I only half understand.\n",
"I’m the co-founder and CTO of\n",
"Nebula.io\n",
". We’re applying AI to a field where it can make a massive, positive impact: helping people discover their potential and pursue their reason for being. Recruiters use our product today to source, understand, engage and manage talent. I’m previously the founder and CEO of AI startup untapt,\n",
"acquired in 2021\n",
".\n",
"We work with groundbreaking, proprietary LLMs verticalized for talent, we’ve\n",
"patented\n",
"our matching model, and our award-winning platform has happy customers and tons of press coverage.\n",
"Connect\n",
"with me for more!\n",
"August 6, 2024\n",
"Outsmart LLM Arena – a battle of diplomacy and deviousness\n",
"June 26, 2024\n",
"Choosing the Right LLM: Toolkit and Resources\n",
"February 7, 2024\n",
"Fine-tuning an LLM on your texts: a simulation of you\n",
"January 31, 2024\n",
"Fine-tuning an LLM on your texts: part 4 – QLoRA\n",
"Navigation\n",
"Home\n",
"Outsmart\n",
"An arena that pits LLMs against each other in a battle of diplomacy and deviousness\n",
"About\n",
"Posts\n",
"Get in touch\n",
"ed [at] edwarddonner [dot] com\n",
"www.edwarddonner.com\n",
"Follow me\n",
"LinkedIn\n",
"Twitter\n",
"Facebook\n",
"Subscribe to newsletter\n",
"Type your email…\n",
"Subscribe\n"
]
}
],
"source": [
"source": [
"# Let's try one out\n",
"# Let's try one out\n",
"\n",
"\n",
@ -132,7 +185,7 @@
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": null,
"execution_count": 39,
"id": "abdb8417-c5dc-44bc-9bee-2e059d162699",
"id": "abdb8417-c5dc-44bc-9bee-2e059d162699",
"metadata": {},
"metadata": {},
"outputs": [],
"outputs": [],
@ -144,7 +197,7 @@
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": null,
"execution_count": 40,
"id": "f0275b1b-7cfe-4f9d-abfa-7650d378da0c",
"id": "f0275b1b-7cfe-4f9d-abfa-7650d378da0c",
"metadata": {},
"metadata": {},
"outputs": [],
"outputs": [],
@ -177,7 +230,7 @@
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": null,
"execution_count": 41,
"id": "0134dfa4-8299-48b5-b444-f2a8c3403c88",
"id": "0134dfa4-8299-48b5-b444-f2a8c3403c88",
"metadata": {},
"metadata": {},
"outputs": [],
"outputs": [],
@ -199,7 +252,7 @@
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": null,
"execution_count": 42,
"id": "905b9919-aba7-45b5-ae65-81b3d1d78e34",
"id": "905b9919-aba7-45b5-ae65-81b3d1d78e34",
"metadata": {},
"metadata": {},
"outputs": [],
"outputs": [],
@ -215,17 +268,28 @@
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": null,
"execution_count": 43,
"id": "05e38d41-dfa4-4b20-9c96-c46ea75d9fb5",
"id": "05e38d41-dfa4-4b20-9c96-c46ea75d9fb5",
"metadata": {},
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"'# Summary of Edward Donner\\'s Website\\n\\nThe website is a personal platform for Ed, who shares his interests in coding, experimenting with large language models (LLMs), DJing, and electronic music production. He is the co-founder and CTO of [Nebula.io](http://nebula.io), a company focused on using AI for talent management. Ed has previous experience as the founder and CEO of the AI startup untapt, which was acquired in 2021.\\n\\n## Key Features:\\n- **Outsmart LLM Arena**: A feature that showcases LLMs competing in scenarios focusing on diplomacy and strategy.\\n- **Resources**: Ed offers posts related to selecting and fine-tuning LLMs for various applications.\\n\\n## Recent Posts:\\n- **August 6, 2024**: \"Outsmart LLM Arena – a battle of diplomacy and deviousness\"\\n- **June 26, 2024**: \"Choosing the Right LLM: Toolkit and Resources\"\\n- **February 7, 2024**: \"Fine-tuning an LLM on your texts: a simulation of you\"\\n- **January 31, 2024**: \"Fine-tuning an LLM on your texts: part 4 – QLoRA\"\\n\\nThe site serves as a hub for Ed\\'s projects and insights related to AI and LLMs, inviting discussions and connections in this innovative field.'"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"source": [
"summarize(\"https://edwarddonner.com\")"
"summarize(\"https://edwarddonner.com\")"
]
]
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": null,
"execution_count": 44,
"id": "3d926d59-450e-4609-92ba-2d6f244f1342",
"id": "3d926d59-450e-4609-92ba-2d6f244f1342",
"metadata": {},
"metadata": {},
"outputs": [],
"outputs": [],
@ -237,30 +301,110 @@
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": null,
"execution_count": 45,
"id": "3018853a-445f-41ff-9560-d925d1774b2f",
"id": "3018853a-445f-41ff-9560-d925d1774b2f",
"metadata": {},
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/markdown": [
"# Summary of Edward Donner's Website\n",
"\n",
"The website belongs to Ed Donner, a co-founder and CTO at Nebula.io, which focuses on using AI to help individuals discover their potential and optimize talent recruitment processes. Ed expresses his passion for coding and experimenting with large language models (LLMs), as well as his interests in DJing and electronic music production.\n",
"\n",
"## Recent Posts\n",
"1. **August 6, 2024** - **Outsmart LLM Arena**: Discusses a platform where LLMs compete in diplomacy and strategy.\n",
"2. **June 26, 2024** - **Choosing the Right LLM**: Provides a toolkit and resources for selecting appropriate LLMs.\n",
"3. **February 7, 2024** - **Fine-tuning an LLM on Your Texts**: Offers insights on personalizing LLMs using individual text samples.\n",
"4. **January 31, 2024** - **Fine-tuning an LLM on Your Texts: Part 4 – QLoRA**: Continuation of the fine-tuning discussion focusing on a specific method. \n",
"\n",
"Ed invites engagement and connection through various platforms."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"source": [
"display_summary(\"https://edwarddonner.com\")"
"display_summary(\"https://edwarddonner.com\")"
]
]
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": null,
"execution_count": 46,
"id": "45d83403-a24c-44b5-84ac-961449b4008f",
"id": "45d83403-a24c-44b5-84ac-961449b4008f",
"metadata": {},
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/markdown": [
"# Summary of CNN Website\n",
"\n",
"The CNN website serves as a comprehensive news platform that provides a variety of content including breaking news articles, in-depth analysis, and videos across multiple categories such as world news, politics, business, health, entertainment, science, and more.\n",
"\n",
"### Current Highlights:\n",
"- **International Conflicts**: Ongoing updates on the Israel-Hamas war and the Ukraine-Russia conflict, including Israel's military actions in response to Iranian missile attacks.\n",
"- **Politics**: Coverage of the 2024 U.S. elections, recent vice presidential debate highlights, and discussions around election certification challenges.\n",
"- **Health**: Emphasis on the importance of flu and COVID-19 vaccinations in the wake of seasonal outbreaks.\n",
"- **Tragedies and Accidents**: Reports on a school bus fire in Thailand that resulted in multiple fatalities and severe impacts from recent weather events in the U.S.\n",
"- **Cultural Events**: Insights into significant events including the opening of the world’s first Nintendo Museum and highlights from Paris Fashion Week.\n",
"\n",
"### Additional Features:\n",
"- **Video Content**: The site includes video segments featuring news updates, expert opinions, and live television options.\n",
"- **Social Issues**: Articles covering societal impacts such as press freedom, AI regulation, and community health initiatives.\n",
"\n",
"In summary, CNN remains a key source for immediate news and diverse storytelling, with a strong focus on global events and pressing issues affecting society."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"source": [
"display_summary(\"https://cnn.com\")"
"display_summary(\"https://cnn.com\")"
]
]
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": null,
"execution_count": 47,
"id": "75e9fd40-b354-4341-991e-863ef2e59db7",
"id": "75e9fd40-b354-4341-991e-863ef2e59db7",
"metadata": {},
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/markdown": [
"# Anthropic Overview\n",
"\n",
"Anthropic is an AI safety and research company based in San Francisco, focused on developing AI systems that prioritize safety and integrity. The organization emphasizes interdisciplinary collaboration among experts in machine learning, physics, policy, and product development.\n",
"\n",
"## Key Features\n",
"\n",
"- **Claude**: A series of advanced AI models, with the latest being **Claude 3.5 Sonnet**, advertised as the most intelligent version yet.\n",
"- **API**: Allows developers to integrate and build applications using Claude.\n",
" \n",
"## Recent Announcements\n",
"\n",
"- **Claude 3.5 Sonnet Launch** (Jun 21, 2024): Introduction of their latest AI model, enhancing intelligence and capabilities.\n",
"- **Research on AI Safety**:\n",
" - **Constitutional AI: Harmlessness from AI Feedback** (Dec 15, 2022): Focused on ensuring AI models operate safely.\n",
" - **Core Views on AI Safety** (Mar 8, 2023): Discusses the principles and rationale behind AI safety practices.\n",
"\n",
"Anthropic is committed to researching and producing reliable AI systems while fostering a safe and beneficial environment for AI development."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"source": [
"display_summary(\"https://anthropic.com\")"
"display_summary(\"https://anthropic.com\")"
]
]
@ -274,6 +418,185 @@
"\n",
"\n",
"You may notice that if you try `display_summary(\"https://openai.com\")` - it doesn't work! That's because OpenAI has a fancy website that uses Javascript. There are many ways around this that some of you might be familiar with. For example, Selenium is a hugely popular framework that runs a browser behind the scenes, renders the page, and allows you to query it. If you have experience with Selenium, Playwright or similar, then feel free to improve the Website class to use them. Please push your code afterwards so I can share it with other students!"
"You may notice that if you try `display_summary(\"https://openai.com\")` - it doesn't work! That's because OpenAI has a fancy website that uses Javascript. There are many ways around this that some of you might be familiar with. For example, Selenium is a hugely popular framework that runs a browser behind the scenes, renders the page, and allows you to query it. If you have experience with Selenium, Playwright or similar, then feel free to improve the Website class to use them. Please push your code afterwards so I can share it with other students!"
]
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "52ae98bb",
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"# Website Summary\n",
"\n",
"The website appears to be currently inaccessible due to requirements for JavaScript and Cookies to be enabled in the user's browser. Consequently, there are no visible contents or announcements available for summary. Users are prompted to adjust their browser settings to gain access to the website's features."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display_summary(\"https://openai.com\")"
]
},
{
"cell_type": "code",
"execution_count": 62,
"id": "5d57e958",
"metadata": {},
"outputs": [],
"source": [
"#Parse webpages which is designed using JavaScript heavely\n",
"# download the chorme driver from here as per your version of chrome - https://developer.chrome.com/docs/chromedriver/downloads\n",
"OpenAI is focused on creating safe Artificial General Intelligence (AGI) that benefits all of humanity. The website features various AI products and research initiatives, highlighting tools developed for both individual users and businesses.\n",
"\n",
"## Recent Announcements and News\n",
"- **OpenAI DevDay:** A new event aimed at bringing developers together to explore new tools and share ideas.\n",
"- **OpenAI o1:** Introduction of a series of AI models that emphasize thoughtfulness in responses.\n",
"- **Partnership with Apple:** Announcement of a collaboration to integrate ChatGPT into Apple experiences.\n",
"- **ChatGPT Enhancements:** New capabilities that allow ChatGPT to see, hear, and speak.\n",
"- **Structured Outputs in the API:** Introduction of new compliance and administrative tools for ChatGPT Enterprise, alongside improvements for data analysis products.\n",
"\n",
"## Key Features\n",
"- **Product Offerings:** Includes ChatGPT for individuals, teams, and enterprise solutions with improved compliance tools.\n",
"- **SearchGPT Prototype:** A prototype featuring new AI search capabilities designed to enhance the search experience.\n",
"- **Improvements in AI Capabilities:** Ongoing enhancements to large language models for better reasoning and understanding.\n",
"\n",
"Overall, OpenAI is committed to advancing AI technology while ensuring it aligns with safety and ethical standards."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display_summary(\"https://openai.com\")"
]
},
{
"cell_type": "code",
"execution_count": 56,
"id": "f2eb9599",
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"# Summary of Edward Donner's Website\n",
"\n",
"Edward Donner's website serves as a platform for his interests in coding, experimenting with language models (LLMs), and creative pursuits like DJing and music production. He is the co-founder and CTO of Nebula.io, a company focused on utilizing AI to help individuals discover their potential in the talent management sector. His previous venture, untapt, was acquired in 2021. \n",
"\n",
"## Key Features:\n",
"- **Outsmart:** A project that involves a competitive arena for LLMs, emphasizing diplomacy and strategy.\n",
"- **Posts:** The website includes a blog with several recent posts about:\n",
" - The Outsmart LLM Arena (August 6, 2024)\n",
" - Resources for choosing the right LLM (June 26, 2024)\n",
" - Fine-tuning LLMs on personal texts, including a simulation and techniques like QLoRA (February 7, 2024; January 31, 2024).\n",
"\n",
"Donner encourages visitors to connect with him and engage in discussions related to AI and programming."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display_summary(\"https://edwarddonner.com\")"
]
},
{
"cell_type": "code",
"execution_count": 57,
"id": "e7ba56c8",
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"# Summary of CNN Website\n",
"\n",
"The CNN website features a wide array of **breaking news, articles, and videos** from various categories, including **US and world news, politics, business, health, entertainment, sports, and science**. \n",
"\n",
"## Recent Major News Highlights:\n",
"- **Escalation in Israel-Lebanon Conflict**: Israel vows retaliation against Iran following missile attacks, leading to significant military activity and troop casualties in Lebanon.\n",
"- **Vice Presidential Debate Insights**: Analysis of a recent debate where JD Vance and Tim Walz presented their views, revealing no clear winner according to a CNN poll.\n",
"- **Global News Updates**: Coverage includes Thai school bus fire resulting in 23 fatalities, and other significant international events.\n",
"- **Health Advisories**: Recommendations have been made for flu and COVID-19 vaccinations as the season approaches.\n",
"\n",
"## Other Notable Stories:\n",
"- **Upcoming Cultural Events**: Mention of fashion highlights from **Paris Fashion Week** and the resurgence of actors like **Daniel Day-Lewis** in new projects.\n",
"- **Weather and Climate Focus**: Reports on severe weather conditions affecting the US, particularly in the southeast due to recent storms.\n",
"\n",
"Overall, the site encapsulates a mix of urgent news alerts and in-depth analyses, catering to a diverse audience seeking current affairs updates."