From the uDemy course on LLM engineering.
https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
518 lines
16 KiB
518 lines
16 KiB
{ |
|
"cells": [ |
|
{ |
|
"cell_type": "markdown", |
|
"id": "d15d8294-3328-4e07-ad16-8a03e9bbfdb9", |
|
"metadata": {}, |
|
"source": [ |
|
"# Welcome to your first assignment!\n", |
|
"\n", |
|
"Instructions are below. Please give this a try, and look in the solutions folder if you get stuck (or feel free to ask me!)" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "ada885d9-4d42-4d9b-97f0-74fbbbfe93a9", |
|
"metadata": {}, |
|
"source": [ |
|
"<table style=\"margin: 0; text-align: left;\">\n", |
|
" <tr>\n", |
|
" <td style=\"width: 150px; height: 150px; vertical-align: middle;\">\n", |
|
" <img src=\"../resources.jpg\" width=\"150\" height=\"150\" style=\"display: block;\" />\n", |
|
" </td>\n", |
|
" <td>\n", |
|
" <h2 style=\"color:#f71;\">Just before we get to the assignment --</h2>\n", |
|
" <span style=\"color:#f71;\">I thought I'd take a second to point you at this page of useful resources for the course. This includes links to all the slides.<br/>\n", |
|
" <a href=\"https://edwarddonner.com/2024/11/13/llm-engineering-resources/\">https://edwarddonner.com/2024/11/13/llm-engineering-resources/</a><br/>\n", |
|
" Please keep this bookmarked, and I'll continue to add more useful links there over time.\n", |
|
" </span>\n", |
|
" </td>\n", |
|
" </tr>\n", |
|
"</table>" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "6e9fa1fc-eac5-4d1d-9be4-541b3f2b3458", |
|
"metadata": {}, |
|
"source": [ |
|
"# HOMEWORK EXERCISE ASSIGNMENT\n", |
|
"\n", |
|
"Upgrade the day 1 project to summarize a webpage to use an Open Source model running locally via Ollama rather than OpenAI\n", |
|
"\n", |
|
"You'll be able to use this technique for all subsequent projects if you'd prefer not to use paid APIs.\n", |
|
"\n", |
|
"**Benefits:**\n", |
|
"1. No API charges - open-source\n", |
|
"2. Data doesn't leave your box\n", |
|
"\n", |
|
"**Disadvantages:**\n", |
|
"1. Significantly less power than Frontier Model\n", |
|
"\n", |
|
"## Recap on installation of Ollama\n", |
|
"\n", |
|
"Simply visit [ollama.com](https://ollama.com) and install!\n", |
|
"\n", |
|
"Once complete, the ollama server should already be running locally. \n", |
|
"If you visit: \n", |
|
"[http://localhost:11434/](http://localhost:11434/)\n", |
|
"\n", |
|
"You should see the message `Ollama is running`. \n", |
|
"\n", |
|
"If not, bring up a new Terminal (Mac) or Powershell (Windows) and enter `ollama serve` \n", |
|
"And in another Terminal (Mac) or Powershell (Windows), enter `ollama pull llama3.2` \n", |
|
"Then try [http://localhost:11434/](http://localhost:11434/) again.\n", |
|
"\n", |
|
"If Ollama is slow on your machine, try using `llama3.2:1b` as an alternative. Run `ollama pull llama3.2:1b` from a Terminal or Powershell, and change the code below from `MODEL = \"llama3.2\"` to `MODEL = \"llama3.2:1b\"`" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "4e2a9393-7767-488e-a8bf-27c12dca35bd", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# imports\n", |
|
"\n", |
|
"import requests\n", |
|
"from bs4 import BeautifulSoup\n", |
|
"from IPython.display import Markdown, display" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "29ddd15d-a3c5-4f4e-a678-873f56162724", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# Constants\n", |
|
"\n", |
|
"OLLAMA_API = \"http://localhost:11434/api/chat\"\n", |
|
"HEADERS = {\"Content-Type\": \"application/json\"}\n", |
|
"MODEL = \"llama3.2\"" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "dac0a679-599c-441f-9bf2-ddc73d35b940", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# Create a messages list using the same format that we used for OpenAI\n", |
|
"\n", |
|
"messages = [\n", |
|
" {\"role\": \"user\", \"content\": \"Describe some of the business applications of Generative AI\"}\n", |
|
"]" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "7bb9c624-14f0-4945-a719-8ddb64f66f47", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"payload = {\n", |
|
" \"model\": MODEL,\n", |
|
" \"messages\": messages,\n", |
|
" \"stream\": False\n", |
|
" }" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "479ff514-e8bd-4985-a572-2ea28bb4fa40", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# Let's just make sure the model is loaded\n", |
|
"\n", |
|
"!ollama pull llama3.2" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "42b9f644-522d-4e05-a691-56e7658c0ea9", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# If this doesn't work for any reason, try the 2 versions in the following cells\n", |
|
"# And double check the instructions in the 'Recap on installation of Ollama' at the top of this lab\n", |
|
"# And if none of that works - contact me!\n", |
|
"\n", |
|
"response = requests.post(OLLAMA_API, json=payload, headers=HEADERS)\n", |
|
"print(response.json()['message']['content'])" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "6a021f13-d6a1-4b96-8e18-4eae49d876fe", |
|
"metadata": {}, |
|
"source": [ |
|
"# Introducing the ollama package\n", |
|
"\n", |
|
"And now we'll do the same thing, but using the elegant ollama python package instead of a direct HTTP call.\n", |
|
"\n", |
|
"Under the hood, it's making the same call as above to the ollama server running at localhost:11434" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "7745b9c4-57dc-4867-9180-61fa5db55eb8", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"import ollama\n", |
|
"\n", |
|
"response = ollama.chat(model=MODEL, messages=messages)\n", |
|
"print(response['message']['content'])" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "a4704e10-f5fb-4c15-a935-f046c06fb13d", |
|
"metadata": {}, |
|
"source": [ |
|
"## Alternative approach - using OpenAI python library to connect to Ollama" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "9f9e22da-b891-41f6-9ac9-bd0c0a5f4f44", |
|
"metadata": {}, |
|
"source": [ |
|
"## Are you confused about why that works?\n", |
|
"\n", |
|
"It seems strange, right? We just used OpenAI code to call Ollama?? What's going on?!\n", |
|
"\n", |
|
"Here's the scoop:\n", |
|
"\n", |
|
"The python class `OpenAI` is simply code written by OpenAI engineers that makes calls over the internet to an endpoint. \n", |
|
"\n", |
|
"When you call `openai.chat.completions.create()`, this python code just makes a web request to the following url: \"https://api.openai.com/v1/chat/completions\"\n", |
|
"\n", |
|
"Code like this is known as a \"client library\" - it's just wrapper code that runs on your machine to make web requests. The actual power of GPT is running on OpenAI's cloud behind this API, not on your computer!\n", |
|
"\n", |
|
"OpenAI was so popular, that lots of other AI providers provided identical web endpoints, so you could use the same approach.\n", |
|
"\n", |
|
"So Ollama has an endpoint running on your local box at http://localhost:11434/v1/chat/completions \n", |
|
"And in week 2 we'll discover that lots of other providers do this too, including Gemini and DeepSeek.\n", |
|
"\n", |
|
"And then the team at OpenAI had a great idea: they can extend their client library so you can specify a different 'base url', and use their library to call any compatible API.\n", |
|
"\n", |
|
"That's it!\n", |
|
"\n", |
|
"So when you say: `ollama_via_openai = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')` \n", |
|
"Then this will make the same endpoint calls, but to Ollama instead of OpenAI." |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "23057e00-b6fc-4678-93a9-6b31cb704bff", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# There's actually an alternative approach that some people might prefer\n", |
|
"# You can use the OpenAI client python library to call Ollama:\n", |
|
"\n", |
|
"from openai import OpenAI\n", |
|
"ollama_via_openai = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')\n", |
|
"\n", |
|
"response = ollama_via_openai.chat.completions.create(\n", |
|
" model=MODEL,\n", |
|
" messages=messages\n", |
|
")\n", |
|
"\n", |
|
"print(response.choices[0].message.content)" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "bc7d1de3-e2ac-46ff-a302-3b4ba38c4c90", |
|
"metadata": {}, |
|
"source": [ |
|
"## Also trying the amazing reasoning model DeepSeek\n", |
|
"\n", |
|
"Here we use the version of DeepSeek-reasoner that's been distilled to 1.5B. \n", |
|
"This is actually a 1.5B variant of Qwen that has been fine-tuned using synethic data generated by Deepseek R1.\n", |
|
"\n", |
|
"Other sizes of DeepSeek are [here](https://ollama.com/library/deepseek-r1) all the way up to the full 671B parameter version, which would use up 404GB of your drive and is far too large for most!" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "cf9eb44e-fe5b-47aa-b719-0bb63669ab3d", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"!ollama pull deepseek-r1:1.5b" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "1d3d554b-e00d-4c08-9300-45e073950a76", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# This may take a few minutes to run! You should then see a fascinating \"thinking\" trace inside <think> tags, followed by some decent definitions\n", |
|
"\n", |
|
"response = ollama_via_openai.chat.completions.create(\n", |
|
" model=\"deepseek-r1:1.5b\",\n", |
|
" messages=[{\"role\": \"user\", \"content\": \"Please give definitions of some core concepts behind LLMs: a neural network, attention and the transformer\"}]\n", |
|
")\n", |
|
"\n", |
|
"print(response.choices[0].message.content)" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "f160ab15-4206-4515-8751-0e5108ff577c", |
|
"metadata": {}, |
|
"source": [ |
|
"# NOW the exercise for you\n", |
|
"\n", |
|
"Take the code from day1 and incorporate it here, to build a website summarizer that uses Llama 3.2 running locally instead of OpenAI; use either of the above approaches." |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "1622d9bb-5c68-4d4e-9ca4-b492c751f898", |
|
"metadata": {}, |
|
"source": [ |
|
"## Imports for Llama using it's package\n", |
|
"\n", |
|
"We'll use the Llama package (from Ollama) to make requests.\n", |
|
"\n", |
|
"Note: everything has already been imported." |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "9b89249a-4ae4-461d-a547-b7c13f98452e", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"#!/usr/bin/env python\n", |
|
"# coding: utf-8\n", |
|
"\n", |
|
"# import os\n", |
|
"# from dotenv import load_dotenv\n", |
|
"# from bs4 import BeautifulSoup\n", |
|
"# from IPython.display import Markdown, display\n", |
|
"# import ollama\n" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "a88db1c7-aeaf-4284-a3b2-55ad6f7cdbc7", |
|
"metadata": {}, |
|
"source": [ |
|
"## Website class" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "87879ba5-a358-4e07-bac5-118f5b309b1a", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# A class to represent a Webpage\n", |
|
"# If you're not familiar with Classes, check out the \"Intermediate Python\" notebook\n", |
|
"\n", |
|
"# Some websites need you to use proper headers when fetching them:\n", |
|
"headers = {\n", |
|
" \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36\"\n", |
|
"}\n", |
|
"\n", |
|
"class Website:\n", |
|
"\n", |
|
" def __init__(self, url):\n", |
|
" \"\"\"\n", |
|
" Create this Website object from the given url using the BeautifulSoup library\n", |
|
" \"\"\"\n", |
|
" self.url = url\n", |
|
" response = requests.get(url, headers=headers)\n", |
|
" soup = BeautifulSoup(response.content, 'html.parser')\n", |
|
" self.title = soup.title.string if soup.title else \"No title found\"\n", |
|
" for irrelevant in soup.body([\"script\", \"style\", \"img\", \"input\"]):\n", |
|
" irrelevant.decompose()\n", |
|
" self.text = soup.body.get_text(separator=\"\\n\", strip=True)" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "f7234905-4295-4f52-a786-79b742112422", |
|
"metadata": {}, |
|
"source": [ |
|
"## System function" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "85d38054-88f8-4981-accd-b5e3d1e5e73e", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# Define our system prompt - you can experiment with this later, changing the last sentence to 'Respond in markdown in Spanish.\"\n", |
|
"\n", |
|
"system_prompt = \"You are an assistant that analyzes the contents of a website \\\n", |
|
"and provides a short summary, ignoring text that might be navigation related. \\\n", |
|
"Respond in markdown.\"" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "57b40e11-3567-4a25-ac6b-6bbcbc5e47e5", |
|
"metadata": {}, |
|
"source": [ |
|
"## User prompt function" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "05e68137-e711-458e-ad11-c184b80a388b", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# A function that writes a User Prompt that asks for summaries of websites:\n", |
|
"\n", |
|
"user_prompt_content = \"\\nThe contents of this website is as follows; \\\n", |
|
"please provide a short summary of this website in markdown. \\\n", |
|
"If it includes news or announcements, then summarize these too.\\n\\n\"\n", |
|
"\n", |
|
"def user_prompt_for(website):\n", |
|
" user_prompt = f\"You are looking at a website titled {website.title}\"\n", |
|
" user_prompt += \"\\nThe contents of this website is as follows; \\\n", |
|
"please provide a short summary of this website in markdown. \\\n", |
|
"If it includes news or announcements, then summarize these too.\\n\\n\"\n", |
|
" user_prompt += website.text\n", |
|
" return user_prompt" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "d7ba76f3-6ef9-4773-ac96-66c758d85e3d", |
|
"metadata": {}, |
|
"source": [ |
|
"## Messages for website function" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "e2008550-d50d-4ea1-81a9-2a317068cda3", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# See how this function creates exactly the format above\n", |
|
"\n", |
|
"def messages_for(website):\n", |
|
" return [\n", |
|
" {\"role\": \"system\", \"content\": system_prompt},\n", |
|
" {\"role\": \"user\", \"content\": user_prompt_for(website)}\n", |
|
" ]" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "921bef16-e40c-412b-9fcd-4841ff02655e", |
|
"metadata": {}, |
|
"source": [ |
|
"## Summarize function" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "f7f12198-a30e-4533-af5c-c12beeecbbce", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# And now: call the OpenAI API. You will get very familiar with this!\n", |
|
"\n", |
|
"def summarize_ollama(url):\n", |
|
" website = Website(url)\n", |
|
" response = ollama.chat(\n", |
|
" model=MODEL,\n", |
|
" messages = messages_for(website)\n", |
|
" )\n", |
|
" return response['message']['content']" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "91639e0a-e4a3-43bb-9f38-ed7bea91be72", |
|
"metadata": {}, |
|
"source": [ |
|
"## Display Summary function" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "b11a6240-65e8-4453-809d-ee51dde5fc25", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"# A function to display this nicely in the Jupyter output, using markdown\n", |
|
"\n", |
|
"def display_summary_ollama(url):\n", |
|
" summary = summarize_ollama(url)\n", |
|
" display(Markdown(summary))" |
|
] |
|
}, |
|
{ |
|
"cell_type": "markdown", |
|
"id": "874fad51-bd82-48d1-ad6f-a63f3db90c19", |
|
"metadata": {}, |
|
"source": [ |
|
"## Test the function" |
|
] |
|
}, |
|
{ |
|
"cell_type": "code", |
|
"execution_count": null, |
|
"id": "7b45d561-0cd0-4d21-b50b-e61edb932498", |
|
"metadata": {}, |
|
"outputs": [], |
|
"source": [ |
|
"test_url = \"https://edwarddonner.com\"\n", |
|
"display_summary_ollama(test_url)" |
|
] |
|
} |
|
], |
|
"metadata": { |
|
"kernelspec": { |
|
"display_name": "Python 3 (ipykernel)", |
|
"language": "python", |
|
"name": "python3" |
|
}, |
|
"language_info": { |
|
"codemirror_mode": { |
|
"name": "ipython", |
|
"version": 3 |
|
}, |
|
"file_extension": ".py", |
|
"mimetype": "text/x-python", |
|
"name": "python", |
|
"nbconvert_exporter": "python", |
|
"pygments_lexer": "ipython3", |
|
"version": "3.11.12" |
|
} |
|
}, |
|
"nbformat": 4, |
|
"nbformat_minor": 5 |
|
}
|
|
|