You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

477 lines
20 KiB

{
"cells": [
{
"cell_type": "markdown",
"id": "e426cd04-c053-43e8-b505-63cee7956a53",
"metadata": {},
"source": [
"# Welcome to a very busy Week 8 folder\n",
"\n",
"## We have lots to do this week!\n",
"\n",
"We'll move at a faster pace than usual, particularly as you're becoming proficient LLM engineers.\n"
]
},
{
"cell_type": "markdown",
"id": "b3cf5389-93c5-4523-bc48-78fabb91d8f6",
"metadata": {},
"source": [
"<table style=\"margin: 0; text-align: left;\">\n",
" <tr>\n",
" <td style=\"width: 150px; height: 150px; vertical-align: middle;\">\n",
" <img src=\"../important.jpg\" width=\"150\" height=\"150\" style=\"display: block;\" />\n",
" </td>\n",
" <td>\n",
" <h2 style=\"color:#900;\">Especially important this week: pull the latest</h2>\n",
" <span style=\"color:#900;\">I'm continually improving these labs, adding more examples and exercises.\n",
" At the start of each week, it's worth checking you have the latest code.<br/>\n",
" First do a <a href=\"https://chatgpt.com/share/6734e705-3270-8012-a074-421661af6ba9\">git pull and merge your changes as needed</a>. Any problems? Try asking ChatGPT to clarify how to merge - or contact me!<br/><br/>\n",
" After you've pulled the code, from the llm_engineering directory, in an Anaconda prompt (PC) or Terminal (Mac), run:<br/>\n",
" <code>conda env update --f environment.yml --prune</code><br/>\n",
" Or if you used virtualenv rather than Anaconda, then run this from your activated environment in a Powershell (PC) or Terminal (Mac):<br/>\n",
" <code>pip install -r requirements.txt</code>\n",
" <br/>Then restart the kernel (Kernel menu >> Restart Kernel and Clear Outputs Of All Cells) to pick up the changes.\n",
" </span>\n",
" </td>\n",
" </tr>\n",
"</table>"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "bc0e1c1c-be6a-4395-bbbd-eeafc9330d7e",
"metadata": {},
"outputs": [],
"source": [
"# Just one import to start with!!\n",
"\n",
"import modal"
]
},
{
"cell_type": "markdown",
"id": "ab5c8533-9f66-448f-b9b2-133d1ff50639",
"metadata": {},
"source": [
"# Setting up the modal tokens\n",
"\n",
"The first time you run this, please uncomment the next line and execute it. \n",
"This is the same as running `modal setup` from the command line. It connects with Modal and installs your tokens.\n",
"\n",
"A student on Windows mentioned that on Windows, you might also need to run this command from a command prompt afterwards: \n",
"`modal token new` \n",
"(Thank you Ed B. for that!)\n",
"\n",
"And I've also heard that in some situations, you might need to restart the Kernel of this jupyter notebook after running this. (Kernel menu >> Restart Kernel and Clear Outputs of All Cells)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "0d240622-8422-4c99-8464-c04d063e4cb6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[2KThe web browser should have opened for you to authenticate and get an API token.\n",
"If it didn't, please copy this URL into your web browser manually:\n",
"\n",
"\u001b[2K\u001b]8;id=988175;https://modal.com/token-flow/tf-4OQWCx9ehoVgotEYIiHbIY\u001b\\\u001b[4;94mhttps://modal.com/token-flow/tf-4OQWCx9ehoVgotEYIiHbIY\u001b[0m\u001b]8;;\u001b\\\n",
"\n",
"\u001b[2K\u001b[32m⠙\u001b[0m Waiting for authentication in the web browser\n",
"\u001b[2K\u001b[32m⠧\u001b[0m Waiting for token flow to complete...omplete...\n",
"\u001b[1A\u001b[2K\u001b[32mWeb authentication finished successfully!\u001b[0m\n",
"\u001b[32mToken is connected to the \u001b[0m\u001b[35mxykong36\u001b[0m\u001b[32m workspace.\u001b[0m\n",
"Verifying token against \u001b[4;34mhttps://api.modal.com\u001b[0m\n",
"\u001b[32mToken verified successfully!\u001b[0m\n",
"\u001b[?25l\u001b[32m⠋\u001b[0m Storing token\n",
"\u001b[1A\u001b[2K\u001b[32mToken written to \u001b[0m\u001b[35m/Users/xiangyu/\u001b[0m\u001b[35m.modal.toml\u001b[0m\u001b[32m in profile \u001b[0m\u001b[35mxykong36\u001b[0m\u001b[32m.\u001b[0m\n"
]
}
],
"source": [
"# Remove the '# ' from the next line and run the cell\n",
"!modal setup"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "3b133701-f550-44a1-a67f-eb7ccc4769a9",
"metadata": {},
"outputs": [],
"source": [
"from hello import app, hello"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "7bf9cecb-f837-458d-afdf-1b925c89b4eb",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[2KThe web browser should have opened for you to authenticate and get an API token.\n",
"If it didn't, please copy this URL into your web browser manually:\n",
"\n",
"\u001b[2K\u001b]8;id=706003;https://modal.com/token-flow/tf-lfGsLFXreoimWW1H3W4E4A\u001b\\\u001b[4;94mhttps://modal.com/token-flow/tf-lfGsLFXreoimWW1H3W4E4A\u001b[0m\u001b]8;;\u001b\\\n",
"\n",
"\u001b[2K\u001b[32m⠋\u001b[0m Waiting for authentication in the web browser\n",
"\u001b[2K\u001b[32m⠙\u001b[0m Waiting for token flow to complete...omplete...\n",
"\u001b[1A\u001b[2K\u001b[32mWeb authentication finished successfully!\u001b[0m\n",
"\u001b[32mToken is connected to the \u001b[0m\u001b[35mxykong36\u001b[0m\u001b[32m workspace.\u001b[0m\n",
"Verifying token against \u001b[4;34mhttps://api.modal.com\u001b[0m\n",
"\u001b[32mToken verified successfully!\u001b[0m\n",
"\u001b[?25l\u001b[32m⠋\u001b[0m Storing token\n",
"\u001b[1A\u001b[2K\u001b[32mToken written to \u001b[0m\u001b[35m/Users/xiangyu/\u001b[0m\u001b[35m.modal.toml\u001b[0m\u001b[32m in profile \u001b[0m\u001b[35mxykong36\u001b[0m\u001b[32m.\u001b[0m\n"
]
}
],
"source": [
"!modal token new"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "8cff4180-24b9-4eba-8198-8c84f7359e72",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ[\"MODAL_TOKEN_ID\"] = \"ak-KaDVhGcPs78noFTPmU0Ef6\"\n",
"os.environ[\"MODAL_TOKEN_SECRET\"] = \"as-xy7sk1qvTN00oBtIJC6zto\""
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "0f3f73ae-1295-49f3-9099-b8b41fc3429b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Hello from Charlotte, North Carolina, US!!'"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with app.run():\n",
" reply=hello.local()\n",
"reply"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "c1d8c6f9-edc7-4e52-9b3a-c07d7cff1ac7",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/opt/anaconda3/envs/llms/lib/python3.11/contextlib.py:210: DeprecationError: 2024-11-20: `show_progress=False` is deprecated (and has no effect)\n",
" return await anext(self.gen)\n"
]
},
{
"ename": "ExecutionError",
"evalue": "Function has not been hydrated with the metadata it needs to run on Modal, because the App it is defined on is not running.",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mExecutionError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[20], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m app\u001b[38;5;241m.\u001b[39mrun(show_progress\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m):\n\u001b[0;32m----> 2\u001b[0m reply\u001b[38;5;241m=\u001b[39m\u001b[43mhello\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mremote\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 3\u001b[0m reply\n",
"File \u001b[0;32m/opt/anaconda3/envs/llms/lib/python3.11/site-packages/synchronicity/synchronizer.py:593\u001b[0m, in \u001b[0;36mSynchronizer._wrap_proxy_method.<locals>.proxy_method\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 591\u001b[0m instance \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__dict__\u001b[39m[synchronizer_self\u001b[38;5;241m.\u001b[39m_original_attr]\n\u001b[1;32m 592\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 593\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mwrapped_method\u001b[49m\u001b[43m(\u001b[49m\u001b[43minstance\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 594\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m UserCodeException \u001b[38;5;28;01mas\u001b[39;00m uc_exc:\n\u001b[1;32m 595\u001b[0m uc_exc\u001b[38;5;241m.\u001b[39mexc\u001b[38;5;241m.\u001b[39m__suppress_context__ \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n",
"File \u001b[0;32m/opt/anaconda3/envs/llms/lib/python3.11/site-packages/synchronicity/combined_types.py:29\u001b[0m, in \u001b[0;36mFunctionWithAio.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 27\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m UserCodeException \u001b[38;5;28;01mas\u001b[39;00m uc_exc:\n\u001b[1;32m 28\u001b[0m uc_exc\u001b[38;5;241m.\u001b[39mexc\u001b[38;5;241m.\u001b[39m__suppress_context__ \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[0;32m---> 29\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m uc_exc\u001b[38;5;241m.\u001b[39mexc\n",
"File \u001b[0;32m/opt/anaconda3/envs/llms/lib/python3.11/site-packages/modal/object.py:251\u001b[0m, in \u001b[0;36mlive_method.<locals>.wrapped\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 249\u001b[0m \u001b[38;5;129m@wraps\u001b[39m(method)\n\u001b[1;32m 250\u001b[0m \u001b[38;5;28;01masync\u001b[39;00m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mwrapped\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m--> 251\u001b[0m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mresolve()\n\u001b[1;32m 252\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m method(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
"File \u001b[0;32m/opt/anaconda3/envs/llms/lib/python3.11/site-packages/modal/object.py:237\u001b[0m, in \u001b[0;36m_Object.resolve\u001b[0;34m(self, client)\u001b[0m\n\u001b[1;32m 235\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m\n\u001b[1;32m 236\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_hydrate_lazily:\n\u001b[0;32m--> 237\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_validate_is_hydrated\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 238\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 239\u001b[0m \u001b[38;5;66;03m# TODO: this client and/or resolver can't be changed by a caller to X.from_name()\u001b[39;00m\n\u001b[1;32m 240\u001b[0m c \u001b[38;5;241m=\u001b[39m client \u001b[38;5;28;01mif\u001b[39;00m client \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m _Client\u001b[38;5;241m.\u001b[39mfrom_env()\n",
"File \u001b[0;32m/opt/anaconda3/envs/llms/lib/python3.11/site-packages/modal/object.py:132\u001b[0m, in \u001b[0;36m_Object._validate_is_hydrated\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 129\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 130\u001b[0m \u001b[38;5;66;03m# Technically possible, but with an ambiguous cause.\u001b[39;00m\n\u001b[1;32m 131\u001b[0m reason \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m--> 132\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m ExecutionError(\n\u001b[1;32m 133\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mobject_type\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m has not been hydrated with the metadata it needs to run on Modal\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mreason\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 134\u001b[0m )\n",
"\u001b[0;31mExecutionError\u001b[0m: Function has not been hydrated with the metadata it needs to run on Modal, because the App it is defined on is not running."
]
}
],
"source": [
"with app.run(show_progress=False):\n",
" reply=hello.remote()\n",
"reply"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "45e59cca-a279-437c-ad29-718d61befde1",
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"\n",
"import modal\n",
"\n",
"app = modal.App(\"example-hello-world\")\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "b2089850-5eb2-4be2-8a1f-398d8cea71c9",
"metadata": {},
"outputs": [],
"source": [
"@app.function()\n",
"def f(i):\n",
" if i % 2 == 0:\n",
" print(\"hello\", i)\n",
" else:\n",
" print(\"world\", i, file=sys.stderr)\n",
"\n",
" return i * i"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "ec242385-7c8a-4732-ba45-487aeaa1571b",
"metadata": {},
"outputs": [],
"source": [
"@app.local_entrypoint()\n",
"def main():\n",
" # run the function locally\n",
" print(f.local(1000))\n",
"\n",
" # run the function remotely on Modal\n",
" print(f.remote(1000))\n",
"\n",
" # run the function in parallel and remotely on Modal\n",
" total = 0\n",
" for ret in f.map(range(200)):\n",
" total += ret\n",
"\n",
" print(total)"
]
},
{
"cell_type": "markdown",
"id": "22e8d804-c027-45fb-8fef-06e7bba6295a",
"metadata": {},
"source": [
"# Before we move on -\n",
"\n",
"## We need to set your HuggingFace Token as a secret in Modal\n",
"\n",
"1. Go to modal.com, sign in and go to your dashboard\n",
"2. Click on Secrets in the nav bar\n",
"3. Create new secret, click on Hugging Face, this new secret needs to be called **hf-secret** because that's how we refer to it in the code\n",
"4. Fill in your HF_TOKEN where it prompts you\n",
"\n",
"### And now back to business: time to work with Llama"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cb8b6c41-8259-4329-b1c4-a1f67d26d1be",
"metadata": {},
"outputs": [],
"source": [
"from llama import app, generate"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "db4a718a-d95d-4f61-9688-c9df21d88fe6",
"metadata": {},
"outputs": [],
"source": [
"with modal.enable_output():\n",
" with app.run():\n",
" result=generate.remote(\"Life is a mystery, everyone must stand alone, I hear\")\n",
"result"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9a9a6844-29ec-4264-8e72-362d976b3968",
"metadata": {},
"outputs": [],
"source": [
"import modal\n",
"from pricer_ephemeral import app, price"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "50e6cf99-8959-4ae3-ba02-e325cb7fff94",
"metadata": {},
"outputs": [],
"source": [
"with modal.enable_output():\n",
" with app.run():\n",
" result=price.remote(\"Quadcast HyperX condenser mic, connects via usb-c to your computer for crystal clear audio\")\n",
"result"
]
},
{
"cell_type": "markdown",
"id": "04d8747f-8452-4077-8af6-27e03888508a",
"metadata": {},
"source": [
"## Transitioning From Ephemeral Apps to Deployed Apps\n",
"\n",
"From a command line, `modal deploy xxx` will deploy your code as a Deployed App\n",
"\n",
"This is how you could package your AI service behind an API to be used in a Production System.\n",
"\n",
"You can also build REST endpoints easily, although we won't cover that as we'll be calling direct from Python."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7f90d857-2f12-4521-bb90-28efd917f7d1",
"metadata": {},
"outputs": [],
"source": [
"!modal deploy pricer_service"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1dec70ff-1986-4405-8624-9bbbe0ce1f4a",
"metadata": {},
"outputs": [],
"source": [
"pricer = modal.Function.lookup(\"pricer-service\", \"price\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "17776139-0d9e-4ad0-bcd0-82d3a92ca61f",
"metadata": {},
"outputs": [],
"source": [
"pricer.remote(\"Quadcast HyperX condenser mic, connects via usb-c to your computer for crystal clear audio\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f56d1e55-2a03-4ce2-bb47-2ab6b9175a02",
"metadata": {},
"outputs": [],
"source": [
"!modal deploy pricer_service2"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9e19daeb-1281-484b-9d2f-95cc6fed2622",
"metadata": {},
"outputs": [],
"source": [
"Pricer = modal.Cls.lookup(\"pricer-service\", \"Pricer\")\n",
"pricer = Pricer()\n",
"reply = pricer.price.remote(\"Quadcast HyperX condenser mic, connects via usb-c to your computer for crystal clear audio\")\n",
"print(reply)"
]
},
{
"cell_type": "markdown",
"id": "9c1b1451-6249-4462-bf2d-5937c059926c",
"metadata": {},
"source": [
"# Optional: Keeping Modal warm\n",
"\n",
"## A way to improve the speed of the Modal pricer service\n",
"\n",
"A student mentioned to me that he was concerned by how slow Modal seems to be. The reason is that Modal puts our service to sleep if we don't use it, and then it takes 2.5 minutes to spin back up.\n",
"\n",
"I've added a utility called `keep_warm.py` that will keep our Modal warm by pinging it every 30 seconds.\n",
"\n",
"To use the utliity, bring up a new Terminal (Mac) or Anaconda prompt (Windows), ensure the environment is activated with `conda activate llms`\n",
"\n",
"Then run: `python keep_warm.py` from within the week8 drectory.\n",
"\n",
"Remember to press ctrl+C or exit the window when you no longer need Modal running.\n"
]
},
{
"cell_type": "markdown",
"id": "3754cfdd-ae28-47c8-91f2-6e060e2c91b3",
"metadata": {},
"source": [
"## And now introducing our Agent class"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ba9aedca-6a7b-4d30-9f64-59d76f76fb6d",
"metadata": {},
"outputs": [],
"source": [
"from agents.specialist_agent import SpecialistAgent"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fe5843e5-e958-4a65-8326-8f5b4686de7f",
"metadata": {},
"outputs": [],
"source": [
"agent = SpecialistAgent()\n",
"agent.price(\"iPad Pro 2nd generation\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f5a3181b-1310-4102-8d7d-52caf4c00538",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}