{ "cells": [ { "cell_type": "markdown", "id": "e426cd04-c053-43e8-b505-63cee7956a53", "metadata": {}, "source": [ "# Welcome to a very busy Week 8 folder\n", "\n", "## We have lots to do this week!\n", "\n", "We'll move at a faster pace than usual, particularly as you're becoming proficient LLM engineers.\n" ] }, { "cell_type": "markdown", "id": "b3cf5389-93c5-4523-bc48-78fabb91d8f6", "metadata": {}, "source": [ "
\n",
" ![]() | \n",
" \n",
" Especially important this week: pull the latest\n", " I'm continually improving these labs, adding more examples and exercises.\n", " At the start of each week, it's worth checking you have the latest code.\n", " First do a git pull and merge your changes as needed. Any problems? Try asking ChatGPT to clarify how to merge - or contact me! \n", " After you've pulled the code, from the llm_engineering directory, in an Anaconda prompt (PC) or Terminal (Mac), run: \n", " conda env update --f environment.yml --prune \n", " Or if you used virtualenv rather than Anaconda, then run this from your activated environment in a Powershell (PC) or Terminal (Mac): \n", " pip install -r requirements.txt \n",
" Then restart the kernel (Kernel menu >> Restart Kernel and Clear Outputs Of All Cells) to pick up the changes.\n", " \n", " | \n",
"
✓ Initialized. View run at https://modal.com/apps/craigprobus/main/ap-a1zKRQGpj8VmuyIJtBeWSk\n", "\n" ], "text/plain": [ "\u001b[32m✓\u001b[0m Initialized. \u001b[38;5;249mView run at \u001b[0m\u001b[4;38;5;249mhttps://modal.com/apps/craigprobus/main/ap-a1zKRQGpj8VmuyIJtBeWSk\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n" ], "text/plain": [] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2c42ac514b904938a52e692f77ab6042", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Output()" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n" ], "text/plain": [] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
✓ Created objects.\n", "├── 🔨 Created mount /Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/llama.py\n", "├── 🔨 Created mount PythonPackage:hello\n", "└── 🔨 Created function generate.\n", "\n" ], "text/plain": [ "\u001b[32m✓\u001b[0m Created objects.\n", "\u001b[38;5;244m├── \u001b[0m🔨 Created mount /Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/llama.py\n", "\u001b[38;5;244m├── \u001b[0m🔨 Created mount PythonPackage:hello\n", "\u001b[38;5;244m└── \u001b[0m🔨 Created function generate.\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ece6417840304983acb4c6a6ec9abfdd", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Output()" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n",
"
\n"
],
"text/plain": [
"\u001b[31m\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1AFetching 4 files: 0%| | 0/4 [00:00<?, ?it/s]
\n"
],
"text/plain": [
"\u001b[31m\u001b[1AFetching 4 files: 0%| | 0/4 [00:00, ?it/s]\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1AFetching 4 files: 25%|██▌ | 1/4 [01:12<03:38, 72.90s/it]Fetching 4 files: 100%|██████████| 4/4 [01:12<00:00, 18.23s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1AFetching 4 files: 25%|██▌ | 1/4 [01:12<03:38, 72.90s/it]Fetching 4 files: 100%|██████████| 4/4 [01:12<00:00, 18.23s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"
\n"
],
"text/plain": [
"\u001b[31m\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 0%| | 0/4 [00:00, ?it/s]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 25%|██▌ | 1/4 [00:04<00:12, 4.10s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 25%|██▌ | 1/4 [00:04<00:12, 4.10s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 50%|█████ | 2/4 [00:08<00:08, 4.46s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 50%|█████ | 2/4 [00:08<00:08, 4.46s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 75%|███████▌ | 3/4 [00:13<00:04, 4.53s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 75%|███████▌ | 3/4 [00:13<00:04, 4.53s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 2.86s/it]Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 3.43s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 2.86s/it]Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 3.43s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
"
\n"
],
"text/plain": [
"\u001b[31mSetting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Stopping app - local entrypoint completed.\n",
"
\n"
],
"text/plain": [
"\u001b[33mStopping app - local entrypoint completed.\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"✓ App completed. View run at https://modal.com/apps/craigprobus/main/ap-a1zKRQGpj8VmuyIJtBeWSk\n", "\n" ], "text/plain": [ "\u001b[32m✓\u001b[0m App completed. \u001b[38;5;249mView run at \u001b[0m\u001b[4;38;5;249mhttps://modal.com/apps/craigprobus/main/ap-a1zKRQGpj8VmuyIJtBeWSk\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "'<|begin_of_text|>Life is a mystery, everyone must stand alone, I hear you call my name,'" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "with modal.enable_output():\n", " with app.run():\n", " result=generate.remote(\"Life is a mystery, everyone must stand alone, I hear\")\n", "result" ] }, { "cell_type": "code", "execution_count": 53, "id": "9a9a6844-29ec-4264-8e72-362d976b3968", "metadata": {}, "outputs": [], "source": [ "import modal\n", "from pricer_ephemeral import app, price" ] }, { "cell_type": "code", "execution_count": 54, "id": "50e6cf99-8959-4ae3-ba02-e325cb7fff94", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e951ef75eef245c8bf36f04fc0d10083", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Output()" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
✓ Initialized. View run at https://modal.com/apps/craigprobus/main/ap-jj4lCu9XtEr4VTT14wMcpK\n", "\n" ], "text/plain": [ "\u001b[32m✓\u001b[0m Initialized. \u001b[38;5;249mView run at \u001b[0m\u001b[4;38;5;249mhttps://modal.com/apps/craigprobus/main/ap-jj4lCu9XtEr4VTT14wMcpK\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n" ], "text/plain": [] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "0aa9f1e2d2f849f997b4edd25555efaa", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Output()" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n" ], "text/plain": [] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
✓ Created objects.\n", "├── 🔨 Created mount /Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_ephemeral.py\n", "├── 🔨 Created mount PythonPackage:hello\n", "├── 🔨 Created mount PythonPackage:llama\n", "└── 🔨 Created function price.\n", "\n" ], "text/plain": [ "\u001b[32m✓\u001b[0m Created objects.\n", "\u001b[38;5;244m├── \u001b[0m🔨 Created mount /Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_ephemeral.py\n", "\u001b[38;5;244m├── \u001b[0m🔨 Created mount PythonPackage:hello\n", "\u001b[38;5;244m├── \u001b[0m🔨 Created mount PythonPackage:llama\n", "\u001b[38;5;244m└── \u001b[0m🔨 Created function price.\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "7b95c170bb954e48b45db743437e9d91", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Output()" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n",
"
\n"
],
"text/plain": [
"\u001b[31m\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1AFetching 4 files: 0%| | 0/4 [00:00<?, ?it/s]
\n"
],
"text/plain": [
"\u001b[31m\u001b[1AFetching 4 files: 0%| | 0/4 [00:00, ?it/s]\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"
\n"
],
"text/plain": [
"\u001b[31m\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1AFetching 4 files: 25%|██▌ | 1/4 [01:03<03:09, 63.06s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1AFetching 4 files: 25%|██▌ | 1/4 [01:03<03:09, 63.06s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1AFetching 4 files: 50%|█████ | 2/4 [01:05<00:54, 27.26s/it]Fetching 4 files: 100%|██████████| 4/4 [01:05<00:00, 16.31s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1AFetching 4 files: 50%|█████ | 2/4 [01:05<00:54, 27.26s/it]Fetching 4 files: 100%|██████████| 4/4 [01:05<00:00, 16.31s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"
\n"
],
"text/plain": [
"\u001b[31m\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 0%| | 0/4 [00:00, ?it/s]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 25%|██▌ | 1/4 [00:04<00:13, 4.64s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 25%|██▌ | 1/4 [00:04<00:13, 4.64s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 50%|█████ | 2/4 [00:09<00:09, 4.58s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 50%|█████ | 2/4 [00:09<00:09, 4.58s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 75%|███████▌ | 3/4 [00:13<00:04, 4.34s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 75%|███████▌ | 3/4 [00:13<00:04, 4.34s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\u001b[1ALoading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 2.77s/it]Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 3.40s/it]\n",
"
\n"
],
"text/plain": [
"\u001b[31m\u001b[1ALoading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 2.77s/it]Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 3.40s/it]\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
"
\n"
],
"text/plain": [
"\u001b[31mSetting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n"
],
"text/plain": []
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Stopping app - local entrypoint completed.\n",
"
\n"
],
"text/plain": [
"\u001b[33mStopping app - local entrypoint completed.\n",
"\u001b[0m"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"✓ App completed. View run at https://modal.com/apps/craigprobus/main/ap-jj4lCu9XtEr4VTT14wMcpK\n", "\n" ], "text/plain": [ "\u001b[32m✓\u001b[0m App completed. \u001b[38;5;249mView run at \u001b[0m\u001b[4;38;5;249mhttps://modal.com/apps/craigprobus/main/ap-jj4lCu9XtEr4VTT14wMcpK\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "220.0" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "with modal.enable_output():\n", " with app.run():\n", " result=price.remote(\"Quadcast HyperX condenser mic, connects via usb-c to your computer for crystal clear audio\")\n", "result" ] }, { "cell_type": "markdown", "id": "04d8747f-8452-4077-8af6-27e03888508a", "metadata": {}, "source": [ "## Transitioning From Ephemeral Apps to Deployed Apps\n", "\n", "From a command line, `modal deploy xxx` will deploy your code as a Deployed App\n", "\n", "This is how you could package your AI service behind an API to be used in a Production System.\n", "\n", "You can also build REST endpoints easily, although we won't cover that as we'll be calling direct from Python.\n", "\n", "## Important note for Windows people:\n", "\n", "On the next line, I call `modal deploy` from within Jupyter lab; I've heard that on some versions of Windows this gives a strange unicode error because modal prints emojis to the output which can't be displayed. If that happens to you, simply use an Anaconda Prompt window or a Powershell instead, with your environment activated, and type `modal deploy pricer_service` there. Follow the same approach the next time we do `!modal deploy` too.\n", "\n", "As an alternative, a few students have mentioned you can run this code within Jupyter Lab if you want to run it from here:\n", "```\n", "# Check the default encoding\n", "print(locale.getpreferredencoding()) # Should print 'UTF-8'\n", "\n", "# Ensure UTF-8 encoding\n", "os.environ['PYTHONIOENCODING'] = 'utf-8'\n", "```" ] }, { "cell_type": "code", "execution_count": 55, "id": "7f90d857-2f12-4521-bb90-28efd917f7d1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[2K\u001b[34m⠸\u001b[0m Creating objects.....\n", "\u001b[37m└── \u001b[0m\u001b[34m⠋\u001b[0m Creating mount \n", "\u001b[37m \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[34m⠦\u001b[0m Creating objects...\n", "\u001b[37m├── \u001b[0m🔨 Created mount \n", "\u001b[37m│ \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[37m│ \u001b[0me.py\n", "\u001b[37m└── \u001b[0m🔨 Created function price.\n", "\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[32m✓\u001b[0m Created objects.\n", "\u001b[37m├── \u001b[0m🔨 Created mount \n", "\u001b[37m│ \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[37m│ \u001b[0me.py\n", "\u001b[37m└── \u001b[0m🔨 Created function price.\n", "\u001b[32m✓\u001b[0m App deployed in 0.902s! 🎉\n", "\n", "View Deployment: \u001b[35mhttps://modal.com/apps/craigprobus/main/deployed/pricer-service\u001b[0m\n" ] } ], "source": [ "!modal deploy -m pricer_service" ] }, { "cell_type": "code", "execution_count": 56, "id": "1dec70ff-1986-4405-8624-9bbbe0ce1f4a", "metadata": {}, "outputs": [], "source": [ "pricer = modal.Function.from_name(\"pricer-service\", \"price\")" ] }, { "cell_type": "code", "execution_count": 57, "id": "17776139-0d9e-4ad0-bcd0-82d3a92ca61f", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "220.0" ] }, "execution_count": 57, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pricer.remote(\"Quadcast HyperX condenser mic, connects via usb-c to your computer for crystal clear audio\")" ] }, { "cell_type": "code", "execution_count": 66, "id": "d5a2d7a8-a88d-4b50-a21f-4d115f4eb2e6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[31m╭─\u001b[0m\u001b[31m Error: Already exists \u001b[0m\u001b[31m─────────────────────────────────────────────────────\u001b[0m\u001b[31m─╮\u001b[0m\n", "\u001b[31m│\u001b[0m Can't overwrite existing volume \u001b[31m│\u001b[0m\n", "\u001b[31m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n" ] } ], "source": [ "!modal volume create hf-hub-cache" ] }, { "cell_type": "code", "execution_count": 69, "id": "f56d1e55-2a03-4ce2-bb47-2ab6b9175a02", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[2K\u001b[34m⠸\u001b[0m Creating objects.....\n", "\u001b[37m└── \u001b[0m\u001b[34m⠋\u001b[0m Creating mount \n", "\u001b[37m \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[34m⠦\u001b[0m Creating objects...\n", "\u001b[37m└── \u001b[0m\u001b[34m⠸\u001b[0m Creating mount \n", "\u001b[37m \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[34m⠏\u001b[0m Creating objects...\n", "\u001b[37m└── \u001b[0m\u001b[34m⠦\u001b[0m Creating mount \n", "\u001b[37m \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[34m⠹\u001b[0m Creating objects...\n", "\u001b[37m└── \u001b[0m\u001b[34m⠏\u001b[0m Creating mount \n", "\u001b[37m \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[34m⠦\u001b[0m Creating objects...\n", "\u001b[37m└── \u001b[0m\u001b[34m⠹\u001b[0m Creating mount \n", "\u001b[37m \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[34m⠏\u001b[0m Creating objects...\n", "\u001b[37m├── \u001b[0m🔨 Created mount \n", "\u001b[37m│ \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[37m│ \u001b[0me2.py\n", "\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[34m⠏\u001b[0m Creating objects...\n", "\u001b[37m├── \u001b[0m🔨 Created mount \n", "\u001b[37m│ \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[37m│ \u001b[0me2.py\n", "\u001b[37m└── \u001b[0m🔨 Created function Pricer.*.\n", "\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[32m✓\u001b[0m Created objects.\n", "\u001b[37m├── \u001b[0m🔨 Created mount \n", "\u001b[37m│ \u001b[0m/Users/craigprobus/Repos/ed-donner-class/llm_engineering/week8/pricer_servic\n", "\u001b[37m│ \u001b[0me2.py\n", "\u001b[37m└── \u001b[0m🔨 Created function Pricer.*.\n", "\u001b[32m✓\u001b[0m App deployed in 1.981s! 🎉\n", "\n", "View Deployment: \u001b[35mhttps://modal.com/apps/craigprobus/main/deployed/pricer-service\u001b[0m\n" ] } ], "source": [ "# You can also run \"modal deploy -m pricer_service2\" at the command line in an activated environment\n", "!modal deploy -m pricer_service2" ] }, { "cell_type": "code", "execution_count": 70, "id": "9e19daeb-1281-484b-9d2f-95cc6fed2622", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "220.0\n" ] } ], "source": [ "Pricer = modal.Cls.from_name(\"pricer-service\", \"Pricer\")\n", "pricer = Pricer()\n", "reply = pricer.price.remote(\"Quadcast HyperX condenser mic, connects via usb-c to your computer for crystal clear audio\")\n", "print(reply)" ] }, { "cell_type": "code", "execution_count": 71, "id": "c29b8c58-4cb7-44b0-ab7e-6469d3a318e8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: modal in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (0.73.155)\n", "Collecting modal\n", " Downloading modal-0.73.165-py3-none-any.whl.metadata (2.5 kB)\n", "Requirement already satisfied: aiohttp in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (3.11.12)\n", "Requirement already satisfied: certifi in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (2025.1.31)\n", "Requirement already satisfied: click>=8.1.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (8.1.8)\n", "Requirement already satisfied: fastapi in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (0.115.8)\n", "Requirement already satisfied: grpclib==0.4.7 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (0.4.7)\n", "Requirement already satisfied: protobuf!=4.24.0,<6.0,>=3.19 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (5.29.3)\n", "Requirement already satisfied: rich>=12.0.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (13.9.4)\n", "Requirement already satisfied: synchronicity~=0.9.10 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (0.9.11)\n", "Requirement already satisfied: toml in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (0.10.2)\n", "Requirement already satisfied: typer>=0.9 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (0.15.1)\n", "Requirement already satisfied: types-certifi in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (2021.10.8.3)\n", "Requirement already satisfied: types-toml in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (0.10.8.20240310)\n", "Requirement already satisfied: watchfiles in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (1.0.4)\n", "Requirement already satisfied: typing_extensions~=4.6 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal) (4.12.2)\n", "Requirement already satisfied: h2<5,>=3.1.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from grpclib==0.4.7->modal) (4.2.0)\n", "Requirement already satisfied: multidict in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from grpclib==0.4.7->modal) (6.1.0)\n", "Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from rich>=12.0.0->modal) (3.0.0)\n", "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from rich>=12.0.0->modal) (2.19.1)\n", "Requirement already satisfied: sigtools>=4.0.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from synchronicity~=0.9.10->modal) (4.0.1)\n", "Requirement already satisfied: shellingham>=1.3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from typer>=0.9->modal) (1.5.4)\n", "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->modal) (2.4.6)\n", "Requirement already satisfied: aiosignal>=1.1.2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->modal) (1.3.2)\n", "Requirement already satisfied: attrs>=17.3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->modal) (25.1.0)\n", "Requirement already satisfied: frozenlist>=1.1.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->modal) (1.5.0)\n", "Requirement already satisfied: propcache>=0.2.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->modal) (0.2.1)\n", "Requirement already satisfied: yarl<2.0,>=1.17.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->modal) (1.18.3)\n", "Requirement already satisfied: starlette<0.46.0,>=0.40.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from fastapi->modal) (0.45.3)\n", "Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from fastapi->modal) (2.10.6)\n", "Requirement already satisfied: anyio>=3.0.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from watchfiles->modal) (4.8.0)\n", "Requirement already satisfied: idna>=2.8 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from anyio>=3.0.0->watchfiles->modal) (3.10)\n", "Requirement already satisfied: sniffio>=1.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from anyio>=3.0.0->watchfiles->modal) (1.3.1)\n", "Requirement already satisfied: hyperframe<7,>=6.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from h2<5,>=3.1.0->grpclib==0.4.7->modal) (6.1.0)\n", "Requirement already satisfied: hpack<5,>=4.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from h2<5,>=3.1.0->grpclib==0.4.7->modal) (4.1.0)\n", "Requirement already satisfied: mdurl~=0.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=12.0.0->modal) (0.1.2)\n", "Requirement already satisfied: annotated-types>=0.6.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi->modal) (0.7.0)\n", "Requirement already satisfied: pydantic-core==2.27.2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi->modal) (2.27.2)\n", "Downloading modal-0.73.165-py3-none-any.whl (558 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m559.0/559.0 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hInstalling collected packages: modal\n", " Attempting uninstall: modal\n", " Found existing installation: modal 0.73.155\n", " Uninstalling modal-0.73.155:\n", " Successfully uninstalled modal-0.73.155\n", "Successfully installed modal-0.73.165\n" ] } ], "source": [ "!pip install --upgrade modal" ] }, { "cell_type": "markdown", "id": "9c1b1451-6249-4462-bf2d-5937c059926c", "metadata": {}, "source": [ "# Optional: Keeping Modal warm\n", "\n", "## A way to improve the speed of the Modal pricer service\n", "\n", "A student mentioned to me that he was concerned by how slow Modal seems to be. The reason is that Modal puts our service to sleep if we don't use it, and then it takes 2.5 minutes to spin back up.\n", "\n", "I've added a utility called `keep_warm.py` that will keep our Modal warm by pinging it every 30 seconds.\n", "\n", "To use the utliity, bring up a new Terminal (Mac) or Anaconda prompt (Windows), ensure the environment is activated with `conda activate llms`\n", "\n", "Then run: `python keep_warm.py` from within the week8 drectory.\n", "\n", "Remember to press ctrl+C or exit the window when you no longer need Modal running.\n" ] }, { "cell_type": "markdown", "id": "3754cfdd-ae28-47c8-91f2-6e060e2c91b3", "metadata": {}, "source": [ "## And now introducing our Agent class" ] }, { "cell_type": "code", "execution_count": 72, "id": "ba9aedca-6a7b-4d30-9f64-59d76f76fb6d", "metadata": {}, "outputs": [], "source": [ "from agents.specialist_agent import SpecialistAgent" ] }, { "cell_type": "code", "execution_count": 74, "id": "fe5843e5-e958-4a65-8326-8f5b4686de7f", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "350.0" ] }, "execution_count": 74, "metadata": {}, "output_type": "execute_result" } ], "source": [ "agent = SpecialistAgent()\n", "agent.price(\"iPad Pro 2nd generation\")" ] }, { "cell_type": "code", "execution_count": null, "id": "f5a3181b-1310-4102-8d7d-52caf4c00538", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.11" } }, "nbformat": 4, "nbformat_minor": 5 }