You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

617 lines
40 KiB

{
"cells": [
{
"cell_type": "markdown",
"id": "5c291475-8c7c-461c-9b12-545a887b2432",
"metadata": {},
"source": [
"# Jupyter Lab\n",
"\n",
"## A Quick Start Guide\n",
"\n",
"Welcome to the wonderful world of Jupyter lab! \n",
"This is a Data Science playground where you can easily write code and investigate the results. It's an ideal environment for: \n",
"- Research & Development\n",
"- Prototyping\n",
"- Learning (that's us!)\n",
"\n",
"It's not typically used for shipping production code, and in Week 8 we'll explore the bridge between Jupyter and python code.\n",
"\n",
"A file in Jupyter Lab, like this one, is called a **Notebook**.\n",
"\n",
"A long time ago, Jupyter used to be called \"IPython\", and so the extensions of notebooks are \".ipynb\" which stands for \"IPython Notebook\".\n",
"\n",
"On the left is a File Browser that lets you navigate around the directories and choose different notebooks. But you probably know that already, or you wouldn't have got here!\n",
"\n",
"The notebook consists of a series of square boxes called \"cells\". Some of them contain text, like this cell, and some of them contain code, like the cell below.\n",
"\n",
"Click in a cell with code and press `Shift + Return` (or `Shift + Enter`) to run the code and print the output.\n",
"\n",
"Do that now for the cell below this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "33d37cd8-55c9-4e03-868c-34aa9cab2c80",
"metadata": {},
"outputs": [],
"source": [
"# Click anywhere in this cell and press Shift + Return\n",
"\n",
"2 + 2"
]
},
{
"cell_type": "markdown",
"id": "9e95df7b-55c6-4204-b8f9-cae83360fc23",
"metadata": {},
"source": [
"## Congrats!\n",
"\n",
"Now run the next cell which sets a value, followed by the cells after it to print the value"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "585eb9c1-85ee-4c27-8dc2-b4d8d022eda0",
"metadata": {},
"outputs": [],
"source": [
"# Set a value for a variable\n",
"\n",
"favorite_fruit = \"bananas\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "07792faa-761d-46cb-b9b7-2bbf70bb1628",
"metadata": {},
"outputs": [],
"source": [
"# The result of the last statement is shown after you run it\n",
"\n",
"favorite_fruit"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a067d2b1-53d5-4aeb-8a3c-574d39ff654a",
"metadata": {},
"outputs": [],
"source": [
"# Use the variable\n",
"\n",
"print(f\"My favorite fruit is {favorite_fruit}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4c5a4e60-b7f4-4953-9e80-6d84ba4664ad",
"metadata": {},
"outputs": [],
"source": [
"# Now change the variable\n",
"\n",
"favorite_fruit = f\"anything but {favorite_fruit}\""
]
},
{
"cell_type": "markdown",
"id": "9442d5c9-f57d-4839-b0af-dce58646c04f",
"metadata": {},
"source": [
"## Now go back and rerun the cell with the print statement, two cells back\n",
"\n",
"See how it prints something different, even though favorite_fruit was changed further down in the notebook? \n",
"\n",
"The order that code appears in the notebook doesn't matter. What matters is the order that the code is **executed**. There's a python process sitting behind this notebook in which the variables are being changed.\n",
"\n",
"This catches some people out when they first use Jupyter."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8e5ec81d-7c5b-4025-bd2e-468d67b581b6",
"metadata": {},
"outputs": [],
"source": [
"# Then run this cell twice, and see if you understand what's going on\n",
"\n",
"print(f\"My favorite fruit is {favorite_fruit}\")\n",
"\n",
"favorite_fruit = \"apples\""
]
},
{
"cell_type": "markdown",
"id": "a29dab2d-bab9-4a54-8504-05e62594cc6f",
"metadata": {},
"source": [
"# Explaining the 'kernel'\n",
"\n",
"Sitting behind this notebook is a Python process which executes each cell when you run it. That Python process is known as the Kernel. Each notebook has its own separate Kernel.\n",
"\n",
"You can go to the Kernel menu and select \"Restart Kernel\".\n",
"\n",
"If you then try to run the next cell, you'll get an error, because favorite_fruit is no longer defined. You'll need to run the cells from the top of the notebook again. Then the next cell should run fine."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "84b1e410-5eda-4e2c-97ce-4eebcff816c5",
"metadata": {},
"outputs": [],
"source": [
"print(f\"My favorite fruit is {favorite_fruit}\")"
]
},
{
"cell_type": "markdown",
"id": "4d4188fc-d9cc-42be-8b4e-ae8630456764",
"metadata": {},
"source": [
"# Adding and moving cells\n",
"\n",
"Click in this cell, then click the \\[+\\] button in the toolbar above to create a new cell immediately below this one. Copy and paste in the code in the prior cell, then run it! There are also icons in the top right of the selected cell to delete it (bin), duplicate it, and move it up and down.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ce258424-40c3-49a7-9462-e6fa25014b03",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "30e71f50-8f01-470a-9d7a-b82a6cef4236",
"metadata": {},
"source": [
"# Cell output\n",
"\n",
"When you execute a cell, the standard output and the result of the last statement is written to the area immediately under the code, known as the 'cell output'. When you save a Notebook from the file menu (or command+S), the output is also saved, making it a useful record of what happened.\n",
"\n",
"You can clean this up by going to Edit menu >> Clear Outputs of All Cells, or Kernel menu >> Restart Kernel and Clear Outputs of All Cells."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a4d021e2-c284-411f-8ab1-030530cfbe72",
"metadata": {},
"outputs": [],
"source": [
"spams = [\"spam\"] * 1000\n",
"print(spams)\n",
"\n",
"# Might be worth clearing output after running this!"
]
},
{
"cell_type": "markdown",
"id": "eac060f2-7a71-46e7-8235-b6ad0a76f5f8",
"metadata": {},
"source": [
"# Using markdown\n",
"\n",
"So what's going on with these areas with writing in them, like this one? Well, there's actually a different kind of cell called a 'Markdown' cell for adding explanations like this. Click the + button to add a cell. Then in the toolbar, click where it says 'Code' and change it to 'Markdown'.\n",
"\n",
"Add some comments using Markdown format, perhaps copying and pasting from here:\n",
"\n",
"```\n",
"# This is a heading\n",
"## This is a sub-head\n",
"### And a sub-sub-head\n",
"\n",
"I like Jupyter Lab because it's\n",
"- Easy\n",
"- Flexible\n",
"- Satisfying\n",
"```\n",
"\n",
"And to turn this into formatted text simply with Shift+Return in the cell.\n",
"Click in the cell and press the Bin icon if you want to remove it."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e1586320-c90f-4f22-8b39-df6865484950",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "1330c83c-67ac-4ca0-ac92-a71699e0c31b",
"metadata": {},
"source": [
"# The exclamation point\n",
"\n",
"There's a super useful feature of jupyter labs; you can type a command with a ! in front of it in a code cell, like:\n",
"\n",
"!pip install \\[some_package\\]\n",
"\n",
"And it will run it at the command line (as if in Windows Powershell or Mac Terminal) and print the result"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "82042fc5-a907-4381-a4b8-eb9386df19cd",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Guide to Jupyter.ipynb day2 EXERCISE.ipynb \u001b[1m\u001b[36msolutions\u001b[m\u001b[m\n",
"Intermediate Python.ipynb day5.ipynb troubleshooting.ipynb\n",
"\u001b[1m\u001b[36mcommunity-contributions\u001b[m\u001b[m diagnostics.py week1 EXERCISE.ipynb\n",
"day1.ipynb notes.mdx\n"
]
}
],
"source": [
"# list the current directory\n",
"\n",
"!ls"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4fc3e3da-8a55-40cc-9706-48bf12a0e20e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"PING cnn.com (151.101.195.5): 56 data bytes\n",
"64 bytes from 151.101.195.5: icmp_seq=0 ttl=56 time=15.914 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=1 ttl=56 time=41.762 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=2 ttl=56 time=82.651 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=3 ttl=56 time=13.553 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=4 ttl=56 time=14.054 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=5 ttl=56 time=13.804 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=6 ttl=56 time=12.579 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=7 ttl=56 time=14.237 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=8 ttl=56 time=11.441 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=9 ttl=56 time=12.085 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=10 ttl=56 time=16.030 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=11 ttl=56 time=16.310 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=12 ttl=56 time=12.209 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=13 ttl=56 time=48.311 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=14 ttl=56 time=95.381 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=15 ttl=56 time=18.853 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=16 ttl=56 time=16.615 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=17 ttl=56 time=13.696 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=18 ttl=56 time=15.165 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=19 ttl=56 time=19.311 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=20 ttl=56 time=15.867 ms\n",
"64 bytes from 151.101.195.5: icmp_seq=21 ttl=56 time=14.687 ms\n",
"^C\n",
"\n",
"--- cnn.com ping statistics ---\n",
"22 packets transmitted, 22 packets received, 0.0% packet loss\n",
"round-trip min/avg/max/stddev = 11.441/24.296/95.381/22.406 ms\n"
]
}
],
"source": [
"# ping cnn.com - press the stop button in the toolbar when you're bored\n",
"\n",
"!ping cnn.com"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "a58e9462-89a2-4b4f-b4aa-51c4bd9f796b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Retrieving notices: done\n",
"Channels:\n",
" - conda-forge\n",
" - defaults\n",
"Platform: osx-arm64\n",
"Collecting package metadata (repodata.json): done\n",
"Solving environment: done\n",
"\n",
"Downloading and Extracting Packages:\n",
"\n",
"Preparing transaction: done\n",
"Verifying transaction: done\n",
"Executing transaction: done\n",
"Installing pip dependencies: \\ Ran pip subprocess with arguments:\n",
"['/opt/anaconda3/envs/llms/bin/python', '-m', 'pip', 'install', '-U', '-r', '/Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt', '--exists-action=b']\n",
"Pip subprocess output:\n",
"Requirement already satisfied: transformers in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (4.46.3)\n",
"Requirement already satisfied: sentence-transformers in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (3.3.1)\n",
"Requirement already satisfied: datasets in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (3.1.0)\n",
"Requirement already satisfied: accelerate in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 4)) (1.1.1)\n",
"Requirement already satisfied: sentencepiece in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 5)) (0.2.0)\n",
"Requirement already satisfied: bitsandbytes in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 6)) (0.42.0)\n",
"Requirement already satisfied: openai in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (1.55.1)\n",
"Collecting openai (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7))\n",
" Downloading openai-1.55.2-py3-none-any.whl.metadata (24 kB)\n",
"Requirement already satisfied: gradio in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (5.6.0)\n",
"Collecting gradio (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8))\n",
" Downloading gradio-5.7.0-py3-none-any.whl.metadata (16 kB)\n",
"Requirement already satisfied: gensim in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 9)) (4.3.3)\n",
"Requirement already satisfied: modal in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (0.66.44)\n",
"Collecting modal (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10))\n",
" Downloading modal-0.67.0-py3-none-any.whl.metadata (2.3 kB)\n",
"Requirement already satisfied: ollama in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 11)) (0.4.1)\n",
"Requirement already satisfied: psutil in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 12)) (6.1.0)\n",
"Requirement already satisfied: setuptools in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 13)) (75.6.0)\n",
"Requirement already satisfied: speedtest-cli in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from -r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 14)) (2.1.3)\n",
"Requirement already satisfied: filelock in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (3.16.1)\n",
"Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (0.26.2)\n",
"Requirement already satisfied: numpy>=1.17 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (1.26.4)\n",
"Requirement already satisfied: packaging>=20.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (24.2)\n",
"Requirement already satisfied: pyyaml>=5.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (6.0.2)\n",
"Requirement already satisfied: regex!=2019.12.17 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (2024.11.6)\n",
"Requirement already satisfied: requests in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (2.32.3)\n",
"Requirement already satisfied: tokenizers<0.21,>=0.20 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (0.20.3)\n",
"Requirement already satisfied: safetensors>=0.4.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (0.4.5)\n",
"Requirement already satisfied: tqdm>=4.27 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (4.67.1)\n",
"Requirement already satisfied: torch>=1.11.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from sentence-transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (2.4.1)\n",
"Requirement already satisfied: scikit-learn in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from sentence-transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (1.5.2)\n",
"Requirement already satisfied: scipy in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from sentence-transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (1.13.1)\n",
"Requirement already satisfied: Pillow in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from sentence-transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (11.0.0)\n",
"Requirement already satisfied: pyarrow>=15.0.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (17.0.0)\n",
"Requirement already satisfied: dill<0.3.9,>=0.3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (0.3.8)\n",
"Requirement already satisfied: pandas in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (2.2.3)\n",
"Requirement already satisfied: xxhash in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (3.5.0)\n",
"Requirement already satisfied: multiprocess<0.70.17 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (0.70.16)\n",
"Requirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (2024.9.0)\n",
"Requirement already satisfied: aiohttp in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (3.11.7)\n",
"Requirement already satisfied: anyio<5,>=3.5.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (4.6.2.post1)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (1.9.0)\n",
"Requirement already satisfied: httpx<1,>=0.23.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (0.27.2)\n",
"Requirement already satisfied: jiter<1,>=0.4.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (0.7.1)\n",
"Requirement already satisfied: pydantic<3,>=1.9.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (2.10.1)\n",
"Requirement already satisfied: sniffio in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (1.3.1)\n",
"Requirement already satisfied: typing-extensions<5,>=4.11 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (4.12.2)\n",
"Requirement already satisfied: aiofiles<24.0,>=22.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (23.2.1)\n",
"Requirement already satisfied: fastapi<1.0,>=0.115.2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.115.5)\n",
"Requirement already satisfied: ffmpy in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.4.0)\n",
"Collecting gradio-client==1.5.0 (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8))\n",
" Downloading gradio_client-1.5.0-py3-none-any.whl.metadata (7.1 kB)\n",
"Requirement already satisfied: jinja2<4.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (3.1.4)\n",
"Requirement already satisfied: markupsafe~=2.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (2.1.5)\n",
"Requirement already satisfied: orjson~=3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (3.10.12)\n",
"Requirement already satisfied: pydub in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.25.1)\n",
"Requirement already satisfied: python-multipart==0.0.12 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.0.12)\n",
"Requirement already satisfied: ruff>=0.2.2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.8.0)\n",
"Requirement already satisfied: safehttpx<1.0,>=0.1.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.1.1)\n",
"Requirement already satisfied: semantic-version~=2.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (2.10.0)\n",
"Requirement already satisfied: starlette<1.0,>=0.40.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.41.3)\n",
"Requirement already satisfied: tomlkit==0.12.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.12.0)\n",
"Requirement already satisfied: typer<1.0,>=0.12 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.13.1)\n",
"Requirement already satisfied: uvicorn>=0.14.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (0.32.1)\n",
"Requirement already satisfied: websockets<13.0,>=10.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gradio-client==1.5.0->gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (12.0)\n",
"Requirement already satisfied: smart-open>=1.8.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from gensim->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 9)) (7.0.5)\n",
"Requirement already satisfied: certifi in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (2024.8.30)\n",
"Requirement already satisfied: click>=8.1.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (8.1.7)\n",
"Requirement already satisfied: grpclib==0.4.7 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (0.4.7)\n",
"Requirement already satisfied: protobuf!=4.24.0,<6.0,>=3.19 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (4.25.3)\n",
"Requirement already satisfied: rich>=12.0.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (13.9.4)\n",
"Requirement already satisfied: synchronicity~=0.9.3 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (0.9.3)\n",
"Requirement already satisfied: toml in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (0.10.2)\n",
"Requirement already satisfied: types-certifi in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (2021.10.8.3)\n",
"Requirement already satisfied: types-toml in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (0.10.8.20240310)\n",
"Requirement already satisfied: watchfiles in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (0.24.0)\n",
"Requirement already satisfied: h2<5,>=3.1.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from grpclib==0.4.7->modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (4.1.0)\n",
"Requirement already satisfied: multidict in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from grpclib==0.4.7->modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (6.1.0)\n",
"Requirement already satisfied: idna>=2.8 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from anyio<5,>=3.5.0->openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (3.10)\n",
"Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (2.4.3)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (1.3.1)\n",
"Requirement already satisfied: attrs>=17.3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (24.2.0)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (1.5.0)\n",
"Requirement already satisfied: propcache>=0.2.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (0.2.0)\n",
"Requirement already satisfied: yarl<2.0,>=1.17.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from aiohttp->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (1.18.0)\n",
"Requirement already satisfied: httpcore==1.* in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from httpx<1,>=0.23.0->openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (1.0.7)\n",
"Requirement already satisfied: h11<0.15,>=0.13 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (0.14.0)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from pandas->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (2.9.0.post0)\n",
"Requirement already satisfied: pytz>=2020.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from pandas->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (2024.1)\n",
"Requirement already satisfied: tzdata>=2022.7 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from pandas->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (2024.2)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from pydantic<3,>=1.9.0->openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.27.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from pydantic<3,>=1.9.0->openai->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 7)) (2.27.1)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from requests->transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (3.4.0)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from requests->transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 1)) (2.2.3)\n",
"Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from rich>=12.0.0->modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (3.0.0)\n",
"Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from rich>=12.0.0->modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (2.18.0)\n",
"Requirement already satisfied: wrapt in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from smart-open>=1.8.1->gensim->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 9)) (1.17.0)\n",
"Requirement already satisfied: sigtools>=4.0.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from synchronicity~=0.9.3->modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (4.0.1)\n",
"Requirement already satisfied: sympy in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from torch>=1.11.0->sentence-transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (1.13.3)\n",
"Requirement already satisfied: networkx in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from torch>=1.11.0->sentence-transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (3.4.2)\n",
"Requirement already satisfied: shellingham>=1.3.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from typer<1.0,>=0.12->gradio->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 8)) (1.5.4)\n",
"Requirement already satisfied: joblib>=1.2.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from scikit-learn->sentence-transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (1.4.2)\n",
"Requirement already satisfied: threadpoolctl>=3.1.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from scikit-learn->sentence-transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (3.5.0)\n",
"Requirement already satisfied: hyperframe<7,>=6.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from h2<5,>=3.1.0->grpclib==0.4.7->modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (6.0.1)\n",
"Requirement already satisfied: hpack<5,>=4.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from h2<5,>=3.1.0->grpclib==0.4.7->modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (4.0.0)\n",
"Requirement already satisfied: mdurl~=0.1 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=12.0.0->modal->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 10)) (0.1.2)\n",
"Requirement already satisfied: six>=1.5 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->datasets->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 3)) (1.16.0)\n",
"Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/anaconda3/envs/llms/lib/python3.11/site-packages (from sympy->torch>=1.11.0->sentence-transformers->-r /Users/xiangyu/repos/llm_engineering/condaenv.u2f__0c0.requirements.txt (line 2)) (1.3.0)\n",
"Downloading openai-1.55.2-py3-none-any.whl (389 kB)\n",
"Downloading gradio-5.7.0-py3-none-any.whl (57.1 MB)\n",
"\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.1/57.1 MB\u001b[0m \u001b[31m76.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n",
"\u001b[?25hDownloading gradio_client-1.5.0-py3-none-any.whl (320 kB)\n",
"Downloading modal-0.67.0-py3-none-any.whl (487 kB)\n",
"Installing collected packages: openai, gradio-client, modal, gradio\n",
" Attempting uninstall: openai\n",
" Found existing installation: openai 1.55.1\n",
" Uninstalling openai-1.55.1:\n",
" Successfully uninstalled openai-1.55.1\n",
" Attempting uninstall: gradio-client\n",
" Found existing installation: gradio_client 1.4.3\n",
" Uninstalling gradio_client-1.4.3:\n",
" Successfully uninstalled gradio_client-1.4.3\n",
" Attempting uninstall: modal\n",
" Found existing installation: modal 0.66.44\n",
" Uninstalling modal-0.66.44:\n",
" Successfully uninstalled modal-0.66.44\n",
" Attempting uninstall: gradio\n",
" Found existing installation: gradio 5.6.0\n",
" Uninstalling gradio-5.6.0:\n",
" Successfully uninstalled gradio-5.6.0\n",
"Successfully installed gradio-5.7.0 gradio-client-1.5.0 modal-0.67.0 openai-1.55.2\n",
"\n",
"done\n",
"#\n",
"# To activate this environment, use\n",
"#\n",
"# $ conda activate llms\n",
"#\n",
"# To deactivate an active environment, use\n",
"#\n",
"# $ conda deactivate\n",
"\n"
]
}
],
"source": [
"# This is a useful command that ensures your Anaconda environment \n",
"# is up to date with any new upgrades to packages;\n",
"# But it might take a minute and will print a lot to output\n",
"\n",
"!conda env update -f ../environment.yml --prune"
]
},
{
"cell_type": "markdown",
"id": "4688baaf-a72c-41b5-90b6-474cb24790a7",
"metadata": {},
"source": [
"# Minor things we encounter on the course\n",
"\n",
"This isn't necessarily a feature of Jupyter, but it's a nice package to know about that is useful in Jupyter Labs, and I use it in the course.\n",
"\n",
"The package `tqdm` will print a nice progress bar if you wrap any iterable."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "2646a4e5-3c23-4aee-a34d-d623815187d2",
"metadata": {},
"outputs": [],
"source": [
"# Here's some code with no progress bar\n",
"# It will take 10 seconds while you wonder what's happpening..\n",
"\n",
"import time\n",
"\n",
"spams = [\"spam\"] * 1000\n",
"\n",
"for spam in spams:\n",
" time.sleep(0.01)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "6e96be3d-fa82-42a3-a8aa-b81dd20563a5",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|███████████████████████████████████████████████████████| 1000/1000 [00:12<00:00, 82.79it/s]\n"
]
}
],
"source": [
"# And now, with a nice little progress bar:\n",
"\n",
"import time\n",
"from tqdm import tqdm\n",
"\n",
"spams = [\"spam\"] * 1000\n",
"\n",
"for spam in tqdm(spams):\n",
" time.sleep(0.01)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "63c788dd-4618-4bb4-a5ce-204411a38ade",
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"# This is a big heading!\n",
"\n",
"- And this is a bullet-point\n",
"- So is this\n",
"- Me, too!"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# On a different topic, here's a useful way to print output in markdown\n",
"\n",
"from IPython.display import Markdown, display\n",
"\n",
"display(Markdown(\"# This is a big heading!\\n\\n- And this is a bullet-point\\n- So is this\\n- Me, too!\"))\n"
]
},
{
"cell_type": "markdown",
"id": "9d14c1fb-3321-4387-b6ca-9af27676f980",
"metadata": {},
"source": [
"# That's it! You're up to speed on Jupyter Lab.\n",
"\n",
"## Want to be even more advanced?\n",
"\n",
"If you want to become a pro at Jupyter Lab, you can read their tutorial [here](https://jupyterlab.readthedocs.io/en/latest/). But this isn't required for our course; just a good technique for hitting Shift + Return and enjoying the result!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}