Narcis Miclaus

8 min read

Python, from the ground up Lesson 47 / 60

Jupyter, notebooks, and when to leave them

Why notebooks are addictive, where they shine, and the moment you should stop and write a real script.

A notebook is a deeply seductive thing. You write a line, hit Shift+Enter, see the output. You write another, see another. An hour later you have a working analysis without ever thinking about file structure, function boundaries, or imports. Notebooks are the closest Python gets to the fluency of a spreadsheet.

They’re also the closest Python gets to a footgun in your foot, because the same property that makes them addictive — the kernel that holds state between cells — is the property that quietly rots your work.

This lesson is about both sides. Where notebooks earn their reputation, where they fail, and the moment to port the code to a .py file.

What a notebook actually is

A Jupyter notebook is a JSON file with the extension .ipynb. It contains an ordered list of cells, each of which is either code or markdown. When you execute a code cell, the text is sent to a kernel — a Python process running in the background — which evaluates it and returns the result. The result, including stdout, stderr, return value, and any rich representation (plots, tables, HTML), is stored back into the JSON.

Three things follow from this design:

  1. The kernel is stateful. A variable defined in cell 3 is visible in cell 7.
  2. You can execute cells in any order. Cell 7 can run before cell 3.
  3. The output saved in the file is the output from whenever you last ran the cell, not necessarily the output from running the whole notebook top to bottom.

Read those three again. Together they explain almost every notebook horror story you’ve ever heard.

Where notebooks shine

Exploration: you have a CSV, you want to know what’s in it. Load it, .head(), .describe(), plot a histogram, slice by a column. The feedback loop of edit-run-see is unbeatable.

import pandas as pd

df = pd.read_csv("sales.csv")
df.head()

Teaching: a notebook with text, code, and output interleaved is a complete artifact. The reader sees what you typed and what came out.

Analysis you’ll hand to a stakeholder: render to HTML or PDF, send the link, done. The recipient sees code, results, and prose in one document.

ML prototyping: training loops you want to inspect mid-run, intermediate plots, hyperparameters you tweak by hand. The kernel staying alive between cells means you don’t reload the model every time you change a learning rate.

Where notebooks fail

Production code: notebooks are nearly impossible to schedule, monitor, or test. You can run a notebook from cron with papermill, but the moment you do you’ve wrapped a stateful interactive tool in a script harness, and now you have two problems.

Version control: the JSON format includes cell outputs, execution counts, and metadata. Two people running the same notebook produce different files even when the code is identical. Git diffs are unreadable. There are tools to strip outputs (nbstripout) but you have to remember to install them in every clone.

Reusable libraries: a function defined in cell 4 is not importable from another notebook without copy-paste. You can write %run other.ipynb to execute another notebook in the current kernel, but now your dependency graph is invisible.

Hidden state: the killer. You define df in cell 1, modify it in cell 3, then go back to cell 2 and re-run it. The variable still has the cell-3 modifications. Six months later someone re-runs the notebook top to bottom and gets different numbers. They think the data changed. The data didn’t change. The cell order changed.

The fix for hidden state is simple in principle: restart and run all before you trust the result, before you commit, before you share. Make it a reflex.

The two IDEs in 2026

Jupyter Lab is the original web-based environment. You start a server (jupyter lab), open a browser, get a file tree on the left and notebook tabs in the center. It’s still the cleanest experience for pure notebook work, especially with extensions like jupyterlab-git and the variable inspector.

VS Code notebooks (also Cursor, Zed, and other VS Code forks) opened the door for a generation of developers who already lived in their editor. You open a .ipynb directly, the editor renders it inline, and you get the full IDE experience: type checking, refactoring, integrated terminal, source control. In 2026 this is the dominant choice for most developers I work with, and the reason is simple: you’re one keyboard shortcut from your .py files in the same window.

Pick whichever you prefer. They’re both fine. The notebook file format is identical, so you can switch between them.

Kernel hygiene

Name the kernel after the project’s virtual environment. From a project’s activated venv:

pip install ipykernel
python -m ipykernel install --user --name myproject --display-name "Python (myproject)"

Now Python (myproject) shows up in the kernel picker. Three reasons this matters:

  1. You won’t accidentally import packages from the wrong environment.
  2. The notebook records which kernel it expects, so collaborators see the right name.
  3. You can have one kernel per project without polluting the global Python.

When something stops working and you can’t tell why, restart the kernel. Half of all “weird” notebook bugs are stale objects in memory.

The percent-cell pattern

The best-kept secret of modern Python tooling: a regular .py file with # %% markers acts as a notebook in VS Code, Cursor, PyCharm, and Spyder.

# %% Imports
import pandas as pd
import numpy as np

# %% Load data
df = pd.read_csv("sales.csv")
df.head()

# %% Quick stats
df.describe()

# %% Plot
df["revenue"].hist(bins=50)

Open this in VS Code, click “Run Cell” above any # %% block, and you get an interactive window with output, exactly like a notebook. The file itself is plain Python. Diffs work. Imports work. You can run it as a script with python file.py. You can write tests for the functions in it.

This is what I default to now. I write .py files with cell markers, drop into the interactive window when I want to explore, and the file stays clean enough to commit. I only reach for .ipynb when I need rendered output for someone else to read.

Tools worth knowing

nbdev treats notebooks as the source of truth for a library. You write the implementation, docs, and tests in the same .ipynb, and nbdev generates a .py package and HTML docs. Some teams swear by it. I’ve seen it work and I’ve seen it become an unmaintainable swamp. The honest answer: if you’re already a notebook-first person, try it; if you’re not, don’t.

papermill parameterizes notebook runs:

papermill analysis.ipynb output.ipynb -p year 2025 -p region "EU"

It injects parameters into a tagged cell and runs the whole notebook headlessly. Useful for batch reports.

jupyter nbconvert turns a notebook into HTML, PDF, or a script:

jupyter nbconvert --to html report.ipynb
jupyter nbconvert --to pdf report.ipynb
jupyter nbconvert --to script report.ipynb  # gives you a .py

The HTML export is the right way to share results with a non-technical stakeholder.

When to leave the notebook

I keep a small mental decision tree. When any of these is true, the notebook needs to become a .py file or a proper module:

  • I find myself running the same notebook on different inputs more than twice.
  • A colleague needs to run it and get the same answer reliably.
  • The logic is being copied into another notebook.
  • It’s slow enough that I want to schedule it.
  • It contains a function I want to use elsewhere.

The migration path is usually: factor the heavy logic into a .py module, then keep a thin notebook that imports it and renders the output.

# analysis.py
def load_clean(path: str) -> pd.DataFrame:
    df = pd.read_csv(path)
    df = df.dropna(subset=["revenue"])
    return df

def summarize(df: pd.DataFrame) -> pd.DataFrame:
    return df.groupby("region")["revenue"].agg(["mean", "median", "count"])
# report.ipynb (or report.py with # %% markers)
from analysis import load_clean, summarize

df = load_clean("sales.csv")
summarize(df)

The notebook becomes a presentation layer. The logic lives in code you can test.

Magics worth knowing

Jupyter’s magic commands are shortcut commands prefixed by % (line) or %% (cell). A handful are worth keeping in muscle memory:

  • %timeit expr and %%timeit measure execution time with proper warmup and repetition. Far better than rolling your own time.time() calls.
  • %load_ext autoreload followed by %autoreload 2 means your imported .py modules get re-imported automatically when you edit them. This is the single biggest quality-of-life setting for the percent-cell workflow.
  • %env VAR=value sets environment variables for the kernel.
  • %%writefile name.py writes a cell’s content to a file, useful when you’ve prototyped a function in a notebook and want to extract it.
  • %debug drops you into a post-mortem debugger after an exception. Combined with breakpoint() from the previous lesson, you have a full debugging story without leaving the kernel.

You don’t need to memorize the rest. %lsmagic lists everything available.

Sharing and reproducibility

A notebook’s biggest weakness as a deliverable is that the recipient needs the same environment to re-run it. Three patterns help:

  1. Include a requirements.txt or pyproject.toml next to the notebook. State your Python version explicitly.
  2. Pin versions of the libraries that drove the analysis, especially pandas and NumPy, since their APIs evolve.
  3. Render to HTML for read-only consumers. They don’t need to run anything; they just need to read.

For collaborators who will re-run, uv (which we covered earlier in the course) makes the setup trivially fast. A README that says “run uv sync && jupyter lab” is enough.

The honest verdict

Notebooks are a brilliant tool for the first half of any project and the wrong tool for the second half. The first half is “what’s in this data, what’s the shape, what’s the story.” The second half is “make this run reliably forever.” Switch tools when you switch phases.

If you’ve been writing notebooks for years and resisting .py files, try the percent-cell pattern for a week. You’ll get the interactive feedback you love and the diffs your future self needs. The next lesson puts all of this — pandas, NumPy, SciPy, and the workflow we just sketched — into a single end-to-end project.

Search