All in One View

Content from Writing Effective Functions

Last updated on 2026-11-06 | Edit this page

Estimated time: 120 minutes

Overview

Questions

  • When should I turn a block of code into a function?
  • How do I design a function so someone else (or future-me) can use it correctly?
  • What belongs in a docstring, and why write one at all?

Objectives

  • Identify repeated or reusable logic that should be extracted into a function
  • Write functions with clear positional arguments, defaults, and *args/**kwargs
  • Add type hints to a function signature
  • Write a docstring that documents a function’s purpose, arguments, and return value

Why write functions?

Open first_analysis.ipynb from the setup materials. It loads a full-year of data from the OOI GI01SUMO (Global Irminger Sea Array Surface Mooring) NSIF (Near-Surface Instrument Frame) and computes 15-minute burst-median statistics for the CTD (temperature, salinity) and the DOSTA (dissolved oxygen). It calculates the same statistics for both instruments, with a six-line block copy-pasted and lightly edited between them.

Challenge

Challenge 1: Spot the repetition

Skim first_analysis.ipynb. Without changing any code yet, identify (you can write a comment at the top of the file) which blocks of code are doing “the same thing” to different data.

The CTD block and the DOSTA block: both convert an xarray Dataset to a dataframe, resample it to 15-minute bins, compute the median and the median absolute deviation (MAD) per bin, and reassemble the two into one table. Only the input dataset changes between them.

A good rule of thumb: if you’ve written the same logic twice, it should be a function. As you add instruments or expand your analysis, a bug fix means finding and fixing it in every copy. Functions turn that into one place.

Function anatomy

We’ll live-code the extraction together, pulling the burst-statistics logic out of the CTD block:

PYTHON 

def resample_burst_stats(df, freq="900s"):
    resampler = df.resample(freq, origin="start_day", offset=pd.Timedelta("3150s"))
    median = resampler.median()

    group_median = resampler.transform("median")
    deviations = (df - group_median).abs()
    mad = deviations.resample(freq, origin="start_day", offset=pd.Timedelta("3150s")).median()

    stats = pd.concat([median, mad], axis=1, keys=["median", "mad"]).swaplevel(axis=1).sort_index(axis=1)
    stats.index = stats.index + pd.Timedelta(freq) / 2
    return stats

Key parts to narrate as you write:

  • Positional arguments (df) vs. keyword arguments with defaults (freq="900s"): the burst interval defines the length of time between the start of collection intervals. It may change between instruments (but not here)
  • The return value: the original script only ever printed or plotted ctd_stats/dosta_stats; returning it is what lets the same function produce both
  • *args/**kwargs would come in if, say, this needed to pass extra options through to .resample() without the function knowing about all of them in advance.
Challenge

Challenge 2: Extract a function

Working from first_analysis.ipynb, extract the repeated resampling block into resample_burst_stats and call it twice: once for the CTD dataframe, once for the DOSTA dataframe.

PYTHON 

def resample_burst_stats(df, freq="900s"):
    resampler = df.resample(freq, origin="start_day", offset=pd.Timedelta("3150s"))
    median = resampler.median()

    group_median = resampler.transform("median")
    deviations = (df - group_median).abs()
    mad = deviations.resample(freq, origin="start_day", offset=pd.Timedelta("3150s")).median()

    stats = pd.concat([median, mad], axis=1, keys=["median", "mad"]).swaplevel(axis=1).sort_index(axis=1)
    stats.index = stats.index + pd.Timedelta(freq) / 2
    return stats

ctd_stats = resample_burst_stats(ctd_df)
dosta_stats = resample_burst_stats(dosta_df)
Callout

Single responsibility

resample_burst_stats computes statistics. It doesn’t load data and it doesn’t plot. The original notebook’s cells did all three in sequence, which is fine for exploring, but bundling them together is exactly what would stop this function from being reused and limits its flexibility. We’ll revisit this when we split code into modules in the next block.

Type hints

Type hints don’t change how Python runs your code. Rather, they’re documentation that your editor and static-analysis tools can check for you.

PYTHON 

def resample_burst_stats(df: pd.DataFrame, freq: str = "900s") -> pd.DataFrame:
    ...
Challenge

Challenge 3: Add type hints

Add type hints to resample_burst_stats. What type should the return value be?

PYTHON 

def resample_burst_stats(df: pd.DataFrame, freq: str = "900s") -> pd.DataFrame:
    resampler = df.resample(freq, origin="start_day", offset=pd.Timedelta("3150s"))
    median = resampler.median()

    group_median = resampler.transform("median")
    deviations = (df - group_median).abs()
    mad = deviations.resample(freq, origin="start_day", offset=pd.Timedelta("3150s")).median()

    stats = pd.concat([median, mad], axis=1, keys=["median", "mad"]).swaplevel(axis=1).sort_index(axis=1)
    stats.index = stats.index + pd.Timedelta(freq) / 2
    return stats

The return type is pd.DataFrame because both median and the final stats table are dataframes, and stats is what gets returned.

Docstrings

A docstring is the first statement inside a function, written as a triple-quoted string. It’s what shows up when someone runs help(resample_burst_stats). first_analysis.ipynb already has one good example to look at: fill_harmonic_gaps, which fills gaps in the burst-median series using a linear-trend-plus-harmonic fit. It’s already written as a clean, documented function. Use it as a reference example.

PYTHON 

def resample_burst_stats(df: pd.DataFrame, freq: str = "900s") -> pd.DataFrame:
    """Compute burst-median statistics for OOI instrument data.

    OOI instruments often sample in short bursts; this
    collapses each burst into a robust summary statistic.

    Parameters
    ----------
    df : pandas.DataFrame
        Time-indexed instrument data, one column per variable, at full sampling resolution.
    freq : str, default "900s"
        Burst interval as a pandas offset string.

    Returns
    -------
    pandas.DataFrame
        Columns are a 2-level MultiIndex (variable, statistic), where
        statistic is one of "median" or "mad" (median absolute deviation).
        Index is the start time of each burst.
    """
    ...

We’re using the NumPy docstring style here since it’s common in scientific Python, but the important content is the same regardless of style: what the function does, what each argument means, and what comes back.

Challenge

Challenge 4: Document your function

Add a docstring to resample_burst_stats. Then, in a notebook cell, run help(resample_burst_stats) and confirm it displays what you expect. Compare it to fill_harmonic_gaps’s docstring already in the file.

It’s worth pointing out here that we’ll rely on these docstrings again in Block 4 when generating project documentation, and thus the investment pays off later in the workshop, not just in some hypothetical future project. fill_harmonic_gaps is deliberately left already-documented in the starter file so learners have a working example to model their own docstring on, rather than writing one from a blank page.

Wrap-up

By the end of this block, learners should have resample_burst_stats extracted, type-hinted, and documented, alongside the already-clean fill_harmonic_gaps. These two functions are what Block 2 organizes into a module.

Key Points
  • Extract a function when you find yourself repeating logic, not just to look tidy
  • Prefer returning values over printing them, so functions can be composed
  • Type hints document expected input/output types for editors and readers
  • A docstring documents purpose, parameters, and return value. Its often best to write it as part of writing the function, not as an afterthought

Content from Organizing Functions into Modules

Last updated on 2026-11-06 | Edit this page

Estimated time: 140 minutes

Overview

Questions

  • When does a collection of functions become a module?
  • How do I use my own modules from a Jupyter notebook, not just a script?
  • How should I decide what becomes a module, and what doesn’t?

Objectives

  • Explain the difference between a script, a module, and a package
  • Import a local module into a running notebook, and pick up edits without restarting the kernel
  • Split a set of related functions into modules by responsibility, not by dataset
  • Recognize when not to make something a module

Why modules?

By the end of Block 1, first_analysis.ipynb has two good functions in it: resample_burst_stats and fill_harmonic_gaps. But they’re still living in notebook cells. That means that if you want to reuse these functions, such as resample_burst_stats, in a different notebook you would have to copy the function out of this one by hand.

Instead, we are going to put the functions into a module. A module is a single .py file that other code can import. This is how we make a function reusable outside the notebook that first needed it.

Challenge

Challenge 1: From notebook cell to module

Create a new file, stats.py, next to first_analysis.ipynb, and move resample_burst_stats and fill_harmonic_gaps into it, along with whatever import lines they need (numpy, pandas).

PYTHON 

# stats.py
"""Burst-statistics and gap-filling for OOI instrument time series."""

import numpy as np
import pandas as pd


def resample_burst_stats(df: pd.DataFrame, freq: str = "900s") -> pd.DataFrame:
    """Compute burst-median statistics for OOI instrument data.

    OOI instruments often sample in short bursts; this
    collapses each burst into a robust summary statistic.

    Parameters
    ----------
    df : pandas.DataFrame
        Time-indexed instrument data, one column per variable, at full sampling resolution.
    freq : str, default "900s"
        Burst interval as a pandas offset string.

    Returns
    -------
    pandas.DataFrame
        Columns are a 2-level MultiIndex (variable, statistic), where
        statistic is one of "median" or "mad" (median absolute deviation).
        Index is the start time of each burst.
    """
    resampler = df.resample(freq, origin="start_day", offset=pd.Timedelta("3150s"))
    median = resampler.median()

    group_median = resampler.transform("median")
    deviations = (df - group_median).abs()
    mad = deviations.resample(freq, origin="start_day", offset=pd.Timedelta("3150s")).median()

    stats = pd.concat([median, mad], axis=1, keys=["median", "mad"]).swaplevel(axis=1).sort_index(axis=1)
    stats.index = stats.index + pd.Timedelta(freq) / 2
    return stats


def fill_harmonic_gaps(series: pd.Series, periods=("365.25D", "182.625D")):
    """Fill gaps in a time-indexed Series with a trend + harmonic fit.

    Parameters
    ----------
    series : pandas.Series
        Time-indexed values, with NaNs marking gaps to fill.
    periods : tuple[str, ...], default ("365.25D", "182.625D")
        Harmonic periods to fit. Defaults are annual and semiannual cycles.

    Returns
    -------
    filled : pandas.Series
    fitted : pandas.Series
    """
    t = (series.index - series.index[0]) / pd.Timedelta("1s")
    omegas = [2 * np.pi / (pd.Timedelta(p) / pd.Timedelta("1s")) for p in periods]

    X = np.column_stack([
        np.ones_like(t),
        t,
        *[f(omega * t) for omega in omegas for f in (np.cos, np.sin)],
    ])

    valid = series.notna().values
    coeffs, *_ = np.linalg.lstsq(X[valid], series.values[valid], rcond=None)

    fitted = pd.Series(X @ coeffs, index=series.index)
    filled = series.where(valid, fitted)
    return filled, fitted

Importing your own module into a notebook

With stats.py sitting next to first_analysis.ipynb, a notebook cell can import it exactly like any other package:

PYTHON 

from stats import resample_burst_stats, fill_harmonic_gaps
Callout

autoreload: editing a module without restarting the kernel

By default, once a notebook has imported a module, editing that module’s .py file and re-running the import cell does nothing because Python caches the module after the first import. Thus any changes you made in the module will not be picked up by the notebook/script/other module you are using.

The fix is two “magic” commands at the top of the notebook, before any of your own imports:

PYTHON 

%load_ext autoreload
%autoreload 2

With this turned on, editing stats.py in your editor and re-running a cell that calls resample_burst_stats(...) picks up the change immediately. This significantly improves the use of notebooks for initial explorations and development, then transfering the changes and results into reuseable functions and modules.

Callout

Note: don’t shadow the standard library

Watch what happens if a module happens to share a name with something in Python’s standard library. In an empty scratch folder:

BASH 

echo "pass" > random.py
uv run --no-project --with numpy python -c "import numpy as np; np.random.rand(3)"

OUTPUT 

ImportError: cannot import name 'SystemRandom' from 'random' (/.../random.py)

In the above code, NumPy tried to import random internally and instead of getting the standard-library modules, it picked up your empty file. Some good module names to avoid are: random.py, csv.py, json.py, types.py, and test.py.

Deciding what becomes a module

It’s tempting to make a module for every conceptual piece of the workflow. Let’s take a look at what a module for loading the data in this workflow would actually contain:

PYTHON 

def load_ooi_dataset(path):
    return xr.open_dataset(path)

One line, wrapping one library call, with nothing project-specific in it. A module built around this doesn’t save future-you any real work because ``xarray.open_dataset``` already is one line. Modularize logic that’s substantial, reusable, or repeated.

What does deserve a module in our notebook? The plotting code. Scroll through first_analysis.ipynb and count how many nearly-identical plotting cells it has. This is the type of repeated code that makes sense as a function and a module.

Challenge

Challenge 2: Extract a plotting module

Pull the repeated plotting logic out of first_analysis.ipynb into plotting.py, with one function per distinct kind of plot (not one function per cell). Aim for something like:

  • plot_raw_timeseries(ctd, dosta, t1=None, t2=None)
    • the raw temperature/salinity/oxygen plots (used at three different zoom levels in the notebook)
  • plot_burst_stats(raw, stats, var, label, color, t1=None, t2=None)
    • raw points with the burst-median ± 2·MAD band

Don’t worry about creating full docstrings for these particular functions. Although you should always add docstrings on your personal code, for this example we can bypass that requirement.

PYTHON 

# plotting.py
"""Plotting helpers for OOI CTD/DOSTA burst-statistics time series."""

import matplotlib.pyplot as plt


def plot_raw_timeseries(ctd, dosta, t1=None, t2=None):
    """Plot raw (unresampled) CTD temperature/salinity and DOSTA oxygen."""

    # Subset the data based on the window to look at
    window = slice(t1, t2)
    ctd_w = ctd.sel(time=window)
    dosta_w = dosta.sel(time=window)

    # Plot the first subplot
    fig, (ax, axb) = plt.subplots(2, 1, figsize=(14, 12), sharex=True)

    ax.plot(ctd_w["time"], ctd_w["sea_water_temperature"], color="tab:red",
            linestyle="", marker=".", label="Temperature (°C)")
    ax.set_ylabel("Temperature (°C)", color="tab:red")
    ax.set_title("Irminger Sea Temperature and Salinity at 12m Depth")

    # Create a second y-axis on the first subplot
    ax2 = ax.twinx()
    ax2.plot(ctd_w["time"], ctd_w["sea_water_practical_salinity"], color="tab:blue",
              linestyle="", marker=".", label="Salinity (PSU)")
    ax2.set_ylabel("Salinity (PSU)", color="tab:blue")
    ax.grid()
    ax2.grid()

    # Plot the second subplot
    axb.plot(dosta_w["time"], dosta_w["oxygen_concentration_corrected"], color="tab:green",
              linestyle="", marker=".", label="Oxygen (µmol/kg)")
    axb.set_ylabel("Oxygen (µmol/kg)", color="tab:green")
    axb.grid()

    fig.autofmt_xdate()
    return fig


def plot_burst_stats(raw, stats, var, label, color, t1=None, t2=None):
    """Plot raw points against burst-median +/- 2*MAD for one variable."""

    # Subset the data based on the window to look at
    window = slice(t1, t2)
    raw_w = raw.sel(time=window)
    median = stats[var]["median"].loc[window]
    mad = stats[var]["mad"].loc[window]

    # Plot the subsetted data
    fig, ax = plt.subplots(1, 1, figsize=(14, 6))
    ax.plot(raw_w["time"], raw_w[var], color=color, linestyle="", marker=".",
            alpha=0.4, label="Raw")
    ax.plot(median.index, median, color=color, label="Burst median")
    ax.fill_between(median.index, median - 2 * mad, median + 2 * mad,
                     color=color, alpha=0.3, linewidth=0, label="±2 MAD")
    ax.set_ylabel(label)
    ax.set_title(f"{label}: burst median ± 2 MAD")
    ax.legend()
    ax.grid()
    fig.autofmt_xdate()
    return fig

In the notebook the plotting cells collapse to:

PYTHON 

plot_raw_timeseries(ctd, dosta)
plot_raw_timeseries(ctd, dosta, t1="2020-09-01", t2="2020-09-08")
plot_raw_timeseries(ctd, dosta, t1="2020-09-01T03:00:00", t2="2020-09-01T04:00:00")

This is much cleaner and easier to read.

Fixing real bugs while modularizing

Scroll to the “Merge and save” section at the bottom of first_analysis.ipynb. It is unfinished, and if you run the notebook from top to bottom it fails:

OUTPUT 

NameError: name 'ctd_ds' is not defined

That one is a plain typo: the variable was named ds, not ctd_ds, and the mistake went unnoticed because the notebook was never re-run start to finish after that line was written.

There are two more problems waiting just past the end of the notebook. The merged table still has 2-level (variable, statistic) column names, and netCDF files can only store variables with plain string names:

OUTPUT 

TypeError: Invalid name ('sea_water_temperature', 'mad') for DataArray or Dataset key: must be either a string or None for serialization to netCDF or zarr files

The notebook starts to flatten the column names, but only for the CTD table. If you merge that flattened CTD table with the DOSTA table, which still has 2-level columns, pandas refuses:

OUTPUT 

MergeError: Not allowed to merge between different levels. (1 levels on the left, 2 on the right)

This is a common and useful side effect of turning exploratory code into functions: writing a function forces you to be explicit about inputs and outputs. If one function flattens a table and we call it for both instruments, the missing step becomes impossible to skip.

Challenge

Challenge 3: Build a merge module and fix the bugs

Write merge.py with the following three functions: 1. A function to flatten a stats table’s columns and add a modeled_flag column 2. A function to merge the flattened CTD and DOSTA tables and attach variable attributes (units, long names) from the original datasets 3. A function to save the result as a netCDF

Use them to merge the CTD variables sea_water_temperature, sea_water_practical_salinity, and sea_water_pressure with the DOSTA variable oxygen_concentration_corrected, and save the result. You will need to fix the bugs described above along the way.

PYTHON 

# merge.py
"""Merge CTD + DOSTA burst statistics into a dataset with metadata."""

import pandas as pd
import xarray as xr

# Attributes that describe each variable
VARIABLE_ATTRS = {
    "sea_water_temperature": {"units": "degrees_Celsius", "long_name": "Sea Water Temperature"},
    "sea_water_practical_salinity": {"units": "1", "long_name": "Sea Water Practical Salinity"},
    "sea_water_pressure": {"units": "dbar", "long_name": "Sea Water Pressure"},
    "oxygen_concentration_corrected": {"units": "umol kg-1", "long_name": "Corrected Dissolved Oxygen Concentration"},
}

# Attributes that describe the statistic that was computed
STAT_ATTRS = {
    "median": {"comment": "Median of each 15-minute burst of samples"},
    "mad": {"comment": "Median absolute deviation of each 15-minute burst of samples"},
}

# Attributes of the modeled flag, which is not in the units of the variable
FLAG_ATTRS = {"units": "1", "comment": "Indicates whether a value was observed (0) or modeled (1)"}



def flag_and_flatten(stats: pd.DataFrame, was_filled: pd.DataFrame, keep_vars: list[str]) -> pd.DataFrame:
    """Restrict to `keep_vars`, add a modeled_flag per variable, and flatten
    the 2-level (variable, stat) columns into single flat names."""
    stats = stats.loc[:, keep_vars].copy()
    for var in keep_vars:
        stats[(var, "modeled_flag")] = was_filled[var].astype(int)

    stats.columns = ["_".join(col).strip() for col in stats.columns.values]
    return stats


def merge_ctd_dosta(ctd_flat, dosta_flat):
    """Outer-merge flattened CTD and DOSTA tables and attach variable attributes."""
    merged = pd.merge(ctd_flat, dosta_flat, left_index=True, right_index=True, how="outer")
    merged.index.name = "time"
    ds = merged.to_xarray()

    for var in ds.data_vars:
        if var.endswith("_modeled_flag"):
            ds[var].attrs.update(FLAG_ATTRS)
            continue
        for stat, stat_attrs in STAT_ATTRS.items():
            if var.endswith(f"_{stat}"):
                base_name = var[: -(len(stat) + 1)]
                ds[var].attrs.update(VARIABLE_ATTRS.get(base_name, {}))
                ds[var].attrs.update(stat_attrs)
                break

    return ds


def save_merged_dataset(ds, path):
    """Write a merged dataset to netCDF, with a couple of dataset-level attrs."""
    ds.attrs.setdefault("source", "OOI GI01SUMO near-surface instrument frame (CTD + DOSTA)")
    ds.attrs.setdefault("processing", "15-minute burst median/MAD; gaps filled with linear trend + harmonic fit")
    ds.to_netcdf(path)

Note that both CTD and DOSTA go through flag_and_flatten now, which is what the original notebook was missing:

PYTHON 

ctd_flat = flag_and_flatten(ctd_stats, was_filled,
                             ["sea_water_temperature", "sea_water_practical_salinity", "sea_water_pressure"])
dosta_flat = flag_and_flatten(dosta_stats, dosta_was_filled,
                               ["oxygen_concentration_corrected"])
merged_ds = merge_ctd_dosta(ctd_flat, dosta_flat)
save_merged_dataset(merged_ds, "gi01sumo_merged.nc")

Turning modules into a package

Two or three loose .py files next to a notebook works for now, but as soon as you want to share this, import this code into a different notebook in a different folder or repo or, even better, put it on PyPi and allow others to pip install your code, you will need to create a package: a directory with an __init__.py file.

mooring_tools/
├── __init__.py
├── stats.py
├── plotting.py
└── merge.py

__init__.py can be empty: its presence is what tells Python mooring_tools is an importable package, and modules inside it are addressed as mooring_tools.stats, mooring_tools.plotting, mooring_tools.merge:

PYTHON 

from mooring_tools.stats import resample_burst_stats, fill_harmonic_gaps
from mooring_tools.plotting import plot_raw_timeseries, plot_burst_stats
from mooring_tools.merge import flag_and_flatten, merge_ctd_dosta, save_merged_dataset
Callout

Module vs. package

A module is a single .py file. A package is a directory of modules with an __init__.py. stats.py is a module; mooring_tools/ (once it has an __init__.py) is a package containing three modules.

Challenge

Challenge 4: Notebook to package

Turn stats.py, plotting.py, and merge.py into a mooring_tools/ package (add the __init__.py), and update first_analysis.ipynb’s import cell to pull from it instead.

Wrap-up

By the end of this section, you should have a mooring_tools/ package with three modules, stats, plotting, merge, and a notebook that imports from it with autoreload active. This package is what the next section will turn into a repository with a managed environment, READMEs, and setup files.

Key Points
  • A module is a single .py file; a package is a directory of modules with an __init__.py
  • %load_ext autoreload + %autoreload 2 lets a notebook pick up edits to your own modules without a kernel restart
  • Python checks your notebook’s own directory before the standard library, such that a module named random.py or csv.py can silently break unrelated imports elsewhere in your code
  • Not everything needs a module. Modularize logic that’s substantial or repeated

Content from Building a Repository and Managing Environments

Last updated on 2026-11-06 | Edit this page

Estimated time: 140 minutes

Overview

Questions

  • What actually belongs in a code repository?
  • Why does each project need its own environment?
  • How do I turn mooring_tools/ into something pip install-able?

Objectives

  • Create a project with uv and lay out a repository with a src/ package, notebooks, and data kept separate
  • Add dependencies to a project and explain what pyproject.toml and uv.lock each record
  • Run a Jupyter notebook inside the project’s environment and import your own package from it
  • Explain why a src/ layout catches packaging bugs that a flat layout hides

What goes in a repo

By the end of Block 2, mooring_tools/ is three modules and an __init__.py, sitting next to first_analysis.ipynb. That’s enough to import from this notebook. It’s not enough to share with a colleague, or use from a notebook anywhere else on your machine. For that, the package needs a proper home:

gi01sumo-mooring-tools/
├── .gitignore
├── .python-version
├── LICENSE
├── README.md
├── pyproject.toml
├── uv.lock
├── src/
│   └── mooring_tools/
│       ├── __init__.py
│       ├── stats.py
│       ├── plotting.py
│       └── merge.py
├── notebooks/
│   └── first_analysis.ipynb
└── data/
    ├── gi01sumo_nsif_ctd.nc
    └── gi01sumo_nsif_dosta.nc

A few things to note about the layout:

  • src/mooring_tools/, not just mooring_tools/ as the repo root.
  • notebooks/ and data/ are separate from the package. The package is code meant to be imported; the notebook is one particular analysis that uses it.
  • pyproject.toml and uv.lock describe the project and the exact packages it was run with. Both are committed. We’ll build them up over this block.
  • .python-version records which version of Python the project is developed with.
  • .gitignore keeps generated/environment files out of version control, such as the venv folder, __pycache__/, and Jupyter checkpoint files.
  • LICENSE states what others are allowed to do with the code.

Creating the project with uv

You could create all of that by hand. Instead, we’ll let uv create the skeleton. uv is one tool that replaces several you may have used separately: it installs Python itself, creates virtual environments, installs packages, and records what it installed. Confirm it’s available:

BASH 

uv --version

Then create the project:

BASH 

uv init --lib --name mooring-tools gi01sumo-mooring-tools
  • --lib asks for a library layout: a src/ folder with a package inside it, ready to be imported by other code.
  • --name mooring-tools is the name the project is installed under. The folder is named gi01sumo-mooring-tools, so we give the name explicitly.
  • uv also runs git init for you, so the folder is already a repository.
Callout

Two names for one thing

The project is called mooring-tools (with a hyphen) but you import it as mooring_tools (with an underscore). The first is the name used to install and publish it; the second is a Python identifier, so it can’t contain a hyphen. uv turns one into the other automatically. If you see ModuleNotFoundError: No module named 'mooring-tools', that’s the mix-up.

Challenge

Challenge 1: Lay out the repository

  1. Run the uv init command above, then cd into the new folder and look at what it created (ls -a, or dir on Windows).
  2. uv init puts a placeholder hello() function in src/mooring_tools/__init__.py. Replace that file with your __init__.py from Block 2, and move stats.py, plotting.py, and merge.py into src/mooring_tools/.
  3. Create notebooks/ and data/, and move first_analysis.ipynb and the two .nc files into them. (Update the data paths in the notebook to ../data/....)
  4. Add a line data/ to the .gitignore file that uv init created, so the large data files are never committed.

Why src/, specifically

Compare two layouts, both without the package actually installed:

BASH 

# flat layout: mooring_tools/ directly at the repo root
cd flat-layout-repo
uv run --no-project python -c "import mooring_tools"

OUTPUT 

(imports successfully — from mooring_tools/__init__.py right there in the folder)

BASH 

# src layout: mooring_tools/ inside src/
cd src-layout-repo
uv run --no-project python -c "import mooring_tools"

OUTPUT 

ModuleNotFoundError: No module named 'mooring_tools'

(--no-project tells uv not to build an environment from a pyproject.toml, so that this really is “without the package installed.”)

With a flat layout, Python finds the package because your current directory is always searched first. That means import mooring_tools can silently “work” from inside the repo even if the package was never actually installed, or if the install is broken. The bug only shows up later, when someone tries to use the package from outside the repo and it’s nowhere to be found. With src/, that failure happens immediately.

Environments

Every Python installation on your machine has one set of installed packages by default. If two projects need different, incompatible versions of the same library, installing globally means only one project can work at a time. A virtual environment gives each project its own isolated set of installed packages.

With uv you don’t create or activate one yourself. The first time you add a dependency (next section), uv creates a .venv folder in the project and installs into it. Every uv run ... command then uses that environment automatically, from the project folder or any folder inside it. There is no activate step to forget, and no way to install into the wrong environment by accident.

Callout

The .venv folder is disposable

Nothing in .venv is precious. If it ever gets into a confusing state, delete the folder and run uv sync. uv will rebuild it from the project files. That is also why .venv is in .gitignore: it’s machine-specific, and fully recreatable.

Describing the Project: pyproject.toml

pyproject.toml, at the repo root, is the file that describes your project to uv and to every other Python packaging tool. It is written in TOML (Tom’s Obvious Minimal Language), a simple key = value format. Open the one uv init created:

TOML 

[project]
name = "mooring-tools"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
authors = [
    { name = "Your Name", email = "you@example.org" }
]
requires-python = ">=3.14"
dependencies = []

[build-system]
requires = ["uv_build>=0.11.32,<0.12.0"]
build-backend = "uv_build"

Two things need your attention before we go on. (Your version numbers and author will differ.)

  • requires-python defaults to the version of Python uv found on your machine. If yours is 3.14, a colleague on 3.11 would be told they cannot install your package, for no real reason. Our code runs fine on older versions, so relax it to ">=3.10".
  • description should say what the package does. Make it "Burst-statistics, plotting, and merge utilities for OOI GI01SUMO CTD/DOSTA data".

The [build-system] table tells tools how to turn src/ into an installable package. uv fills it in; you don’t need to edit it.

Adding dependencies

Our code needs xarray, netCDF4, pandas, numpy, and matplotlib. Add them with:

BASH 

uv add xarray netcdf4 pandas numpy matplotlib

That one command does three things:

  1. creates .venv if it doesn’t exist yet,
  2. installs the packages (and everything they depend on) into it
  3. records them in two places.

Look at pyproject.toml again:

TOML 

dependencies = [
    "matplotlib>=3.10.9",
    "netcdf4>=1.7.3",
    "numpy>=2.2.6",
    "pandas>=2.3.3",
    "xarray>=2025.6.1",
]

uv has written a minimum version for each one: the version it installed, or the oldest version that works with the oldest Python you allow. This says what your code needs to run at all, loose enough not to fight with other projects.

For notebooks, add JupyterLab as a development dependency. This is something you need to work on the project, but that someone else who is only using mooring_tools does not:

BASH 

uv add --dev jupyterlab

TOML 

[dependency-groups]
dev = [
    "jupyterlab>=4.6.5",
]
Challenge

Challenge 2: Set up the project’s dependencies

In the project folder, fix requires-python and description in pyproject.toml, then run both uv add commands above. Open pyproject.toml and confirm the dependencies and [dependency-groups] tables were filled in for you. Which new files and folders has uv created?

.venv/ (the environment, git-ignored) and uv.lock (the exact record of what was installed, which we look at shortly). pyproject.toml itself was edited by uv add.

Your minimum versions will differ from the ones above, because they reflect whatever was current when you ran the command.

Running code in the project

To run a command inside the project’s environment, put uv run in front of it:

BASH 

uv run python -c "import mooring_tools; print(mooring_tools.__file__)"

The printed path should point into your repo’s src/mooring_tools/. uv installs your own package into .venv in editable mode: Python imports it directly from your src/ files rather than from a copy, so edits you make take effect immediately without reinstalling (This is the persistent cousin of Block 2’s %autoreload).

Now start JupyterLab inside the project:

BASH 

uv run jupyter lab

Open notebooks/first_analysis.ipynb. The notebook’s kernel is the project’s .venv Python, so import mooring_tools simply works.

Challenge

Challenge 3: Use the package from a notebook

Launch JupyterLab with uv run jupyter lab from the repo root. In a new notebook inside notebooks/, run:

PYTHON 

import mooring_tools
from mooring_tools.stats import resample_burst_stats
print(mooring_tools.__file__)

Then change the body of a function in src/mooring_tools/stats.py (for example, the default freq), and call it again. Do you need to restart the kernel? (Remember what %autoreload 2 did in Block 2.)

The printed path points at your repo’s src/mooring_tools/. Whether the edit shows up without a restart depends on whether you ran %load_ext autoreload and %autoreload 2: the editable install makes the package findable from anywhere, while autoreload makes already-imported code refresh in a running kernel.

If import mooring_tools raises ModuleNotFoundError, the notebook is almost certainly running in a different Python than the project’s. Check that you started JupyterLab with uv run jupyter lab, not with a jupyter installed elsewhere on your machine.

Package management: the lock file

pyproject.toml has loose, minimum versions. For exact reproducibility look at the other file uv created: uv.lock. It records the exact version of every package in the environment, including the ones you never asked for directly (xarray pulls in packaging, pandas pulls in python-dateutil, and so on):

[[package]]
name = "xarray"
version = "2026.9.0"
...

(You don’t write or edit uv.lock; uv maintains it every time you uv add, uv remove, or uv lock.)

The difference between pyproject.toml and uv.lock is that the former declares what is needed to run your code, whereas the latter records exactly what was installed, version by version, the last time uv resolved your dependencies. A colleague reproducing your result wants the second one. The lock file covers every platform and Python version your project supports, so a lock file made on a Mac also works on Windows and Linux.

Commit both files. Then reproducing your environment on a new machine is:

BASH 

git clone <repo-url>
cd gi01sumo-mooring-tools
uv sync

uv sync creates .venv and installs exactly what uv.lock says. If the right Python isn’t installed on the new machine, uv downloads it too.

Challenge

Challenge 4: Reproduce your environment

Commit your work (git add . then git commit). Then, in a different folder, clone your own repo and run uv sync. Compare the output of uv run python -c "import xarray; print(xarray.__version__)" in the clone and in your original. What do you expect?

When that works, delete the .venv folder in your original project and run uv sync again. How long does it take compared to the first time?

The versions match, because both environments were built from the same uv.lock.

Rebuilding is much faster the second time: uv keeps a cache of everything it has already downloaded, so a rebuild is mostly linking files from that cache into a new .venv.

Changing your mind later

You’ll rarely get the dependency list right the first time. The same commands work whenever you need them:

BASH 

uv add scipy           # new dependency
uv remove scipy        # no longer needed
uv lock --upgrade      # move to the newest versions allowed, then...
uv sync                # ...bring .venv in line with uv.lock

Each of these edits pyproject.toml and/or uv.lock for you, so the two files and the environment stay consistent. Commit the changed files together.

Callout

.gitignore and what to commit

uv init already wrote a .gitignore containing .venv, __pycache__/, *.py[oc], build/, dist/, wheels/, and *.egg-info. Add one line for Jupyter:

.ipynb_checkpoints/

Commit: pyproject.toml, uv.lock, .python-version, README.md, LICENSE, and everything under src/ and notebooks/. Don’t commit .venv/.

We already added data/ to .gitignore. Our files are far larger than GitHub accepts, and large or sensitive data in general rarely belongs in a code repository. Instead, the README says where to get the data.

Callout

What about conda?

Many scientists use conda or mamba, and for good reason: it can install non-Python libraries (compilers, GDAL, and so on) that uv cannot. Every package in this workshop is available as a pre-built wheel for macOS, Windows, and Linux, so uv is enough here. If a future project needs non-Python system libraries, conda remains a sensible choice for that project. The ideas in this block — one environment per project, minimum versions in the project file, exact versions in a lock file, commit both — apply either way.

Two practical points. First, uv must be on each learner’s PATH in a new terminal window; “command not found: uv” right after installing is almost always an old terminal. Second, the first uv add downloads several hundred megabytes (xarray, pandas, numpy, matplotlib, netCDF4, and JupyterLab with its dependencies). On workshop wifi, have learners run Challenge 2 at the same time rather than one at a time, or pre-warm uv’s cache by asking them to run the commands from learners/setup.md before the session.

Wrap-up

By the end of this block, mooring_tools is a real, installable package: src/ layout, a pyproject.toml declaring what it needs, a uv.lock recording exactly what was installed, and an environment that uv builds and uses without any activation. A colleague can reproduce all of it with git clone and uv sync. Block 4 adds the README and docstring pass that makes this repo usable by someone who isn’t you.

Key Points
  • Separate the installable package (src/mooring_tools/) from notebooks and data
  • A src/ layout turns “the package isn’t actually installed” from a silent bug into a raised error
  • uv init --lib creates the project skeleton; uv add installs a dependency and records it in pyproject.toml and uv.lock
  • uv run runs a command in the project’s environment with nothing to activate; the .venv folder is disposable and never committed
  • pyproject.toml declares what your code needs (minimum versions); uv.lock records exactly what’s installed: commit both, and reproduction is git clone plus uv sync

Content from Documentation and Reuse

Last updated on 2026-11-06 | Edit this page

Estimated time: 140 minutes

Overview

Questions

  • What does a README need to contain to actually be useful?
  • How do docstrings work together across a whole project, not just one function?
  • How do I extend this package for new work without breaking what already works?

Objectives

  • Write a README that lets a new user install and run the project
  • Audit and complete docstrings and type hints across a small project
  • Extend the package to support a new variable without editing existing functions

Writing a README

A README’s job is to let someone who isn’t you install the project and run the code successfully, without opening an issue or emailing you first. For mooring_tools, that means including the following: what it does, how to install it, an example, and where the data comes from.

Challenge

Challenge 1: Draft the README

Write README.md for mooring-tools, covering: what the package does, install instructions (using uv sync), and a usage example a new user could actually run against data/gi01sumo_nsif_ctd.nc.

MARKDOWN 

# mooring tools

Burst-statistics, plotting, and merging utilities for OOI CTD/DOSTA
data (Global Irminger Sea Array, near-surface instrument frame).

## What this does

Many sensors on OOI moorings sample in short bursts. This
package takes raw CTD (temperature, salinity, pressure, density) and DOSTA
(dissolved oxygen) records and:

- computes robust burst-median statistics (median and median absolute
  deviation) for each variable
- fills gaps in the burst-median series with a linear-trend and harmonic fit
- plots raw vs. burst-median data, and observed vs. modeled series
- merges CTD and DOSTA into one annotated, saveable netCDF dataset, flagging
  which values are observed vs. gap-filled

## Install

This project uses [uv](https://docs.astral.sh/uv/) to manage Python and its
packages. [Install uv](https://docs.astral.sh/uv/getting-started/installation/),
then:

```bash
git clone <this-repo-url>
cd gi01sumo-mooring-tools
uv sync
```

`uv sync` creates a `.venv` folder and installs the exact package versions
recorded in `uv.lock`, plus `mooring_tools` itself. Nothing needs to be
activated: put `uv run` in front of any command to run it in this
environment.

To work in a notebook:

```bash
uv run jupyter lab
```

## Usage

```python
import numpy as np
import xarray as xr
from mooring_tools.stats import resample_burst_stats

ctd = xr.open_dataset("data/gi01sumo_nsif_ctd.nc")
scalar_vars = [v for v in ctd.data_vars if ctd[v].dims == ("time",) and np.issubdtype(ctd[v].dtype, np.number)]
ctd_stats = resample_burst_stats(ctd[scalar_vars].to_dataframe())
```

See `notebooks/first_analysis.ipynb` for a full worked example.

## Data

The `data/` folder is not part of this repository because the files are
too large. It should contain `gi01sumo_nsif_ctd.nc` and
`gi01sumo_nsif_dosta.nc`, a full deployment of real OOI GI01SUMO data. Get
them from the workshop's starter-files download (see the lesson's Getting
Started page), or download similar data from the OOI Data Explorer.

## License

MIT — see `LICENSE`.

Docstrings at the project level

Block 1 introduced docstrings at the level of a single function, with fill_harmonic_gaps as the model. Now that there are three modules, it’s worth revisiting any functions which we didn’t finish/complete the docstrings for.

Open merge.py. Compare its docstrings to stats.py’s:

PYTHON 

# merge.py, as it stands after Block 2
def flag_and_flatten(stats, was_filled, keep_vars):
    """Restrict to `keep_vars`, add a modeled_flag per variable, and flatten
    the 2-level (variable, stat) columns into single flat names."""
    ...

No type hints, no Parameters/Returns sections, only a one-line summary only. Meanwhile stats.py’s fill_harmonic_gaps has both. This is a common issue that crops up during development and exploration.

Challenge

Challenge 2: Docstring and type-hint audit

Go through plotting.py and merge.py and bring every function up to the same standard as stats.py: type hints on the signature, and a full NumPy-style docstring (summary, Parameters, Returns).

PYTHON 

# merge.py, after the audit
import pandas as pd
import xarray as xr

# Attributes that describe each variable
VARIABLE_ATTRS = {
    "sea_water_temperature": {"units": "degrees_Celsius", "long_name": "Sea Water Temperature"},
    "sea_water_practical_salinity": {"units": "1", "long_name": "Sea Water Practical Salinity"},
    "sea_water_pressure": {"units": "dbar", "long_name": "Sea Water Pressure"},
    "oxygen_concentration_corrected": {"units": "umol kg-1", "long_name": "Corrected Dissolved Oxygen Concentration"},
}

# Attributes that describe the statistic that was computed
STAT_ATTRS = {
    "median": {"comment": "Median of each 15-minute burst of samples"},
    "mad": {"comment": "Median absolute deviation of each 15-minute burst of samples"},
}

# Attributes of the modeled flag, which is not in the units of the variable
FLAG_ATTRS = {"units": "1", "comment": "Indicates whether a value was observed (0) or modeled (1)"}


def flag_and_flatten(stats: pd.DataFrame, was_filled: pd.DataFrame, keep_vars: list[str]) -> pd.DataFrame:
    """Restrict to `keep_vars`, add a modeled_flag per variable, and flatten
    the 2-level (variable, stat) columns into single flat names.

    Parameters
    ----------
    stats : pandas.DataFrame
        Output of `resample_burst_stats` (2-level column MultiIndex of
        (variable, stat)).
    was_filled : pandas.DataFrame
        Boolean frame with one column per variable, True where that burst
        was gap-filled rather than observed.
    keep_vars : list[str]
        Which variables (first-level column names) to keep.

    Returns
    -------
    pandas.DataFrame
        Flat columns like "sea_water_temperature_median",
        "sea_water_temperature_mad", "sea_water_temperature_modeled_flag".
    """
    stats = stats.loc[:, keep_vars].copy()
    for var in keep_vars:
        stats[(var, "modeled_flag")] = was_filled[var].astype(int)

    stats.columns = ["_".join(col).strip() for col in stats.columns.values]
    return stats


def merge_ctd_dosta(ctd_flat: pd.DataFrame, dosta_flat: pd.DataFrame) -> xr.Dataset:
    """Outer-merge flattened CTD and DOSTA tables and attach variable attributes.

    Parameters
    ----------
    ctd_flat, dosta_flat : pandas.DataFrame
        Output of `flag_and_flatten` for each instrument. Both must already
        be flattened to single-level columns -- merging one flattened and
        one still-MultiIndex table raises a `MergeError`.

    Returns
    -------
    xarray.Dataset
        Merged on the shared burst-median time index. Each data variable gets
        the `units` and `long_name` listed for it in `VARIABLE_ATTRS`, plus a
        comment describing the statistic. To support a new variable, add it to
        `VARIABLE_ATTRS` and include it in the `keep_vars` passed to
        `flag_and_flatten` -- no changes needed here.
    """
    merged = pd.merge(ctd_flat, dosta_flat, left_index=True, right_index=True, how="outer")
    merged.index.name = "time"
    ds = merged.to_xarray()

    for var in ds.data_vars:
        if var.endswith("_modeled_flag"):
            ds[var].attrs.update(FLAG_ATTRS)
            continue
        for stat, stat_attrs in STAT_ATTRS.items():
            if var.endswith(f"_{stat}"):
                base_name = var[: -(len(stat) + 1)]
                ds[var].attrs.update(VARIABLE_ATTRS.get(base_name, {}))
                ds[var].attrs.update(stat_attrs)
                break

    return ds


def save_merged_dataset(ds: xr.Dataset, path: str) -> None:
    """Write a merged dataset to netCDF, with a couple of dataset-level attrs.

    Parameters
    ----------
    ds : xarray.Dataset
        Typically the output of `merge_ctd_dosta`.
    path : str
        Output file path.
    """
    ds.attrs.setdefault("source", "OOI GI01SUMO near-surface instrument frame (CTD + DOSTA)")
    ds.attrs.setdefault("processing", "15-minute burst median/MAD; gaps filled with linear trend + harmonic fit")
    ds.to_netcdf(path)

Worth naming explicitly: merge_ctd_dosta’s docstring now states the “add a new variable” extension path directly in its Returns section. This is documenting a design property Block 2’s callout and this block’s reuse challenge both rely on. A good docstring doesn’t just describe current behavior; it tells the next person (including future-you) what’s safe to change without reading the implementation.

Reusing and updating the repo

The real test of a well-organized package is what happens when the scientific scope changes. Here’s an example: add sea_water_density as a variable to data/gi01sumo_nsif_ctd.nc results. resample_burst_stats has been computing burst statistics for it since Block 1, since it works generically over every numeric column. But merge.py’s keep_vars lists (in Block 2’s Challenge 3, and VARIABLE_ATTRS) never included it, so it’s never made it into the final dataset.

Challenge

Challenge 3: Add sea_water_density to the dataset

Add sea_water_density as a supported merged-output variable. You should not need to edit resample_burst_stats, flag_and_flatten, or merge_ctd_dosta at all.

PYTHON 

from mooring_tools.merge import VARIABLE_ATTRS

VARIABLE_ATTRS["sea_water_density"] = {"units": "kg m-3", "long_name": "Sea Water Density"}

keep_vars = ["sea_water_temperature", "sea_water_practical_salinity",
             "sea_water_pressure", "sea_water_density"]
ctd_flat = flag_and_flatten(ctd_stats, was_filled_ctd, keep_vars)

resample_burst_stats already computed statistics for every numeric column in the input dataframe, including density, since it was never told to look for specific variable names. flag_and_flatten and merge_ctd_dosta are equally generic; they only needed VARIABLE_ATTRS extended and a longer keep_vars list at the call site. If this doesn’t work you may need to revisit the functions from Block 2.

Versioning changes

As the package evolves, with new variables, bug fixes, new functions, etc., here are how you should handle these changes:

  • Bump the version in pyproject.toml (version = "0.1.0" → "0.2.0") when you make a change someone else relying on the package would care about. uv version --bump minor edits the version line in pyproject.toml for you (use patch for bug fixes, major for breaking changes); uv version on its own prints the current version.
  • Tag the commit in git (git tag v0.2.0) so “the version I ran this analysis with” is always recoverable later, even after further changes.

Which number to bump follows a widely used convention called semantic versioning: version numbers look like MAJOR.MINOR.PATCH.

  • Patch (0.1.0 → 0.1.1, uv version --bump patch): bug fixes that don’t change how anyone uses the package.
  • Minor (0.1.0 → 0.2.0, uv version --bump minor): new features that don’t break existing code, such as adding a new function or supporting density.
  • Major (0.2.0 → 1.0.0, uv version --bump major): changes that can break code that used the old version, such as renaming a function or changing what it returns.

Commit the changed pyproject.toml and uv.lock together with the tag, so the version number and the recorded environment never disagree.

Picking the work back up months later is the same two commands a new user runs: uv sync to rebuild .venv from uv.lock, then uv run jupyter lab.

Wrap-up

Across all four blocks: a repeated block became a function (Block 1), the function joined others in a package (Block 2), the package became installable with a managed environment (Block 3), and now it’s documented well, and generic, to extend for new scientific questions without touching working code (Block 4).

Key Points
  • A README’s job is to let someone else install and run the project without asking you
  • Docstring consistency matters more as a project grows past one file
  • A well-documented function’s docstring tells the reader what’s safe to extend, not just what the function currently does
  • Reusing a repo well means extending generic functions with new data/config, not copy-pasting or editing working code
  • Bumping a version number and tagging the commit is enough to keep “what did I run this with” answerable later