Configuration of the JULES land surface model¶

import marimo as mo

JULES (Joint UK Land Environment Simulator) is a community land surface model developed by the UK Centre for Ecology & Hydrology (UKCEH) and the Met Office. It simulates the exchange of energy, water, momentum, carbon and other trace gases between the land surface and the atmosphere.¹ ²

Documentation is available at https://jules-lsm.github.io.

Note

This example was developed and tested using version 7.9 of JULES. It's possible that the way JULES is configured will change in future.

from os import PathLike
from pathlib import Path

import dirconf

Namelists¶

Fortran namelists are a standard mechanism for passing configuration parameters to a program. They consist of named blocks (&block_name ... /) containing key-value pairs. JULES uses roughly 30 separate namelist files to organize its many parameters by physical process (e.g., soil, snow, vegetation) and model component (e.g., output, grid, timesteps).

The JULES executable is run by passing the path to a directory containing these namelists as its sole positional argument,

jules.exe /path/to/namelists/

where namelists/ must contain all of the required namelists files,

namelists/
├── ancillaries.nml
├── crop_params.nml
├── drive.nml
├── fire.nml
├── imogen.nml
├── initial_conditions.nml
├── jules_deposition.nml
├── jules_hydrology.nml
├── jules_irrig.nml
├── jules_prnt_control.nml
├── jules_radiation.nml
├── jules_rivers.nml
├── jules_snow.nml
├── jules_soil_biogeochem.nml
├── jules_soil.nml
├── jules_surface.nml
├── jules_surface_types.nml
├── jules_vegetation.nml
├── jules_water_resources.nml
├── model_environment.nml
├── model_grid.nml
├── nveg_params.nml
├── output.nml
├── pft_params.nml
├── prescribed_data.nml
├── science_fixes.nml
├── timesteps.nml
├── triffid_params.nml
└── urban.nml

Note

There is no freedom to use different file names for the namelist files; they must be present exactly as specified above.

Namelist file handler¶

We will make use of the f90nml Python package³ for reading and writing namelist files.

import json

import f90nml

class NamelistFileHandler:
    def read(self, path: str | PathLike) -> dict:
        """Read a Fortran namelist file and return a dict of its contents."""
        data = f90nml.read(path)

        # f90nml converts namelists to OrderedDict by default. However, since Python 3.7
        # regular dicts guarantee insertion order. We will use json to cast to regular
        # dict here, purely because they pretty-print whereas OrderedDict doesn't.
        return json.loads(json.dumps(data.todict()))

    def write(
        self, path: str | PathLike, data: dict, *, overwrite_ok: bool = False
    ) -> None:
        """Write a dict to a Fortran namelist file."""
        f90nml.write(data, path, force=overwrite_ok)

Namelists directory config¶

We now construct a DirConfig-based Handler for a namelists directory, in which each Node corresponds to a single .nml file with a fixed path and handler.

_jules_namelists = [
    "ancillaries",
    "crop_params",
    "drive",
    "fire",
    "imogen",
    "initial_conditions",
    "jules_deposition",
    "jules_hydrology",
    "jules_irrig",
    "jules_prnt_control",
    "jules_radiation",
    "jules_rivers",
    "jules_snow",
    "jules_soil_biogeochem",
    "jules_soil",
    "jules_surface",
    "jules_surface_types",
    "jules_vegetation",
    "jules_water_resources",
    "model_environment",
    "model_grid",
    "nveg_params",
    "output",
    "pft_params",
    "prescribed_data",
    "science_fixes",
    "timesteps",
    "triffid_params",
    "urban",
]

NamelistConfig = dirconf.make_dirconfig(
    cls_name="NamelistConfig",
    spec={
        name: {"path": f"{name}.nml", "handler": NamelistFileHandler}
        for name in _jules_namelists
    },
)

This produces a subclass of DirConfig which is instantiated without any arguments.

namelist_config = NamelistConfig()

print(namelist_config)

dirconf.config.NamelistConfig
├──ancillaries -------------- (path='ancillaries.nml', handler=NamelistFileHandler)
├──crop_params -------------- (path='crop_params.nml', handler=NamelistFileHandler)
├──drive -------------------- (path='drive.nml', handler=NamelistFileHandler)
├──fire --------------------- (path='fire.nml', handler=NamelistFileHandler)
├──imogen ------------------- (path='imogen.nml', handler=NamelistFileHandler)
├──initial_conditions ------- (path='initial_conditions.nml', handler=NamelistFileHandler)
├──jules_deposition --------- (path='jules_deposition.nml', handler=NamelistFileHandler)
├──jules_hydrology ---------- (path='jules_hydrology.nml', handler=NamelistFileHandler)
├──jules_irrig -------------- (path='jules_irrig.nml', handler=NamelistFileHandler)
├──jules_prnt_control ------- (path='jules_prnt_control.nml', handler=NamelistFileHandler)
├──jules_radiation ---------- (path='jules_radiation.nml', handler=NamelistFileHandler)
├──jules_rivers ------------- (path='jules_rivers.nml', handler=NamelistFileHandler)
├──jules_snow --------------- (path='jules_snow.nml', handler=NamelistFileHandler)
├──jules_soil_biogeochem ---- (path='jules_soil_biogeochem.nml', handler=NamelistFileHandler)
├──jules_soil --------------- (path='jules_soil.nml', handler=NamelistFileHandler)
├──jules_surface ------------ (path='jules_surface.nml', handler=NamelistFileHandler)
├──jules_surface_types ------ (path='jules_surface_types.nml', handler=NamelistFileHandler)
├──jules_vegetation --------- (path='jules_vegetation.nml', handler=NamelistFileHandler)
├──jules_water_resources ---- (path='jules_water_resources.nml', handler=NamelistFileHandler)
├──model_environment -------- (path='model_environment.nml', handler=NamelistFileHandler)
├──model_grid --------------- (path='model_grid.nml', handler=NamelistFileHandler)
├──nveg_params -------------- (path='nveg_params.nml', handler=NamelistFileHandler)
├──output ------------------- (path='output.nml', handler=NamelistFileHandler)
├──pft_params --------------- (path='pft_params.nml', handler=NamelistFileHandler)
├──prescribed_data ---------- (path='prescribed_data.nml', handler=NamelistFileHandler)
├──science_fixes ------------ (path='science_fixes.nml', handler=NamelistFileHandler)
├──timesteps ---------------- (path='timesteps.nml', handler=NamelistFileHandler)
├──triffid_params ----------- (path='triffid_params.nml', handler=NamelistFileHandler)
└──urban -------------------- (path='urban.nml', handler=NamelistFileHandler)

Dry-run¶

We can now use this handler to read an entire namelists directory into a Python dict.

Let's do this now, and check all expected keys are present.

namelist_config_dict = namelist_config.read("config/namelists")

list(namelist_config_dict.keys())

['ancillaries',
 'crop_params',
 'drive',
 'fire',
 'imogen',
 'initial_conditions',
 'jules_deposition',
 'jules_hydrology',
 'jules_irrig',
 'jules_prnt_control',
 'jules_radiation',
 'jules_rivers',
 'jules_snow',
 'jules_soil_biogeochem',
 'jules_soil',
 'jules_surface',
 'jules_surface_types',
 'jules_vegetation',
 'jules_water_resources',
 'model_environment',
 'model_grid',
 'nveg_params',
 'output',
 'pft_params',
 'prescribed_data',
 'science_fixes',
 'timesteps',
 'triffid_params',
 'urban']

The drive namelist controls the meteorological forcing data: start and end dates, time step, variable names, and the path to the driving data file.

namelist_config_dict["drive"]

{'jules_drive': {'data_end': '1997-12-31 23:00:00',
                 'data_period': 1800,
                 'data_start': '1996-12-31 23:00:00',
                 'diff_frac_const': 0.4,
                 'file': 'inputs/Loobos_1997.dat',
                 'interp': ['nf', 'nf', 'nf', 'nf', 'nf', 'nf', 'nf', 'nf'],
                 'nvars': 8,
                 't_for_con_rain': 293.15,
                 'var': ['sw_down',
                         'lw_down',
                         'tot_rain',
                         'tot_snow',
                         't',
                         'wind',
                         'pstar',
                         'q'],
                 'z1_tq_in': 10.0,
                 'z1_uv_in': 10.0}}

Input data¶

Ascii file handler¶

In addition to namelist files, JULES requires input data such as meteorological forcing ("driving") data, initial conditions, and spatial maps like tile fractions. These are typically provided as plain-text ASCII files or in the NetCDF format.

Initial conditions specify the starting state of soil moisture, temperature, and other prognostic variables at each grid point.
Tile fractions define the fractional coverage of each surface type (e.g., broadleaf trees, C3 grass, urban) within a grid cell.
Driving data contains the time series of meteorological variables (temperature, precipitation, radiation, etc.) that force the model.

We can use numpy to read and write floating point ASCII data.

from typing import TypedDict

import numpy

@dirconf.filter(write=lambda path, data, **_: not path.is_absolute())
@dirconf.filter_missing()
class AsciiFileHandler:
    class AsciiData(TypedDict):
        values: numpy.ndarray
        comment: str

    def read(self, path: str | PathLike) -> AsciiData:
        comment_lines = []
        num_lines = 0
        with open(path) as file:
            for line in file:
                line = line.strip()
                if line.startswith(("#", "!")):
                    comment_lines.append(line)
                    continue
                elif line:  # non-comment data line
                    num_lines = num_lines + 1
                    if num_lines > 1:
                        break
        comment = "\n".join(comment_lines)  # join all comment header lines
        values = numpy.loadtxt(str(path), comments=("#", "!"))
        if num_lines == 1:
            assert values.ndim == 1  # we just need to know if it's >1
            values = values.reshape(1, -1)
        return self.AsciiData(values=values, comment=comment)

    def write(
        self, path: str | PathLike, data: AsciiData, *, overwrite_ok: bool = False
    ) -> None:
        numpy.savetxt(
            str(path),
            data["values"],
            fmt="%.5f",
            header=data["comment"],
            comments="#",
        )
        # NOTE: Unfortunately numpy.loadtxt/savetxt does not correctly round-trip
        # single-row data. We need to catch it here and add an extra dimension.

NetCDF file handler¶

ASCII files (.dat, .txt) are simple and human-readable, making them suitable for small datasets like initial conditions or tile fraction maps. However, for large multidimensional time series such as meteorological driving data, the NetCDF format is strongly preferred: it is compact, self-describing, and supports metadata and coordinate labels natively.

We will use xarray to read and write input data in the netCDF format.

import xarray

@dirconf.filter(
    read=lambda path: not path.is_absolute(),
    write=lambda path, data, **_: not path.is_absolute(),
)
@dirconf.filter_missing()
class NetcdfFileHandler:
    def read(self, path: str | PathLike) -> xarray.Dataset:
        dataset = xarray.load_dataset(path)
        return dataset

    def write(
        self,
        path: str | PathLike,
        data: xarray.Dataset,
        *,
        overwrite_ok: bool = False,
    ) -> None:
        if not overwrite_ok and Path(path).is_file():
            raise FileExistsError(f"There is already a file at '{path}'")
        data.to_netcdf(path)

Putting it all together¶

We have now defined handlers for three file types: namelists (via f90nml), ASCII data (via numpy), and NetCDF (via xarray). Before composing them into a unified configuration, we register the ASCII and NetCDF handlers with dirconf so they can be selected automatically by file extension.

The @dirconf.filter decorator attaches predicates that determine when a handler is applicable (e.g., based on path properties). The @dirconf.filter_missing() decorator ensures that missing files are handled gracefully rather than raising an error immediately.

dirconf.register_handler("ascii", AsciiFileHandler, [".txt", ".dat", ".asc"])
dirconf.register_handler("netcdf", NetcdfFileHandler, [".nc", ".cdf"])

InputFilesConfig = dirconf.make_dirconfig(
    cls_name="InputFilesConfig",
    spec={
        "initial_conditions": {
            "handler": AsciiFileHandler,
        },
        "tile_fractions": {
            "handler": AsciiFileHandler,
        },
        "driving_data": {},  # handler resolved by file extension at runtime
    },
)

JulesConfig = dirconf.make_dirconfig(
    cls_name="JulesConfig",
    spec={
        "inputs": {},  # we will fix this upon instantiation
        "namelists": {"handler": NamelistConfig},  # fully fixed
    },
)

With the input file handlers defined, we can now compose the top-level JulesConfig. Note that the inputs node uses a lambda factory function rather than a direct handler instance. This defers instantiation of InputFilesConfig until the handler is constructed, allowing us to bind specific file paths at that time. The namelists node, by contrast, is fully fixed since all namelist file paths are known in advance.

Reading an existing configuration¶

from dirconf.utils import tree

print(tree("./config"))

./config
├──inputs
│  ├──Loobos_1997.dat
│  ├──Loobos_1997.nc
│  ├──initial_conditions.dat
│  └──tile_fractions.dat
└──namelists
   ├──ancillaries.nml
   ├──crop_params.nml
   ├──drive.nml
   ├──fire.nml
   ├──imogen.nml
   ├──initial_conditions.nml
   ├──jules_deposition.nml
   ├──jules_hydrology.nml
   ├──jules_irrig.nml
   ├──jules_prnt_control.nml
   ├──jules_radiation.nml
   ├──jules_rivers.nml
   ├──jules_snow.nml
   ├──jules_soil.nml
   ├──jules_soil_biogeochem.nml
   ├──jules_surface.nml
   ├──jules_surface_types.nml
   ├──jules_vegetation.nml
   ├──jules_water_resources.nml
   ├──model_environment.nml
   ├──model_grid.nml
   ├──nveg_params.nml
   ├──output.nml
   ├──pft_params.nml
   ├──prescribed_data.nml
   ├──science_fixes.nml
   ├──timesteps.nml
   ├──triffid_params.nml
   └──urban.nml

The directory tree above shows the complete JULES configuration layout: a namelists/ directory with all required .nml files, and an inputs/ directory with the driving data, initial conditions, and tile fraction files.

Reading with ascii¶

config_ascii = JulesConfig(
    namelists="namelists",
    inputs={
        "path": "inputs",
        "handler": lambda: InputFilesConfig(
            initial_conditions="initial_conditions.dat",
            tile_fractions="tile_fractions.dat",
            driving_data="Loobos_1997.dat",
        ),
    },
)

print(config_ascii)

dirconf.config.JulesConfig
├──inputs ------- (path='inputs', handler=InputFilesConfig)
│  ├──initial_conditions ---- (path='initial_conditions.dat', handler=AsciiFileHandler)
│  ├──tile_fractions -------- (path='tile_fractions.dat', handler=AsciiFileHandler)
│  └──driving_data ---------- (path='Loobos_1997.dat', handler=AsciiFileHandler)
└──namelists ---- (path='namelists', handler=NamelistConfig)
   ├──ancillaries -------------- (path='ancillaries.nml', handler=NamelistFileHandler)
   ├──crop_params -------------- (path='crop_params.nml', handler=NamelistFileHandler)
   ├──drive -------------------- (path='drive.nml', handler=NamelistFileHandler)
   ├──fire --------------------- (path='fire.nml', handler=NamelistFileHandler)
   ├──imogen ------------------- (path='imogen.nml', handler=NamelistFileHandler)
   ├──initial_conditions ------- (path='initial_conditions.nml', handler=NamelistFileHandler)
   ├──jules_deposition --------- (path='jules_deposition.nml', handler=NamelistFileHandler)
   ├──jules_hydrology ---------- (path='jules_hydrology.nml', handler=NamelistFileHandler)
   ├──jules_irrig -------------- (path='jules_irrig.nml', handler=NamelistFileHandler)
   ├──jules_prnt_control ------- (path='jules_prnt_control.nml', handler=NamelistFileHandler)
   ├──jules_radiation ---------- (path='jules_radiation.nml', handler=NamelistFileHandler)
   ├──jules_rivers ------------- (path='jules_rivers.nml', handler=NamelistFileHandler)
   ├──jules_snow --------------- (path='jules_snow.nml', handler=NamelistFileHandler)
   ├──jules_soil_biogeochem ---- (path='jules_soil_biogeochem.nml', handler=NamelistFileHandler)
   ├──jules_soil --------------- (path='jules_soil.nml', handler=NamelistFileHandler)
   ├──jules_surface ------------ (path='jules_surface.nml', handler=NamelistFileHandler)
   ├──jules_surface_types ------ (path='jules_surface_types.nml', handler=NamelistFileHandler)
   ├──jules_vegetation --------- (path='jules_vegetation.nml', handler=NamelistFileHandler)
   ├──jules_water_resources ---- (path='jules_water_resources.nml', handler=NamelistFileHandler)
   ├──model_environment -------- (path='model_environment.nml', handler=NamelistFileHandler)
   ├──model_grid --------------- (path='model_grid.nml', handler=NamelistFileHandler)
   ├──nveg_params -------------- (path='nveg_params.nml', handler=NamelistFileHandler)
   ├──output ------------------- (path='output.nml', handler=NamelistFileHandler)
   ├──pft_params --------------- (path='pft_params.nml', handler=NamelistFileHandler)
   ├──prescribed_data ---------- (path='prescribed_data.nml', handler=NamelistFileHandler)
   ├──science_fixes ------------ (path='science_fixes.nml', handler=NamelistFileHandler)
   ├──timesteps ---------------- (path='timesteps.nml', handler=NamelistFileHandler)
   ├──triffid_params ----------- (path='triffid_params.nml', handler=NamelistFileHandler)
   └──urban -------------------- (path='urban.nml', handler=NamelistFileHandler)

config_dict_ascii = config_ascii.read("./config")

Reading with netcdf¶

config_netcdf = JulesConfig(
    namelists="namelists",
    inputs={
        "path": "inputs",
        "handler": lambda: InputFilesConfig(
            initial_conditions="initial_conditions.dat",
            tile_fractions="tile_fractions.dat",
            driving_data="Loobos_1997.nc",
        ),
    },
)
print(config_netcdf)

dirconf.config.JulesConfig
├──inputs ------- (path='inputs', handler=InputFilesConfig)
│  ├──initial_conditions ---- (path='initial_conditions.dat', handler=AsciiFileHandler)
│  ├──tile_fractions -------- (path='tile_fractions.dat', handler=AsciiFileHandler)
│  └──driving_data ---------- (path='Loobos_1997.nc', handler=NetcdfFileHandler)
└──namelists ---- (path='namelists', handler=NamelistConfig)
   ├──ancillaries -------------- (path='ancillaries.nml', handler=NamelistFileHandler)
   ├──crop_params -------------- (path='crop_params.nml', handler=NamelistFileHandler)
   ├──drive -------------------- (path='drive.nml', handler=NamelistFileHandler)
   ├──fire --------------------- (path='fire.nml', handler=NamelistFileHandler)
   ├──imogen ------------------- (path='imogen.nml', handler=NamelistFileHandler)
   ├──initial_conditions ------- (path='initial_conditions.nml', handler=NamelistFileHandler)
   ├──jules_deposition --------- (path='jules_deposition.nml', handler=NamelistFileHandler)
   ├──jules_hydrology ---------- (path='jules_hydrology.nml', handler=NamelistFileHandler)
   ├──jules_irrig -------------- (path='jules_irrig.nml', handler=NamelistFileHandler)
   ├──jules_prnt_control ------- (path='jules_prnt_control.nml', handler=NamelistFileHandler)
   ├──jules_radiation ---------- (path='jules_radiation.nml', handler=NamelistFileHandler)
   ├──jules_rivers ------------- (path='jules_rivers.nml', handler=NamelistFileHandler)
   ├──jules_snow --------------- (path='jules_snow.nml', handler=NamelistFileHandler)
   ├──jules_soil_biogeochem ---- (path='jules_soil_biogeochem.nml', handler=NamelistFileHandler)
   ├──jules_soil --------------- (path='jules_soil.nml', handler=NamelistFileHandler)
   ├──jules_surface ------------ (path='jules_surface.nml', handler=NamelistFileHandler)
   ├──jules_surface_types ------ (path='jules_surface_types.nml', handler=NamelistFileHandler)
   ├──jules_vegetation --------- (path='jules_vegetation.nml', handler=NamelistFileHandler)
   ├──jules_water_resources ---- (path='jules_water_resources.nml', handler=NamelistFileHandler)
   ├──model_environment -------- (path='model_environment.nml', handler=NamelistFileHandler)
   ├──model_grid --------------- (path='model_grid.nml', handler=NamelistFileHandler)
   ├──nveg_params -------------- (path='nveg_params.nml', handler=NamelistFileHandler)
   ├──output ------------------- (path='output.nml', handler=NamelistFileHandler)
   ├──pft_params --------------- (path='pft_params.nml', handler=NamelistFileHandler)
   ├──prescribed_data ---------- (path='prescribed_data.nml', handler=NamelistFileHandler)
   ├──science_fixes ------------ (path='science_fixes.nml', handler=NamelistFileHandler)
   ├──timesteps ---------------- (path='timesteps.nml', handler=NamelistFileHandler)
   ├──triffid_params ----------- (path='triffid_params.nml', handler=NamelistFileHandler)
   └──urban -------------------- (path='urban.nml', handler=NamelistFileHandler)

config_dict_netcdf = config_netcdf.read("./config")

Writing a new configuration¶

We will now demonstrate how to generate a new JULES configuration based on a modification of the existing one.

Here we modify the output_period parameter in the output namelist from its default value to 3600 seconds, which means JULES will write output every hour. We then write the modified configuration to a temporary directory using tempfile, which is automatically cleaned up on exit. In practice you would use a persistent directory.

import tempfile

print(
    "current output period: ",
    config_dict_netcdf["namelists"]["output"]["jules_output_profile"][
        "output_period"
    ],
)
config_dict_netcdf["namelists"]["output"]["jules_output_profile"][
    "output_period"
] = 3600
with tempfile.TemporaryDirectory() as temp_dir:
    config_netcdf.write(temp_dir, config_dict_netcdf)

current output period:  1800

Best, M. J., et al. (2011). The Joint UK Land Environment Simulator (JULES), model description – Part 1: Energy and water fluxes, Geosci. Model Dev., 4, 677–699, 10.5194/gmd-4-677-2011. ↩
Clark, D. B., et al. (2011). The Joint UK Land Environment Simulator (JULES), model description – Part 2: Carbon fluxes and vegetation dynamics, Geosci. Model Dev., 4, 701–722, 10.5194/gmd-4-701-2011. ↩
Marshall Ward. (2019). marshallward/f90nml: Version 1.1.2 (v1.1.2). Zenodo. 10.5281/zenodo.3245482 ↩