Configuration of the JULES land surface model

Some background about the JULES land surface model.

Documentation is available at https://jules-lsm.github.io.

from os import PathLike
from pathlib import Path

import metaconf

Namelists

The JULES executable is run by passing the path to a directory containing these namelists as its sole positional argument,

jules.exe /path/to/namelists/

where namelists/ must contain all of the required namelists files,

namelists/
├── ancillaries.nml
├── crop_params.nml
├── drive.nml
├── fire.nml
├── imogen.nml
├── initial_conditions.nml
├── jules_deposition.nml
├── jules_hydrology.nml
├── jules_irrig.nml
├── jules_prnt_control.nml
├── jules_radiation.nml
├── jules_rivers.nml
├── jules_snow.nml
├── jules_soil_biogeochem.nml
├── jules_soil.nml
├── jules_surface.nml
├── jules_surface_types.nml
├── jules_vegetation.nml
├── jules_water_resources.nml
├── model_environment.nml
├── model_grid.nml
├── nveg_params.nml
├── output.nml
├── pft_params.nml
├── prescribed_data.nml
├── science_fixes.nml
├── timesteps.nml
├── triffid_params.nml
└── urban.nml

Note

There is no freedom to use different file names for the namelist files; they must be present exactly as specified above.

We will make use of the f90nml Python package¹ for reading and writing namelist files.

import f90nml

class NamelistFileHandler:
    def read(self, path: str | PathLike) -> dict:
        """Read a Fortran namelist file and return a dict of its contents."""
        data = f90nml.read(path)
        return data.todict()

    def write(
        self, path: str | PathLike, data: dict, *, overwrite_ok: bool = False
    ) -> None:
        """Write a dict to a Fortran namelist file."""
        f90nml.write(data, path, force=overwrite_ok)

We now construct a MetaConfig-based Handler for a namelists directory, in which each Node corresponds to a single .nml file with a fixed path and handler.

_jules_namelists = [
    "ancillaries",
    "crop_params",
    "drive",
    "fire",
    "imogen",
    "initial_conditions",
    "jules_deposition",
    "jules_hydrology",
    "jules_irrig",
    "jules_prnt_control",
    "jules_radiation",
    "jules_rivers",
    "jules_snow",
    "jules_soil_biogeochem",
    "jules_soil",
    "jules_surface",
    "jules_surface_types",
    "jules_vegetation",
    "jules_water_resources",
    "model_environment",
    "model_grid",
    "nveg_params",
    "output",
    "pft_params",
    "prescribed_data",
    "science_fixes",
    "timesteps",
    "triffid_params",
    "urban",
]

NamelistDirectoryHandler = metaconf.make_metaconfig(
    cls_name="NamelistDirectoryHandler",
    spec={
        name: {"path": f"{name}.nml", "handler": NamelistFileHandler}
        for name in _jules_namelists
    },
)

This produces a subclass of MetaConfig which is instantiated without any arguments.

namelists_handler = NamelistDirectoryHandler()

print(namelists_handler)

metaconf.config.NamelistDirectoryHandler
├──ancillaries -------------- (path='ancillaries.nml', handler=NamelistFileHandler)
├──crop_params -------------- (path='crop_params.nml', handler=NamelistFileHandler)
├──drive -------------------- (path='drive.nml', handler=NamelistFileHandler)
├──fire --------------------- (path='fire.nml', handler=NamelistFileHandler)
├──imogen ------------------- (path='imogen.nml', handler=NamelistFileHandler)
├──initial_conditions ------- (path='initial_conditions.nml', handler=NamelistFileHandler)
├──jules_deposition --------- (path='jules_deposition.nml', handler=NamelistFileHandler)
├──jules_hydrology ---------- (path='jules_hydrology.nml', handler=NamelistFileHandler)
├──jules_irrig -------------- (path='jules_irrig.nml', handler=NamelistFileHandler)
├──jules_prnt_control ------- (path='jules_prnt_control.nml', handler=NamelistFileHandler)
├──jules_radiation ---------- (path='jules_radiation.nml', handler=NamelistFileHandler)
├──jules_rivers ------------- (path='jules_rivers.nml', handler=NamelistFileHandler)
├──jules_snow --------------- (path='jules_snow.nml', handler=NamelistFileHandler)
├──jules_soil_biogeochem ---- (path='jules_soil_biogeochem.nml', handler=NamelistFileHandler)
├──jules_soil --------------- (path='jules_soil.nml', handler=NamelistFileHandler)
├──jules_surface ------------ (path='jules_surface.nml', handler=NamelistFileHandler)
├──jules_surface_types ------ (path='jules_surface_types.nml', handler=NamelistFileHandler)
├──jules_vegetation --------- (path='jules_vegetation.nml', handler=NamelistFileHandler)
├──jules_water_resources ---- (path='jules_water_resources.nml', handler=NamelistFileHandler)
├──model_environment -------- (path='model_environment.nml', handler=NamelistFileHandler)
├──model_grid --------------- (path='model_grid.nml', handler=NamelistFileHandler)
├──nveg_params -------------- (path='nveg_params.nml', handler=NamelistFileHandler)
├──output ------------------- (path='output.nml', handler=NamelistFileHandler)
├──pft_params --------------- (path='pft_params.nml', handler=NamelistFileHandler)
├──prescribed_data ---------- (path='prescribed_data.nml', handler=NamelistFileHandler)
├──science_fixes ------------ (path='science_fixes.nml', handler=NamelistFileHandler)
├──timesteps ---------------- (path='timesteps.nml', handler=NamelistFileHandler)
├──triffid_params ----------- (path='triffid_params.nml', handler=NamelistFileHandler)
└──urban -------------------- (path='urban.nml', handler=NamelistFileHandler)

We can now use this handler to read an entire namelists directory into a Python dict.

namelists_dict = namelists_handler.read("config/namelists")

namelists_dict.keys()

dict_keys(['ancillaries', 'crop_params', 'drive', 'fire', 'imogen', 'initial_conditions', 'jules_deposition', 'jules_hydrology', 'jules_irrig', 'jules_prnt_control', 'jules_radiation', 'jules_rivers', 'jules_snow', 'jules_soil_biogeochem', 'jules_soil', 'jules_surface', 'jules_surface_types', 'jules_vegetation', 'jules_water_resources', 'model_environment', 'model_grid', 'nveg_params', 'output', 'pft_params', 'prescribed_data', 'science_fixes', 'timesteps', 'triffid_params', 'urban'])

namelists_dict["drive"]

OrderedDict([('jules_drive',
              OrderedDict([('file', 'inputs/Loobos_1997.dat'),
                           ('data_start', '1996-12-31 23:00:00'),
                           ('data_end', '1997-12-31 23:00:00'),
                           ('data_period', 1800),
                           ('nvars', 8),
                           ('var',
                            ['sw_down',
                             'lw_down',
                             'tot_rain',
                             'tot_snow',
                             't',
                             'wind',
                             'pstar',
                             'q']),
                           ('interp',
                            ['nf', 'nf', 'nf', 'nf', 'nf', 'nf', 'nf', 'nf']),
                           ('diff_frac_const', 0.4),
                           ('t_for_con_rain', 293.15),
                           ('z1_uv_in', 10.0),
                           ('z1_tq_in', 10.0)]))])

Input data

Ascii input data

JULES also requires

We can use numpy to read and write floating point ascii data.

from typing import TypedDict
import numpy

@metaconf.filter(
    write=lambda path, data, **_: not path.is_absolute()
)
@metaconf.filter_missing()
class AsciiFileHandler:
    class AsciiData(TypedDict):
        values: numpy.ndarray
        comment: str

    def read(self, path: str | PathLike) -> AsciiData:
        comment_lines = []
        num_lines = 0

        with open(path, "r") as file:
            for line in file:
                line = line.strip()

                if line.startswith(("#", "!")):  # comment line
                    comment_lines.append(line)
                    continue

                elif line:  # non-empty line
                    num_lines += 1

                    if num_lines > 1:  # we just need to know if it's >1
                        break

        comment = "\n".join(comment_lines)

        values = numpy.loadtxt(str(path), comments=("#", "!"))

        # NOTE: Unfortunately numpy.loadtxt/savetxt does not correctly round-trip
        # single-row data. We need to catch it here and add an extra dimension.
        if num_lines == 1:
            assert values.ndim == 1
            values = values.reshape(1, -1)

        return self.AsciiData(values=values, comment=comment)

    def write(
        self,
        path: str | PathLike,
        data: AsciiData,
        *,
        overwrite_ok: bool = False,
    ) -> None:
        numpy.savetxt(
            str(path),
            data["values"],
            fmt="%.5f",
            header=data["comment"],
            comments="#",
        )

NetCDF input data

We will use xarray to read and write input data in the netCDF format.

import xarray

@metaconf.filter(
    read=lambda path: not path.is_absolute(),
    write=lambda path, data, **_: not path.is_absolute()
)
@metaconf.filter_missing()
class NetcdfFileHandler:
    def read(self, path: str | PathLike) -> xarray.Dataset:
        dataset = xarray.load_dataset(path)
        return dataset

    def write(
        self, path: str | PathLike, data: xarray.Dataset, *, overwrite_ok: bool = False
    ) -> None:
        if not overwrite_ok and Path(path).is_file():
            raise FileExistsError(f"There is already a file at '{path}'")
        data.to_netcdf(path)

Putting it together

metaconf.register_handler("ascii", AsciiFileHandler, [".txt", ".dat", ".asc"])
metaconf.register_handler("netcdf", NetcdfFileHandler, [".nc", ".cdf"])

InputFilesConfig = metaconf.make_metaconfig(
    cls_name="InputFilesConfig",
    spec={
        "initial_conditions": {
            "handler": AsciiFileHandler,
        },
        "tile_fractions": {
            "handler": AsciiFileHandler,
        },
        "driving_data": {},
    },
)

JulesConfigHandler = metaconf.make_metaconfig(
    cls_name="JulesConfigHandler",
    spec={
        "inputs": {},  # we will fix this upon instantiation
        "namelists": {"handler": NamelistDirectoryHandler},  # fully fixed

    },
)

Reading an existing configuration

from metaconf.utils import tree

print(tree("./config"))

./config
├──inputs
│  ├──Loobos_1997.dat
│  ├──Loobos_1997.nc
│  ├──initial_conditions.dat
│  └──tile_fractions.dat
└──namelists
   ├──ancillaries.nml
   ├──crop_params.nml
   ├──drive.nml
   ├──fire.nml
   ├──imogen.nml
   ├──initial_conditions.nml
   ├──jules_deposition.nml
   ├──jules_hydrology.nml
   ├──jules_irrig.nml
   ├──jules_prnt_control.nml
   ├──jules_radiation.nml
   ├──jules_rivers.nml
   ├──jules_snow.nml
   ├──jules_soil.nml
   ├──jules_soil_biogeochem.nml
   ├──jules_surface.nml
   ├──jules_surface_types.nml
   ├──jules_vegetation.nml
   ├──jules_water_resources.nml
   ├──model_environment.nml
   ├──model_grid.nml
   ├──nveg_params.nml
   ├──output.nml
   ├──pft_params.nml
   ├──prescribed_data.nml
   ├──science_fixes.nml
   ├──timesteps.nml
   ├──triffid_params.nml
   └──urban.nml

handler = JulesConfigHandler(
    namelists="namelists",
    inputs={
        "path": "inputs",
        "handler": lambda: InputFilesConfig(
            initial_conditions="initial_conditions.dat",
            tile_fractions="tile_fractions.dat",
            driving_data="Loobos_1997.dat",
        )
    }
)

print(handler)

metaconf.config.JulesConfigHandler
├──inputs ------- (path='inputs', handler=InputFilesConfig)
│  ├──initial_conditions ---- (path='initial_conditions.dat', handler=AsciiFileHandler)
│  ├──tile_fractions -------- (path='tile_fractions.dat', handler=AsciiFileHandler)
│  └──driving_data ---------- (path='Loobos_1997.dat', handler=AsciiFileHandler)
└──namelists ---- (path='namelists', handler=NamelistDirectoryHandler)
   ├──ancillaries -------------- (path='ancillaries.nml', handler=NamelistFileHandler)
   ├──crop_params -------------- (path='crop_params.nml', handler=NamelistFileHandler)
   ├──drive -------------------- (path='drive.nml', handler=NamelistFileHandler)
   ├──fire --------------------- (path='fire.nml', handler=NamelistFileHandler)
   ├──imogen ------------------- (path='imogen.nml', handler=NamelistFileHandler)
   ├──initial_conditions ------- (path='initial_conditions.nml', handler=NamelistFileHandler)
   ├──jules_deposition --------- (path='jules_deposition.nml', handler=NamelistFileHandler)
   ├──jules_hydrology ---------- (path='jules_hydrology.nml', handler=NamelistFileHandler)
   ├──jules_irrig -------------- (path='jules_irrig.nml', handler=NamelistFileHandler)
   ├──jules_prnt_control ------- (path='jules_prnt_control.nml', handler=NamelistFileHandler)
   ├──jules_radiation ---------- (path='jules_radiation.nml', handler=NamelistFileHandler)
   ├──jules_rivers ------------- (path='jules_rivers.nml', handler=NamelistFileHandler)
   ├──jules_snow --------------- (path='jules_snow.nml', handler=NamelistFileHandler)
   ├──jules_soil_biogeochem ---- (path='jules_soil_biogeochem.nml', handler=NamelistFileHandler)
   ├──jules_soil --------------- (path='jules_soil.nml', handler=NamelistFileHandler)
   ├──jules_surface ------------ (path='jules_surface.nml', handler=NamelistFileHandler)
   ├──jules_surface_types ------ (path='jules_surface_types.nml', handler=NamelistFileHandler)
   ├──jules_vegetation --------- (path='jules_vegetation.nml', handler=NamelistFileHandler)
   ├──jules_water_resources ---- (path='jules_water_resources.nml', handler=NamelistFileHandler)
   ├──model_environment -------- (path='model_environment.nml', handler=NamelistFileHandler)
   ├──model_grid --------------- (path='model_grid.nml', handler=NamelistFileHandler)
   ├──nveg_params -------------- (path='nveg_params.nml', handler=NamelistFileHandler)
   ├──output ------------------- (path='output.nml', handler=NamelistFileHandler)
   ├──pft_params --------------- (path='pft_params.nml', handler=NamelistFileHandler)
   ├──prescribed_data ---------- (path='prescribed_data.nml', handler=NamelistFileHandler)
   ├──science_fixes ------------ (path='science_fixes.nml', handler=NamelistFileHandler)
   ├──timesteps ---------------- (path='timesteps.nml', handler=NamelistFileHandler)
   ├──triffid_params ----------- (path='triffid_params.nml', handler=NamelistFileHandler)
   └──urban -------------------- (path='urban.nml', handler=NamelistFileHandler)

config = handler.read("./config")

config.keys()

dict_keys(['inputs', 'namelists'])

Reading with netcdf

handler = JulesConfigHandler(
    namelists="namelists",
    inputs={
        "path": "inputs",
        "handler": lambda: InputFilesConfig(
            initial_conditions="initial_conditions.dat",
            tile_fractions="tile_fractions.dat",
            driving_data="Loobos_1997.nc",
        )
    }
)

print(handler)

metaconf.config.JulesConfigHandler
├──inputs ------- (path='inputs', handler=InputFilesConfig)
│  ├──initial_conditions ---- (path='initial_conditions.dat', handler=AsciiFileHandler)
│  ├──tile_fractions -------- (path='tile_fractions.dat', handler=AsciiFileHandler)
│  └──driving_data ---------- (path='Loobos_1997.nc', handler=NetcdfFileHandler)
└──namelists ---- (path='namelists', handler=NamelistDirectoryHandler)
   ├──ancillaries -------------- (path='ancillaries.nml', handler=NamelistFileHandler)
   ├──crop_params -------------- (path='crop_params.nml', handler=NamelistFileHandler)
   ├──drive -------------------- (path='drive.nml', handler=NamelistFileHandler)
   ├──fire --------------------- (path='fire.nml', handler=NamelistFileHandler)
   ├──imogen ------------------- (path='imogen.nml', handler=NamelistFileHandler)
   ├──initial_conditions ------- (path='initial_conditions.nml', handler=NamelistFileHandler)
   ├──jules_deposition --------- (path='jules_deposition.nml', handler=NamelistFileHandler)
   ├──jules_hydrology ---------- (path='jules_hydrology.nml', handler=NamelistFileHandler)
   ├──jules_irrig -------------- (path='jules_irrig.nml', handler=NamelistFileHandler)
   ├──jules_prnt_control ------- (path='jules_prnt_control.nml', handler=NamelistFileHandler)
   ├──jules_radiation ---------- (path='jules_radiation.nml', handler=NamelistFileHandler)
   ├──jules_rivers ------------- (path='jules_rivers.nml', handler=NamelistFileHandler)
   ├──jules_snow --------------- (path='jules_snow.nml', handler=NamelistFileHandler)
   ├──jules_soil_biogeochem ---- (path='jules_soil_biogeochem.nml', handler=NamelistFileHandler)
   ├──jules_soil --------------- (path='jules_soil.nml', handler=NamelistFileHandler)
   ├──jules_surface ------------ (path='jules_surface.nml', handler=NamelistFileHandler)
   ├──jules_surface_types ------ (path='jules_surface_types.nml', handler=NamelistFileHandler)
   ├──jules_vegetation --------- (path='jules_vegetation.nml', handler=NamelistFileHandler)
   ├──jules_water_resources ---- (path='jules_water_resources.nml', handler=NamelistFileHandler)
   ├──model_environment -------- (path='model_environment.nml', handler=NamelistFileHandler)
   ├──model_grid --------------- (path='model_grid.nml', handler=NamelistFileHandler)
   ├──nveg_params -------------- (path='nveg_params.nml', handler=NamelistFileHandler)
   ├──output ------------------- (path='output.nml', handler=NamelistFileHandler)
   ├──pft_params --------------- (path='pft_params.nml', handler=NamelistFileHandler)
   ├──prescribed_data ---------- (path='prescribed_data.nml', handler=NamelistFileHandler)
   ├──science_fixes ------------ (path='science_fixes.nml', handler=NamelistFileHandler)
   ├──timesteps ---------------- (path='timesteps.nml', handler=NamelistFileHandler)
   ├──triffid_params ----------- (path='triffid_params.nml', handler=NamelistFileHandler)
   └──urban -------------------- (path='urban.nml', handler=NamelistFileHandler)

config = handler.read("./config")

config.keys()

dict_keys(['inputs', 'namelists'])

config["inputs"]["driving_data"]

<xarray.Dataset> Size: 1MB
Dimensions:     (time: 17520)
Coordinates:
  * time        (time) datetime64[ns] 140kB 1996-12-31T23:00:00 ... 1997-12-3...
Data variables:
    sw_down     (time) float64 140kB 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    lw_down     (time) float64 140kB 187.8 186.9 186.7 ... 307.4 313.8 312.3
    precip      (time) float64 140kB 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    snow        (time) float64 140kB 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    air_temp    (time) float64 140kB 259.8 259.7 259.6 ... 279.8 279.6 279.5
    wind_speed  (time) float64 140kB 2.017 3.77 4.29 4.42 ... 1.62 1.83 1.79
    pressure    (time) float64 140kB 1.024e+05 1.024e+05 ... 1.005e+05 1.005e+05
    humidity    (time) float64 140kB 0.001384 0.001384 ... 0.005731 0.005697

Writing a new configuration

We will now demonstrate how to generate a new JULES configuration based on a modification of the existing one.

Here we will use tempfile to write to a temporary directory that is automatically deleted upon exit from the context handler. In practice one would create a persistent directory.

import tempfile

config = handler.read("./config")

print("current output period: ", config["namelists"]["output"]["jules_output_profile"]["output_period"])

config["namelists"]["output"]["jules_output_profile"]["output_period"] = 3600

with tempfile.TemporaryDirectory() as temp_dir:

    handler.write(temp_dir, config)

    print(tree(temp_dir))

current output period:  1800
/tmp/tmprwqhov6s
├──inputs
│  ├──Loobos_1997.nc
│  ├──initial_conditions.dat
│  └──tile_fractions.dat
└──namelists
   ├──ancillaries.nml
   ├──crop_params.nml
   ├──drive.nml
   ├──fire.nml
   ├──imogen.nml
   ├──initial_conditions.nml
   ├──jules_deposition.nml
   ├──jules_hydrology.nml
   ├──jules_irrig.nml
   ├──jules_prnt_control.nml
   ├──jules_radiation.nml
   ├──jules_rivers.nml
   ├──jules_snow.nml
   ├──jules_soil.nml
   ├──jules_soil_biogeochem.nml
   ├──jules_surface.nml
   ├──jules_surface_types.nml
   ├──jules_vegetation.nml
   ├──jules_water_resources.nml
   ├──model_environment.nml
   ├──model_grid.nml
   ├──nveg_params.nml
   ├──output.nml
   ├──pft_params.nml
   ├──prescribed_data.nml
   ├──science_fixes.nml
   ├──timesteps.nml
   ├──triffid_params.nml
   └──urban.nml

---

1. Marshall Ward. (2019). marshallward/f90nml: Version 1.1.2 (v1.1.2). Zenodo. https://doi.org/10.5281/zenodo.3245482 ↩