libpipe documentation#

libpipe is the execution layer of the NenuFAR Cosmic Dawn and LOFAR EoR pipelines. It runs the many independent commands of a processing step, one per MeasurementSet, image or cube, in parallel on the current machine, on other machines over SSH, or on a Slurm cluster. It also provides the TOML settings files of the pipelines.

pspipe and NenuCal use it for every task they run: the [worker] section of their configuration files is passed to a libpipe worker pool. You do not need to use libpipe directly to run them, but this documentation explains how the tasks are executed and what each [worker] option does.

Overview#

A worker pool receives tasks, shell commands with an optional log file, and runs them on its hosts:

  • localhost: the current machine;

  • other host names (nancep[1-4,6]): machines reached over SSH;

  • slurm: a Slurm cluster, each task started with srun.

The pool runs up to max_tasks_per_worker tasks per host at a time, retries a task that could not start, enforces a time limit, writes the output of each task to its log file and to the console, and reports which tasks succeeded. On Slurm, the tasks run either as one job each, or as job steps packed on the nodes of one allocation, from the CPUs and memory of their resource category. Each task on the current machine or on Slurm is measured: wall time, CPU time and peak memory, summarised per task type at the end.

The three ways a worker pool runs its tasks

The three ways a worker pool runs its tasks: on this machine or SSH hosts, as one Slurm job per task, or as job steps packed on the nodes of one Slurm allocation.#

User guide

Reference

Installation#

$ pip install libpipe

libpipe requires Python 3.10 or later. The Slurm modes need the Slurm commands (srun, sbatch, squeue, scancel) on the machine running the pool, and Slurm 20.11 or later for job steps.