Python SDK

A program described in Python (see the Describing a Program in Python Section) is made of the classes below, which the file of the description imports from the package debasher:

from debasher import Process, Program, in_outdir

Only what a description uses is listed: the classes it creates, the methods it calls and the functions it uses. debasher_build (see the DeBasher Tools Section) loads the file and writes the program; the last section describes what the Python functions of a process (the tasks of an array or a generator, and the skip method) receive when the engine calls them.

Program

class debasher.Program(name: str, description: str = '', program_options: dict[str, str] | None = None, preamble: str = '', env_vars: dict[str, str] | None = None, shared_dirs: list[str] | None = None, scheduler: str = 'BUILTIN', builtin_sched_cpus: str = '', builtin_sched_mem: str = '', dflt_nodes: str = '', dflt_throttle: str = '', rerun_outdated_procs: bool = False, conda_support: bool = False, docker_support: bool = False, resident: bool = False, snapshot_every_secs: str = '')

A program: its name, which names its module (<name>.sh) and the file that defines it (<name>.py), its description, the options of its command line that a run is given (program_options, label to value, what “Set program options” of the editor holds), and what a module holds beside its processes: the preamble (Bash before the functions, such as load_debasher_module lines), the environment variables (env_vars, of which only DEBASHER_MOD_DIR is read) and the shared directories. The execution options of the editor are given by name: scheduler, builtin_sched_cpus, builtin_sched_mem, dflt_nodes, dflt_throttle, rerun_outdated_procs, conda_support, docker_support. A resident program (resident=True) adds nodes and a Supervisor instead of processes, and may ask for periodic snapshots of its runs (snapshot_every_secs).

add(process: Process, cpus: float | None = None, mem: float | None = None, time: str | None = None, processdeps: str | None = None, force: bool = False, position: tuple[float, float] | None = None, **node_specs) → Process: Adds process to the program with the computational specifications of its use there (cpus, mem in megabytes, time as hh:mm:ss; 1 CPU, 256 megabytes and 01:00:00 for those not given, as the editor gives a new process), its explicit dependencies (processdeps, as the engine writes them, afterok:count), whether it is rerun on every run (force), and where it goes on the canvas (position, else the layout of the editor). A node of a resident program takes the specifications of a node too, under the names of the engine (input_log_max_mb, heartbeat_timeout_s, …). Returns the process.

property processes: list[Process]: The processes added to the program, in the order they were added.

write(home_dir: str | Path) → Path: Writes the program into home_dir, as Save does in the editor: its artifact, the topology file program.yaml and the process files, the tasks files and the method files where the description had them, relative to its directory (each written only when its content differs, so a program written into the directory of the description leaves its files as they are), the generated module <name>.sh, and a copy of every script (external alias), replacing an older copy. Returns the path of the module. A program that the SDK or script generation refuses leaves the directory as it was.

Processes

A process: its name, qualified with dot-separated namespaces if wanted, its description, and where its code is, as one of:

code, a file in the language of the process, told by its extension (.sh, .py, .pl, .r or .R, .groovy) or by language: a Bash function named as the process, with its helper functions, or the whole script of another language. The topology file names the file, which the editor shows and edits as the code of the process; the module embeds it when it is Bash, and runs it as it is (an external alias of the file) in any other language.
script, a file that the module runs as it is, with the interpreter of its extension (the external alias of the engine): the file stays outside the module, copied beside it when the program is written elsewhere.
alias, the name of another process whose code this one runs under its own name and options.
none of them: the code of a process of that name that a module loaded by the preamble of the program provides.

language (bash, python, perl, r or groovy) gives the language of the process when the extension of its file does not tell it. A script or an alias may rename the options that the process passes on with alias_opt_map, the name of each option of this process to the one that the code expects ({"-x": "-y"}), as the alias option map of a module does.

The additional methods of the process are given as files too, each with the body that the editor takes: skip (whether a task is skipped), a Bash body, or a skip file in Python (.py) that defines skip(task); post (run after the process), reset_outfiles (what to remove before a rerun) and outdir_basename (the name of the process output directory), Bash bodies. The environments that the process declares are data: conda_envs, a dict of the name of each Conda environment to its yml file, and docker_imgs, a list of Docker images; a path given instead is a file with the Bash body of the method.

A relative path is taken from the directory of the Python file that creates the process (base_dir names another directory).

Process.__getitem__(label: str) → Option: The option of the process with the label label, such as generate["-outf"], to give as the source of an input; KeyError when the process has none.

class debasher.Array(name: str, *args, tasks: str | Path | None = None, **kwargs): An array process: tasks is a Bash file that builds the array of its tasks, under the name array, from the command line and the files of the run (local w=$(debasher::read_opt_value_from_line "${cmdline}" "-w"), then array+=(...)), or a Python file whose tasks(run) returns the elements (range(int(run.value("-w")))); every option is defined once per element, with ${task_idx} and ${array[$task_idx]} at hand in its value. It is the array mode of the options handler of the editor. It takes the arguments of a Process, and tasks.

class debasher.Generator(name: str, *args, tasks: str | Path | None = None, **kwargs): An option generator: tasks is a Bash file that prints the number of tasks, from the command line and the files of the run, or a Python file whose tasks(run) returns it; the options of each task are defined from its index, ${task_idx}, in their values. It is the generator mode of the options handler of the editor. It takes the arguments of a Process, and tasks.

Options

Every option is declared by a method of its process, which returns it.

Process.cmdline_opt(label: str, data_type: str, description: str, mandatory: bool = True, shaping: bool = False) → Option: An input whose value comes from the command line of the run; a task shaping option (shaping) is read by the options handler only.

Process.cmdline_flag(label: str, description: str) → Option: A flag of the command line of the run, which has no value.

Process.input(label: str, data_type: str, description: str, value: str = '', source: Option | tuple[str, str] | list[Union[Option, tuple[str, str]]] | None = None, count_source: Option | str | None = None, display: str = 'line') → Option: An input: connected to the output of another process (source, one or, for a fan-in, several), or with a literal value, or, for a family of inputs (a label ending in “-ith”), counted by an option of this process (count_source) or by the tasks of the process it reads.

Process.output(label: str, data_type: str, description: str, value: str = '', count_source: Option | str | None = None) → Option: An output, with the value that names what the process produces (see in_outdir), or a family of outputs counted by count_source.

Process.fifo_output(label: str, data_type: str, description: str, name: str, mirror: bool = False, count_source: Option | str | None = None) → Option: An output written to a FIFO named name, mirrored into a log file for the canvas to watch if asked; or a family of FIFOs, one per value of the index ("dispatch_fifo_${i}"), counted by count_source.

Process.fifo_input(label: str, data_type: str, description: str, name: str, external: bool = False) → Option: An input read from a FIFO named name, which the process opens itself rather than receives through a connection; in a resident program, an external input (external), written from outside the program.

Process.outdir_opt(label: str, description: str, subpath: str = '') → Option: An output that names the process output directory of the process, or a task subdirectory of it (subpath, a Bash word such as "${task_idx}"), for another process to read through a connection.

Process.shared_dir_opt(label: str, data_type: str, description: str, shared_dir: str, subpath: str = '', source: Option | tuple[str, str] | list[Union[Option, tuple[str, str]]] | None = None) → Option: An option whose value is a shared directory of the program, named by shared_dir (one of the shared directories of the Program), or a shared subdirectory of it (subpath). A source only draws the sharing on the canvas: the engine orders the processes by the directory itself.

Process.procspec_opt(label: str, data_type: str, description: str, attribute: str) → Option: An input whose value is an attribute of the specifications of the process in the program, such as "cpus".

Process.value_desc_output(label: str, data_type: str, description: str) → Option: An output delivered as a value descriptor, which the engine synthesizes for the process.

Process.opt(label: str, data_type: str = 'string', description: str = '', **fields) → Option: An option with the fields of the program model given by name (value, channel, subpath, mirror, fifo_tag, command_line, mandatory, from_process_spec, task_shaping, count_source, source, display); the other methods are this one with the fields of one case set. A flag has the data type “None”.

debasher.in_outdir(name: str = '') → str: The value of an output kept in the process output directory: the path of name under it, or the directory itself when name is empty, as a Bash word that the engine expands when the option is defined.

class debasher.Option

One option of a process, with the fields of the program model that script generation reads. Created through the methods of Process, never directly.

Its attributes are the process it belongs to (process), its label (label), its data type (data_type), its description (description), its value (value), its option channel (channel) and its direction (direction, “input” or “output”, from its label), and the other fields that Process.opt takes, under the same names (sources for source), with their defaults when the method that created it gives none.

Resident programs

A resident program (Program(..., resident=True)) adds nodes and a Supervisor instead of processes.

class debasher.Node(name: str, code: str | Path, initiator: bool = False, description: str = '', base_dir: str | Path | None = None): A node of a resident program: a process whose code is a Python file with the class of the node, derived from a class of the runtime library (FBPProcess, ProgramLauncher or DirectoryWatcher), as the generated module writes it: the import of the node kind, the node preamble, the class with its hooks, and the line that runs it. The SDK splits the file into the parts of the program model with the same code as import, and refuses what the web UI could not hold. An initiator is a node where a round starts.

class debasher.Supervisor(name: str, description: str = '', base_dir: str | Path | None = None): The Supervisor of a resident program, which watches its nodes and relaunches one that goes down. It has no code and no options of its own: script generation writes its whole class and its wiring.

Errors

class debasher.DefinitionError: A program that the SDK cannot write: what it says is contradictory, incomplete, or something the engine would refuse. The message says what, and names the process or the option concerned.

Python functions of a process

The function tasks(run) of a tasks file in Python receives a Run, and the function skip(task) of a skip file in Python receives a Task. The file imports nothing to receive them.

class debasher_run_py_func.Run

What tasks(run) receives: the command line of the run (args), the name of the process (process) and its process output directory (outdir).

given(label): Whether the option or flag label is given.

value(label, default=None): The value given to the option label, or default when the option is not given or is given without a value.

class debasher_run_py_func.Task

What skip(task) receives: the options of the task (args), and the name of its process (process).

given(label): Whether the option or flag label is given.

value(label, default=None): The value given to the option label, as for any options, except that the value of an input option that is a value descriptor is the value that the descriptor holds, as read_opt_value_from_func_args reads it.