.. _exec: Executing DeBasher Programs =========================== This section explains how DeBasher programs are executed. Additionally, other important aspects related to program execution are also described, such as the structure of the output directory or how to visualize the program execution status. .. _prg_exec: Program Execution ----------------- DeBasher incorporates a specific tool to execute programs called ``debasher_exec``. The command provides help about its available options if executed as follows: :: $ debasher_exec --help ``debasher_exec`` only has two mandatory input options: * ``--pfile ``: allows to define the DeBasher file specifying the program to be executed. * ``--outdir ``: specifies the name of the output directory. Visualizing Command Line Options for a Program ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The command line options for the DeBasher program defined by the ``--pfile`` option should be provided to ``debasher_exec``. In order to visualize such command line options, we can use the ``--show-cmdline-opts`` option. Assuming that we are working with the "Hello World!" program described in the :ref:`quickstart_example` Section, we can execute the following: :: $ debasher_exec --pfile debasher_hello_world.sh --outdir out --show-cmdline-opts The output of the previous command incorporates a section related to the program's input options. For the ``debasher_hello_world`` program, this section contains the following: :: # Command line options for the program... CATEGORY: GENERAL -s String to be displayed ('Hello World!' by default) [hello_world] Command line options can be divided into different categories, being the "GENERAL" category the default one. In this case, the "GENERAL" category contains the ``-s`` option, that is used to provide the string to be displayed by the ``debasher_hello_world`` program. Additionally, the process receiving the ``-s`` option (``hello_world``) is specified within square brackets. Troubleshooting Process Options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When working with complex programs, composed of many processes and connections between them, it is useful to ensure that all the options are passed correctly. ``debasher_exec`` incorporates two options that are useful for that purpose: * ``--check-proc-opts``: when providing this option, ``debasher_exec`` only checks if all the options are available for the different processes that compose a program. If there are options missing, an error message is shown. Otherwise, the options are shown and the tool finishes its execution. * ``--debug``: this option carries out all the necessary steps to execute a DeBasher program, with the exception of the execution itself. This includes the option checking process that can be done using the ``--check-proc-opts`` option, but also the current status of the different processes involved in the program. Executing Programs Using the Built-In Scheduler ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``debasher_exec`` incorporates the ``--sched`` option to specify which scheduler is used to orchestrate program execution. DeBasher incorporates a built-in scheduler that can be activated for the "Hello Word!" program as follows: :: $ debasher_exec --pfile debasher_hello_world.sh --outdir out --sched BUILTIN When using the built-in scheduler, it is possible to specify the number of CPUs and the amount of RAM that is available. For this purpose, the following two options can be used: * ``--builtinsched-cpus ``: this option allows to specify the number CPUs available for the built-in scheduler. A value of zero means unlimited CPUs. * ``--builtinsched-mem ``: indicates the amount of RAM in MB that can be used by the built-in scheduler. A value of zero means unlimited memory. Executing Programs Using External Schedulers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It is also possible to combine ``debasher_exec`` with external schedulers. Currently, DeBasher provides support for the well known `Slurm scheduler `__. Slurm """"" To make Slurm work in combination with DeBasher, it is necessary to install and properly configure the scheduler in the machine where DeBasher will be executed. Once Slurm is installed, we can proceed with the installation of DeBasher, which will automatically detect Slurm's availability. If Slurm and DeBasher are correctly configured, then we can combine them by means of the following command: :: $ debasher_exec --pfile debasher_hello_world.sh --outdir out --sched SLURM .. _outdstruct: Structure of the Output Directory --------------------------------- When executing ``debasher_exec``, it is necessary to specify the output directory of the program to be executed. Assuming that we executed a DeBasher program composed of two processes ``a`` and ``b``, and that the name of output directory generated by ``debasher_exec`` was ``out``, its content will be: .. code-block:: text out ├── a │ ├── file1 │ ├── file2 │ └── ... ├── b │ ├── file1 │ ├── file2 │ └── ... ├── __exec__ │ ├── a │ │ ├── a │ │ ├── a.finished (depends on execution status) │ │ ├── a.id (depends on execution status) │ │ ├── a.sched_out (depends on execution status) │ │ └── a.stdout (depends on execution status) │ └── b │ ├── b │ ├── b.finished (depends on execution status) │ ├── b.id (depends on execution status) │ ├── b.sched_out (depends on execution status) │ └── b.stdout (depends on execution status) ├── __fifos__ │ ├── a │ │ ├── fifo1 │ │ ├── fifo2 │ │ └── ... │ └── b │ ├── fifo1 │ ├── fifo2 │ └── ... ├── __graphs__ │ ├── dependency_graph.dot │ ├── dependency_graph.eps │ ├── dependency_graph.pdf │ ├── process_graph.dot (optional) │ ├── process_graph.eps (optional) │ └── process_graph.pdf (optional) ├── command_line.sh ├── program.fifos ├── program.opts └── program.procspec As it is shown in the previous diagram, ``debasher_exec`` generates the following directories: * ``a``: stores the output files of process ``a``. It is not mandatory that process ``a`` stores any output files, but the directory is automatically created by ``debasher_exec``. * ``b``: it is the output directory of process ``b``. * ``__exec__``: stores the execution information for the different processes. Contains one directory per process. For a given process, the following files are created: * ````: contains the code to be executed. * ``.finished``: this file is created to signal that the execution of the process has finished. * ``.id``: this file is created when the process starts its execution and contains an identifier of the process created by the scheduler being used. * ``.sched_out``: this file contains all the output related to process execution, including the standard and error outputs and also some additional information generated by the scheduler. * ``.stdout``: this file contains the standard output generated by the execution of the process. * ``__fifos__``: stores the FIFOs used by each process if there are any. See :ref:`implem` Section for more details about using FIFOs in DeBasher programs. * ``__graphs__``: contains certain graphical representations of the program being executed. ``debasher_exec`` always generates a dependency graph showing process dependencies (``dependency_graph.eps`` and``dependency_graph.pdf`` file). Optionally, it can also generate a process graph (``process_graph.eps`` and ``process_graph.pdf``), where the relationship between process options is shown. The process graph is generated if the option ``--gen-proc-graph`` is provided to ``debasher_exec``. For both diagrams, a version in `Graphviz `__ format is also provided (files with ``dot`` extension). In addition to this, the output directory will contain the following files automatically generated by ``debasher_exec``: * ``comand_line.sh``: stores the command line used to execute the program. * ``program.fifos``: contains information about the FIFOs used by the program. See :ref:`implem` Section for more details about using FIFOs in DeBasher programs. * ``program.opts``: contains an exhaustive list of all the options provided to the processes that compose the program to be executed. * ``program.procspec``: contains a specification for each process executed within a given program. Such specification incorporates the resources utilized by each process, as well as its process dependencies (e.g. the execution of a process may depend on the successful completion of another process). Process Status Visualization ---------------------------- During or after the execution of a DeBasher program, it is possible to use the ``debasher_status`` tool to get information about the execution status of each process involved in the program. We can apply ``debasher_status`` to check the status of the process involved in the "Hello World!" program previously executed: :: $ debasher_status -d out The output of the tool is: :: PROCESS: hello_world ; STATUS: FINISHED * SUMMARY: num_processes= 1 ; finished= 1 ; inprogress= 0 ; unfinished= 0 ; unfinished_but_runnable= 0 ; todo= 0 ``debasher_status`` show the status of the ``hello_world`` process as ``FINISHED``. The tool also shows a summary of the all of the processes statuses involved in the program. DeBasher works with the following process statuses: * ``finished``: the process successfully completed execution. * ``inprogress``: the process is currently being executed. * ``unfinished``: the process did not successfully complete execution. * ``unfinished_but_runnable``: the process has not completed execution yet, and is not being executed. However, it can resume its execution. * ``todo``: the process has not yet started execution. The ``debasher_status`` tool can also show the status of an individual process using the ``-p`` option: :: $ debasher_status -d out -p hello_world Program Stop ------------ When a DeBasher program is being executed, it is possible to stop a given process or the whole program by means of the ``debasher_stop`` tool. For this purpose, the name of the output directory and, optionally, the name of a specific process should be provided. The following command stops the ``hello_word`` process that compose the "Hello World!" program that was mentioned above: :: $ debasher_stop -d out -p hello_world To stop the whole program, the ``-p`` option is omitted: :: $ debasher_stop -d out Program Statistics Generation ----------------------------- The ``debasher_stats`` tool allows to generate statistics for a DeBasher program. Specifically, the tool report process statuses and the elapsed time in seconds until completion (if the process was executed). Again, the execution of this tool requires to provide the output directory of the program and, optionally, the name of the process for which we want to obtain the statistics. This command shows statistics for the ``hello_world`` process that belongs to the "Hello World!" program that we executed at the :ref:`prg_exec` Section: :: $ debasher_stats -d out -p hello_world To get statistics for all processes, we can simply type: :: $ debasher_stats -d out