Example

This example walks through a common usage of the package with some mock executables.

Suppose now we want to build a workflow of jobs to analyze some simulated data. The workflow consists of

  1. Generate simulated data with the executable generate-data.

  2. Analyze the simulated data with the executable analyze-data.

  3. Post-process the analysis results with the executable postprocess-data, and at the same time generate plots of the analysis results with the executable plot.

  4. Summerize the post-processed results and plots in a PDF file with the executable summary.

Be reminded that the above are all mock executables which serve for illustration only. You should not try to find those executables in your machine.

Outline of the walk-through:

from slurmpter import SlurmJob, Slurm

# Define the error, output and submit directories.
error = "slurm/error"
output = "slurm/output"
submit = "slurm/submit"

# Instantiate a Slurm object which defines the graph of workflow.
slurm = Slurm(name="slurm", submit=submit)

################
# Generate data
################

# Instantiate a SlurmJob object for generating simulated data.
job_gen = SlurmJob(name="generate_data",
                   executable="generate-data",
                   submit=submit,
                   output=output,
                   error=error,
                   slurm=slurm)
# We would like to generate two sets of data.
job_gen.add_arg("--output data_0.txt --input input_0.txt")
job_gen.add_arg("--output data_1.txt --input input_1.txt")

###############
# Analyze data
###############

# Instantiate a SlurmJob object for analyzing the simulated data.
job_analyze = SlurmJob(name="analyze_data",
                       executable="analyze-data",
                       submit=submit,
                       output=output,
                       error=error,
                       slurm=slurm)
# Since we must have the simulated data to be generated before doing the analysis, the job_gen is the parent job of job_analyze.
job_analyze.add_parent(job_gen)
# We analyze the two sets of data and get the result outputs.
job_analyze.add_arg("--output result_0.txt --input data_0.txt")
job_analyze.add_arg("--output result_1.txt --input data_1.txt")

#######################
# Post-process results
#######################

# Instantiate a SlurmJob object for post-processing the analysis results, e.g. merging the results etc..
job_postprocess = SlurmJob(name="postprocess_data",
                           executable="postprocess-data",
                           submit=submit,
                           output=output,
                           error=error,
                           slurm=slurm)
# The result outputs have to be ready before we do the post-processing, so the job_analyze is the parent job of job_postprocess.
job_postprocess.add_parent(job_analyze)
# We post-process the result outputs i.e. result_0.txt and result_1.txt.
job_postprocess.add_arg("--output proprocessed_results.txt --input result_0.txt result_1.txt")

###################
# Plot the results
###################

# Instantiate a SlurmJob object for plotting the analysis results.
job_plot = SlurmJob(name="plot",
                    executable="plot",
                    submit=submit,
                    output=output,
                    error=error,
                    slurm=slurm)
# Plotting only needs the analysis results, so it can happen concurrently with post-processing.
job_plot.add_parent(job_analyze)
# Generate plots of each of the analysis results.
job_plot.add_arg("--output result_0_plot.pdf --input result_0.txt")
job_plot.add_arg("--output result_1_plot.pdf --input result_1.txt")

#####################
# Generate a summary
#####################

# Instantiate a SlurmJob object for generating summary.
job_summary = SlurmJob(name="summary",
                       executable="generate-summary",
                       submit=submit,
                       output=output,
                       error=error,
                       slurm=slurm)
# The summary file needs the post-processed result and the plots.
job_summary.add_parents([job_postprocess,job_plot])
# Generate the summary as a PDF document.
job_summary.add_arg("--output summary.pdf --input processed_results.txt result_0_plot.pdf result_1_plot.pdf")

# Build the submit files.
slurm.build()
# Call build_submit() if you want to submit the jobs immediately after the build.
# slurm.build_submit()

You should see the following submit files in slurm/submit if they are successfully built.

slurm_<date>_01.submit:

#!/bin/bash
#SBATCH --job-name=slurm_<date>_01
#SBATCH --output=slurm/submit/slurm_<date>_01.output
#SBATCH --error=slurm/submit/slurm_<date>_01.error

jid0=($(sbatch slurm/submit/generate_data_<date>_01.submit))
jid1=($(sbatch --dependency=afterok:${jid0[-1]} slurm/submit/analyze_data_<date>_01.submit))
jid2=($(sbatch --dependency=afterok:${jid1[-1]} slurm/submit/postprocess_data_<date>_01.submit))
jid3=($(sbatch --dependency=afterok:${jid1[-1]} slurm/submit/plot_<date>_01.submit))
jid4=($(sbatch --dependency=afterok:${jid2[-1]}:${jid3[-1]} slurm/submit/summary_<date>_01.submit))

generate_data_<date>_01.submit:

#!/bin/bash
#SBATCH --job-name=generate_data_<date>_01
#SBATCH --output=slurm/output/generate_data_<date>_01.output
#SBATCH --error=slurm/error/generate_data_<date>_01.error

srun --ntasks=1 --exclusive generate-data --output data_0.txt --input input_0.txt &
srun --ntasks=1 --exclusive generate-data --output data_1.txt --input input_1.txt &
wait

analyze_data_<date>_01.submit:

#!/bin/bash
#SBATCH --job-name=analyze_data_<date>_01
#SBATCH --output=slurm/output/analyze_data_<date>_01.output
#SBATCH --error=slurm/error/analyze_data_<date>_01.error

srun --ntasks=1 --exclusive analyze-data --output result_0.txt --input data_0.txt &
srun --ntasks=1 --exclusive analyze-data --output result_1.txt --input data_1.txt &
wait

postprocess_data_<date>_01.submit:

#!/bin/bash
#SBATCH --job-name=postprocess_data_<date>_01
#SBATCH --output=slurm/output/postprocess_data_<date>_01.output
#SBATCH --error=slurm/error/postprocess_data_<date>_01.error

srun --ntasks=1 --exclusive postprocess-data --output proprocessed_results.txt --input result_0.txt result_1.txt &
wait

plot_<date>_01.submit:

#!/bin/bash
#SBATCH --job-name=plot_<date>_01
#SBATCH --output=slurm/output/plot_<date>_01.output
#SBATCH --error=slurm/error/plot_<date>_01.error

srun --ntasks=1 --exclusive plot --output result_0_plot.pdf --input result_0.txt &
srun --ntasks=1 --exclusive plot --output result_1_plot.pdf --input result_1.txt &
wait

summary_<date>_01.submit:

#!/bin/bash
#SBATCH --job-name=summary_<date>_01
#SBATCH --output=slurm/output/summary_<date>_01.output
#SBATCH --error=slurm/error/summary_<date>_01.error

srun --ntasks=1 --exclusive summary --output summary.pdf --input processed_results.txt result_0_plot.pdf result_1_plot.pdf &
wait

You can either call slurm.build_submit() to squentially build and submit the jobs or call slurm.submit_slurm() to submit the jobs after the submit files are built. Alternatively, you can type the following command to submit the jobs after the submit files are built:

sbatch slurm_<date>_01.submit

SlurmJob and Slurm objects

from slurmpter import SlurmJob, Slurm

Slurm is a collection of SlurmJob which could have different priorities to run using Slurm.

Job directories

error = "slurm/error"
output = "slurm/output"
submit = "slurm/submit"

The submit files are built in the submit directory, and also the standard output and standard error from Slurm are written into the submit directory. The standard output and standard error from the SlurmJob are written into the output and error directories respectively.

Construct a Slurm object

# Instantiate a Slurm object which defines the graph of workflow.
slurm = Slurm(name="slurm", submit=submit)

You have to first construct a Slurm which will hold all the SlurmJob. The name of the object defines the prefix of the ouput submit file, and therefore you should use different names for all Slurm and SlurmJob objects.

Construct a SlurmJob object for generating simulated data

################
# Generate data
################

# Instantiate a SlurmJob object for generating simulated data.
job_gen = SlurmJob(name="generate_data",
                   executable="generate-data",
                   submit=submit,
                   output=output,
                   error=error,
                   slurm=slurm)
# We would like to generate two sets of data.
job_gen.add_arg("--output data_0.txt --input input_0.txt")
job_gen.add_arg("--output data_1.txt --input input_1.txt")

Usage of generate-data:

generate-data --output <output> --input <input>``

generate-data ingests the input file <input> and writes the simulated data to <output>.

name defines the name of the job. executable is the name of the executable. You should make sure executable can be found in PATH. slurm=slurm adds this SlurmJob object to the Slurm object slurm that we created above.

Two arguments are added to the job via job_gen.add_arg(). The SlurmJob then defines two independent runs with the commands

generate-data --output data_0.txt --input input_0.txt

generate-data --output data_1.txt --input input_1.txt

respectively. The two runs can occur concurrently.

Construct a SlurmJob object for analyzing data

###############
# Analyze data
###############

# Instantiate a SlurmJob object for analyzing the simulated data.
job_analyze = SlurmJob(name="analyze_data",
                       executable="analyze-data",
                       submit=submit,
                       output=output,
                       error=error,
                       slurm=slurm)
# Since we must have the simulated data to be generated before doing the analysis, the job_gen is the parent job of job_analyze.
job_analyze.add_parent(job_gen)
# We analyze the two sets of data and get the result outputs.
job_analyze.add_arg("--output result_0.txt --input data_0.txt")
job_analyze.add_arg("--output result_1.txt --input data_1.txt")

Usage of analyze-data:

analyze-data --output <output> --input <input>

analyze-data ingests the simulated data <input> and outputs the analysis result to <output>.

Here the job_analyze ingests the output files data_0.txt and data_1.txt from job_gen and writes the anlysis results to `result_0.txt and result_1.txt respectively. job_analyze.add_parent(job_gen) forces the job_analyze to start after job_gen has completed all the runs and all exit normally.

Construct a SlurmJob object for postprocessing

#######################
# Post-process results
#######################

# Instantiate a SlurmJob object for post-processing the analysis results, e.g. merging the results etc..
job_postprocess = SlurmJob(name="postprocess_data",
                           executable="postprocess-data",
                           submit=submit,
                           output=output,
                           error=error,
                           slurm=slurm)
# The result outputs have to be ready before we do the post-processing, so the job_analyze is the parent job of job_postprocess.
job_postprocess.add_parent(job_analyze)
# We post-process the result outputs i.e. result_0.txt and result_1.txt.
job_postprocess.add_arg("--output proprocessed_results.txt --input result_0.txt result_1.txt")

Usage of postprocess-data:

postprocess-data --output <output> --input <intput>

postprocess-data ingests the result files <input> and writes the processed data to <output>.

The job must occur after job_analyze completes all the runs to have the result files ready for post-processing.

Construct a SlurmJob object for plotting result

###################
# Plot the results
###################

# Instantiate a SlurmJob object for plotting the analysis results.
job_plot = SlurmJob(name="plot",
                    executable="plot",
                    submit=submit,
                    output=output,
                    error=error,
                    slurm=slurm)
# Plotting only needs the analysis results, so it can happen concurrently with post-processing.
job_plot.add_parent(job_analyze)
# Generate plots of each of the analysis results.
job_plot.add_arg("--output result_0_plot.pdf --input result_0.txt")
job_plot.add_arg("--output result_1_plot.pdf --input result_1.txt")

Usage of plot:

plot --output <output> --input <input>

plot ingests a result file <input> and generates the plot to <output>.

Similar to job_postprocess, job_plot must occur after job_analyze to have all the result files ready for plotting.

Notice that the parents of job_postprocess and job_plot are both job_analyze, but job_postprocess and job_plot do not depend on each other. job_postprocess and job_plot can occur concurrently after job_analyze finishes.

Construct a SlurmJob object for generating a summary

#####################
# Generate a summary
#####################

# Instantiate a SlurmJob object for generating summary.
job_summary = SlurmJob(name="summary",
                       executable="generate-summary",
                       submit=submit,
                       output=output,
                       error=error,
                       slurm=slurm)
# The summary file needs the post-processed result and the plots.
job_summary.add_parents([job_postprocess,job_plot])
# Generate the summary as a PDF document.
job_summary.add_arg("--output summary.pdf --input processed_results.txt result_0_plot.pdf result_1_plot.pdf")

Usage of summary:

summary --output <output> --input <intput>

summary ingests the post-processed data and plots <input> and writes a summary file to <output>.

job_summary depends on two jobs i.e. job_postprocess and job_plot. You can add multiple jobs as the parents at the same time by passing a list of jobs to add_parents.

Build and submit the jobs

# Build the submit files.
slurm.build()
# Call build_submit() if you want to submit the jobs immediately after the build.
# slurm.build_submit()

You can either call build() to build the submit files but do not submit the jobs yet, or call build_submit() to build the submit files and then submit the jobs immediately.

Visualize the workflow

slurm.visualize("workflow.pdf")

You can generate a figure of the workflow by calling visualize() of the Slurm object.

_images/workflow.png