Example¶
This example walks through a common usage of the package with some mock executables.
Suppose now we want to build a workflow of jobs to analyze some simulated data. The workflow consists of
Generate simulated data with the executable
generate-data
.Analyze the simulated data with the executable
analyze-data
.Post-process the analysis results with the executable
postprocess-data
, and at the same time generate plots of the analysis results with the executableplot
.Summerize the post-processed results and plots in a PDF file with the executable
summary
.
Be reminded that the above are all mock executables which serve for illustration only. You should not try to find those executables in your machine.
Outline of the walk-through:
from slurmpter import SlurmJob, Slurm
# Define the error, output and submit directories.
error = "slurm/error"
output = "slurm/output"
submit = "slurm/submit"
# Instantiate a Slurm object which defines the graph of workflow.
slurm = Slurm(name="slurm", submit=submit)
################
# Generate data
################
# Instantiate a SlurmJob object for generating simulated data.
job_gen = SlurmJob(name="generate_data",
executable="generate-data",
submit=submit,
output=output,
error=error,
slurm=slurm)
# We would like to generate two sets of data.
job_gen.add_arg("--output data_0.txt --input input_0.txt")
job_gen.add_arg("--output data_1.txt --input input_1.txt")
###############
# Analyze data
###############
# Instantiate a SlurmJob object for analyzing the simulated data.
job_analyze = SlurmJob(name="analyze_data",
executable="analyze-data",
submit=submit,
output=output,
error=error,
slurm=slurm)
# Since we must have the simulated data to be generated before doing the analysis, the job_gen is the parent job of job_analyze.
job_analyze.add_parent(job_gen)
# We analyze the two sets of data and get the result outputs.
job_analyze.add_arg("--output result_0.txt --input data_0.txt")
job_analyze.add_arg("--output result_1.txt --input data_1.txt")
#######################
# Post-process results
#######################
# Instantiate a SlurmJob object for post-processing the analysis results, e.g. merging the results etc..
job_postprocess = SlurmJob(name="postprocess_data",
executable="postprocess-data",
submit=submit,
output=output,
error=error,
slurm=slurm)
# The result outputs have to be ready before we do the post-processing, so the job_analyze is the parent job of job_postprocess.
job_postprocess.add_parent(job_analyze)
# We post-process the result outputs i.e. result_0.txt and result_1.txt.
job_postprocess.add_arg("--output proprocessed_results.txt --input result_0.txt result_1.txt")
###################
# Plot the results
###################
# Instantiate a SlurmJob object for plotting the analysis results.
job_plot = SlurmJob(name="plot",
executable="plot",
submit=submit,
output=output,
error=error,
slurm=slurm)
# Plotting only needs the analysis results, so it can happen concurrently with post-processing.
job_plot.add_parent(job_analyze)
# Generate plots of each of the analysis results.
job_plot.add_arg("--output result_0_plot.pdf --input result_0.txt")
job_plot.add_arg("--output result_1_plot.pdf --input result_1.txt")
#####################
# Generate a summary
#####################
# Instantiate a SlurmJob object for generating summary.
job_summary = SlurmJob(name="summary",
executable="generate-summary",
submit=submit,
output=output,
error=error,
slurm=slurm)
# The summary file needs the post-processed result and the plots.
job_summary.add_parents([job_postprocess,job_plot])
# Generate the summary as a PDF document.
job_summary.add_arg("--output summary.pdf --input processed_results.txt result_0_plot.pdf result_1_plot.pdf")
# Build the submit files.
slurm.build()
# Call build_submit() if you want to submit the jobs immediately after the build.
# slurm.build_submit()
You should see the following submit files in slurm/submit
if they are successfully built.
slurm_<date>_01.submit
:
#!/bin/bash
#SBATCH --job-name=slurm_<date>_01
#SBATCH --output=slurm/submit/slurm_<date>_01.output
#SBATCH --error=slurm/submit/slurm_<date>_01.error
jid0=($(sbatch slurm/submit/generate_data_<date>_01.submit))
jid1=($(sbatch --dependency=afterok:${jid0[-1]} slurm/submit/analyze_data_<date>_01.submit))
jid2=($(sbatch --dependency=afterok:${jid1[-1]} slurm/submit/postprocess_data_<date>_01.submit))
jid3=($(sbatch --dependency=afterok:${jid1[-1]} slurm/submit/plot_<date>_01.submit))
jid4=($(sbatch --dependency=afterok:${jid2[-1]}:${jid3[-1]} slurm/submit/summary_<date>_01.submit))
generate_data_<date>_01.submit
:
#!/bin/bash
#SBATCH --job-name=generate_data_<date>_01
#SBATCH --output=slurm/output/generate_data_<date>_01.output
#SBATCH --error=slurm/error/generate_data_<date>_01.error
srun --ntasks=1 --exclusive generate-data --output data_0.txt --input input_0.txt &
srun --ntasks=1 --exclusive generate-data --output data_1.txt --input input_1.txt &
wait
analyze_data_<date>_01.submit
:
#!/bin/bash
#SBATCH --job-name=analyze_data_<date>_01
#SBATCH --output=slurm/output/analyze_data_<date>_01.output
#SBATCH --error=slurm/error/analyze_data_<date>_01.error
srun --ntasks=1 --exclusive analyze-data --output result_0.txt --input data_0.txt &
srun --ntasks=1 --exclusive analyze-data --output result_1.txt --input data_1.txt &
wait
postprocess_data_<date>_01.submit
:
#!/bin/bash
#SBATCH --job-name=postprocess_data_<date>_01
#SBATCH --output=slurm/output/postprocess_data_<date>_01.output
#SBATCH --error=slurm/error/postprocess_data_<date>_01.error
srun --ntasks=1 --exclusive postprocess-data --output proprocessed_results.txt --input result_0.txt result_1.txt &
wait
plot_<date>_01.submit
:
#!/bin/bash
#SBATCH --job-name=plot_<date>_01
#SBATCH --output=slurm/output/plot_<date>_01.output
#SBATCH --error=slurm/error/plot_<date>_01.error
srun --ntasks=1 --exclusive plot --output result_0_plot.pdf --input result_0.txt &
srun --ntasks=1 --exclusive plot --output result_1_plot.pdf --input result_1.txt &
wait
summary_<date>_01.submit
:
#!/bin/bash
#SBATCH --job-name=summary_<date>_01
#SBATCH --output=slurm/output/summary_<date>_01.output
#SBATCH --error=slurm/error/summary_<date>_01.error
srun --ntasks=1 --exclusive summary --output summary.pdf --input processed_results.txt result_0_plot.pdf result_1_plot.pdf &
wait
You can either call slurm.build_submit()
to squentially build and submit the jobs or call slurm.submit_slurm()
to submit the jobs after the submit files are built. Alternatively, you can type the following command to submit the jobs after the submit files are built:
sbatch slurm_<date>_01.submit
SlurmJob and Slurm objects¶
from slurmpter import SlurmJob, Slurm
Slurm
is a collection of SlurmJob
which could have different priorities to run using Slurm.
Job directories¶
error = "slurm/error"
output = "slurm/output"
submit = "slurm/submit"
The submit files are built in the submit
directory, and also the standard output and standard error from Slurm
are written into the submit
directory. The standard output and standard error from the SlurmJob
are written into the output
and error
directories respectively.
Construct a Slurm object¶
# Instantiate a Slurm object which defines the graph of workflow.
slurm = Slurm(name="slurm", submit=submit)
You have to first construct a Slurm
which will hold all the SlurmJob
. The name
of the object defines the prefix of the ouput submit file, and therefore you should use different names for all Slurm
and SlurmJob
objects.
Construct a SlurmJob object for generating simulated data¶
################
# Generate data
################
# Instantiate a SlurmJob object for generating simulated data.
job_gen = SlurmJob(name="generate_data",
executable="generate-data",
submit=submit,
output=output,
error=error,
slurm=slurm)
# We would like to generate two sets of data.
job_gen.add_arg("--output data_0.txt --input input_0.txt")
job_gen.add_arg("--output data_1.txt --input input_1.txt")
Usage of generate-data
:
generate-data --output <output> --input <input>``
generate-data
ingests the input file <input>
and writes the simulated data to <output>
.
name
defines the name of the job. executable
is the name of the executable. You should make sure executable
can be found in PATH
. slurm=slurm
adds this SlurmJob
object to the Slurm
object slurm
that we created above.
Two arguments are added to the job via job_gen.add_arg()
. The SlurmJob
then defines two independent runs with the commands
generate-data --output data_0.txt --input input_0.txt
generate-data --output data_1.txt --input input_1.txt
respectively. The two runs can occur concurrently.
Construct a SlurmJob object for analyzing data¶
###############
# Analyze data
###############
# Instantiate a SlurmJob object for analyzing the simulated data.
job_analyze = SlurmJob(name="analyze_data",
executable="analyze-data",
submit=submit,
output=output,
error=error,
slurm=slurm)
# Since we must have the simulated data to be generated before doing the analysis, the job_gen is the parent job of job_analyze.
job_analyze.add_parent(job_gen)
# We analyze the two sets of data and get the result outputs.
job_analyze.add_arg("--output result_0.txt --input data_0.txt")
job_analyze.add_arg("--output result_1.txt --input data_1.txt")
Usage of analyze-data
:
analyze-data --output <output> --input <input>
analyze-data
ingests the simulated data <input>
and outputs the analysis result to <output>
.
Here the job_analyze
ingests the output files data_0.txt
and data_1.txt
from job_gen
and writes the anlysis results to `result_0.txt
and result_1.txt
respectively. job_analyze.add_parent(job_gen)
forces the job_analyze
to start after job_gen
has completed all the runs and all exit normally.
Construct a SlurmJob object for postprocessing¶
#######################
# Post-process results
#######################
# Instantiate a SlurmJob object for post-processing the analysis results, e.g. merging the results etc..
job_postprocess = SlurmJob(name="postprocess_data",
executable="postprocess-data",
submit=submit,
output=output,
error=error,
slurm=slurm)
# The result outputs have to be ready before we do the post-processing, so the job_analyze is the parent job of job_postprocess.
job_postprocess.add_parent(job_analyze)
# We post-process the result outputs i.e. result_0.txt and result_1.txt.
job_postprocess.add_arg("--output proprocessed_results.txt --input result_0.txt result_1.txt")
Usage of postprocess-data
:
postprocess-data --output <output> --input <intput>
postprocess-data
ingests the result files <input>
and writes the processed data to <output>
.
The job must occur after job_analyze
completes all the runs to have the result files ready for post-processing.
Construct a SlurmJob object for plotting result¶
###################
# Plot the results
###################
# Instantiate a SlurmJob object for plotting the analysis results.
job_plot = SlurmJob(name="plot",
executable="plot",
submit=submit,
output=output,
error=error,
slurm=slurm)
# Plotting only needs the analysis results, so it can happen concurrently with post-processing.
job_plot.add_parent(job_analyze)
# Generate plots of each of the analysis results.
job_plot.add_arg("--output result_0_plot.pdf --input result_0.txt")
job_plot.add_arg("--output result_1_plot.pdf --input result_1.txt")
Usage of plot
:
plot --output <output> --input <input>
plot
ingests a result file <input>
and generates the plot to <output>
.
Similar to job_postprocess
, job_plot
must occur after job_analyze
to have all the result files ready for plotting.
Notice that the parents of job_postprocess
and job_plot
are both job_analyze
, but job_postprocess
and job_plot
do not depend on each other. job_postprocess
and job_plot
can occur concurrently after job_analyze
finishes.
Construct a SlurmJob object for generating a summary¶
#####################
# Generate a summary
#####################
# Instantiate a SlurmJob object for generating summary.
job_summary = SlurmJob(name="summary",
executable="generate-summary",
submit=submit,
output=output,
error=error,
slurm=slurm)
# The summary file needs the post-processed result and the plots.
job_summary.add_parents([job_postprocess,job_plot])
# Generate the summary as a PDF document.
job_summary.add_arg("--output summary.pdf --input processed_results.txt result_0_plot.pdf result_1_plot.pdf")
Usage of summary
:
summary --output <output> --input <intput>
summary
ingests the post-processed data and plots <input>
and writes a summary file to <output>
.
job_summary
depends on two jobs i.e. job_postprocess
and job_plot
. You can add multiple jobs as the parents at the same time by passing a list of jobs to add_parents
.
Build and submit the jobs¶
# Build the submit files.
slurm.build()
# Call build_submit() if you want to submit the jobs immediately after the build.
# slurm.build_submit()
You can either call build()
to build the submit files but do not submit the jobs yet, or call build_submit()
to build the submit files and then submit the jobs immediately.
Visualize the workflow¶
slurm.visualize("workflow.pdf")
You can generate a figure of the workflow by calling visualize()
of the Slurm
object.