> For the complete documentation index, see [llms.txt](https://doane-ccla.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doane-ccla.gitbook.io/docs/using-the-hpc/execute-a-job.md). # Execute a Job The tutorial below shows you how to run a basic job on Doane's Onyx supercomputer. This tutorial is intended for users who are new to the HPC environment and leverages a Slurm batch (sbatch) script. Additional examples can be found in [C++](/docs/using-the-hpc/execute-a-job/cpp.md), [Fortran](/docs/using-the-hpc/execute-a-job/fortran.md) or [Python](/docs/using-the-hpc/execute-a-job/python.md) sections. **Table of Contents** * [Step 1: Access the Onyx HPC](/docs/using-the-hpc/execute-a-job.md#step-1-access-the-onyx-hpc) * [Step 2: Create an sbatch Script](/docs/using-the-hpc/execute-a-job.md#step-2-create-a-sbatch-script) * [Example sbatch Script](/docs/using-the-hpc/execute-a-job.md#example-sbatch-script) * [sbatch Script Breakdown](/docs/using-the-hpc/execute-a-job.md#sbatch-script-breakdown) * [sbatch Procedure](/docs/using-the-hpc/execute-a-job.md#sbatch-procedure) * [Step 3: Run the Job](/docs/using-the-hpc/execute-a-job.md#step-3-run-the-job) 📝 **Note:** Do not execute jobs on the login nodes; only use the login nodes to access your compute nodes. Processor-intensive, memory-intensive, or otherwise disruptive processes running on login nodes will be killed without warning. ## Step 1: Access the Onyx HPC 1. Open a Bash terminal (or MobaXterm for Windows users). 2. Execute `ssh doaneusername@onyx.doane.edu`. 3. If promted by a security message, type `yes` to continue connection. 4. When prompted, enter your password. Once you have connected to the head node, you can proceed to Step 2 and begin assembling your sbatch script. ## Step 2: Create an sbatch Script Below is the sbatch script we are using to run an MPI "hostname" program as a batch job. sbatch scripts use variables to specify things like the number of nodes and cores used to execute your job, estimated walltime for your job, and which compute resources to use (e.g., GPU *vs.* CPU). The sections below feature an example sbatch script for HPC resources, show you how to create and save your own sbatch script, and show you how store the sbatch script on an HPC file system. Consult the [official Slurm documentation](https://slurm.schedmd.com/sbatch.html) for a complete list of sbatch variables. ### Example sbatch Script Here is an example sbatch script for running a batch job on Onyx. We break down each command in the section below. ```bash #!/bin/bash #SBATCH -n 16 #SBATCH -o test_%A.out #SBATCH --error test_%A.err #SBATCH --mail-user $CHANGE_TO_YOUR_EMAIL #SBATCH --mail-type ALL srun -l hostname module avail module purge module load gnu/5.4.0 module load spack-apps module load openmpi-3.0.0-gcc-5.4.0-clbdgmf module list mpirun -np 16 hostname ``` ### sbatch Script Breakdown Here, we break down the essential elements of the above PBS script. * `#!/bin/bash`: sets the script type * `#SBATCH -n 8`: sets the number of processors that you want to use to run your job; use -N to specify nodes instead of processors * `#SBATCH -o test_%A.out`: sets the name of the output file; here `%A` will be replaced by slurm with the job number; * `#SBATCH --error test_%A.err`: sets the name of the error output file; here `%A` will be replaced by slurm with the job number; * `#SBATCH --mail-user $CHANGE_TO_YOUR_EMAIL`: add your email address if you would like your job's status to be emailed to you * `#SBATCH --mail-type ALL`: specifies which job status changes you want to be notified about; options include: NONE, BEGIN, END, FAIL, REQUEUE, ALL * `srun -l hostname`: Run a parallel job on cluster managed by Slurm, in this case run `hostname` in parallel with `-l` prepending task numbers to lines of output stdout/err. * `module avail`: Lists the currently available software modules. * `module purge`: Clears any modules currently loaded that might result in a conflict. * `module load gnu`: Loads the gnu module version 5.4.0. * `module load spack-apps`: Loads additional applications managed by the spack software. * `module load openmpi-3.0.0-gcc-5.4.0-clbdgmf`: Loads the openmpi module. * `module list`: confirms the modules that were loaded. * `mpirun -np 16 hostname`: `hostname` is called again, this time Slurm calls OpenMPI to run `hostname` with the number of processors we specified earlier. ### sbatch Procedure Now that we have covered the basics of a sbatch script in the context of an HPC, we will now talk about actually creating and using the script on Onyx. When creating and editing your sbatch script, we will be working on the head node using the text editor, nano. 1. From the login node, change your working directory to your home directory. ```bash cd ``` 2. Use nano to create and edit your sbatch script. ```bash nano slurm_example.job ``` 3. Write your sbatch script within nano or paste the contents of your sbatch script into nano. * Copy sbacth script from this page * Hit Control/Command + p key to paste into nano from Windows/MacOS 4. When finished, hit `^X` (control + x key) to exit. 5. Enter `Y` to save your changes, and press `Return` to save your file and return to the Bash shell. With the sbatch script in place, you can now move on to running the script in Step 3. ## Step 3: Run the Job 1. Before proceeding, ensure that you are still in your working directory (using `pwd`). * We need to be in the same path/directory as our sbatch script. Use `ls -al` to confirm its presence. 2. Use `sbatch` to schedule your batch job in the queue. ```bash sbatch slurm_example.job ``` This command will automatically queue your job using Slurm and produce a job number. You can check the status of your job at any time with the `squeue` command. ```bash squeue --job ``` `squeue --job ` is likely to not return any information, as this test job takes only a second to complete. You can also stop your job at any time with the `scancel` command. ```bash scancel --job ``` 3. View your results.\ Once your job completes, slurm will produce an output/data file. This output/data file, unless otherwise specified in the sbatch script, are placed in the same path as your binary.\ The file (`test_.out`) contains the results of the binary you just executed.\ Replace "myscript" with the name of your script and "\" with your job number.\ You can view the contents of these files using the `more` command followed by the file name.
```bash more test_.out ``` Your output should look something like this: ```bash 2: compute-1 10: compute-3 7: compute-2 13: compute-4 1: compute-1 3: compute-1 0: compute-1 8: compute-3 9: compute-3 11: compute-3 5: compute-2 6: compute-2 4: compute-2 15: compute-4 12: compute-4 14: compute-4 --------------------- /opt/ohpc/pub/moduledeps/gnu-openmpi --------------------- scipy/0.19.1 ------------------------- /opt/ohpc/pub/moduledeps/gnu ------------------------- R_base/3.3.3 openblas/0.2.20 python3/3.7.1 numpy/1.12.1 openmpi/1.10.7 (L) -------------------------- /opt/ohpc/pub/modulefiles --------------------------- cmake/3.11.1 ohpc (L) singularity/2.5.1 gnu/5.4.0 (L) pmix/2.1.1 spack-apps/0.11.2 gnu7/7.3.0 prun/1.2 (L) valgrind/3.13.0 Where: L: Module is loaded Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". Currently Loaded Modules: 1) gnu/5.4.0 2) openmpi/1.10.7 compute-1 compute-1 compute-1 compute-3 compute-1 compute-3 compute-3 compute-3 compute-2 compute-2 compute-2 compute-2 compute-4 compute-4 compute-4 compute-4 ``` 📝 **Note:** The number and order of the hostnames will be different for you. If you see any errors, try typing in the sbatch script by hand instead of copying and pasting it. Sometimes the clipboard of your OS will bring along extra hidden characters that confuse Bash and Slurm. 1. Download your results (using the `scp` command or an SFTP client) or move them to persistent storage. See our [moving data](https://github.com/Doane-CCLA/docs/tree/754804d05501a0d68096ef3d76dc511cad3e97e8/data-transfer-storage/moving-data.md) section for help. #### Additional Examples * [Working with C++](/docs/using-the-hpc/execute-a-job/cpp.md) * [Working with Fortran](/docs/using-the-hpc/execute-a-job/fortran.md) * [Working with Python](/docs/using-the-hpc/execute-a-job/python.md) * [Working with Makefiles](/docs/using-the-hpc/execute-a-job/makefile.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://doane-ccla.gitbook.io/docs/using-the-hpc/execute-a-job.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.