Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
habrok:advanced_job_management:job_arrays [2024/03/14 14:32] – [Output and job information for job arrays] pedrohabrok:advanced_job_management:job_arrays [2026/03/27 14:09] (current) – pedro
Line 5: Line 5:
 In order to create a job array, start by creating the job script that you would need to run just one instance of the job. For instance, consider the following job script for running R: In order to create a job array, start by creating the job script that you would need to run just one instance of the job. For instance, consider the following job script for running R:
  
-<code>+<code bash>
 #!/bin/bash #!/bin/bash
 #SBATCH --job-name=R_job #SBATCH --job-name=R_job
Line 17: Line 17:
 Now suppose you want this to be run 100 times. You can simply add the array definition by adding the following line somewhere at the top of the script: Now suppose you want this to be run 100 times. You can simply add the array definition by adding the following line somewhere at the top of the script:
  
-<code>+<code bash>
 #SBATCH --array=1-100 #SBATCH --array=1-100
 </code> </code>
Line 27: Line 27:
 So let us take it a step further: suppose we have 100 different R scripts that have to be run, which are named myscript1.r, myscript2.r, …, myscript100.r. Now we can use the aforementioned environment variable to pick the right R script for each job: So let us take it a step further: suppose we have 100 different R scripts that have to be run, which are named myscript1.r, myscript2.r, …, myscript100.r. Now we can use the aforementioned environment variable to pick the right R script for each job:
  
-<code>+<code bash>
 #!/bin/bash #!/bin/bash
 #SBATCH --job-name=R_job #SBATCH --job-name=R_job
Line 78: Line 78:
 A job array will get just one main job id, just like a regular job. However, the index values of the range will be used as suffix for the job id: <jobid>_1, <jobid>_2, etcetera. Furthermore, each job will produce its own output file with a filename like slurm_<jobid>_<index>.out. It is also possible to to provide a custom name for the slurm output file with ''$SBATCH --ouput=''. In normal circumstances a name such as ''R_job.out'' would be fine, however, with job arrays that would result in every job writing to the same output file, thus overwriting the previous ones. We can get around his by using ''%j'' and ''%a'', which here will be replaced with the job ID and array index, which would look like: ''$SBATCH --ouput=R_job_%j_%a.out'' A job array will get just one main job id, just like a regular job. However, the index values of the range will be used as suffix for the job id: <jobid>_1, <jobid>_2, etcetera. Furthermore, each job will produce its own output file with a filename like slurm_<jobid>_<index>.out. It is also possible to to provide a custom name for the slurm output file with ''$SBATCH --ouput=''. In normal circumstances a name such as ''R_job.out'' would be fine, however, with job arrays that would result in every job writing to the same output file, thus overwriting the previous ones. We can get around his by using ''%j'' and ''%a'', which here will be replaced with the job ID and array index, which would look like: ''$SBATCH --ouput=R_job_%j_%a.out''
  
-The same kind of job ids will also be used in the output of SLURM tools like squeue and sacct. The squeue command will usually try to combine the jobs in the array into a single line, e.g.:+The same kind of job ids will also be used in the output of SLURM tools like ''squeue'' and ''sacct''. The ''squeue'' command will usually try to combine the jobs in the array into a single line, e.g.:
  
 <code> <code>
Line 85: Line 85:
 </code> </code>
 \\ \\
-If you want to get each job of the array to appear on a separate line, you can pass the ''-r'' or ''—array'' option to squeue.+If you want to get each job of the array to appear on a separate line, you can pass the ''-r'' or ''—array'' option to ''squeue''.
  
 ===== Cancelling job arrays ===== ===== Cancelling job arrays =====