Running advanced jobs on the HEC

The following sections highlight some of the more advanced features of the LSF commands. For an exhaustive list of all the features offered, please see the relevant online manual pages by logging on to the HEC and typing:

man command

This will display the man page for the specified command.

• Large memory jobs
• Job arrays: Submitting multiple jobs

Large memory jobs

As the job scheduler cannot predict in advance how much memory your jobs will consume, it is important for you to declare the amount of memory required so that the scheduler can make sure that it is sent to a compute node with enough memory to support it. Jobs over 500M in size are classed as large memory jobs and must specify their memory requirements. It's important to declare memory usage as closely as possible - specifying far more memory than a job requires means that the scheduler reserves that memory for you whether you use it or not, resulting in it not being available to other users. Bear in mind that in order to guard against jobs with run-away memory leaks which might adversely affect other users, the scheduler will terminate any job which exceeds its requested memory requirement. As a rule of thumb: request slightly more memory than your job requires, but not too much.

Quick Guide:

For jobs greater than 500 megabytes, calculate the amount of memory your job requires as closely as possible in megabytes then then for memory size X submit the job with:

Alternatively, you can add a similar directive directly into the job script by adding the line:

Job arrays: Submitting multiple jobs

For tasks such as Monte Carlo simulations and parameter studies, it is often necessary to run the same program multiple times, often with slightly different input parameters. Rather than create a unique job file for each run and submit each of them separately, LSF offers a job array option. When combined with a tailored job script, this allows you to submit multiple similar jobs with a single command.

Job arrays can be submitted by appending a job array directive to the end of the job's name directive, as in the following example:

Submits the job script myjob.com as a number of tasks, each task having its own unique index number. This index number can be used by the job script to perform slightly different actions each time, e.g. reading from a different input file (as in the above example), or passing a different set of parameters to your program for each task.

The number of submissions, and the values of the index numbers, are controlled by the extra arguments within square brackets after the job name. The format is x-y:z, where x is the first index number, y the last, and the optional :z gives the step increment. The above example submits the job script 4 times, with index numbers of 4, 6, 8 and 10 (ie, first = 4, last = 10, step = 2). NOTE: Index numbers must always be positive integers.

The index number is available to the job script via the environment variable $LSB_JOBINDEX, and can be used by the job script to alter what exactly is run for each task. In the above example, it is used to change the input file sent to the user application my_program. Successive tasks will read input.4.dat, up to input.10.dat

Each task in the job array can be given a unique output filename using the token %I in the output file directives. In the above directive, each array output file will be given a unique name through use of both %J and %I - job output filenames will be created using both the index number (via %I) AND the job id (via %J). NOTE: It's very important for task array elements to have unique output file names as in the example above. If they're not unique, then every element will try to write to the same file in an uncontrolled manner, which will result in output from one job element being overwritten by another.

Managing job arrays

Once you get the hang of writing flexible job scripts, job arrays make job submission much easier. They also make job management easier too. All tasks within the same job are given a different index number, but all have the same job id. An example output from bjobs for a job array is given below. As with the job submission script format, each task's index number appears in square brackets after the job name,

JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
7002    paceytm RUN   normal     login01     enc01       myjob[4]   Oct 15 14:21
7002    paceytm RUN   normal     login01     enc09       myjob[5]   Oct 15 14:21
7002    paceytm RUN   normal     login01     enc07       myjob[6]   Oct 15 14:21
7002    paceytm RUN   normal     login01     enc01       myjob[7]   Oct 15 14:21
7002    paceytm RUN   normal     login01     enc09       myjob[8]   Oct 15 14:21
7002    paceytm RUN   normal     login01     enc08       myjob[9]   Oct 15 14:21

If you want to stop all the tasks at once, you can use the normal bkill command:

Note: For large job arrays, it may take several minutes to kill all jobs

If you want to stop individual jobs, you can suffix the job id with the individual task id. To stop just task 4, we do:

To stop the first three jobs, do:

Managing short jobs

Care should be taken to avoid very short jobs - on the order of a few seconds to a few minutes - as these make very inefficient use of the cluster. It takes the system several seconds both to start and finish a job, and the scheduler itself works on 40 second cycles. Very short job therefore end up causing a lot of idle time on the system. To avoid this, consider bunching several short tasks together into a single job array element.

The example below gives a template for this type of solution. A job array originally of 10000 individual elements each of which ran for only a few seconds has been converted into one containing just 10 elements, with each element looping over the 1000 tasks, depending on the job array ID it receives:

How it works: The -J BSUB directive is set up the job array to run from 1 to 9001 in steps of 1000. This will result in 10 separate task array elements, with LSB_JOBINDEX containing values of 1, 1001, 2001, up to 9001. The shell variable x is set to the LSB_JOBINDEX value, and shell variable y is set to the last value in the set of tasks to be run, using some simple shell arithemtic and LSF's LSB_JOBINDEX_STEP shell variable, which is automatically set to the stepping size of current array (1000 in this case). The for loop sets shell variable z to all values between x and y in sequence using the standard unix tool seq. Each iteration of the loop will run myprogram using unique input and output files, based upon the value of z.

Note that while this example still produces 10,000 output files named output.$z.data for all values of z between 1 and 10,000, there are only 10 job array elements so there will only be 10 stdout and stderr files as specified in the -oo and -eo BSUB directives.

©Lancaster University   Computer User Agreement   Privacy Statement