ARGO: Running Jobs

• Overview
• Torque
• Batch and interactive jobs
• Queues
• Environmental Variables
• Commands
• Submitting a job for execution
• Job Output and Management
• Node Selection and Properties
• My job does not run
• How do I see what's going on on a compute node
• What programs I've run on argo
• Web Monitor

How you run a program on the argo cluster is VERY DIFFERENT from how you run a job on a single machine with one or multiple CPUs (for example, tigger). And, for the matter, how you would run a program on a PC/laptop in either a Windows or Linux operating system.

To begin with, you do not run your executable on the machine (the master) where you create the executable. The following point cannot be emphasized enough:

The master node is ONLY for job creation and not job execution.

There are monitors that alert systems to user programs running on the master. Running a program on the master is a violation of ACCC policy and can result in suspension and termination of your argo account.

There are two types of programs that may be executed on the cluster:

• Sequential (also known as serial)
• Parallel

For the purposes of the ACCC cluster, a sequential job is a single instance program that runs on one and only one node.

A parallel job is composed of:

• Multiple instances of the same program running on different nodes with no internodal communication among the program instances (unfortunately, this model of execution is sometimes derisively referred to as embarrassingly parallel), or
• One or multiple instances of different programs running on different nodes with internodal communication among the instances.

Serial version of the classic hello_world program - source in C

#include <stdio.h>
void main(int argc, char** argv) {
    printf("Hello-world\n");
  }

Parallel version of the classic hello_world program using MPI - source in C

#include <stdio.h>
#include "mpi.h"

void main(int argc, char **argv) {
    int rank;
    int size;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    printf("Hello-world, I'm rank %d; Size is %d\n", rank, size);
    MPI_Finalize();
  }

Torque is a networked subsystem for submitting, monitoring, and controlling a workload of jobs on the cluster.

ALL USER JOBS MUST be run via torque.

Years ago, only batch jobs could execute on the cluster. THAT IS NOT THE CASE NOW: torque does not restrict jobs to just batch execution; interactive jobs with GUIs and users interacting with the GUI may be run.

• student_short
• student_medium
• student_long
• staff
• dedicated

For more information about how to tell your job which queue to use, click here.
For more information about how long a job may run on a particular queue, click here.

There are two environments, each with its own define variables, available to you:

• Your shell environment variables, and
• torque environmental variables

Shell environmental variables

Torque environmetal variables

Every user job has the following torque enviromental variables available to it:

The command for submitting a job on the cluster is:

qsub -V my_script

• For staff, the default is the staff queue.
• For students, the default is student_short.

• Include the queue name as an operand to the qsub command:
  
  
  qsub -V -q staff my_script


• Put the queue name in your script file:
  
  
  #PBS -q staff
  
  and submit the script without the queue name on the qsub line:
  
  
  qsub -V my_script

/home/homes51/mhoma/a.out

After submitting a program for execution, it is assigned an identifying number called a job id:

XXX.argo.cc.uic.edu

where XXX is the number (you can disregard the argo.cc.uic.edu portion).

To see the status of your job, use the qstat command:

qstat job id

For stdout and stderr, torque creates two files. (If you are not familar with stdout/stderr files, then click here.) The names of the two files are constructed from the job name, the letter e (for stderr) or o (for stdout), and the job number. So for the sample hello world program (listed earlier) that had job id 338, you would have the following files:

hello.o338 <-- this is stdout
hello.e338 <-- this is stderr

Let's take a look:

    -rw------- 1 mhoma sys   0 Jan 17 13:43 hello.e338
    -rw------- 1 mhoma sys 110 Jan 17 13:43 hello.o338

An empty error file is a good sign. Let's see what we have in stdout:

cat hello.o338

Gives:

My process rank ==> 4
My process rank ==> 3
My process rank ==> 2
My process rank ==> 1

And, that's what we should have.

Every node has multiple properties associated with it. The property that clients are most familiar is the node name. Properties may be used to identify and use a particular node. The properties of a particular node can be displayed by using the qmgr command. For example, to see the properties associated with argo1-1:

qmgr -c "list node argo1-1 | grep properties"

properties = Linux2.i86pc,amd,smp,staff

The property amd tells you that argo1-1 contains an Opteron processor. The property smp tells you that argo1-1 has dual processors (both of which, by virtue of the amd property, are Opterons).

The generic sytax of the qsub command is:

qsub -l nodes=node_spec[+node_spc...]

where node_spec is:

number | property[:property...] | number:property[:property...]

A series of examples follows:

November 9, 2011. This section no longer reflects current information. Systems will re-write it. Please do not use.

Before you read this section, make sure you understand the information presented regarding qsub. If not, click here

The two most likely causes of a job not running are:

• Requesting too many resources, and
• Requesting resources that are not available.

The two commands that are most useful to diagnose problems pertaining to jobs not running:

• tracejob
• checkjob

The operand for both commands is the job id.

The output of the checkjob can be very cryptic but the reason why the job is not running is there. For example, suppose a student issues the following command:

qsub -V -l nodes=4:ppn=3 -q student_short my_script

The job is assigned id 1277 but remains queued which is indicated by the capital Q in the S column (status) in the output of the qstat 1277 command:

  Job id  Name       User    Time Use S Queue
  ------  ---------- ----    ---- --- - ------- 
  1277    my_script  jsmith     0     Q student_short

If the student issues the checkjob 1277 command, the output will include the following (the output is too long and has been abbreviated HERE for the purposes of brevity):

Holds: Batch (hold reason: PolicyViolation)
Messages: procs too high (12 > 8)
PE: 12.00 StartPriority: 7
cannot select job 1277 for partition DEFAULT (job hold active)

Look closely: PolicyViolation: procs too high (12 >8)." The student is asking for twelve processors. Go back and take a look at the qsub command:

qsub -V -l nodes=4:ppn=3 -q student_short my_script

Four nodes multiplied by three processors per node results in twelve processors. But, the student is limited to a total of eight processors on the student_short queue:

qmgr -c "list queue student_short" | grep resources_max.ncpus

resources_max.ncpus = 8

qsub -V -l nodes=4:ppn=2 -q student_short my_script

Remember to delete the queued job:

qdel 1277

Though you can't login to a compute node, you will see what's going on via the rsh command. For example, suppose you want to see what processes you are running. For processes on the master, it's the basic ps command:

ps -ef | grep netid

For a compute node, you would do the following:

rsh -l netid compute_node-name "ps -ef | grep netid"

rsh -l jsmith argo1-1 "ps -ef | grep jsmith"

Make sure you enclose the entire command (after the node name in double quotation marks:

rsh -l jsmith argo1-1 "ps -ef | grep jsmith"

All the shell commands are available. Suppose user jsmith had used the /tmp filesystem on the compute node as a scratch area for a temporary file and, upon completion of the job wants to erase it:

$ rsh -l jsmith argo1-1 "ls -al /tmp/junk1"

-rw-r--r-- 1 jsmith users 0 May 19 13:31 /tmp/junk1"
$ rsh -l jsmith argo1-1 "rm -f /tmp/junk1"
$ rsh -l jsmith argo1-1 "ls -al /tmp/junk1"

ls: /tmp/junk1: No such file or directory

Your job history is available. Click here to get a listing of all the months for which statistics are available. Or, you can go directly to the month by entering the URL:

http://www.uic.edu/depts/accc/hardware/argo/YYYYMM.html

where YYYY is replaced with the year and MM is replaced with the two-digit month.

Examples:
• http://www.uic.edu/depts/accc/hardware/argo/201001.html
• http://www.uic.edu/depts/accc/hardware/argo/200911.html
• http://www.uic.edu/depts/accc/hardware/argo/200805.html

http://www.uic.edu/depts/accc/hardware/argo/YYYYMM.stats.html

Examples:
• http://www.uic.edu/depts/accc/hardware/argo/201010.stats.html
• http://www.uic.edu/depts/accc/hardware/argo/200910.stats.html
• http://www.uic.edu/depts/accc/hardware/argo/200905.stats.html

There is also a very nice web-based tool to view the system. To access it, point your browser to the following URL:

https://argo.cc.uic.edu

Notice that you must use https and not the insecure http. You will be prompted for your netid and password before you are allowed in. If you can't access the tool, then it means you are not on campus or, if off campus, not using running a VPN on your computer. For more information about using a VPN, click here.