hpc2: PBS job submittal

Steps in PBS job creation, submital, monitoring, and job deletion if necessary.

   
1) Login in to hpc2 using the ssh command
     All PBS jobs submitted on hpc2 will run on hpc2.

2) Make sure that you have the correct PATH. 
     The files /var/sysadm/config/default.cshrc  and /var/sysadm/config/default.login
     can be copied to your ~/.cshrc an ~/.login files respectively and will
     set an appropriate path.
    
3) Create PBS script (assume it is called myjob)
     An easy way is to use PBSwriter and copy the web page text to a file. 
     You can use this script on either hpc1 or hpc2.
     If you need temoporary disk space, use $TMPDIR in your script, this comes from
     51 Gbytes of fast striped disk. A separate $TMPDIR is used for each job.

4) qsub myjob
     This command submits the PBS script in the file myjob. You may submit several
     jobs in succession if they use different output files. Jobs will be scheduled
     for queues based on the resources requested. In addition, jobs by HPC partners
     and class accounts will have priority over other jobs which request similar 
     resources. Further, there are limits on each queue regarding the maximum
     number of simultaneous jobs by one user or group.

5) qstat -a
    This shows all jobs in the system.  If you don't see your job, it has completed 
    execution. Look at the output file and error files (PBSwriter names the output 
    file BATCH_OUTPUT and the error file BATCH_ERRORS).


 * Restrictions and gotchas :
   1) It is best to explicitly cd to the appropriate directory before
      execution. E.g. cd /scratch2/my_username .
      This ensures that you are in the directory of your choice.

   2) The output files named on the #PBS -o and #PBS -e commands need fully 
      qualified names.  These files must be different, or you will end up 
      with empty files.