Egate
From CCMSTWiki
Contents |
Getting Started with Egate
Egate can be accessed using ssh software only. The cluster is accessed by logging into the master node: ssh username@egate.chemistry.gatech.edu. The internal name of the master node is control. Cluster nodes can be accessed from the master node using ssh. Nodes are named c1-1 through c1-79. Here's the information on Egate nodes:
| Host Names | Type | Specs | Type of use |
|---|---|---|---|
| c1-1 -- c1-69 | Thin nodes | Two 3.2 GHz Xeons, 4 GB RAM, 150 GB disk | LSF |
| c1-70 -- c1-77 | Fat nodes | Two 3.2 GHz Xeons, 8 GB RAM, 270 GB disk | LSF |
Running Jobs with LSF
Platform LSF is a batch system, like PBS, LoadLeveler, and others. LSF is the only legal way to run jobs on Egate. Look at ~evaleev/LSF/test.cmd for a sample LSF command file. Note that your priority in the queue is inversely proportional to the computer time you have used recently.
Most useful LSF commands: (See man pages for more information)
- bsub
- Submit a job to the queue. This must be used with the redirect when using a command file, i.e. bsub < test.cmd
- bjobs
- Monitor your own jobs in the queue. To see everyone's jobs use bjobs -u all
- blist
- Nicer version of bjobs (Perl script by David Sherrill); takes most bjobs arguments.
- bkill
- Kill a job in the queue with associated job number
- bmod
- Changes job resource requirements once a job has been submitted to a queue. Eg. bmod "-W 200:00" 12345 would change the time limit of job id 12345 to 200 hours.
- bhpart
- Shows the usage summary of the whole cluster by users. It sorts users according to their priority. If you notice that your jobs are not picking up, it is likely that people with higher priority have jobs pending.
- bhist
- Shows a summary of the amount of time your recent jobs have spent in various states (waiting, running, etc).
Here is a handy reference card of LSF commands: Image:Lsf user qrefcard 60.pdf.
A sample LSF script
#!/bin/csh # This specifies a job name #BSUB -J s_mp2_qz # This specifies a stdout logfile name #BSUB -o s_mp2_qz.stdout # This specifies max runtime in hours:minutes #BSUB -W 24:00 # This reserves one scratch and 3300 MB per processor # Memory specifications are mandatory, use Scratch only if you need it. #BSUB -R "rusage[Scratch=1:Memory=3300]" runmolprop s.in
Notes
- Memory keyword
- The old mem keyword has been replaced by Memory, which is properly accounted as a resource. The old mem specification actually checked whether or not that much memory was currently available on the machine, which would usually fail. To see current Memory allotments use "bhosts -s Memory" or a more complete listing of all resources with "bhosts -l".
- All LSF resources requirements are specified on a per processor basis.
- This is particularly important to remember when running parallel jobs. For example the above sample script is requesting a total of 13200 MB of memory, 3300 for each process. In the case of conflicts, the job will simply sit in the queue indefinitely. I.e., if the span[ptile=2] option was also specified, this job would never run because there is only one Scratch resource per node. Unfortunately, LSF does not reject jobs for which resource requests can't be met.
-
#BSUB -x- If you wish to reserve exclusive use of a node, you can do so using the -x option. This reserves the node only for you, and a second job will not pick up, even if you only use one processor.
- If you want to run parallel jobs under LSF -- come to see your system administrator.
- Selecting a Fat Node
-
#BSUB -R "select[defined(fatnode)] rusage[Memory=7200]"
-
- Be courteous to other users
- If you submit a serial job with >3500 MB of memory, most probably you will commandeer an entire thin node preventing almost everyone else from using the remaining processor. Of course there are exceptions, but perhaps consider requesting one of the fat nodes designed for this type of job (yes, even if it means you have to wait longer). At the very least check the queue and try to limit your hog factor if the machine is heavily loaded.
- LSF schedules future jobs based on the amount of time and resources each need to run. Therefore please try to be somewhat accurate with the -W specification.
Executing Command On Many Nodes At Once
The command name is cluster-fork. A quick example: cluster-fork "ps auxww | grep psi3"
will show where Psi3 is running. cluster-fork -h shows how to use the command
WARNING periodically nodes go down. Depending on how hard the failure was, cluster-fork may think
that the node is still available, try to run the command on it, and hang indefinitely. To avoid this,
there is a new command called cluster-fork-nohang which will skip the downed nodes. We recommend you
always use cluster-fork-nohang.
Cleaning Up After Yourself
If your job crashes on Egate, it may leave behind scratch files in /scratch/[username] on the node. We have automated scripts that clean these up periodically, but they are not always quick enough. If enough stale scratch files accumulate on a node, they can fill the disk, and then all jobs on that node afterward (if they use any disk) will crash. If you use significant disk and your job crashes, please log into the node ssh compute-x-x (where x-x is the node number) and remove your files inside /scratch/[username]. Do not remove the directory /scratch/[username] itself, you will need it.
If you want to check if you are using much scratch space, you can issue the following command which will check all the nodes for you
cluster-fork-nohang du -sk /scratch/[username]
This reports your usage of /scratch in kilobytes.
Parallel jobs using MPI can fail to clean up their Memory Usage. If a node is mysteriously using up memory with no active jobs running, you may need to clean up the memory. The ipcs command lists shared memory segments, and the ipcrm command removes them.
How To Run Specific Programs
ADF
The ADF license for Egate permits to execute up to 64 tasks at a time (reminder: a parallel job has several tasks). To keep track of the total number of tasks running ADF, you must submit all ADF jobs to queue "q_adf". To do that, add
#BSUB -q q_adf
to the command file.
To examine the number of ADF tasks currently running, do
bhosts -s ADF
Note that only nodes 1-32 can execute ADF jobs.
Jaguar
In order to run Jaguar, your job has to request Jaguar licenses. Here's how:
#BSUB -R "select[defined(Jaguar)] rusage[Jaguar=1]"
The first part selects hosts which have Jaguar license resource defined, the second part grabs one unit of the resource.
Here's an example of how to combine the resource requests:
#BSUB -R "select[defined(Jaguar)] rusage[Memory=200,Jaguar=1]"
i.e. I requested 200 MB and 1 Jaguar license.
In order to look up how many Jaguar licenses are available, do
bhosts -s Jaguar
A sample Jaguar command file is at /theoryfs/common/home/evaleev/tests/jaguar_test.
MPQC
MolPro
- Molpro never dies! If you use bkill to terminate a job, the process will be removed from the queue but keep running on the nodes. It is currently the user's responsibility to manage such instances. Please log in to the node(s) on which it was running and clean up.
NAMD
- NAMD 2.7 is now available (NEW!)
To run a serial version of NAMD, your script should be like this one:
#!/bin/csh # This specifies a job name #BSUB -J myJobName # This specifies an error logfile name #BSUB -e %J.err # This specifies a stdout logfile name #BSUB -o %J.out # This specifies max runtime in hours:minutes #BSUB -W 24:00 # This reserves 200 MB per processor # Memory specifications are MANDATORY. #BSUB -R "rusage[Memory=200]" /theoryfs/common/software/NAMD_2.7b1/namd2 config_file(s)
This script starts a parallel calculation with NAMD 2.7:
#!/bin/bash
#
# job name (enter your job name here)
#
#BSUB -J job_name
#
# stdout logfile name (edit with your logfile name)
#
#BSUB -o job_name.log
#
# error file name name (edit with your logfile name)
#
#BSUB -e job_name.err
#
# Number of processors (enter your desired number of processors)
#
#BSUB -n 4
#
# max runtime in hours:minutes (enter your limit, or delete for unlimited time)
#
#BSUB -W 00:20
#
# memory request (MB) per processor (substitute your estimate here. NOTE: this is mandatory!)
#
#BSUB -R "rusage[Memory=100]"
#
# Job proper starts here
#
# Configure NAMD environment
#
NAMDDIR=/theoryfs/common/software/NAMD_2.7b1
#
# use ssh for remote execution (the default would be rsh, which does not work)
export CONV_RSH=ssh
#
# set number of processors and hosts file. Some bash mumbo jumbo is used here
# (there should be no need of editing this part)
#
nprocs=0
echo "group main" > ${LSB_JOBID}_hosts
for host in ${LSB_HOSTS}; do
((nprocs= ${nprocs} + 1))
echo "host ${host}" >> ${LSB_JOBID}_hosts
done
echo "number of processes = ${nprocs}"
cat ${LSB_JOBID}_hosts
#
# run job
#
${NAMDDIR}/charmrun ${NAMDDIR}/namd2 +p${nprocs} ++nodelist ${LSB_JOBID}_hosts config_file(s)
This script runs a parallel calculation using the (old) 2.6 version of NAMD:
# This specifies a job name #BSUB -J myJobName # This specifies an error logfile name #BSUB -e %J.err # This specifies a stdout logfile name #BSUB -o %J.out # This specifies max runtime in hours:minutes #BSUB -W 24:00 # This reserves 200 MB per processor # Memory specifications are MANDATORY. #BSUB -R "rusage[Memory=200]" # This specifies a type of mpi: # use either mvapich #BSUB -a mvapich # or mpich2 (the next line is commented out) ##BSUB -a mpich2 # This specifies a number of processors #BSUB -n 4 /theoryfs/common/software/NAMD_2.6_Linux-i686-TCP/namd2mpi config_file(s)
(mvapich doesn't improve the overall performance of NAMD too much compared to mpich2).
Q-Chem
To run Q-Chem:
- add the following line to the command file or .cshrc:
source /theoryfs/common/software/etc/cshrc.qchem30
- use the script
runqchem <inputfile> <outfile>
A simpler way to run Q-Chem is to use the bsubqchem3 script located in /theoryfs/common/software/etc/. This script will create a command file for you, so all you need is a Q-Chem input file. The syntax is straightforward:
bsubqchem3 <inputfile> <timelimit>
<timelimit> is an optional time limit for the job, in hours. The default is 2 weeks.
