Seventh Sense » ROCKS

BASH – Wrappers For qstat In NPACI ROCKS 5.2.2

Gowtham — Thu, 10 Dec 2009 20:51:33 +0000

What is ROCKS?

Rocks Cluster Distribution (originally called NPACI Rocks) is a Linux distribution intended for high-performance computing clusters. It was started by NPACI and the SDSC in 2000, and was initially funded in part by an NSF grant (2000-2007) but is currently funded by the followup NSF grant. Rocks was initially based on the Red Hat Linux distribution, however modern versions of Rocks are now based on CentOS, with a modified Anaconda installer that simplifies mass installation onto many computers. Rocks includes many tools (such as MPI) which are not part of CentOS but are integral components that make a group of computers into a cluster. Installations can be customized with additional software packages at install-time by using special user-supplied CDs (called Roll CDs). The Rolls extend the system by integrating seamlessly and automatically into the management and packaging mechanisms used by base software, greatly simplifying installation and configuration of large numbers of computers. Over a dozen Rolls have been created, including the SGE roll, the Condor roll, the Lustre roll, the Java roll, and the ganglia roll.

Why wrappers?

Functionality of qstat [part of the SGE Roll] changed in ROCKS 5.2.2 [it might have changed earlier than that, but that's when I first noticed]. To keep cluster users, supervisors and systems-administrators happy, I ended up writing these two wrappers:

General Wrapper for qstat

#! /bin/bash
 
# A wrapper for 'qstat' command to show 
# the total number processors used/in use.
# ROCKS 5.2.2
 
# Thu, 10 Dec 2009 12:27:59 -0500
 
export PROC_TOTAL="142"
 
export USERS_LIST="
  user0
  user1
  user2
  user3
  user4"
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat -u $USERS_LIST | \
                  awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
                  tail -1 | awk '{print $NF}'`
 
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
echo '-----------------------------------------------------------------------------------------------------------------' 
/opt/gridengine/bin/lx26-amd64/qstat -u $USERS_LIST
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "   $PROC_TOTAL total processors :: $PROC_INUSE is/are in use :: $PROC_AVAIL is/are available"
echo '-----------------------------------------------------------------------------------------------------------------'
echo

User Specific Wrapper for qstat

#! /bin/bash
 
# A wrapper for 'qstat' command to show only the jobs
# submitted by a given user but show the total number
# processors used/in use.
# ROCKS 5.2.2
#
# Usage:
# qu [USERNAME]
# In the absence of any jobs submitted by USERNAME, or if
# USERNAME is not specified at all, it just displays the jobs, 
# if any, by the currently logged in user.
#
# Thu, 10 Dec 2009 12:27:59 -0500
 
export EXPECTED_ARGS="1"
export PROC_TOTAL="142"
 
export USERS_LIST="
  user0
  user1
  user2
  user3
  user4"
 
if [ $# -ne $EXPECTED_ARGS ];
then 
  export MYUSER="$USER"
else
  export MYUSER="$1"
fi
 
export PROC_USER=`/opt/gridengine/bin/lx26-amd64/qstat -u $MYUSER | \
                 awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
                 tail -1 | awk '{print $NF}'`
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat -u $USERS_LIST | \
                  awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
                  tail -1 | awk '{print $NF}'`
 
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
if [[ $PROC_USER -gt 0 ]] ;
then
  echo '-----------------------------------------------------------------------------------------------------------------' 
  /opt/gridengine/bin/lx26-amd64/qstat -u $MYUSER
  echo '-----------------------------------------------------------------------------------------------------------------' 
  echo " $PROC_TOTAL total processors :: $PROC_INUSE is/are in use :: $PROC_USER is/are use by $MYUSER :: $PROC_AVAIL is/are available "
  echo '-----------------------------------------------------------------------------------------------------------------'
  echo
else
  echo
  echo "  `hostname` @ `date -R`"
  echo
  echo "  $PROC_TOTAL total processors"
  echo "  $PROC_INUSE is/are in use "
  echo "  0 is/are use by $MYUSER"
  echo "  $PROC_AVAIL is/are available "
  echo
fi

BASH – Wrappers For qstat In NPACI ROCKS

Gowtham — Mon, 13 Oct 2008 22:34:07 +0000

BASH is a free software UNIX shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 UNIX circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

AWK is a general purpose programming language that is designed for processing text-based data, either in files or data streams, and was created at Bell Labs in the 1970s. The name AWK is derived from the family names of its authors — Alfred Aho, Peter Weinberger, and Brian Kernighan; however, it is not commonly pronounced as a string of separate letters but rather to sound the same as the name of the bird, auk. awk, when written in all lowercase letters, refers to the UNIX or Plan 9 program that runs other programs written in the AWK programming language. AWK is an example of a programming language that extensively uses the string data type, associative arrays (that is, arrays indexed by key strings), and regular expressions. The power, terseness, and limitations of AWK programs and sed scripts inspired Larry Wall to write PERL. Because of their dense notation, all these languages are often used for writing one-liner programs. AWK is one of the early tools to appear in Version 7 UNIX and gained popularity as a way to add computational features to a UNIX pipeline. A version of the AWK language is a standard feature of nearly every modern UNIX-like operating system.

What is ROCKS?

General Wrapper for qstat

#! /bin/bash
 
# Save this as /home/local/bin/qstat
# A wrapper for 'qstat' command to show all currently submitted jobs along
# with the total number processors used/in use
#
# Check the path to 'qstat'
#
# Thu, 02 Oct 2008 21:06:05 -0400
 
# Modify this variable to meet the cluster's requirements
export PROC_TOTAL="128"
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat | \
       awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
       tail -1 | \
       awk '{print $NF}'`
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
export JOBS_LIST=`/opt/gridengine/bin/lx26-amd64/qstat`
 
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "$JOBS_LIST"
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "   $PROC_TOTAL total processors :: $PROC_INUSE are in use :: $PROC_AVAIL are available"
echo '-----------------------------------------------------------------------------------------------------------------'
echo

User Specific Wrapper for qstat

#! /bin/bash
 
# Save this as /home/local/bin/qu
# A wrapper for 'qstat' command to show all currently submitted jobs by a given 
# user ($USER) along with the total number processors used/in use
# If the $USER has no jobs, displays all currently submitted jobs.
#
# Check the path to 'qstat'
#
# Thu, 02 Oct 2008 21:35:04 -0400
 
 
# Modify this variable to meet the cluster's requirements
export PROC_TOTAL="128"
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat | \
       awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
       tail -1 | \
       awk '{print $NF}'`
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
export JOBS_LIST=`/opt/gridengine/bin/lx26-amd64/qstat -u $USER`
 
if [ "$JOBS_LIST" == "" ];
then
  export JOBS_LIST=`/opt/gridengine/bin/lx26-amd64/qstat`
fi
 
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "$JOBS_LIST"
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "   $PROC_TOTAL total processors :: $PROC_INUSE are in use :: $PROC_AVAIL are available"
echo '-----------------------------------------------------------------------------------------------------------------'
echo

VASP, MPICH, Intel Compilers And NPACI Rocks

Gowtham — Thu, 22 Dec 2005 18:44:41 +0000

Much of the weekend was spent in some relaxation, shoveling snow (I have come to realize that this is a very good exercise – it can make one sweat even when it’s 10 F outside!) and trying to debug the errors associated with execution of parallel version of VASP 4.6.28. However, same errors persisted and I didn’t go too far ahead with it.

I got in touch with few of Intel’s authorized resellers (one in Bloomington, MN – who transferred the call to someone else in Boston, MA) to see if I can get a copy of version 8.x of FORTRAN and C compilers. Though the latter person promised to get back to me with some favorable information soon, I got directly in touch with an Intel Support Technician – hoping that he/she wouldn’t be from a call center in some other country. Fortunately, it was somebody in California and unfortunately, he redirected me back to their re-sellers. Last call, pretty much out of desperation, did what I wanted : I just had to register the non-commercially downloaded products for Premier Support and I would be entitled for previous versions too. If only Intel explicitly mentioned what Premier Support actually is, my worries would have ended a long time ago. By the way, if you are now wondering what the error message was (running on 2 processors), here it is:

running on    2 nodes
distr:  one band on    1 nodes,    2 groups
vasp.4.6.28 25Jul05 complex
POSCAR found :  4 types and   18 ions
LDA part: xc-table for Ceperly-Alder, standard interpolation
found WAVECAR, reading the header
POSCAR, INCAR and KPOINTS ok, starting setup
WARNING: wrap around errors must be expected
FFT: planning ...           1
reading WAVECAR
the WAVECAR file was read sucessfully
LAPACK: Routine ZPOTRF failed!           8
LAPACK: Routine ZPOTRF failed!           8

Having managed to get version 8.x and 7.x of Intel compilers, situation only got worse as the error message remained the same At this point, I must thank the help offered by VASP Tech Support and Andri Arnaldsson (from University of Washington) – they have been pretty quick in their responses, sent their copies of Makefiles along with several tips and tricks. Changing compiler versions, using a previous version of MPICH (1.2.7p1 to be precise), repeating compilation many times with different BLAS, LAPACK libraries … nothing helped.

Taking a break for an hour and watching an episode of South Park seemed to have helped. A modification in the key words used for Google! search and reading some discussion forum lot more carefully, I found that adding three lines at the end of VASP Makefile (what this does is to reduce the level of optimization for mpi.F), the error vanished and the calculations started running smoothly. I repeated the same calculation using 2, 4, 6 and 8 processors and noticed a slightly strange behavior – when the number of processors was 2, 4 or 8, energy optimization is exactly same as in a serial calculation but when the number of processors is 2*N (N=3 in this case), energy optimization route is different – final result is still exactly the same. Though I have to do more trials (say 3, 5, 7, 9, 10 processors) to completely convince myself, it appears to me that using 2^N processors does the trick. Like Dave Kraus mentioned once before – knowing what trick works is certainly important, but knowing why that trick works is even more important.