Seventh Sense » Scripts

HPCC 1.4.1 Benchmark With GCC 4.1.2 On Rocks 5.4.2

Gowtham — Tue, 28 Feb 2012 18:27:14 +0000

Disclaimer

The instructions/steps given below worked for me (and Michigan Technological University) running NPACI Rocks 5.4.2 (with CentOS 5.5) – as has been a common practice for several years now, a full version of Operating System was installed. These instructions may very well work for you (or your institution), on Rocks-like or other linux clusters. Please note that if you decide to use these instructions on your machine, you are doing so entirely at your very own discretion and that neither this site, sgowtham.net, nor its author (or Michigan Technological University) is responsible for any/all damage – intellectual and/or otherwise.

A Bit About HPCC

Citing HPCC website,

HPCC benchmark suite has been released by DARPA High Productivity Computing Systems (HPCS) program to help define the performance boundaries of future Petascale computing systems. HPCC is a suite of tests that examine the performance of HPC architectures using kernels with memory access patterns more challenging than those of the HPL benchmark used in Top500. This suite is designed to augment the list, providing benchmarks that bound the performance of many real applications as a function of memory access characteristics. For e.g., spatial and temporal locality, and providing a framework for including additional tests. HPCC consists of seven tests that attempt to span high and low spatial and temporal locality space.

By design, the HPCC tests are scalable with the size of data sets being a function of the largest HPL matrix tested for the system. Since HPCC kernels consist of simple mathematical operations, it provides a unique opportunity to look at language and parallel programming issues. To characterize the architecture of the system, the following three scenarios are considered:

Local – only a single processor is performing the computations

Embarrassingly Parallel – each processor in the entire system is performing the computations but without explicit communication with each other

Global – all processors in the system are performing the computations with explicit communication with each other

The seven tests that make up HPCC benchmark are as follows:

HPL – the Linpack TPP benchmark which measures the floating point rate of execution for solving a linear system of equations

DGEMM – measures the floating point rate of execution of double precision real matrix-matrix multiplication

STREAM – a simple synthetic benchmark program that measures sustainable memory bandwidth (in GB/s) and the corresponding computation rate for simple vector kernel

PTRANS (parallel matrix transpose) – exercises the communications where pairs of processors communicate with each other simultaneously. It is a useful test of the total communications capacity of the network

RandomAccess – measures the rate of integer random updates of memory (GUPS)

FFT – measures the floating point rate of execution of double precision complex one-dimensional Discrete Fourier Transform (DFT)

Communication bandwidth and latency – a set of tests to measure latency and bandwidth of a number of simultaneous communication patterns; based on b_eff (effective bandwidth benchmark)

Pre-requisites: MPI & Goto BLAS

Please refer to the corresponding sections in HPL 2.0 Benchmark With GCC 4.1.2 On Rocks 5.4.2 to learn more about required pre-requisites as well as instructions for installing/compiling them. Following Rocks recommendations, pre-requisites will be installed under /share/apps/; software installed by me, in clusters at Michigan Tech, have the following template for their folder structure:

/share/apps/Software/Software_Version/Compiler/Compiler_Version

HPCC Installation/Compilation

With MPICH2 and Goto BLAS2 in place, HPCC 1.4.1 will be installed under

/share/apps/hpcc/1.4.1/mpich2/1.4.1p1/gcc/4.1.2

The following script, used in installation, will assume that one has downloaded HPCC 1.4.1 from here & the necessary Make.MPICH2141p1_GCC412.HPCC (listed below; it does look identical to the one used when compiling HPL 2.0 – the only difference is in the definition of TOPdir variable) and placed them in /share/apps/tmp/

#
# Makefile (Make.MPICH2141p1_GCC412) used to compile HPCC 
# on HPC clusters running NPACI Rocks (5.4.2) with CentOS (5.5)
# at Michigan Technological University.
# 
# Disclaimer:
# Please note that you are using these instructions 
# at your very own risk and that Michigan Technological
# University is not responsible for any/all damage caused to 
# your property, intellectual or otherwise.
# 
# For additional help and/or comments, questions, suggestions,
# please contact 
#
# Gowtham
# Information Technology Services & Security
# Michigan Technological University
# g@mtu.edu
#
# * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
#
#
#  -- High Performance Computing Linpack Benchmark (HPL)                
#     HPL - 2.0 - September 10, 2008                          
#     Antoine P. Petitet                                                
#     University of Tennessee, Knoxville                                
#     Innovative Computing Laboratory                                 
#     (C) Copyright 2000-2008 All Rights Reserved                       
#                                                                       
#  -- Copyright notice and Licensing terms:                             
#                                                                       
#  Redistribution and use in source and binary forms, with or without
#  modification, are permitted provided that the following conditions
#  are met:                                                             
#                                                                       
#  1. Redistributions of source code must retain the above copyright
#  notice, this list of conditions and the following disclaimer.        
#                                                                       
#  2. Redistributions in binary form must reproduce the above copyright
#  notice, this list of conditions, and the following disclaimer in the
#  documentation and/or other materials provided with the distribution. 
#                                                                       
#  3. All advertising materials mentioning features or use of this
#  software must display the following acknowledgement:                 
#  This product includes software developed at the University of
#  Tennessee, Knoxville, Innovative Computing Laboratory.             
#                                                                       
#  4. The name of the University, the name of the Laboratory, or the
#  names of its contributors may not be used to endorse or promote
#  products derived from this software without specific written
#  permission.
#
#  -- Disclaimer:
#                                                                       
#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
#  ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
#  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
#  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY
#  OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
#  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
#  DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
#  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
#  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#  
 
#
# Shell Details
SHELL        = /bin/sh
CD           = cd
CP           = cp
LN_S         = ln -s
MKDIR        = mkdir
RM           = /bin/rm -f
TOUCH        = touch
 
#
# Platform Identifier
ARCH         = MPICH2141p1_GCC412
 
#
# HPL Directory Structure / HPL library
TOPdir       = ../../..
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
HPLlib       = $(LIBdir)/libhpl.a 
 
#
# Message Passing library (MPI)
MPdir        = /share/apps/mpich2/1.4.1p1/gcc/4.1.2
MPinc        = -I$(MPdir)/include
MPlib        = $(MPdir)/lib/libmpich.a
 
#
# Linear Algebra library (BLAS)
LAlib        = /share/apps/gotoblas2/1.13/gcc/4.1.2/libgoto2.a
 
#
# F77 / C Interface
# You can skip this section if and only if you are not planning to use
# a BLAS library featuring a Fortran 77 interface. Otherwise, it is
# necessary to fill out the F2CDEFS variable with the appropriate
# options. **One and only one** option should be chosen in **each** of
# the 3 following categories:
#
# 1) name space (How C calls a Fortran 77 routine)
#
# -DAdd_              : all lower case and a suffixed underscore (Suns,
#                       Intel, ...) [default]
# -DNoChange          : all lower case (IBM RS6000)
# -DUpCase            : all upper case (Cray)
# -DAdd__             : the FORTRAN compiler in use is f2c
#
# 2) C and Fortran 77 integer mapping
#
# -DF77_INTEGER=int   : Fortran 77 INTEGER is a C int [default]
# -DF77_INTEGER=long  : Fortran 77 INTEGER is a C long
# -DF77_INTEGER=short : Fortran 77 INTEGER is a C short
#
# 3) Fortran 77 string handling
#
# -DStringSunStyle    : The string address is passed at the string location
#                       on the stack, and the string length is then
#                       passed as an F77_INTEGER after all explicit
#                       stack arguments [default]
# -DStringStructPtr   : The address of a structure is passed by a
#                       Fortran 77 string, and the structure is of the
#                       form: struct {char *cp; F77_INTEGER len;}
# -DStringStructVal   : A structure is passed by value for each Fortran
#                       77 string, and the structure is of the form:
#                       struct {char *cp; F77_INTEGER len;}
# -DStringCrayStyle   : Special option for Cray machines, which uses
#                       Cray fcd (fortran character descriptor) for
#                       interoperation
F2CDEFS      =  -DAdd_
 
#
# HPL Includes / Libraries / Specifics
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
 
#
# HPL Compile Time Options
# -DHPL_COPY_L           force the copy of the panel L before bcast
# -DHPL_CALL_CBLAS       call the cblas interface
# -DHPL_CALL_VSIPL       call the vsip library
# -DHPL_DETAILED_TIMING  enable detailed timers
#
# By default HPL will:
#    *) not copy L before broadcast
#    *) call the BLAS Fortran 77 interface
#    *) not display detailed timing information
HPL_OPTS     = -DHPL_COPY_L -DHPL_CALL_CBLAS -DHPL_DETAILED_TIMING 
 
#
# HPL Definitions
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
 
#
# Compilers / linkers - Optimization Flags
CC           = mpicc
CCNOOPT      = $(HPL_DEFS)
CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall
 
#
# On some platforms, it is necessary to use the Fortran linker 
# to find the Fortran internals used in the BLAS library
LINKER       = mpif77
LINKFLAGS    = $(CCFLAGS)
ARCHIVER     = ar
ARFLAGS      = r
RANLIB       = echo
#

#! /bin/bash
#
# install_hpcc.sh
# BASH script to install HPCC 1.4.1 (compiled against MPICH2 1.4.1p1) on a
# Rocks 5.4.2 cluster's front end
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  # Set necessary variables
  export HPCC_VERSION="1.4.1"
  export MPICH2_VERSION="1.4.1p1"
  export GCC_VERSION="4.1.2"
  export HPCC_INSTALL="/share/apps/hpcc/${HPCC_VERSION}/mpich2/${MPICH2_VERSION}/gcc/${GCC_VERSION}"
  export MAKEFILE_ARCH="MPICH2141p1_GCC412"
 
  # Begin HPCC-${HPCC_VERSION}.tar.gz check IF
  if [ -e "/share/apps/tmp/hpcc-${HPCC_VERSION}.tar.gz" ]
  then
 
    mkdir /share/apps/hpcc/${HPCC_VERSION}/mpich2/${MPICH2_VERSION}/gcc/
    cd /share/apps/hpcc/${HPCC_VERSION}/mpich2/${MPICH2_VERSION}/gcc/
    tar -zxvf /share/apps/tmp/hpcc-${HPCC_VERSION}.tar.gz
    mv hpcc-${HPCC_VERSION} ${GCC_VERSION}
 
    echo
    echo "  Step #0: copy the makefile, make clean and make"
    cd ${HPCC_INSTALL}
    cp /share/apps/tmp/Make.${MAKEFILE_ARCH}.HPCC ./hpl/Make.${MAKEFILE_ARCH}
 
    make clean arch=${MAKEFILE_ARCH}
    make arch=${MAKEFILE_ARCH}
 
    echo
    echo "  Step #1: Update $HOME/.bashrc"
    cat <
 
  fi
  # End hpcc-${HPCC_VERSION}.tar.gz check IF
  echo
 
fi 
# End root-check IF

Running HPCC Benchmark

Please refer to the corresponding sections in HPL 2.0 Benchmark With GCC 4.1.2 On Rocks 5.4.2 to learn how to get a fix on N, NB, etc., given the hardware configuration of compute nodes. Running HPCC benchmark is very similar to running HPL 2.0 benchmark.

HPL 2.0 Benchmark With GCC 4.1.2 On Rocks 5.4.2

Gowtham — Mon, 27 Feb 2012 23:44:17 +0000

Disclaimer

A Bit About HPL

LINPACK is a software library for performing numerical linear algebra on computers. It was written in FORTRAN by Jack Dongarra, Jim Bunch, Cleve Moler and Gilbert Stewart. LINPACK makes use of the BLAS libraries for performing basic vector and matrix operations. It has largely been superseded by LAPACK, which runs more efficiently on modern architectures.

The LINPACK benchmarks are a measure of a system’s floating point computing power. Introduced by Jack Dongarra, they measure how fast a computer solves a dense N * N system of linear equations, Ax = b. The solution is obtained by Gaussian Elimination with Partial Pivoting, with

floating point iterations. The result is often expressed in billions of floating point operations per second (GFLOPS). HPL, a portable implementation of High Performance LINPACK Benchmark, is used as a performance measure for ranking the supercomputers in the Top500 list.

Pre-requisite #1: MPI

Following Rocks recommendations, this and other pre-requisites will be installed under /share/apps/; software installed by me, in clusters at Michigan Tech, have the following template for their folder structure:

/share/apps/Software/Software_Version/Compiler/Compiler_Version

Rocks 5.4.2 installation has an instance (of few flavors) of MPI but I prefer to compile MPICH2 using GCC 4.1.2. At the time of writing this post, the latest stable version of MPICH2 is 1.4.1p1 and it may be downloaded from here. Following folder structure/template mentioned above, it will be installed under

/share/apps/mpich2/1.4.1p1/gcc/4.1.2

To avoid confusion and/or missed steps leading to undesired results, steps associated with installation of MPICH2 have been put in the following script:

#! /bin/bash
#
# install_mpich2.sh
# BASH script to install MPICH2 (compiled against GCC 4.1.2) on a
# Rocks 5.4.2 cluster's front end
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  # Set necessary variables
  export CC="gcc"
  export CXX="g++"
  export FC="gfortran"
  export F77="gfortran"
  export MPICH2_VERSION="1.4.1p1"
  export GCC_VERSION="4.1.2"
  export MPICH2_INSTALL="/share/apps/mpich2/${MPICH2_VERSION}/gcc/${GCC_VERSION}"
  export ANL="http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs"
 
  echo
  echo "  Step #0: Download MPICH2 to /share/apps/tmp"
  cd /share/apps/tmp/
  wget ${ANL}/${MPICH2_VERSION}/mpich2-${MPICH2_VERSION}.tar.gz
 
  # Begin mpich2-${MPICH2_VERSION}.tar.gz check IF
  if [ -e "mpich2-${MPICH2_VERSION}.tar.gz" ]
  then
    echo
    echo "  Step #1: configure, make clean, make and make install"
 
    tar -zxpf mpich2-${MPICH2_VERSION}.tar.gz
    cd mpich2-${MPICH2_VERSION}/
    ./configure --prefix=${MPICH2_INSTALL}
    make clean
    make
    make install
 
    echo
    echo "  Step #2: Update $HOME/.bashrc"
    cat <
 
  fi
  # End mpich2-${MPICH2_VERSION}.tar.gz check IF
  echo
 
fi 
# End root-check IF

A good test of a successful installation is that which mpicc, which mpif77, etc. return the respective commands located in ${MPICH2_INSTALL}.

Pre-requisite #2: Goto BLAS

As in the case of pre-requisite #1, this one will be installed under

/share/apps/gotoblas2/1.13/gcc/4.1.2

The following script, used in installation, will assume that one has downloaded Goto BLAS2 1.13 from here and placed it in
/share/apps/tmp/

#! /bin/bash
#
# install_gotoblas2.sh
# BASH script to install Goto BLAS2 (compiled against GCC 4.1.2) on a
# Rocks 5.4.2 cluster's front end
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  # Set necessary variables
  export CC="gcc"
  export CXX="g++"
  export FC="gfortran"
  export F77="gfortran"
  export GOTOBLAS2_VERSION="1.13"
  export GCC_VERSION="4.1.2"
  export GOTOBLAS2_INSTALL="/share/apps/gotoblas2/${GOTOBLAS2_VERSION}/gcc/${GCC_VERSION}"
 
  # Begin GotoBLAS2-${GOTOBLAS2_VERSION}.tar.gz check IF
  if [ -e "/share/apps/tmp/GotoBLAS2-${GOTOBLAS2_VERSION}.tar.gz" ]
  then
 
    mkdir /share/apps/gotoblas2/${GOTOBLAS2_VERSION}/gcc/
    cd /share/apps/gotoblas2/${GOTOBLAS2_VERSION}/gcc/
    tar -zxvf /share/apps/tmp/GotoBLAS2-${GOTOBLAS2_VERSION}.tar.gz
    mv GotoBLAS2 ${GCC_VERSION}
 
    echo
    echo "  Step #0: make clean and make"
    cd ${GOTOBLAS2_INSTALL}
    make clean
    make BINARY=64
 
    echo
    echo "  Step #1: Update $HOME/.bashrc"
    cat <
 
  fi
  # End GotoBLAS2-${GOTOBLAS2_VERSION}.tar.gz check IF
  echo
 
fi 
# End root-check IF

HPL Installation/Compilation

With MPICH2 and Goto BLAS2 in place, HPL 2.0 will be installed under

/share/apps/hpl/2.0/mpich2/1.4.1p1/gcc/4.1.2

The following script, used in installation, will assume that one has downloaded HPL 2.0 from here & the necessary Make.MPICH2141p1_GCC412.HPL (listed below) and placed them in /share/apps/tmp/

#
# Makefile (Make.MPICH2141p1_GCC412) used to compile HPL 
# on HPC clusters running NPACI Rocks (5.4.2) with CentOS (5.5)
# at Michigan Technological University.
# 
# Disclaimer:
# Please note that you are using these instructions 
# at your very own risk and that Michigan Technological
# University is not responsible for any/all damage caused to 
# your property, intellectual or otherwise.
# 
# For additional help and/or comments, questions, suggestions,
# please contact 
#
# Gowtham
# Information Technology Services & Security
# Michigan Technological University
# g@mtu.edu
#
# * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
#
#
#  -- High Performance Computing Linpack Benchmark (HPL)                
#     HPL - 2.0 - September 10, 2008                          
#     Antoine P. Petitet                                                
#     University of Tennessee, Knoxville                                
#     Innovative Computing Laboratory                                 
#     (C) Copyright 2000-2008 All Rights Reserved                       
#                                                                       
#  -- Copyright notice and Licensing terms:                             
#                                                                       
#  Redistribution and use in source and binary forms, with or without
#  modification, are permitted provided that the following conditions
#  are met:                                                             
#                                                                       
#  1. Redistributions of source code must retain the above copyright
#  notice, this list of conditions and the following disclaimer.        
#                                                                       
#  2. Redistributions in binary form must reproduce the above copyright
#  notice, this list of conditions, and the following disclaimer in the
#  documentation and/or other materials provided with the distribution. 
#                                                                       
#  3. All advertising materials mentioning features or use of this
#  software must display the following acknowledgement:                 
#  This product includes software developed at the University of
#  Tennessee, Knoxville, Innovative Computing Laboratory.             
#                                                                       
#  4. The name of the University, the name of the Laboratory, or the
#  names of its contributors may not be used to endorse or promote
#  products derived from this software without specific written
#  permission.
#
#  -- Disclaimer:
#                                                                       
#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
#  ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
#  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
#  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY
#  OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
#  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
#  DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
#  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
#  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#  
 
#
# Shell Details
SHELL        = /bin/sh
CD           = cd
CP           = cp
LN_S         = ln -s
MKDIR        = mkdir
RM           = /bin/rm -f
TOUCH        = touch
 
#
# Platform Identifier
ARCH         = MPICH2141p1_GCC412
 
#
# HPL Directory Structure / HPL library
TOPdir       = /share/apps/hpl/2.0/mpich2/1.4.1p1/gcc/4.1.2
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
HPLlib       = $(LIBdir)/libhpl.a 
 
#
# Message Passing library (MPI)
MPdir        = /share/apps/mpich2/1.4.1p1/gcc/4.1.2
MPinc        = -I$(MPdir)/include
MPlib        = $(MPdir)/lib/libmpich.a
 
#
# Linear Algebra library (BLAS)
LAlib        = /share/apps/gotoblas2/1.13/gcc/4.1.2/libgoto2.a
 
#
# F77 / C Interface
# You can skip this section if and only if you are not planning to use
# a BLAS library featuring a Fortran 77 interface. Otherwise, it is
# necessary to fill out the F2CDEFS variable with the appropriate
# options. **One and only one** option should be chosen in **each** of
# the 3 following categories:
#
# 1) name space (How C calls a Fortran 77 routine)
#
# -DAdd_              : all lower case and a suffixed underscore (Suns,
#                       Intel, ...) [default]
# -DNoChange          : all lower case (IBM RS6000)
# -DUpCase            : all upper case (Cray)
# -DAdd__             : the FORTRAN compiler in use is f2c
#
# 2) C and Fortran 77 integer mapping
#
# -DF77_INTEGER=int   : Fortran 77 INTEGER is a C int [default]
# -DF77_INTEGER=long  : Fortran 77 INTEGER is a C long
# -DF77_INTEGER=short : Fortran 77 INTEGER is a C short
#
# 3) Fortran 77 string handling
#
# -DStringSunStyle    : The string address is passed at the string location
#                       on the stack, and the string length is then
#                       passed as an F77_INTEGER after all explicit
#                       stack arguments [default]
# -DStringStructPtr   : The address of a structure is passed by a
#                       Fortran 77 string, and the structure is of the
#                       form: struct {char *cp; F77_INTEGER len;}
# -DStringStructVal   : A structure is passed by value for each Fortran
#                       77 string, and the structure is of the form:
#                       struct {char *cp; F77_INTEGER len;}
# -DStringCrayStyle   : Special option for Cray machines, which uses
#                       Cray fcd (fortran character descriptor) for
#                       interoperation
F2CDEFS      =  -DAdd_
 
#
# HPL Includes / Libraries / Specifics
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
 
#
# HPL Compile Time Options
# -DHPL_COPY_L           force the copy of the panel L before bcast
# -DHPL_CALL_CBLAS       call the cblas interface
# -DHPL_CALL_VSIPL       call the vsip library
# -DHPL_DETAILED_TIMING  enable detailed timers
#
# By default HPL will:
#    *) not copy L before broadcast
#    *) call the BLAS Fortran 77 interface
#    *) not display detailed timing information
HPL_OPTS     = -DHPL_COPY_L -DHPL_CALL_CBLAS -DHPL_DETAILED_TIMING 
 
#
# HPL Definitions
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
 
#
# Compilers / linkers - Optimization Flags
CC           = mpicc
CCNOOPT      = $(HPL_DEFS)
CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall
 
#
# On some platforms, it is necessary to use the Fortran linker 
# to find the Fortran internals used in the BLAS library
LINKER       = mpif77
LINKFLAGS    = $(CCFLAGS)
ARCHIVER     = ar
ARFLAGS      = r
RANLIB       = echo
#

#! /bin/bash
#
# install_hpl.sh
# BASH script to install HPL 2.0 (compiled against MPICH2 1.4.1p1) on a
# Rocks 5.4.2 cluster's front end
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  # Set necessary variables
  export HPL_VERSION="2.0"
  export MPICH2_VERSION="1.4.1p1"
  export GCC_VERSION="4.1.2"
  export HPL_INSTALL="/share/apps/hpl/${HPL_VERSION}/mpich2/${MPICH2_VERSION}/gcc/${GCC_VERSION}"
  export MAKEFILE_ARCH="MPICH2141p1_GCC412"
 
  # Begin hpl-${HPL_VERSION}.tar.gz check IF
  if [ -e "/share/apps/tmp/hpl-${HPL_VERSION}.tar.gz" ]
  then
 
    mkdir /share/apps/hpl/${HPL_VERSION}/mpich2/${MPICH2_VERSION}/gcc/
    cd /share/apps/hpl/${HPL_VERSION}/mpich2/${MPICH2_VERSION}/gcc/
    tar -zxvf /share/apps/tmp/hpl-${HPL_VERSION}.tar.gz
    mv hpl-${HPL_VERSION} ${GCC_VERSION}
 
    echo
    echo "  Step #0: copy the makefile, make clean and make"
    cd ${HPL_INSTALL}
    cp /share/apps/tmp/Make.${MAKEFILE_ARCH}.HPC ./Make.${MAKEFILE_ARCH}
 
    if [ -d "${HPL_INSTALL}/bin/${MAKEFILE_ARCH}"]
    then
      make clean arch=${MAKEFILE_ARCH}
    fi
 
    if [ "$(ls -A ${HPL_INSTALL}/bin)"];
    then
      make clean_arch_all arch=${MAKEFILE_ARCH} 
    fi
 
    make arch=${MAKEFILE_ARCH}
 
    echo
    echo "  Step #1: Update $HOME/.bashrc"
    cat <
 
  fi
  # End hpl-${HPL_VERSION}.tar.gz check IF
  echo
 
fi 
# End root-check IF

Running HPL Benchmark

The amount of memory used by HPL is essentially the size of the co-efficient matrix, A. Following standard binary definition, 1 GB is 1024 * 1024 * 1024 bytes (TOTAL_MEMORY_BYTES). Most scientific/engineering computations use double precision numbers, with each such double precision number taking 8 bytes of memory. Thus, 1 GB can accommodate 134,217,728 double precision entities (TOTAL_DP_ELEMENTS = TOTAL_MEMORY_BYTES/8). Theoretically, sqrt(TOTAL_DP_ELEMENTS) represents the maximum possible value of N. However, operating system needs some memory to perform some necessary operations. As such, HPL benchmark is usually performed for the following values of N – with m representing the fraction of total memory – making sure that swapping did not occur (which would result in reduced performance).

HPL uses the block size (NB) for the data distribution as well as for the computational granularity. From a data distribution perspective, the smaller NB, the better the load balance. From a computational perspective, too small of a value for NB may limit the computational performance by a large factor since almost no data re-use will occur in the highest level of the memory hierarchy. The number of messages will also increase. In my case, this benchmark was performed for NB values of 128, 256 and 512.

The results so obtained are compared with the theoretical peak value, , computed as follows:

For e.g., for a cluster with 16 identical recent Intel architecture compute nodes, each compute node with dual hex cores @ 3.00 GHz, will be

If each of these nodes had 24 GB RAM, then TOTAL_MEMORY_BYTES = 1024 * 1024 * 1024 * 24 * 16 = 412316860416
and TOTAL_DP_ELEMENTS = 412316860416/8 = 51539607552.

As such, N values will be

What if the cluster has heterogeneous compute nodes?

Computing isn’t easy in this case; becomes even more so when these compute nodes belong to different generations as one has to account for aging factor. It has been a practice at Michigan Tech, in such cases, to split the cluster into different queues – one for each generation/type of compute nodes – and run the HPL benchmark separately.

Thanks be to

Rocks mailing list and its participants.

BASH – Smart Extract Utility

Gowtham — Wed, 04 May 2011 19:38:34 +0000

Why This Script?

As a systems administrator or as someone who dabbles in compilation technologies, it often becomes necessary to extract one or more variety of compressed files/folders. This script will automagically handle the options necessary for uncompressing different kinds of compressions.

The Script

#! /bin/bash
#
# BASH script to automagically uncompress different kinds of compressed files/folders
#
# Usage: extract.sh [COMPRESSED_FILENAME]
# First written: Wed, 04 May 2011 05:14:10 -0400
# Last modified: Wed, 04 May 2011 05:14:10 -0400
#
 
E_BADARGS=65
 
if [ $# != 1 ];
then
  echo
  echo "  Usage: `basename $0` [COMPRESSED_FILENAME]"
  echo
  exit $E_BADARGS
else
  if [ -f $1 ];
  then
    case $1 in
      *.tar)       tar -xvf  $1  ;;
      *.tgz)       tar -xvzf $1  ;;
      *.tar.gz)    tar -xvzf $1  ;;
      *.tbz2)      tar -xvjf $1  ;;
      *.tar.bz2)   tar -xvjf $1  ;;
      *.gz)        gunzip    $1  ;;
      *.bz2)       bunzip2   $1  ;;
      *.zip)       unzip     $1  ;;
      *)           echo "  '$1' file type unknown" ;;
    esac
  else
    echo "  '$1' is not a regular file"
    echo
  fi
fi

Use it as a function!

Add the following lines to ${HOME}/.bashrc (and remember to source it).

# Smart extract function
extract () {
  if [ $# != 1 ];
  then
    echo
    echo "  Usage: extract [COMPRESSED_FILENAME]"
    echo
  else
    if [ -f $1 ];
    then
      case $1 in
        *.tar)       tar -xvf  $1  ;;
        *.tgz)       tar -xvzf $1  ;;
        *.tar.gz)    tar -xvzf $1  ;;
        *.tbz2)      tar -xvjf $1  ;;
        *.tar.bz2)   tar -xvjf $1  ;;
        *.gz)        gunzip    $1  ;;
        *.bz2)       bunzip2   $1  ;;
        *.zip)       unzip     $1  ;;
        *)           echo "  '$1' file type unknown" ;;
      esac
    else
      echo "  '$1' is not a regular file"
      echo
    fi
  fi
}

BASH – Router’s Public IP Address For Mac

Gowtham — Tue, 19 Jan 2010 16:47:09 +0000

Why This Script?

While in its simplest form of usage, this script will only print out the public (& private) IP address along with the SSID to which the machine is connected to, the results therefrom can be used for a variety of other purposes – I will let your imagination fly you to your dream destination!

The Script

#! /bin/bash
 
# BASH script to display the IP Address - one that the machine has, 
# as well as router's public IP Address, if any.
# Tested on Apple Mac Snow Leopard 10.6.2
 
# First written : Gowtham, Thu Dec 31 12:05:35 EST 2005
# Last modified : Gowtham, Thu Dec 31 12:05:35 EST 2005
 
# en1 is the wireless interface. 
# Change appropriately, if different
export WIFI_INTERFACE="en1"
 
# Details about the ${WIFI_INTERFACE}
ifconfig ${WIFI_INTERFACE} > public_ip.txt
export STATUS=`cat public_ip.txt | grep "status" | awk '{print $4}'`
 
# Analyze the details and display appropriate information
if [ "$STATUS" = "active" ];
then
  export IPADDRESS=`cat public_ip.txt | grep broadcast | awk '{print $2}'`
  export SSID=`system_profiler SPAirPortDataType | \
               grep -A1 "Current Network Information:" | \
               tail -1 | awk -F ':' '{print $1}' | \
               sed 's/^[ \t]*//;s/[ \t]*$//'`
else
  echo
  echo " `date`"
  echo " You are not connected to the internet"
  echo
  exit 0
fi
 
# Get the public IP address, $NSIP
export NSIP=`curl -s http://checkip.dyndns.org | awk '{print $6}' | \
             awk -F '<' '{print $1}'`
 
# Compare $IPADDRESS & $NSIP
# Display appropriate information along with SSID
if [ "$IPADDRESS" = "$NSIP" ];
then
  echo
  echo "  `date`"
  echo "  SSID       : $SSID"
  echo "  IP Address : $IPADDRESS"
  echo
else
  echo
  echo "  `date`"
  echo "  SSID                        : $SSID"
  echo "  Machine's static IP Address : $IPADDRESS"
  echo "  Router's  public IP Address : $NSIP"
  echo
fi
 
# Delete the scratch file
rm -f public_ip.txt

Screenshots

Directly connected to the internet

Configured behind a router

I must thank Jon DeVree [@nuxi] for help with a previous version of this script [for linux] as well as Sarah for graciously letting me use her WiFi network.

BASH – Wrappers For qstat In NPACI ROCKS 5.2.2

Gowtham — Thu, 10 Dec 2009 20:51:33 +0000

What is ROCKS?

Rocks Cluster Distribution (originally called NPACI Rocks) is a Linux distribution intended for high-performance computing clusters. It was started by NPACI and the SDSC in 2000, and was initially funded in part by an NSF grant (2000-2007) but is currently funded by the followup NSF grant. Rocks was initially based on the Red Hat Linux distribution, however modern versions of Rocks are now based on CentOS, with a modified Anaconda installer that simplifies mass installation onto many computers. Rocks includes many tools (such as MPI) which are not part of CentOS but are integral components that make a group of computers into a cluster. Installations can be customized with additional software packages at install-time by using special user-supplied CDs (called Roll CDs). The Rolls extend the system by integrating seamlessly and automatically into the management and packaging mechanisms used by base software, greatly simplifying installation and configuration of large numbers of computers. Over a dozen Rolls have been created, including the SGE roll, the Condor roll, the Lustre roll, the Java roll, and the ganglia roll.

Why wrappers?

Functionality of qstat [part of the SGE Roll] changed in ROCKS 5.2.2 [it might have changed earlier than that, but that's when I first noticed]. To keep cluster users, supervisors and systems-administrators happy, I ended up writing these two wrappers:

General Wrapper for qstat

#! /bin/bash
 
# A wrapper for 'qstat' command to show 
# the total number processors used/in use.
# ROCKS 5.2.2
 
# Thu, 10 Dec 2009 12:27:59 -0500
 
export PROC_TOTAL="142"
 
export USERS_LIST="
  user0
  user1
  user2
  user3
  user4"
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat -u $USERS_LIST | \
                  awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
                  tail -1 | awk '{print $NF}'`
 
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
echo '-----------------------------------------------------------------------------------------------------------------' 
/opt/gridengine/bin/lx26-amd64/qstat -u $USERS_LIST
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "   $PROC_TOTAL total processors :: $PROC_INUSE is/are in use :: $PROC_AVAIL is/are available"
echo '-----------------------------------------------------------------------------------------------------------------'
echo

User Specific Wrapper for qstat

#! /bin/bash
 
# A wrapper for 'qstat' command to show only the jobs
# submitted by a given user but show the total number
# processors used/in use.
# ROCKS 5.2.2
#
# Usage:
# qu [USERNAME]
# In the absence of any jobs submitted by USERNAME, or if
# USERNAME is not specified at all, it just displays the jobs, 
# if any, by the currently logged in user.
#
# Thu, 10 Dec 2009 12:27:59 -0500
 
export EXPECTED_ARGS="1"
export PROC_TOTAL="142"
 
export USERS_LIST="
  user0
  user1
  user2
  user3
  user4"
 
if [ $# -ne $EXPECTED_ARGS ];
then 
  export MYUSER="$USER"
else
  export MYUSER="$1"
fi
 
export PROC_USER=`/opt/gridengine/bin/lx26-amd64/qstat -u $MYUSER | \
                 awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
                 tail -1 | awk '{print $NF}'`
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat -u $USERS_LIST | \
                  awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
                  tail -1 | awk '{print $NF}'`
 
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
if [[ $PROC_USER -gt 0 ]] ;
then
  echo '-----------------------------------------------------------------------------------------------------------------' 
  /opt/gridengine/bin/lx26-amd64/qstat -u $MYUSER
  echo '-----------------------------------------------------------------------------------------------------------------' 
  echo " $PROC_TOTAL total processors :: $PROC_INUSE is/are in use :: $PROC_USER is/are use by $MYUSER :: $PROC_AVAIL is/are available "
  echo '-----------------------------------------------------------------------------------------------------------------'
  echo
else
  echo
  echo "  `hostname` @ `date -R`"
  echo
  echo "  $PROC_TOTAL total processors"
  echo "  $PROC_INUSE is/are in use "
  echo "  0 is/are use by $MYUSER"
  echo "  $PROC_AVAIL is/are available "
  echo
fi

PERL – Retrieving Data From MySQL Databases In Different Servers

Gowtham — Wed, 04 Mar 2009 08:07:04 +0000

Let us consider the following (hypothetical/real) case: we have a fruit basket with an assortment of fruits in it. Over a period of time, let us suppose we accumulate so much fruit [will they not stink? may be - but that's a problem beyond the scope of PERL & MySQL] that one basket can no longer hold all the fruits. One possible [and meaningful] way to solve this problem is to introduce another basket and segregate fruits into either one of the two baskets. To make it bit more elegant, we should probably make sure that a given kind of fruit is stored in only one of the baskets. Then, when someone asks us for a fruit – say an Apple – we know which basket contains apples and we can readily pick them out; when someone else asks for an Orange, we can repeat the process; and so on.

What does fruit basket have anything to do with PERL/MySQL?

Analogically speaking, we can consider fruits as some form of data/information while baskets can be considered as different servers/MySQL databases. Also, our thought/action process – of deciding whether to choose from one or both baskets – can be formulated as a PERL script. While some parts of transforming this problem into a script were relatively easy, the task of merging the information acquired from different servers/databases and presenting them as one unit [consider this as packaging fruits] bugged me for a really long time.

Tweet [and Email] to find answers!

After many sleepless nights & weekends, several trips to Princeton, long discussions with myself and friends, I decided to seek help through couple of very trusted sources: Linux Users Group @ Michigan Tech and Twitter. And needless to say, the answer wasn’t too far along – thanks mainly to Derek LaHousse (@ManoftheSea), Jon DeVree (@nuxi) and Adam Jaskiewicz (@ajjaskie) took time out of their busy schedules to help out with blocks of code. Owing to the kind of problem at hand, Jaskie’s code block fit the requirements perfectly.

PERL script to query MySQL databases in different servers

For completeness sake, let us suppose that the two servers are named server1.domain.com and server2.domain.com; the two databases are named database1 and database2; these two databases have the exact same table names/structure but data pertaining to uniquely different sets of fruits. MySQL userid and password could be same or different, but for sake of generalization, let us assume username1/password1 applies to database1 while username2/password2 applies to database2. Needless to say, this set up can be generalized to N servers.

#! /usr/bin/perl -wT
 
use DBI;
use CGI;
use CGI::Carp qw ( fatalsToBrowser );
use File::Basename;
 
# Server Name, Database, UserID and Password
$hostname1 = "server1.domain.com";
$database1 = "database1";
$userid1   = "userid1";
$password1 = "password1";
 
$hostname2 = "server2.domain.com";
$database2 = "database2";
$userid2   = "userid2";
$password2 = "password2";
 
# Sample Query Statements
# $stmt="SELECT * FROM FruitBasket";
# $stmt = "SELECT * FROM FruitBasket WHERE FruitName='Apple'";
# $stmt = "SELECT * FROM FruitBasket WHERE FruitName='Orange'";
$stmt="SELECT * FROM FruitBasket WHERE FruitName='Apple' OR FruitName='Orange'";
 
# Uncomment for debugging purposes only
# print "$stmt\n";
 
if($stmt =~ m/(FruitName)/) {
 
  # Uncomment for debugging purposes only
  # print "FruitName found.\n";
 
  if($stmt =~ m/(Apple|Banana|Pineapple)/) {
 
    # Uncomment for debugging purposes only
    # print "Apple|Banana|Pineapple :: Query server1.domain.com\n";
 
    $dbh = DBI->connect("dbi:mysql:$database1:$hostname1","$userid1","$password1")
           or die "Error!";
    $h1  = $dbh->prepare($stmt);
    $h1->execute;
    $r1  = $h1->fetchall_arrayref();
 
    @final_result=(@$r1);
  }
 
  if($stmt =~ m/(Grape|Orange|Peach|Tomato)/) {
 
    # Uncomment for debugging purposes only
    # print "Grape|Orange|Peach|Tomato :: Query server2.domain.com\n";
 
    $dbh = DBI->connect("dbi:mysql:$database2:$hostname2","$userid2","$password2")
           or die "Error!";
    $h2  = $dbh->prepare($stmt);
    $h2->execute;
    $r2  = $h2->fetchall_arrayref();
 
    @final_result=(@$r2);
  }
 
} else {
# FruitType not present in the query statement.
# Case where someone asks 'bring me some fruits'
 
  # Uncomment for debugging purposes only
  # print "Query server1.domain.com and server2.domain.com\n";
 
  $dbh = DBI->connect("dbi:mysql:$database1:$hostname1","$userid1","$password1")
         or die "Error!";
  $h1  = $dbh->prepare($stmt);
  $h1->execute;
  $r1  = $h1->fetchall_arrayref();
 
  $dbh = DBI->connect("dbi:mysql:$database2:$hostname2","$userid2","$password2")
         or die "Error!";
  $h2  = $dbh->prepare($stmt);
  $h2->execute;
  $r2  = $h2->fetchall_arrayref();
 
  @final_result=(@$r1, @$r2);
}
 
# Loop through the @final_result array and print the results
for $array_ref ( @final_result ) {
  print "@$array_ref \n";
}

Is there a better way of doing the same thing?

There probably is one – one could keep the information about fruits & server/database names in a flat text file (as a mapping file), read that in and compare it with $stmt. For now, this seems to be doing the job and as I find more time, I will look into it. If you find this useful and/or find bugs/mistakes and/or have other elegant methods of accomplishing the same thing, please do share them – readers, including me, will greatly appreciate it.

BASH – GZIP or BZIP2?

Gowtham — Sun, 19 Oct 2008 16:27:39 +0000

BASH is a free software Unix shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 Unix circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

AWK is a general purpose programming language that is designed for processing text-based data, either in files or data streams, and was created at Bell Labs in the 1970s. The name AWK is derived from the family names of its authors — Alfred Aho, Peter Weinberger, and Brian Kernighan; however, it is not commonly pronounced as a string of separate letters but rather to sound the same as the name of the bird, auk. awk, when written in all lowercase letters, refers to the UNIX or Plan 9 program that runs other programs written in the AWK programming language. AWK is an example of a programming language that extensively uses the string data type, associative arrays (that is, arrays indexed by key strings), and regular expressions. The power, terseness, and limitations of AWK programs and sed scripts inspired Larry Wall to write PERL. Because of their dense notation, all these languages are often used for writing one-liner programs. AWK is one of the early tools to appear in Version 7 UNIX and gained popularity as a way to add computational features to a UNIX pipeline. A version of the AWK language is a standard feature of nearly every modern UNIX-like operating system.

The Script

Often times, it becomes necessary to compress files to save (or make) space (for more files??). With the availability of gzip and bzip2 on most Linux distributions, often times one is left to ponder – which mechanism is better for a given file? Although it’s a common understanding that bigger the file size, bzip2 performs better, the following script is expected to help one in that matter. It compresses a given file with both options, compares the resulting filesize with the original (uncompressed) file and then makes a decision to retain one of the three (uncompressed, gzipped or bzipped).

#! /bin/bash
 
# BASH script that takes ONE filename as a (mandatory) argument
# and compresses them using both 'GZIP' and 'BZIP2'. Then, compares
# the filesizes and retains the one with the smallest value.
# 
# Usage: ./optimal_compression.sh [FILENAME]
#
# 01 September, 2006
# Tue, 14 Oct 2008 12:53:11 -0400
# Sun, 19 Oct 2008 12:08:36 -0400
 
if [[ $# -ne 1 ]];
then
  # Display error message when no (or more than one) files are specified as arguments
  echo
  echo " You must specify a filename"
  echo
  exit
else
 
  # Assign the supplied filename to a variable
  FILENAME=$1
 
  # Other useful variables
  TODAY=`date +"%Y%m%d_%H%M%S"`
  BZIP=`which bzip2`
  GZIP=`which gzip`
 
  # Temporary files that contain compressed data
  BZIP_FILE="/tmp/tmp_$TODAY.bz2"
  GZIP_FILE="/tmp/tmp_$TODAY.gz"
 
 
  # Compress the files with gzip and bzip2
  # One may add '&' at the end of the lines below and
  # uncomment 'wait' line, if really big files are being
  # compressed. It may save time. However, the definition
  # OPTIMUM might need some modification if done so.
  $BZIP < $FILENAME > $BZIP_FILE
  $GZIP < $FILENAME > $GZIP_FILE
  # wait
 
  # Compare file size - of normal, gzipped and bzipped files
  # and determine which one has the smallest (optimal) value.
  # Remember, 'ls -l' sorts filenames alphabetically
  OPTIMUM=`ls -ltr $FILENAME $BZIP_FILE $GZIP_FILE | \
           awk '{print $5":"NR}' | sort -n | \
           awk -F ':' '{if ($1 != "") { print $2 }}' | head -1`
 
  case "$OPTIMUM" in
    1 ) echo
        echo " $FILENAME not compressed."
        echo
        ;;
    2 ) echo
        echo " $FILENAME compressed with $BZIP => $FILENAME.bz2"
        echo
        mv $BZIP_FILE "$FILENAME.bz2" 
        ;;
    3 ) echo
        echo " $FILENAME compressed with $GZIP => $FILENAME.gz"
        echo
        mv $GZIP_FILE "$FILENAME.gz" 
        ;;
  esac
 
  # Remove temporary files 
  rm -f $FILENAME $BZIP_FILE $GZIP_FILE
fi

PERL – Mapping Elements To Their Respective Arrays

Gowtham — Wed, 15 Oct 2008 16:19:25 +0000

Often times, a need arises where one has to search a bunch of arrays to find out to which one of them a given set of elements belong. Of course, life becomes a lot simpler if those arrays are unique (meaning any given element appears only in one of the arrays) and life may not always be that simple. Some common examples of such simple case include, but not limited to, the following:

Decide which fruit belongs to which basket
Decide which database to query based on a set criterion. Certainly helps if the array names are the same as respective database names. There are potentially very many such database driven applications
And so on.

If you are looking for methods to map just one element (in question) to its array, you may refer to my previous write up.

The Script/Module

Save the following into a file and call it, elements_arrays_map.pl (or elements_arrays_map.pm, as the need may be). It provides a function/sub routine that takes the element names as the input and returns the array names to which the requested elements belong as an array. The sub-routine makes a hash of all the arrays and uses foreach loop to sift through them. The array name (depending on the application), can subsequently used appropriately. (Re)naming the variables and functions to describe the specific application may also be useful.

#! /usr/bin/perl -wT
 
#
# PERL module/script to decide to which array an element belongs to
# when there are multiple arrays.
# Assumes that the arrays contain UNIQUE element names
# Element_Array_Map() takes in elementNames as input and returns the 
# name of arrays they are in.
# 
 
#
# sub Elements_Arrays_Map() BEGINS
 
sub Elements_Arrays_Map (@) {
 
#
# Requested elementNames
 
  my @elements = @_;
 
#
# Uncomment the following lines for debugging purposes
# foreach my $element (@{elements}) {
#   print "Requested element is $element\n";
# }
 
#
# A hash of arrays
# Each array contains several element names
# Each array can have different number of elements 
# Elements of each array are UNIQUE
# Hash definition can be modified to include 'N' arrays
 
  my %HASH_OF_ARRAYS = (
      ARRAY1 => ['Element1', 'Element2'], 
      ARRAY2 => ['Element3', 'Element4', 'Element5', 'Element6'],
      ARRAY3 => ['Element7', 'Element8', 'Element9'],
      ARRAY4 => ['Element0'],
      );
 
#
# Search the hash of arrays to find out to which arrays do the 
# requested elements belong
 
  my $i = 0;
  foreach my $element (@{elements}) {
    foreach my $element_array (keys %HASH_OF_ARRAYS) {
      foreach (@{$HASH_OF_ARRAYS{$element_array}}) {
        if ( $element =~ m/$_/ ) {  
          print "$element is in $element_array\n";
          $output_array[$i] = $element_array;
        }
      }
    }
    $i++;
  }
  return @output_array;
} 
# sub Elements_Arrays_Map() ENDS
 
#
# Call it after the function to avoid the following error
# 'main::Elements_Arrays_Map() called too early to check prototype'
# If not using '-wT' flags, this function can be called before its 
# definition. The value '@output_array' can be passed on to other 
# modules/scripts, if necessary.
 
my @elementNames = ("Element1", "Element0", "Element4");
my @output_array = Elements_Arrays_Map(@elementNames);
 
# Uncomment the following line for debugging purposes
# print "$element_array\n";

BASH – Wrappers For qstat In NPACI ROCKS

Gowtham — Mon, 13 Oct 2008 22:34:07 +0000

BASH is a free software UNIX shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 UNIX circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

What is ROCKS?

General Wrapper for qstat

#! /bin/bash
 
# Save this as /home/local/bin/qstat
# A wrapper for 'qstat' command to show all currently submitted jobs along
# with the total number processors used/in use
#
# Check the path to 'qstat'
#
# Thu, 02 Oct 2008 21:06:05 -0400
 
# Modify this variable to meet the cluster's requirements
export PROC_TOTAL="128"
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat | \
       awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
       tail -1 | \
       awk '{print $NF}'`
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
export JOBS_LIST=`/opt/gridengine/bin/lx26-amd64/qstat`
 
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "$JOBS_LIST"
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "   $PROC_TOTAL total processors :: $PROC_INUSE are in use :: $PROC_AVAIL are available"
echo '-----------------------------------------------------------------------------------------------------------------'
echo

User Specific Wrapper for qstat

#! /bin/bash
 
# Save this as /home/local/bin/qu
# A wrapper for 'qstat' command to show all currently submitted jobs by a given 
# user ($USER) along with the total number processors used/in use
# If the $USER has no jobs, displays all currently submitted jobs.
#
# Check the path to 'qstat'
#
# Thu, 02 Oct 2008 21:35:04 -0400
 
 
# Modify this variable to meet the cluster's requirements
export PROC_TOTAL="128"
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat | \
       awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
       tail -1 | \
       awk '{print $NF}'`
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
export JOBS_LIST=`/opt/gridengine/bin/lx26-amd64/qstat -u $USER`
 
if [ "$JOBS_LIST" == "" ];
then
  export JOBS_LIST=`/opt/gridengine/bin/lx26-amd64/qstat`
fi
 
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "$JOBS_LIST"
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "   $PROC_TOTAL total processors :: $PROC_INUSE are in use :: $PROC_AVAIL are available"
echo '-----------------------------------------------------------------------------------------------------------------'
echo

PERL – Mapping An Element To Its Array

Gowtham — Sat, 11 Oct 2008 17:28:50 +0000

Disclaimer

Often times, a need arises where one has to search a bunch of arrays to find out to which one of them a given element belongs. Of course, life becomes a lot simpler if those arrays are unique (meaning any given element appears only in one of the arrays) and life may not always be that simple. Some common examples of such simple case include, but not limited to, the following:

Decide which fruit belongs to which basket
Decide which database to query based on a set criterion. Certainly helps if the array names are the same as respective database names. There are potentially very many such database driven applications
And so on.

The Script/Module

Save the following into a file and call it, element_array_map.pl (or element_array_map.pm, as the need may be). It provides a function/sub routine that takes the element name as the input and returns the array name to which the requested element belongs as output. The sub-routine makes a hash of all the arrays and uses foreach loop to sift through them. The array name (depending on the application), can subsequently used appropriately. (Re)naming the variables and functions to describe the specific application may also be useful.

#! /usr/bin/perl -wT
 
#
# PERL module/script to decide to which array an element belongs to
# when there are multiple arrays.
# Assumes that the arrays contain UNIQUE element names
# Element_Array_Map() takes in elementName as input and returns the name of array it's in.
# 
 
#
# sub Element_Array_Map() BEGINS
 
sub Element_Array_Map($) {
 
#
# Requested elementName
 
  local ($element) = $_[0];
 
#
# Uncomment the following line for debugging purposes
# print("Requested element is $element\n");
 
#
# A hash of arrays
# Each array contains several element names
# Each array can have different number of elements 
# Elements of each array are UNIQUE
# Hash definition can be modified to include 'N' arrays
 
  my %HASH_OF_ARRAYS = (
      ARRAY1 => ['Element1', 'Element2'], 
      ARRAY2 => ['Element3', 'Element4', 'Element5', 'Element6'],
      ARRAY3 => ['Element7', 'Element8', 'Element9'],
      ARRAY4 => ['Element0'],
      );
 
#
# Search the hash of arrays to find out to which array does the 
# requested element belong
 
  foreach my $element_array (keys %HASH_OF_ARRAYS) {
    foreach (@{$HASH_OF_ARRAYS{$element_array}}) {
      if ( $element =~ m/$_/ ) {  
        print "$element is in $element_array\n";
        return $element_array;
      }
    }
  }
 
} 
# sub Element_Array_Map() ENDS
 
#
# Call it after the function to avoid the following error
# 'main::Element_Array_Map() called too early to check prototype'
# If not using '-wT' flags, this function can be called before its 
# definition. The value '$element_array' can be passed on to other 
# modules/scripts, if necessary.
 
my $elementName   = "Element1";
my $element_array = Element_Array_Map($elementName);
 
# Uncomment the following line for debugging purposes
# print "$element_array\n";

Expect – Remote SSH Login

Gowtham — Fri, 12 Sep 2008 21:03:04 +0000

Expect is a UNIX automation and testing tool, written by Don Libes as an extension to the Tcl scripting language, for interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, ssh, and others. It uses UNIX pseudo terminals to wrap up sub-processes transparently, allowing the automation of arbitrary applications that are accessed over a terminal. With Tk, interactive applications can be wrapped in X11 GUIs. Expect has regular expression pattern matching and general program capabilities, allowing simple scripts to intelligently control programs such as telnet, ftp, and ssh, all of which lack a programming language, macros, or any other program mechanism. The result is that Expect scripts provide old tools with significant new power and flexibility.

The Script

When a server keeps prompting for password at SSH attempts in spite of setting up RSA/DSA keys, this script can be used to overcome that issue. Make sure that the script has 700 permission as it will contain your password in plain text.

#! /usr/bin/expect
 
# Fri, 12 Sep 2008 12:58:10 -0400
 
# Edit the following line - $USER@$SERVER
spawn ssh your-useid@your-server-name
 
# First time connection will print out some text for
# which one needs to type 'yes' to continue
# Comment these two lines after the first attempt
expect "*Are you sure you want to continue connecting*"
send   "yes\r"
 
# Put the password here
expect "*assword*"
send   "YOUR-PASSWORD\r"
 
# Start interacting
interact

BASH – Sum of All File/Folder Sizes

Gowtham — Wed, 09 Apr 2008 21:14:08 +0000

Disclaimer

The instructions/steps/scripts/methods given below worked for me running CentOS. It may very well work for you on other linux distributions, Red Hat-like or otherwise. Please note that if you decide to use these instructions on your machine, you are doing so entirely at your very own discretion and that neither this site, sgowtham.net, nor its author is responsible for any/all damage – intellectual or otherwise.

What is BASH and AWK?

The Script

#! /bin/bash
 
# Calculates the sum of all files/folders (in bytes) 
# Wed, 09 Apr 2008 10:44:02 -0400
 
echo
echo "Sum of all requested files/folders"
ls -al $1 | awk '{sum = sum + $5} END {print sum}'
echo

BASH – Read A File, One Line At A Time

Gowtham — Wed, 23 Jan 2008 23:15:24 +0000

Disclaimer

The instructions/steps given below worked for me running CentOS. It may very well work for you on other linux distributions, Red Hat-like or otherwise. Please note that if you decide to use these instructions on your machine, you are doing so entirely at your very own discretion and that neither this site, sgowtham.net, nor its author is responsible for any/all damage – intellectual or otherwise.

What is BASH?

BASH is a free software Unix shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 UNIX circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

The Script

#! /bin/bash
 
# Read one line at a time from a specified file. 
# Can be used to extract required information from those lines.
#
# Usage: ./read-lines.sh FILENAME 
# Mon, 23 Jun 2008 22:24:24 -0400
 
echo
exec<$1
 
while read line
do
  echo $line
done
 
echo

BASH – Replace Space In Filenames With Underscore

Gowtham — Fri, 19 Oct 2007 02:20:51 +0000

Disclaimer

What is BASH?

BASH is a free software Unix shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 UNIX circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

The Script

#! /bin/bash
 
# Removes spaces in filenames (replaces them with underscore)
# Thu, 18 Oct 2007 21:34:45 -0400
 
for file in *;
do
  echo Converting "$file" to "${file// /_}"
  mv "$file" "${file// /_}"
done

PERL – Password Encryption

Gowtham — Sat, 14 Jul 2007 00:16:24 +0000

Disclaimer

What is PERL?

In computer programming, PERL is a high-level, general-purpose, interpreted, dynamic programming language. PERL was originally developed by Larry Wall, a linguist working as a systems administrator for NASA, in 1987, as a general purpose UNIX scripting language to make report processing easier. Since then, it has undergone many changes and revisions and became widely popular among programmers. Larry Wall continues to oversee development of the core language. PERL borrows features from other programming languages including C, shell scripting (sh), AWK, sed and Lisp. The language provides powerful text processing facilities without the arbitrary data length limits of many contemporary UNIX tools, making it the ideal language for manipulating text files. It is also used for graphics programming, system administration, network programming, applications that require database access and CGI programming on the Web. PERL is nicknamed the Swiss Army knife of programming languages due to its flexibility and adaptability.

The Script

#! /usr/bin/perl
 
# PERL script to quickly generate a password (one could use mkpasswd as well). 
# The password generated with this script has its first two characters as the 'salt' 
# defined below, easily indicating whether the user has changed the temporary 
# password or not.
#
# Patrick Krogel, Michigan Technological University
 
# Salt - common practice is to choose System Administrator's initials
$salt = "ab";
 
sub encrypt($) {
   my $pass  = shift;
   my $epass = crypt($pass,$salt);
   return $epass;
}
 
# Step 1: Get Password (twice)
system("stty -echo");
print "Enter password: ";
$pass1 = <STDIN>;
chomp ($pass1);
 
print "\nConfirm password: ";
$pass2 = <STDIN>;
chomp ($pass2);
 
system("stty echo");
 
# Compare the user entry and process it
if (! -z $pass1 && $pass1 eq $pass2) {
   $epass = encrypt($pass1);
   print "\nPassword: $epass\n";
} elsif ($pass1 ne $pass2) {
   print "\nPasswords do not match.\n";
} else {
   print "\nPassword is blank.\n";
}

BASH – Sum Of All Integers Between Given Two Integers

Gowtham — Sat, 07 Jul 2007 01:23:57 +0000

Disclaimer

What is BASH and AWK?

The Script

#! /bin/bash
 
# BASH/AWK utility to find the sum of all integers between any two given integers.
# In the absence of second integer, it calculates the sum of all integers, starting from zero,
# till the given number
# 
# Usage: sum.sh INTEGER1 INTEGER2
# Fri, 06 Jul 2007 20:46:42 -0400
 
seq $1 $2 | awk '{x+=$1} END {print x}'

BASH – Filename Toggle Case

Gowtham — Thu, 18 May 2006 03:34:08 +0000

Disclaimer

What is BASH?

BASH is a free software Unix shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 UNIX circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

The Script

#! /bin/bash
 
# BASH script to convert upper (lower) case filenames to lower (upper) case.
# Converts all files in a given directory, including any directory.
#
# Wed, 17 May 2006 22:09:15 -0400
 
# Uppercase to lowercase
for file in `ls *.*`;
do
  lfile=$(echo $file | tr [A-Z] [a-z])
  echo "  Converting ${file} to ${lfile} "
  mv ${file} ${lfile}
done
 
# Lowercase to Uppercase
# for file in `ls *.*`; 
# do
#   ufile=$(echo $file | tr [a-z] [A-Z])
#   echo "  Converting ${file} to ${ufile} "
#   mv ${file} ${ufile}
# done

BASH – Center Align A File

Gowtham — Sun, 09 Apr 2006 23:39:48 +0000

Disclaimer

What is BASH and SED?

BASH is a free software Unix shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 UNIX circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

sed (Stream EDitor) refers to a UNIX utility which parses text files and implements a programming language which can apply textual transformations to such files. It reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line. It was developed from 1973 to 1974 as a UNIX utility by Lee E. McMahon of Bell Labs, and is available for most operating systems.

The Script

#! /bin/bash
 
# BASH script to center all text in the middle of 79-column width.
# Leading white spaces are insignificant and no trailing spaces appear at the end.
#
# Sun, 09 Apr 2006 08:47:20 -0400
 
if [ "$1" != "" ];
then
  echo
  echo "Center aligning the text in $1"
  export FILENAME=`echo $1 | awk -F '.' '{print $1}'`
  export EXTN=`echo $1 | awk -F '.' '{print $2}'`
  sed  -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' $1 > $FILENAME_calign.$EXTN
  echo "Open $FILENAME_calign.$EXTN"
  echo
fi

PERL – Disk Usage Reminder

Gowtham — Fri, 24 Feb 2006 23:46:55 +0000

Disclaimer

What is PERL?

The Script

#! /usr/bin/perl -wT
 
# Perl script to check disk usage and send emails to users hogging more than a specified limit. 
# Needs to be run as root and a good way to do it is via a cron job.
# 
# First written : Pat Krogel, Fri Feb 24 11:51:35 EST 2006
# Last modified : Gowtham, Fri Feb 24 15:37:40 EST 2006
 
# Variable initializations
$date    = localtime;
$dulimit = 20;   # Disk Usage Limit, in GB
 
# Get the disk usage, in GB, and separate out space and uids 
open PIPE, "du -s -B 1024M /home/* |" or die "Can't get disk usage.\n";
while ()
{
   chomp;
   my ($space, $user) = split /\s+/;
   $user =~ s/^.*\/(\w+)$/\1/;
   if ($space > $dulimit)
   {
      # Debugging purposes only! 
      # print "$user is taking up $space GB\n";
      open MAIL, "| mail -s 'Disk Usage Notification' $user\@domain.name"
        or die "Can't send email.\n";
      print MAIL "$user,\n\n";
      print MAIL "Your home folder,\n\n";
      print MAIL "/home/$user ($space GB)\n\n";
      print MAIL "in 'host.domain.name' has exceeded the present limit.\n";
      print MAIL "Please back it up elsewhere (some other machine or\n";
      print MAIL "external hard drives) and clean up the home folder.\n\n";
      print MAIL "root\@host.domain.name\n";
      print MAIL "$date\n";
      close MAIL;
   }
}
close PIPE;

BASH – Router’s Public IP Address

Gowtham — Wed, 04 Jan 2006 23:58:01 +0000

BASH, AWK and SED

BASH is a free software Unix shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 UNIX circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

sed (stream editor) is a Unix utility that parses text files and implements a programming language which can apply textual transformations to such files. It reads input files line by line, applying the operation which has been specified via the command line (or a sed script), and then outputs the line. It was developed in 1973-74 as a Unix utility by Lee McMahon of Bell Labs, and is available for UNIX and most flavors of Linux operating system.

The Script

#! /bin/bash
 
# POSIX SH shell script to display the IP Address - one that the machine has, 
# as well as router's public IP Address, if any.
 
# First written : Gowtham,  Wed Dec 28 22:05:35 EST 2005
# Last modified : Gowtham,  Fri Dec 30 06:57:50 EST 2005
#                 Gowtham,  Tue Jan  3 09:52:47 EST 2006
#                 Jon DeVree, Tue Jan  3 22:28:15 EST 2006 
 
IFACE=`netstat -rn | grep ^0.0.0.0 | awk '{print $8}'`
 
if [ "$IFACE" = "" ];
then
  echo
  echo " You are not connected to the internet"
  echo
  exit 0
fi
 
IP=`/sbin/ifconfig $IFACE | grep Bcast | awk '{print $2}' | awk -F ':' '{print $2}'`
NSIP=`curl -s http://checkip.dyndns.org | awk '{print $6}' | \
            awk -F '<' '{print $1}'`
 
if [ "$IP" = "$NSIP" ];
then
  echo
  echo " Your machine is directly connected to the internet"
  echo " IP Address : $IP"
  echo
else
  echo
  echo " Your machine is configured behind a router"
  echo " Machine's static IP Address : $IP"
  echo " Router's  public IP Address : $NSIP"
  echo
fi

BASH – Login Counter

Gowtham — Sat, 24 Sep 2005 00:20:34 +0000

Disclaimer

What is BASH and AWK?

BASH is a free software Unix shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 UNIX circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

The Script

#! /bin/bash
# 
# Displays the number of login attempts by users
# Gowtham, 2005.09.23
 
echo
echo " `hostname` login attempts by users in this month"
echo
last | awk '{print $1}' | sort | uniq -c | sort -nr