Seventh Sense » Rocks http://sgowtham.net/blog Rambling about life's little things, in 7 ≡ 1 (mod 6) fashion Tue, 28 Feb 2012 23:25:54 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 Seventh Sense http://sgowtham.net/images/sg_logo_rss.png http://sgowtham.net/blog 32 32 HPCC 1.4.1 Benchmark With GCC 4.1.2 On Rocks 5.4.2 http://sgowtham.net/blog/2012/02/28/hpcc-1-4-1-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/ http://sgowtham.net/blog/2012/02/28/hpcc-1-4-1-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/#comments Tue, 28 Feb 2012 18:27:14 +0000 Gowtham http://sgowtham.net/blog/?p=3411 Disclaimer

The instructions/steps given below worked for me (and Michigan Technological University) running NPACI Rocks 5.4.2 (with CentOS 5.5) – as has been a common practice for several years now, a full version of Operating System was installed. These instructions may very well work for you (or your institution), on Rocks-like or other linux clusters. Please note that if you decide to use these instructions on your machine, you are doing so entirely at your very own discretion and that neither this site, sgowtham.net, nor its author (or Michigan Technological University) is responsible for any/all damage – intellectual and/or otherwise.


A Bit About HPCC

Citing HPCC website,

HPCC benchmark suite has been released by DARPA High Productivity Computing Systems (HPCS) program to help define the performance boundaries of future Petascale computing systems. HPCC is a suite of tests that examine the performance of HPC architectures using kernels with memory access patterns more challenging than those of the HPL benchmark used in Top500. This suite is designed to augment the list, providing benchmarks that bound the performance of many real applications as a function of memory access characteristics. For e.g., spatial and temporal locality, and providing a framework for including additional tests. HPCC consists of seven tests that attempt to span high and low spatial and temporal locality space.

By design, the HPCC tests are scalable with the size of data sets being a function of the largest HPL matrix tested for the system. Since HPCC kernels consist of simple mathematical operations, it provides a unique opportunity to look at language and parallel programming issues. To characterize the architecture of the system, the following three scenarios are considered:

  1. Local – only a single processor is performing the computations
  2. Embarrassingly Parallel – each processor in the entire system is performing the computations but without explicit communication with each other
  3. Global – all processors in the system are performing the computations with explicit communication with each other

The seven tests that make up HPCC benchmark are as follows:

  1. HPL – the Linpack TPP benchmark which measures the floating point rate of execution for solving a linear system of equations
  2. DGEMM – measures the floating point rate of execution of double precision real matrix-matrix multiplication
  3. STREAM – a simple synthetic benchmark program that measures sustainable memory bandwidth (in GB/s) and the corresponding computation rate for simple vector kernel
  4. PTRANS (parallel matrix transpose) – exercises the communications where pairs of processors communicate with each other simultaneously. It is a useful test of the total communications capacity of the network
  5. RandomAccess – measures the rate of integer random updates of memory (GUPS)
  6. FFT – measures the floating point rate of execution of double precision complex one-dimensional Discrete Fourier Transform (DFT)
  7. Communication bandwidth and latency – a set of tests to measure latency and bandwidth of a number of simultaneous communication patterns; based on beff (effective bandwidth benchmark)


Pre-requisites: MPI & Goto BLAS

Please refer to the corresponding sections in HPL 2.0 Benchmark With GCC 4.1.2 On Rocks 5.4.2 to learn more about required pre-requisites as well as instructions for installing/compiling them. Following Rocks recommendations, pre-requisites will be installed under /share/apps/; software installed by me, in clusters at Michigan Tech, have the following template for their folder structure:

/share/apps/Software/Software_Version/Compiler/Compiler_Version


HPCC Installation/Compilation

With MPICH2 and Goto BLAS2 in place, HPCC 1.4.1 will be installed under

/share/apps/hpcc/1.4.1/mpich2/1.4.1p1/gcc/4.1.2



The following script, used in installation, will assume that one has downloaded HPCC 1.4.1 from here & the necessary Make.MPICH2141p1_GCC412.HPCC (listed below; it does look identical to the one used when compiling HPL 2.0 – the only difference is in the definition of TOPdir variable) and placed them in /share/apps/tmp/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175

#
# Makefile (Make.MPICH2141p1_GCC412) used to compile HPCC 
# on HPC clusters running NPACI Rocks (5.4.2) with CentOS (5.5)
# at Michigan Technological University.
# 
# Disclaimer:
# Please note that you are using these instructions 
# at your very own risk and that Michigan Technological
# University is not responsible for any/all damage caused to 
# your property, intellectual or otherwise.
# 
# For additional help and/or comments, questions, suggestions,
# please contact 
#
# Gowtham
# Information Technology Services & Security
# Michigan Technological University
# g@mtu.edu
#
# * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
#
#
#  -- High Performance Computing Linpack Benchmark (HPL)                
#     HPL - 2.0 - September 10, 2008                          
#     Antoine P. Petitet                                                
#     University of Tennessee, Knoxville                                
#     Innovative Computing Laboratory                                 
#     (C) Copyright 2000-2008 All Rights Reserved                       
#                                                                       
#  -- Copyright notice and Licensing terms:                             
#                                                                       
#  Redistribution and use in source and binary forms, with or without
#  modification, are permitted provided that the following conditions
#  are met:                                                             
#                                                                       
#  1. Redistributions of source code must retain the above copyright
#  notice, this list of conditions and the following disclaimer.        
#                                                                       
#  2. Redistributions in binary form must reproduce the above copyright
#  notice, this list of conditions, and the following disclaimer in the
#  documentation and/or other materials provided with the distribution. 
#                                                                       
#  3. All advertising materials mentioning features or use of this
#  software must display the following acknowledgement:                 
#  This product includes software developed at the University of
#  Tennessee, Knoxville, Innovative Computing Laboratory.             
#                                                                       
#  4. The name of the University, the name of the Laboratory, or the
#  names of its contributors may not be used to endorse or promote
#  products derived from this software without specific written
#  permission.
#
#  -- Disclaimer:
#                                                                       
#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
#  ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
#  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
#  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY
#  OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
#  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
#  DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
#  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
#  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#  
 
#
# Shell Details
SHELL        = /bin/sh
CD           = cd
CP           = cp
LN_S         = ln -s
MKDIR        = mkdir
RM           = /bin/rm -f
TOUCH        = touch
 
#
# Platform Identifier
ARCH         = MPICH2141p1_GCC412
 
#
# HPL Directory Structure / HPL library
TOPdir       = ../../..
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
HPLlib       = $(LIBdir)/libhpl.a 
 
#
# Message Passing library (MPI)
MPdir        = /share/apps/mpich2/1.4.1p1/gcc/4.1.2
MPinc        = -I$(MPdir)/include
MPlib        = $(MPdir)/lib/libmpich.a
 
#
# Linear Algebra library (BLAS)
LAlib        = /share/apps/gotoblas2/1.13/gcc/4.1.2/libgoto2.a
 
#
# F77 / C Interface
# You can skip this section if and only if you are not planning to use
# a BLAS library featuring a Fortran 77 interface. Otherwise, it is
# necessary to fill out the F2CDEFS variable with the appropriate
# options. **One and only one** option should be chosen in **each** of
# the 3 following categories:
#
# 1) name space (How C calls a Fortran 77 routine)
#
# -DAdd_              : all lower case and a suffixed underscore (Suns,
#                       Intel, ...) [default]
# -DNoChange          : all lower case (IBM RS6000)
# -DUpCase            : all upper case (Cray)
# -DAdd__             : the FORTRAN compiler in use is f2c
#
# 2) C and Fortran 77 integer mapping
#
# -DF77_INTEGER=int   : Fortran 77 INTEGER is a C int [default]
# -DF77_INTEGER=long  : Fortran 77 INTEGER is a C long
# -DF77_INTEGER=short : Fortran 77 INTEGER is a C short
#
# 3) Fortran 77 string handling
#
# -DStringSunStyle    : The string address is passed at the string location
#                       on the stack, and the string length is then
#                       passed as an F77_INTEGER after all explicit
#                       stack arguments [default]
# -DStringStructPtr   : The address of a structure is passed by a
#                       Fortran 77 string, and the structure is of the
#                       form: struct {char *cp; F77_INTEGER len;}
# -DStringStructVal   : A structure is passed by value for each Fortran
#                       77 string, and the structure is of the form:
#                       struct {char *cp; F77_INTEGER len;}
# -DStringCrayStyle   : Special option for Cray machines, which uses
#                       Cray fcd (fortran character descriptor) for
#                       interoperation
F2CDEFS      =  -DAdd_
 
#
# HPL Includes / Libraries / Specifics
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
 
#
# HPL Compile Time Options
# -DHPL_COPY_L           force the copy of the panel L before bcast
# -DHPL_CALL_CBLAS       call the cblas interface
# -DHPL_CALL_VSIPL       call the vsip library
# -DHPL_DETAILED_TIMING  enable detailed timers
#
# By default HPL will:
#    *) not copy L before broadcast
#    *) call the BLAS Fortran 77 interface
#    *) not display detailed timing information
HPL_OPTS     = -DHPL_COPY_L -DHPL_CALL_CBLAS -DHPL_DETAILED_TIMING 
 
#
# HPL Definitions
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
 
#
# Compilers / linkers - Optimization Flags
CC           = mpicc
CCNOOPT      = $(HPL_DEFS)
CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall
 
#
# On some platforms, it is necessary to use the Fortran linker 
# to find the Fortran internals used in the BLAS library
LINKER       = mpif77
LINKFLAGS    = $(CCFLAGS)
ARCHIVER     = ar
ARFLAGS      = r
RANLIB       = echo
#


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

#! /bin/bash
#
# install_hpcc.sh
# BASH script to install HPCC 1.4.1 (compiled against MPICH2 1.4.1p1) on a
# Rocks 5.4.2 cluster's front end
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  # Set necessary variables
  export HPCC_VERSION="1.4.1"
  export MPICH2_VERSION="1.4.1p1"
  export GCC_VERSION="4.1.2"
  export HPCC_INSTALL="/share/apps/hpcc/${HPCC_VERSION}/mpich2/${MPICH2_VERSION}/gcc/${GCC_VERSION}"
  export MAKEFILE_ARCH="MPICH2141p1_GCC412"
 
  # Begin HPCC-${HPCC_VERSION}.tar.gz check IF
  if [ -e "/share/apps/tmp/hpcc-${HPCC_VERSION}.tar.gz" ]
  then
 
    mkdir /share/apps/hpcc/${HPCC_VERSION}/mpich2/${MPICH2_VERSION}/gcc/
    cd /share/apps/hpcc/${HPCC_VERSION}/mpich2/${MPICH2_VERSION}/gcc/
    tar -zxvf /share/apps/tmp/hpcc-${HPCC_VERSION}.tar.gz
    mv hpcc-${HPCC_VERSION} ${GCC_VERSION}
 
    echo
    echo "  Step #0: copy the makefile, make clean and make"
    cd ${HPCC_INSTALL}
    cp /share/apps/tmp/Make.${MAKEFILE_ARCH}.HPCC ./hpl/Make.${MAKEFILE_ARCH}
 
    make clean arch=${MAKEFILE_ARCH}
    make arch=${MAKEFILE_ARCH}
 
    echo
    echo "  Step #1: Update $HOME/.bashrc"
    cat <<EOF
 
  Add the following lines to $HOME/.bashrc and remember to source it
 
  # HPCC (${HPCC_VERSION}) settings
  export HPCC="${HPCC_INSTALL}"
  export PATH="\${PATH}:\${HPCC}/bin/${MAKEFILE_ARCH}"
  export LD_LIBRARY_PATH="\${LD_LIBRARY_PATH}:\${HPCC}/${MAKEFILE_ARCH}"
 
EOF
 
  fi
  # End hpcc-${HPCC_VERSION}.tar.gz check IF
  echo
 
fi 
# End root-check IF


Running HPCC Benchmark

Please refer to the corresponding sections in HPL 2.0 Benchmark With GCC 4.1.2 On Rocks 5.4.2 to learn how to get a fix on N, NB, etc., given the hardware configuration of compute nodes. Running HPCC benchmark is very similar to running HPL 2.0 benchmark.

Share

]]> http://sgowtham.net/blog/2012/02/28/hpcc-1-4-1-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/feed/ 0 HPL 2.0 Benchmark With GCC 4.1.2 On Rocks 5.4.2 http://sgowtham.net/blog/2012/02/27/hpl-2-0-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/ http://sgowtham.net/blog/2012/02/27/hpl-2-0-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/#comments Mon, 27 Feb 2012 23:44:17 +0000 Gowtham http://sgowtham.net/blog/?p=3361 Disclaimer

The instructions/steps given below worked for me (and Michigan Technological University) running NPACI Rocks 5.4.2 (with CentOS 5.5) – as has been a common practice for several years now, a full version of Operating System was installed. These instructions may very well work for you (or your institution), on Rocks-like or other linux clusters. Please note that if you decide to use these instructions on your machine, you are doing so entirely at your very own discretion and that neither this site, sgowtham.net, nor its author (or Michigan Technological University) is responsible for any/all damage – intellectual and/or otherwise.


A Bit About HPL

LINPACK is a software library for performing numerical linear algebra on computers. It was written in FORTRAN by Jack Dongarra, Jim Bunch, Cleve Moler and Gilbert Stewart. LINPACK makes use of the BLAS libraries for performing basic vector and matrix operations. It has largely been superseded by LAPACK, which runs more efficiently on modern architectures.

The LINPACK benchmarks are a measure of a system’s floating point computing power. Introduced by Jack Dongarra, they measure how fast a computer solves a dense N * N system of linear equations, Ax = b. The solution is obtained by Gaussian Elimination with Partial Pivoting, with

\frac{2}{3}\:N^3 \:+\: 2\:N^2 \:+\: \mathcal{O}\left(N\right)

floating point iterations. The result is often expressed in billions of floating point operations per second (GFLOPS). HPL, a portable implementation of High Performance LINPACK Benchmark, is used as a performance measure for ranking the supercomputers in the Top500 list.


Pre-requisite #1: MPI

Following Rocks recommendations, this and other pre-requisites will be installed under /share/apps/; software installed by me, in clusters at Michigan Tech, have the following template for their folder structure:

/share/apps/Software/Software_Version/Compiler/Compiler_Version



Rocks 5.4.2 installation has an instance (of few flavors) of MPI but I prefer to compile MPICH2 using GCC 4.1.2. At the time of writing this post, the latest stable version of MPICH2 is 1.4.1p1 and it may be downloaded from here. Following folder structure/template mentioned above, it will be installed under

/share/apps/mpich2/1.4.1p1/gcc/4.1.2



To avoid confusion and/or missed steps leading to undesired results, steps associated with installation of MPICH2 have been put in the following script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

#! /bin/bash
#
# install_mpich2.sh
# BASH script to install MPICH2 (compiled against GCC 4.1.2) on a
# Rocks 5.4.2 cluster's front end
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  # Set necessary variables
  export CC="gcc"
  export CXX="g++"
  export FC="gfortran"
  export F77="gfortran"
  export MPICH2_VERSION="1.4.1p1"
  export GCC_VERSION="4.1.2"
  export MPICH2_INSTALL="/share/apps/mpich2/${MPICH2_VERSION}/gcc/${GCC_VERSION}"
  export ANL="http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs"
 
  echo
  echo "  Step #0: Download MPICH2 to /share/apps/tmp"
  cd /share/apps/tmp/
  wget ${ANL}/${MPICH2_VERSION}/mpich2-${MPICH2_VERSION}.tar.gz
 
  # Begin mpich2-${MPICH2_VERSION}.tar.gz check IF
  if [ -e "mpich2-${MPICH2_VERSION}.tar.gz" ]
  then
    echo
    echo "  Step #1: configure, make clean, make and make install"
 
    tar -zxpf mpich2-${MPICH2_VERSION}.tar.gz
    cd mpich2-${MPICH2_VERSION}/
    ./configure --prefix=${MPICH2_INSTALL}
    make clean
    make
    make install
 
    echo
    echo "  Step #2: Update $HOME/.bashrc"
    cat <<EOF
 
  Add the following lines to $HOME/.bashrc and remember to source it
 
  # MPICH2 (${MPICH2_VERSION}) settings
  export MPICH2="${MPICH2_INSTALL}"
  export PATH="\${PATH}:\${MPICH2}/bin:\${MPICH2}/sbin"
  export MANPATH="\${MANPATH}:\${MPICH2}/man"
  export LD_LIBRARY_PATH="\${LD_LIBRARY_PATH}:\${MPICH2}/lib"
 
EOF
 
  fi
  # End mpich2-${MPICH2_VERSION}.tar.gz check IF
  echo
 
fi 
# End root-check IF



A good test of a successful installation is that which mpicc, which mpif77, etc. return the respective commands located in ${MPICH2_INSTALL}.


Pre-requisite #2: Goto BLAS

As in the case of pre-requisite #1, this one will be installed under

/share/apps/gotoblas2/1.13/gcc/4.1.2



The following script, used in installation, will assume that one has downloaded Goto BLAS2 1.13 from here and placed it in
/share/apps/tmp/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

#! /bin/bash
#
# install_gotoblas2.sh
# BASH script to install Goto BLAS2 (compiled against GCC 4.1.2) on a
# Rocks 5.4.2 cluster's front end
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  # Set necessary variables
  export CC="gcc"
  export CXX="g++"
  export FC="gfortran"
  export F77="gfortran"
  export GOTOBLAS2_VERSION="1.13"
  export GCC_VERSION="4.1.2"
  export GOTOBLAS2_INSTALL="/share/apps/gotoblas2/${GOTOBLAS2_VERSION}/gcc/${GCC_VERSION}"
 
  # Begin GotoBLAS2-${GOTOBLAS2_VERSION}.tar.gz check IF
  if [ -e "/share/apps/tmp/GotoBLAS2-${GOTOBLAS2_VERSION}.tar.gz" ]
  then
 
    mkdir /share/apps/gotoblas2/${GOTOBLAS2_VERSION}/gcc/
    cd /share/apps/gotoblas2/${GOTOBLAS2_VERSION}/gcc/
    tar -zxvf /share/apps/tmp/GotoBLAS2-${GOTOBLAS2_VERSION}.tar.gz
    mv GotoBLAS2 ${GCC_VERSION}
 
    echo
    echo "  Step #0: make clean and make"
    cd ${GOTOBLAS2_INSTALL}
    make clean
    make BINARY=64
 
    echo
    echo "  Step #1: Update $HOME/.bashrc"
    cat <<EOF
 
  Add the following lines to $HOME/.bashrc and remember to source it
 
  # GOTOBLAS2 (${GOTOBLAS2_VERSION}) settings
  export GOTOBLAS2="${GOTOBLAS2_INSTALL}"
  export LD_LIBRARY_PATH="\${LD_LIBRARY_PATH}:\${GOTOBLAS2}"
 
EOF
 
  fi
  # End GotoBLAS2-${GOTOBLAS2_VERSION}.tar.gz check IF
  echo
 
fi 
# End root-check IF


HPL Installation/Compilation

With MPICH2 and Goto BLAS2 in place, HPL 2.0 will be installed under

/share/apps/hpl/2.0/mpich2/1.4.1p1/gcc/4.1.2



The following script, used in installation, will assume that one has downloaded HPL 2.0 from here & the necessary Make.MPICH2141p1_GCC412.HPL (listed below) and placed them in /share/apps/tmp/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175

#
# Makefile (Make.MPICH2141p1_GCC412) used to compile HPL 
# on HPC clusters running NPACI Rocks (5.4.2) with CentOS (5.5)
# at Michigan Technological University.
# 
# Disclaimer:
# Please note that you are using these instructions 
# at your very own risk and that Michigan Technological
# University is not responsible for any/all damage caused to 
# your property, intellectual or otherwise.
# 
# For additional help and/or comments, questions, suggestions,
# please contact 
#
# Gowtham
# Information Technology Services & Security
# Michigan Technological University
# g@mtu.edu
#
# * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
#
#
#  -- High Performance Computing Linpack Benchmark (HPL)                
#     HPL - 2.0 - September 10, 2008                          
#     Antoine P. Petitet                                                
#     University of Tennessee, Knoxville                                
#     Innovative Computing Laboratory                                 
#     (C) Copyright 2000-2008 All Rights Reserved                       
#                                                                       
#  -- Copyright notice and Licensing terms:                             
#                                                                       
#  Redistribution and use in source and binary forms, with or without
#  modification, are permitted provided that the following conditions
#  are met:                                                             
#                                                                       
#  1. Redistributions of source code must retain the above copyright
#  notice, this list of conditions and the following disclaimer.        
#                                                                       
#  2. Redistributions in binary form must reproduce the above copyright
#  notice, this list of conditions, and the following disclaimer in the
#  documentation and/or other materials provided with the distribution. 
#                                                                       
#  3. All advertising materials mentioning features or use of this
#  software must display the following acknowledgement:                 
#  This product includes software developed at the University of
#  Tennessee, Knoxville, Innovative Computing Laboratory.             
#                                                                       
#  4. The name of the University, the name of the Laboratory, or the
#  names of its contributors may not be used to endorse or promote
#  products derived from this software without specific written
#  permission.
#
#  -- Disclaimer:
#                                                                       
#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
#  ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
#  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
#  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY
#  OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
#  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
#  DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
#  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
#  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#  
 
#
# Shell Details
SHELL        = /bin/sh
CD           = cd
CP           = cp
LN_S         = ln -s
MKDIR        = mkdir
RM           = /bin/rm -f
TOUCH        = touch
 
#
# Platform Identifier
ARCH         = MPICH2141p1_GCC412
 
#
# HPL Directory Structure / HPL library
TOPdir       = /share/apps/hpl/2.0/mpich2/1.4.1p1/gcc/4.1.2
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
HPLlib       = $(LIBdir)/libhpl.a 
 
#
# Message Passing library (MPI)
MPdir        = /share/apps/mpich2/1.4.1p1/gcc/4.1.2
MPinc        = -I$(MPdir)/include
MPlib        = $(MPdir)/lib/libmpich.a
 
#
# Linear Algebra library (BLAS)
LAlib        = /share/apps/gotoblas2/1.13/gcc/4.1.2/libgoto2.a
 
#
# F77 / C Interface
# You can skip this section if and only if you are not planning to use
# a BLAS library featuring a Fortran 77 interface. Otherwise, it is
# necessary to fill out the F2CDEFS variable with the appropriate
# options. **One and only one** option should be chosen in **each** of
# the 3 following categories:
#
# 1) name space (How C calls a Fortran 77 routine)
#
# -DAdd_              : all lower case and a suffixed underscore (Suns,
#                       Intel, ...) [default]
# -DNoChange          : all lower case (IBM RS6000)
# -DUpCase            : all upper case (Cray)
# -DAdd__             : the FORTRAN compiler in use is f2c
#
# 2) C and Fortran 77 integer mapping
#
# -DF77_INTEGER=int   : Fortran 77 INTEGER is a C int [default]
# -DF77_INTEGER=long  : Fortran 77 INTEGER is a C long
# -DF77_INTEGER=short : Fortran 77 INTEGER is a C short
#
# 3) Fortran 77 string handling
#
# -DStringSunStyle    : The string address is passed at the string location
#                       on the stack, and the string length is then
#                       passed as an F77_INTEGER after all explicit
#                       stack arguments [default]
# -DStringStructPtr   : The address of a structure is passed by a
#                       Fortran 77 string, and the structure is of the
#                       form: struct {char *cp; F77_INTEGER len;}
# -DStringStructVal   : A structure is passed by value for each Fortran
#                       77 string, and the structure is of the form:
#                       struct {char *cp; F77_INTEGER len;}
# -DStringCrayStyle   : Special option for Cray machines, which uses
#                       Cray fcd (fortran character descriptor) for
#                       interoperation
F2CDEFS      =  -DAdd_
 
#
# HPL Includes / Libraries / Specifics
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
 
#
# HPL Compile Time Options
# -DHPL_COPY_L           force the copy of the panel L before bcast
# -DHPL_CALL_CBLAS       call the cblas interface
# -DHPL_CALL_VSIPL       call the vsip library
# -DHPL_DETAILED_TIMING  enable detailed timers
#
# By default HPL will:
#    *) not copy L before broadcast
#    *) call the BLAS Fortran 77 interface
#    *) not display detailed timing information
HPL_OPTS     = -DHPL_COPY_L -DHPL_CALL_CBLAS -DHPL_DETAILED_TIMING 
 
#
# HPL Definitions
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
 
#
# Compilers / linkers - Optimization Flags
CC           = mpicc
CCNOOPT      = $(HPL_DEFS)
CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall
 
#
# On some platforms, it is necessary to use the Fortran linker 
# to find the Fortran internals used in the BLAS library
LINKER       = mpif77
LINKFLAGS    = $(CCFLAGS)
ARCHIVER     = ar
ARFLAGS      = r
RANLIB       = echo
#


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

#! /bin/bash
#
# install_hpl.sh
# BASH script to install HPL 2.0 (compiled against MPICH2 1.4.1p1) on a
# Rocks 5.4.2 cluster's front end
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  # Set necessary variables
  export HPL_VERSION="2.0"
  export MPICH2_VERSION="1.4.1p1"
  export GCC_VERSION="4.1.2"
  export HPL_INSTALL="/share/apps/hpl/${HPL_VERSION}/mpich2/${MPICH2_VERSION}/gcc/${GCC_VERSION}"
  export MAKEFILE_ARCH="MPICH2141p1_GCC412"
 
  # Begin hpl-${HPL_VERSION}.tar.gz check IF
  if [ -e "/share/apps/tmp/hpl-${HPL_VERSION}.tar.gz" ]
  then
 
    mkdir /share/apps/hpl/${HPL_VERSION}/mpich2/${MPICH2_VERSION}/gcc/
    cd /share/apps/hpl/${HPL_VERSION}/mpich2/${MPICH2_VERSION}/gcc/
    tar -zxvf /share/apps/tmp/hpl-${HPL_VERSION}.tar.gz
    mv hpl-${HPL_VERSION} ${GCC_VERSION}
 
    echo
    echo "  Step #0: copy the makefile, make clean and make"
    cd ${HPL_INSTALL}
    cp /share/apps/tmp/Make.${MAKEFILE_ARCH}.HPC ./Make.${MAKEFILE_ARCH}
 
    if [ -d "${HPL_INSTALL}/bin/${MAKEFILE_ARCH}"]
    then
      make clean arch=${MAKEFILE_ARCH}
    fi
 
    if [ "$(ls -A ${HPL_INSTALL}/bin)"];
    then
      make clean_arch_all arch=${MAKEFILE_ARCH} 
    fi
 
    make arch=${MAKEFILE_ARCH}
 
    echo
    echo "  Step #1: Update $HOME/.bashrc"
    cat <<EOF
 
  Add the following lines to $HOME/.bashrc and remember to source it
 
  # HPL (${HPL_VERSION}) settings
  export HPL="${HPL_INSTALL}"
  export PATH="\${PATH}:\${HPL}/bin/${MAKEFILE_ARCH}"
  export LD_LIBRARY_PATH="\${LD_LIBRARY_PATH}:\${HPL}/${MAKEFILE_ARCH}"
 
EOF
 
  fi
  # End hpl-${HPL_VERSION}.tar.gz check IF
  echo
 
fi 
# End root-check IF


Running HPL Benchmark

The amount of memory used by HPL is essentially the size of the co-efficient matrix, A. Following standard binary definition, 1 GB is 1024 * 1024 * 1024 bytes (TOTAL_MEMORY_BYTES). Most scientific/engineering computations use double precision numbers, with each such double precision number taking 8 bytes of memory. Thus, 1 GB can accommodate 134,217,728 double precision entities (TOTAL_DP_ELEMENTS = TOTAL_MEMORY_BYTES/8). Theoretically, sqrt(TOTAL_DP_ELEMENTS) represents the maximum possible value of N. However, operating system needs some memory to perform some necessary operations. As such, HPL benchmark is usually performed for the following values of N – with m representing the fraction of total memory – making sure that swapping did not occur (which would result in reduced performance).

N \:=\: m\:\sqrt{\mathrm{TOTAL\_DP\_ELEMENTS}} \hspace{0.50in} m: 0.50\:(0.10)\:0.80



HPL uses the block size (NB) for the data distribution as well as for the computational granularity. From a data distribution perspective, the smaller NB, the better the load balance. From a computational perspective, too small of a value for NB may limit the computational performance by a large factor since almost no data re-use will occur in the highest level of the memory hierarchy. The number of messages will also increase. In my case, this benchmark was performed for NB values of 128, 256 and 512.

The results so obtained are compared with the theoretical peak value, \mathrm{GFLOPS}_\mathrm{Theory}, computed as follows:


\mbox{\# of Nodes} \;\times\; \mbox{\# of Sockets/Node} \;\times\; \mbox{\# of Cores/Socket} \;\times\; \\\\\mbox{CPU Frequency (Cycles/second)} \;\times\; \mbox{\# of Floating Point Operations/Cycle}



For e.g., for a cluster with 16 identical recent Intel architecture compute nodes, each compute node with dual hex cores @ 3.00 GHz, \mathrm{GFLOPS}_\mathrm{Theory} will be


\mbox{16 (\# of Nodes)} \;\times\; \mbox{2 (\# of Sockets/Node)} \;\times\; \mbox{6 (\# of Cores/Socket)} \;\times\; \\\\\mbox{3 G [CPU Frequency (Cycles/second)]} \;\times\; \mbox{4 (\# of Floating Point Operations/Cycle)}\\\\ = 2304 \mbox{ GFLOPS} \\\\ \approx 2.3 \mbox{ TFLOPS}



If each of these nodes had 24 GB RAM, then TOTAL_MEMORY_BYTES = 1024 * 1024 * 1024 * 24 * 16 = 412316860416
and TOTAL_DP_ELEMENTS = 412316860416/8 = 51539607552.

As such, N values will be


N \:=\: m\:\times\: \sqrt{51539607552} \:\approx\: m\:\times\: 227020 \hspace{0.50in} m: 0.50\:(0.10)\:0.80


What if the cluster has heterogeneous compute nodes?

Computing \mathrm{GFLOPS}_\mathrm{Theory} isn’t easy in this case; becomes even more so when these compute nodes belong to different generations as one has to account for aging factor. It has been a practice at Michigan Tech, in such cases, to split the cluster into different queues – one for each generation/type of compute nodes – and run the HPL benchmark separately.


Thanks be to

Rocks mailing list and its participants.

Share

]]> http://sgowtham.net/blog/2012/02/27/hpl-2-0-benchmark-with-gcc-4-1-2-on-rocks-5-4-2/feed/ 1 Ganglia – gmond Python Module For Monitoring NVIDIA GPU On Rocks 5.4.2 http://sgowtham.net/blog/2012/02/17/ganglia-gmond-python-module-for-monitoring-nvidia-gpu-on-rocks-5-4-2/ http://sgowtham.net/blog/2012/02/17/ganglia-gmond-python-module-for-monitoring-nvidia-gpu-on-rocks-5-4-2/#comments Fri, 17 Feb 2012 23:24:43 +0000 Gowtham http://sgowtham.net/blog/?p=3238 Disclaimer

The instructions/steps given below worked for me (and Michigan Technological University) running NPACI Rocks 5.4.2 (with CentOS 5.5) – as has been a common practice for several years now, a full version of Operating System was installed. These instructions may very well work for you (or your institution), on Rocks-like or other linux clusters. Please note that if you decide to use these instructions on your machine, you are doing so entirely at your very own discretion and that neither this site, sgowtham.net, nor its author (or Michigan Technological University) is responsible for any/all damage – intellectual and/or otherwise.


A Bit About Ganglia (gmond & gmetad)

Citing Ganglia website,

Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRD tool for data storage and visualization. It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency. The implementation is robust, has been ported to an extensive set of operating systems and processor architectures, and is currently in use on thousands of clusters around the world. It has been used to link clusters across university campuses and around the world and can scale to handle clusters with 2000 nodes.

Further, citing Wikipedia,

gmond (Ganglia Monitoring Daemon) is a multi-threaded daemon which runs on each cluster node that needs to be monitored. Installation does not require having a common NFS file system or a database back-end, install special accounts or maintain configuration files. It has four main responsibilities: monitor changes in host state; announce relevant changes; listen to the state of all other ganglia nodes via a unicast or multicast channel; answer requests for an XML description of the cluster state.

Each gmond transmits in information in two different ways: unicasting or multicasting host state in external data representation (XDR) format using UDP messages OR sending XML over a TCP connection.

Federation in Ganglia is achieved using a tree of point-to-point connections amongst representative cluster nodes to aggregate the state of multiple clusters. At each node in the tree, a Ganglia Meta Daemon (gmetad) periodically polls a collection of child data sources, parses the collected XML, saves all numeric, volatile metrics to round-robin databases and exports the aggregated XML over a TCP sockets to clients. Data sources may be either gmond daemons, representing specific clusters, or other gmetad daemons, representing sets of clusters. Data sources use source IP addresses for access control and can be specified using multiple IP addresses for fail over. The latter capability is natural for aggregating data from clusters since each gmond daemon contains the entire state of its cluster.

The Ganglia web front-end provides a view of the gathered information via real-time dynamic web pages. Most importantly, it displays Ganglia data in a meaningful way for system administrators and computer users. Although the web front-end to ganglia started as a simple HTML view of the XML tree, it has evolved into a system that keeps a colourful history of all collected data. The Ganglia web front-end caters to system administrators and users (for e.g., one can view the CPU utilization over the past hour, day, week, month, or year). The web front-end shows similar graphs for memory usage, disk usage, network statistics, number of running processes, and all other Ganglia metrics. The web front-end depends on the existence of the gmetad which provides it with data from several Ganglia sources.

Specifically, the web front-end will open the local port 8651 (by default) and expects to receive a Ganglia XML tree. The web pages themselves are highly dynamic; any change to the Ganglia data appears immediately on the site. This behaviour leads to a very responsive site, but requires that the full XML tree be parsed on every page access. Therefore, the Ganglia web front-end should run on a fairly powerful, dedicated machine if it presents a large amount of data. The Ganglia web front-end is written in the PHP scripting language, and uses graphs generated by gmetad to display history information.


Installation & Configuration

Rocks 5.4.2 installation in itself takes care of almost everything pertaining to installing and configuring Ganglia, gmond, gmetad and Ganglia web interface. However, by default & design, a Rocks cluster’s web interface is not publicly accessible. To fix this, following commands were run:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

#! /bin/bash
#
# update_web_firewall.sh
# BASH script to run necessary 'rocks' commands to update the firewall rules on a
# Rocks 5.4.2 cluster's front end to make the web interface accessible from anywhere
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  echo
  echo "  Step #0: display current firewall rules"
  /opt/rocks/bin/rocks report host firewall localhost
 
  echo "  Step #1: removing the current rule for www"
  /opt/rocks/bin/rocks remove host firewall localhost chain=INPUT \
    flags="-m state --state NEW --source &Kickstart_PublicNetwork;/&Kickstart_PublicNetmask;" \
    protocol=tcp service=www action=ACCEPT network=public
 
  /opt/rocks/bin/rocks sync host firewall localhost
 
  echo "  Step #2: removing the current rule for https"
  /opt/rocks/bin/rocks remove host firewall localhost chain=INPUT \
    flags="-m state --state NEW --source &Kickstart_PublicNetwork;/&Kickstart_PublicNetmask;" \
    protocol=tcp service=https action=ACCEPT network=public
 
  /opt/rocks/bin/rocks sync host firewall localhost
 
  echo "  Step #3: adding new rule for www"
  /opt/rocks/bin/rocks add host firewall localhost chain=INPUT \
    flags="-m state --state NEW --source 0.0.0.0/0.0.0.0" \
    protocol=tcp service=www action=ACCEPT network=public
 
  /opt/rocks/bin/rocks sync host firewall localhost
 
  echo "  Step #4: adding new rule for https"
  /opt/rocks/bin/rocks add host firewall localhost chain=INPUT \
    flags="-m state --state NEW --source 0.0.0.0/0.0.0.0" \
    protocol=tcp service=https action=ACCEPT network=public
 
  /opt/rocks/bin/rocks sync host firewall localhost
 
  echo "  Step #5: display current firewall rules"
  /opt/rocks/bin/rocks report host firewall localhost
  echo
 
fi 
# End root-check IF



Upon pointing the browser to the http://FQDN/ganglia/, the web page should display the relevant information.


Monitoring NVIDIA GPU

The aforementioned set up works fine and as expected but it doesn’t necessarily provide any information about GPU(s) that may be part of the hardware. For e.g., the cluster used in this case has two NVIDIA GeForce GTX 260 cards in each compute node. For testing purposes, only one compute node was installed – also, one of the GTX 260 cards was replaced with a NVIDIA Quadro 6000. With more and more scientific & engineering computations tending towards GPU based computing, it’d be useful to include their status/usage information in Ganglia’s web portal. To this effect, NVIDIA released gmond Python module for GPUs (made aware of it by one of Michigan Tech ITSS directors). The instructions given in the NVIDIA-linked pages do work as described – however, Rocks 5.4.2 uses python 2.4 while one requires python 2.5 (or higher) to get the GPU metrics to show up in Ganglia.


Rebuilding Rocks Distribution with Python ctypes Library

I downloaded python-ctypes-1.0.2-2.el5.x86_64.rpm from http://ftp.osuosl.org/pub/fedora-epel/5/x86_64/ and placed it in
/export/rocks/install/contrib/5.4/x86_64/RPMS/ – rebuilding of the distribution, with following commands as usual, was uneventful.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#! /bin/bash
#
# update_rocks_distribution.sh
# BASH script to download python-ctypes-1.0.2-2.el5.x86_64.rpm from 
# http://ftp.osuosl.org/pub/fedora-epel/5/x86_64/ and rebuild the
# rocks distribution
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  echo
  cd /export/rocks/install/contrib/5.4/x86_64/RPMS/
  wget http://ftp.osuosl.org/pub/fedora-epel/5/x86_64/python-ctypes-1.0.2-2.el5.x86_64.rpm
 
  cd /export/rocks/install/
  rocks create distro
 
fi
# End root-check IF



Re-install the compute node(s) [in this case, compute-0-0]. Without and with ctypes library, gmond (when run in debug mode, i.e. gmond -d9 -f), results in the following message.


ganglia_gpu_41 ganglia_gpu_42




NVIDIA Driver Installation

Once the compute node(s) are re-installed, NVIDIA driver, NVIDIA-Linux-x86_64-285.05.33.run, was installed using the following script.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

#! /bin/bash
#
# install_nvidia_driver.sh
# BASH script to install NVIDIA driver in compute node(s) - save this in /share/apps/bin/
# Assumes that NVIDIA-Linux-x86_64-285.05.33.run is located in /share/apps/src/nvidia_cuda/
# Also, assumes that CUDA SDK 4.1.28 has been installed on front end in /share/apps/cuda/
# Must be root to run this script and run this in all compute nodes from the front end via
# the command
#
# rocks run host '/share/apps/bin/install_nvidia_driver.sh'
#
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  echo
  # 1. Install NVIDIA driver
  /share/apps/src/nvidia_cuda/NVIDIA-Linux-x86_64-285.05.33.run --silent
 
  # 2: Updating /etc/ld.so.config
  echo "/share/apps/cuda/lib64" >> /etc/ld.so.conf
  echo "/share/apps/cuda/lib" >> /etc/ld.so.conf
  /sbin/ldconfig
 
  # 3: Creating missing symbolic links to necessary libraries
  cd /usr/lib64/
  ln -sf libXmu.so.6.2.0 libXmu.so
  ln -sf libXi.so.6.0.0  libXi.so
 
fi
# End root-check IF


Python Bindings for the NVIDIA Management Library

This provides Python access to static information and monitoring data for NVIDIA GPUs, as well as management capabilities. It exposes the functionality of the NVML and one may download these from here – as before, the necessary steps are included in a BASH script.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#! /bin/bash
#
# install_python_nvml_bindings.sh
# BASH script to install Python Bindings for the NVML in compute node(s) - save this in /share/apps/bin/
# Assumes that nvidia-ml-py-2.285.01.tar.gz is in /share/apps/src/nvidia_ganglia/
# Must be root to run this script and run this in all compute nodes from the front end via the command
#
# rocks run host '/share/apps/bin/install_python_nvml_bindings.sh'
#
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  #
  # Download and install
  cd /tmp/
  cp /share/apps/src/nvidia_ganglia/nvidia-ml-py-2.285.01.tar.gz .
 
  tar -zxvpf nvidia-ml-py-2.285.01.tar.gz
  cd nvidia-ml-py-2.285.01
  python setup.py install
 
  # Copy nvidia_smi.py & pynvml.py to /opt/ganglia/lib64/ganglia/python_modules/
  cp nvidia_smi.py /opt/ganglia/lib64/ganglia/python_modules/
  cp pynvml.py /opt/ganglia/lib64/ganglia/python_modules/
 
fi
# End root-check IF


gmond Python Module For Monitoring NVIDIA GPUs using NVML

After downloading ganglia-gmond_python_modules-3dfa553.tar.gz from GitHub for ganglia / gmond_python_modules to
/share/apps/src/nvidia_ganglia/, the following steps need to be performed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#! /bin/bash
#
# copy_ganglia_gmond_python_computenodes.sh
# BASH script to copy relevant files from ganglia-gmond_python_modules to Ganglia, 
# and restart gmond - save this in /share/apps/bin/
# Assumes that ganglia-gmond_python_modules-3dfa553.tar.gz is in /share/apps/src/nvidia_ganglia/
# Must be root to run this script and run this in all compute nodes from the front end via the command
#
# rocks run host '/share/apps/bin/copy_ganglia_gmond_python_computenodes.sh'
#
 
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
 
  # Copy relevant files to Ganglia
  cd /tmp/
  tar -zxvpf /share/apps/src/nvidia_ganglia/ganglia-gmond_python_modules-3dfa553.tar.gz
  cd ganglia-gmond_python_modules-3dfa553/gpu/nvidia/
  cp python_modules/nvidia.py /opt/ganglia/lib64/ganglia/python_modules/
  cp conf.d/nvidia.pyconf /etc/ganglia/conf.d/
 
  #
  # Restart gmond
  /etc/init.d/gmond restart
 
fi
# End root-check IF


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#! /bin/bash
#
# copy_ganglia_gmond_python_frontend.sh
# BASH script to copy relevant files from ganglia-gmond_python_modules to Ganglia, 
# apply patch for Ganglia web interface and restart necessary services 
# Assumes that ganglia-gmond_python_modules-3dfa553.tar.gz is in /share/apps/src/nvidia_ganglia/
# Must be root (or at least have sudo privilege) to run this script and run this only on front end
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
 
  #
  # Apply web patch for Ganglia to display custom graphs
  cd /tmp/
  tar -zxvpf /share/apps/src/nvidia_ganglia/ganglia-gmond_python_modules-3dfa553.tar.gz
  cd ganglia-gmond_python_modules-3dfa553/gpu/nvidia/
  cp graph.d/*.php /var/www/html/ganglia/graph.d/
 
  cd /var/www/html/ganglia/
  patch -p0 < /tmp/ganglia-gmond_python_modules-3dfa553/gpu/nvidia/ganglia_web.patch
 
  #
  # Restart necessary services
  /etc/init.d/gmetad restart
  /etc/init.d/gmond restart
 
fi
# End root-check IF



Upon pointing the browser to the http://FQDN/ganglia/ (e.g., http://paracuda.math.mtu.edu/ganglia/ – the link will probably die or be changed to something else in due course), the display should include information about GPU as well, as shown in screenshots below:


ganglia_gpu_51 ganglia_gpu_52 ganglia_gpu_53 ganglia_gpu_54 ganglia_gpu_55 ganglia_gpu_56 ganglia_gpu_57





With little more work, the rather unaesthetic looking Ganglia web interface can be made to look like a given institution’s theme:

ganglia_gpu_58 ganglia_gpu_59 ganglia_gpu_60 ganglia_gpu_61 ganglia_gpu_62 ganglia_gpu_63 ganglia_gpu_64

ganglia_gpu_65 ganglia_gpu_66 ganglia_gpu_67 ganglia_gpu_68




Thanks be to

Dr. Allan Struthers for letting his paracuda.math be used for this purpose; my friendly neighbors for their kindness in letting me borrow a NVIDIA Quadro 6000 card; Robert Alexander of NVIDIA (http://developer.nvidia.com/ganglia-monitoring-system/), Bernard Li of Lawrence Berkeley National Laboratory and Jeremy Enos of National Center for Supercomputing Applications for developing this gmond Python module as well as making time to answer my questions.



Near Future Work

Work is currently underway to include all of the compute node related steps in the above described procedure in the local Rocks distribution, so that the compute nodes get them as soon as they are installed.

Share

]]> http://sgowtham.net/blog/2012/02/17/ganglia-gmond-python-module-for-monitoring-nvidia-gpu-on-rocks-5-4-2/feed/ 0 Ganglia – gmond Python Module For Monitoring NVIDIA GPU On RHEL 6.2 http://sgowtham.net/blog/2012/02/11/ganglia-gmond-python-module-for-monitoring-nvidia-gpu-on-rhel-6-2/ http://sgowtham.net/blog/2012/02/11/ganglia-gmond-python-module-for-monitoring-nvidia-gpu-on-rhel-6-2/#comments Sat, 11 Feb 2012 22:27:29 +0000 Gowtham http://sgowtham.net/blog/?p=3154 Disclaimer

The instructions/steps given below worked for me (and Michigan Technological University) running site licensed Red Hat Enterprise Linux 6.2 – as has been a common practice for several years now, a full version of Operating System was installed and all necessary patches/upgrades have been applied. These instructions may very well work for you (or your institution), on Red Hat-like or other linux distributions. Please note that if you decide to use these instructions on your machine, you are doing so entirely at your very own discretion and that neither this site, sgowtham.net, nor its author (or Michigan Technological University) is responsible for any/all damage – intellectual and/or otherwise.


A Bit About Ganglia (gmond & gmetad)

Citing Ganglia website,

Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRD tool for data storage and visualization. It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency. The implementation is robust, has been ported to an extensive set of operating systems and processor architectures, and is currently in use on thousands of clusters around the world. It has been used to link clusters across university campuses and around the world and can scale to handle clusters with 2000 nodes.

Further, citing Wikipedia,

gmond (Ganglia Monitoring Daemon) is a multi-threaded daemon which runs on each cluster node that needs to be monitored. Installation does not require having a common NFS file system or a database back-end, install special accounts or maintain configuration files. It has four main responsibilities: monitor changes in host state; announce relevant changes; listen to the state of all other ganglia nodes via a unicast or multicast channel; answer requests for an XML description of the cluster state.

Each gmond transmits in information in two different ways: unicasting or multicasting host state in external data representation (XDR) format using UDP messages OR sending XML over a TCP connection.

Federation in Ganglia is achieved using a tree of point-to-point connections amongst representative cluster nodes to aggregate the state of multiple clusters. At each node in the tree, a Ganglia Meta Daemon (gmetad) periodically polls a collection of child data sources, parses the collected XML, saves all numeric, volatile metrics to round-robin databases and exports the aggregated XML over a TCP sockets to clients. Data sources may be either gmond daemons, representing specific clusters, or other gmetad daemons, representing sets of clusters. Data sources use source IP addresses for access control and can be specified using multiple IP addresses for fail over. The latter capability is natural for aggregating data from clusters since each gmond daemon contains the entire state of its cluster.

The Ganglia web front-end provides a view of the gathered information via real-time dynamic web pages. Most importantly, it displays Ganglia data in a meaningful way for system administrators and computer users. Although the web front-end to ganglia started as a simple HTML view of the XML tree, it has evolved into a system that keeps a colourful history of all collected data. The Ganglia web front-end caters to system administrators and users (for e.g., one can view the CPU utilization over the past hour, day, week, month, or year). The web front-end shows similar graphs for memory usage, disk usage, network statistics, number of running processes, and all other Ganglia metrics. The web front-end depends on the existence of the gmetad which provides it with data from several Ganglia sources.

Specifically, the web front-end will open the local port 8651 (by default) and expects to receive a Ganglia XML tree. The web pages themselves are highly dynamic; any change to the Ganglia data appears immediately on the site. This behaviour leads to a very responsive site, but requires that the full XML tree be parsed on every page access. Therefore, the Ganglia web front-end should run on a fairly powerful, dedicated machine if it presents a large amount of data. The Ganglia web front-end is written in the PHP scripting language, and uses graphs generated by gmetad to display history information.


Installation

In order to make sure that none of the required steps are missed when performing similar installation on other machines (or repeating on the same machine), a BASH script was written.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#! /bin/bash
#
# install_ganglia.sh
# BASH script to install Ganglia on RHEL 6.2
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  #
  # Enable EPEL repository
  # EPEL: Extra Packages for Enterprise Linux
  cd /tmp/
  wget http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm
  rpm -ivh epel-release-6-5.noarch.rpm
 
  #
  # Install Ganglia
  yum install ganglia ganglia-gmetad ganglia-gmond ganglia-web ganglia-gmond-python
 
  #
  # Make sure httpd, gmond and gmetad automatically start after each reboot
  chkconfig --level 345 httpd on
  chkconfig --level 345 gmond on
  chkconfig --level 345 gmetad on
 
fi 
# End root-check IF


Configuration

Assuming all went well so far, one can expect to have the following files: /etc/ganglia/gmetad.conf, /etc/ganglia/gmond.conf and
/etc/httpd/conf.d/ganglia.conf



Edit /etc/ganglia/gmetad.conf to have the following line:

1

data_source "dirac.dcs" localhost:8649



/etc/ganglia/gmond.conf will have the following edits:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

/*
 * The cluster attributes specified will be used as part of the <CLUSTER>
 * tag that will wrap all hosts collected by this instance.
 */
cluster {
  name = "dirac.dcs"
  owner = "Michigan Technological University"
  latlong = "N47.11 W88.57"
  url = "http://www.it.mtu.edu/"
}
 
/* Feel free to specify as many udp_send_channels as you like.
 * Gmond used to only support having a single channel 
*/
udp_send_channel {
  bind_hostname = yes  # Highly recommended, soon to be default.
                       # This option tells gmond to use a source address
                       # that resolves to the machine's hostname.  Without
                       # this, the metrics may appear to come from any
                       # interface and the DNS names associated with
                       # those IPs will be used to create the RRDs.
  mcast_join = 239.2.11.71
  host = dirac.dcs.it.mtu.edu
  port = 8649
  ttl = 1
}
 
/* You can specify as many udp_recv_channels as you like as well. */
udp_recv_channel {
  mcast_join = 239.2.11.71
  port = 8649
  bind = 239.2.11.71
}
 
/* You can specify as many tcp_accept_channels as you like to share
 * an xml description of the state of the cluster 
*/
tcp_accept_channel {
  port = 8649
  acl {
    default = "deny"
 
    access {
      ip = 127.0.0.1
      mask = 32
      action = "allow"
    }
  }
}



/etc/httpd/conf.d/ganglia.conf will have the following edits:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

  #
  # Ganglia monitoring system php web frontend
  #
 
  Alias /ganglia /usr/share/ganglia
 
  <Location /ganglia>
    Order deny,allow
    # Deny from all
    # Allow from 127.0.0.1
    # Allow from ::1
    # # Allow from .example.com
    Allow from all
  </Location>



The firewall needs to be modified so that it accepts UDP & TCP requests on port 8649. To that effect, /etc/sysconfig/iptables will have the following lines:

1
2
3
4

# Ganglia gmond/gmetad
-A INPUT -m udp -p udp --dport 8649 -j ACCEPT
-A INPUT -m tcp -p tcp --dport 8649 -j ACCEPT
#



(Re)Start the necessary services:

1
2
3
4

/etc/init.d/iptables restart
/etc/init.d/gmetad restart
/etc/init.d/gmond restart
/etc/init.d/httpd restart



After a few minutes of collecting the data and upon pointing the browser to the http://FQDN/ganglia/, the web page should display the relevant information.

If, instead of relevant information, the web page displays the following error message

There was an error collecting ganglia data (127.0.0.1:8652): fsockopen error: Permission denied

then, more often than not, it hints to a selinux related issue. Edit the file, /etc/sysconfig/selinux, to look like:

1
2
3
4
5
6
7
8
9
10

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted



Restart the machine and point the browser to the http://FQDN/ganglia/, the web page should display the relevant information.


Monitoring NVIDIA GPU

The aforementioned set up works fine and as expected but it doesn’t necessarily provide any information about GPU(s) that may be part of the hardware. For e.g., the test machine used in our case has two NVIDIA GeForce GTX 570 cards. With more and more scientific & engineering computations tending towards GPU based computing, it’d be useful to include their status/usage information in Ganglia’s web portal. To this effect, NVIDIA released gmond Python module for GPUs (made aware of it by one of Michigan Tech ITSS directors). The instructions given in the NVIDIA-linked pages do work as described and are included here for the sake of completeness.


Python Bindings for the NVIDIA Management Library

This provides Python access to static information and monitoring data for NVIDIA GPUs, as well as management capabilities. It exposes the functionality of the NVML and one may download these from here – as before, the necessary steps are included in a BASH script.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

#! /bin/bash
#
# install_python_nvml_bindings.sh
# BASH script to download and install Python Bindings for the NVML
# Must be root (or at least have sudo privilege) to run this script
# Does not work with Python 2.4 - needs higher/more recent version
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  #
  # Download and install
  cd /tmp/
  wget http://pypi.python.org/packages/source/n/nvidia-ml-py/nvidia-ml-py-2.285.01.tar.gz
 
  tar -zxvpf nvidia-ml-py-2.285.01.tar.gz
  cd nvidia-ml-py-2.285.01
  python setup.py install
 
fi
# End root-check IF


gmond Python Module For Monitoring NVIDIA GPUs using NVML

After downloading ganglia-gmond_python_modules-3dfa553.tar.gz from GitHub for ganglia / gmond_python_modules to /tmp/, the following steps need to be performed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

#! /bin/bash
#
# copy_ganglia_gmond_python.sh
# BASH script to copy Copy nvidia_smi.py & pynvml.py to 
# /usr/lib64/ganglia/python_modules/, relevant files from 
# ganglia-gmond_python_modules to Ganglia, apply patch
# for Ganglia web interface and restart necessary services
# Must be root (or at least have sudo privilege) to run this script
 
# Begin root-check IF
if [ $UID != 0 ];
then
  clear
  echo
  echo "  You must be logged in as root!"
  echo "  Exiting..."
  echo
  exit
else
  #
  # Copy nvidia_smi.py & pynvml.py to /usr/lib64/ganglia/python_modules/
  cp /tmp/nvidia-ml-py-2.285.01/nvidia_smi.py /usr/lib64/ganglia/python_modules/
  cp /tmp/nvidia-ml-py-2.285.01/pynvml.py /usr/lib64/ganglia/python_modules/
 
  #
  # Copy relevant files to Ganglia
  cd /tmp/
  tar -zxvpf ganglia-gmond_python_modules-3dfa553.tar.gz
  cd ganglia-gmond_python_modules-3dfa553
 
  cd gpu/nvidia/
  cp python_modules/nvidia.py /usr/lib64/ganglia/python_modules/
  cp conf.d/nvidia.pyconf /etc/ganglia/conf.d/
  cp graph.d/*.php /usr/share/ganglia/graph.d/
 
  #
  # Apply web patch for Ganglia to display custom graphs
  cd /usr/share/ganglia/
  patch -p0 < /tmp/ganglia-gmond_python_modules-3dfa553/gpu/nvidia/ganglia_web.patch
 
  #
  # Restart necessary services
  /etc/init.d/gmetad restart
  /etc/init.d/gmond restart
 
fi
# End root-check IF



Upon pointing the browser to the http://FQDN/ganglia/ (e.g., http://dirac.dcs.it.mtu.edu/ganglia/ – the link will probably die or be changed to something else in due course), the display should include information about GPU as well, as shown in screenshots below:


ganglia_gpu_00 ganglia_gpu_01 ganglia_gpu_02 ganglia_gpu_03 ganglia_gpu_04 ganglia_gpu_05 ganglia_gpu_06 ganglia_gpu_07



With little more work, the rather unaesthetic looking Ganglia web interface can be made to look like a given institution’s theme:

ganglia_gpu_08 ganglia_gpu_09 ganglia_gpu_10 ganglia_gpu_11 ganglia_gpu_12 ganglia_gpu_13 ganglia_gpu_14 ganglia_gpu_15

ganglia_gpu_16 ganglia_gpu_17 ganglia_gpu_18 ganglia_gpu_19 ganglia_gpu_20



Near Future Work

Work is currently underway, most certainly with help from the ever-awesome Rocks mailing list, to integrate the above into a NPACI Rocks 5.4.2 cluster with compute nodes having one or more GPUs. Another post will come along as and when this work has been completed and tested.

Share

]]> http://sgowtham.net/blog/2012/02/11/ganglia-gmond-python-module-for-monitoring-nvidia-gpu-on-rhel-6-2/feed/ 0 BASH – Wrappers For qstat In NPACI ROCKS 5.2.2 http://sgowtham.net/blog/2009/12/10/bash-wrappers-for-qstat-in-npaci-rocks-5-2-2/ http://sgowtham.net/blog/2009/12/10/bash-wrappers-for-qstat-in-npaci-rocks-5-2-2/#comments Thu, 10 Dec 2009 20:51:33 +0000 Gowtham http://sgowtham.net/blog/?p=2536 What is ROCKS?

Rocks Cluster Distribution (originally called NPACI Rocks) is a Linux distribution intended for high-performance computing clusters. It was started by NPACI and the SDSC in 2000, and was initially funded in part by an NSF grant (2000-2007) but is currently funded by the followup NSF grant. Rocks was initially based on the Red Hat Linux distribution, however modern versions of Rocks are now based on CentOS, with a modified Anaconda installer that simplifies mass installation onto many computers. Rocks includes many tools (such as MPI) which are not part of CentOS but are integral components that make a group of computers into a cluster. Installations can be customized with additional software packages at install-time by using special user-supplied CDs (called Roll CDs). The Rolls extend the system by integrating seamlessly and automatically into the management and packaging mechanisms used by base software, greatly simplifying installation and configuration of large numbers of computers. Over a dozen Rolls have been created, including the SGE roll, the Condor roll, the Lustre roll, the Java roll, and the ganglia roll.


Why wrappers?

Functionality of qstat [part of the SGE Roll] changed in ROCKS 5.2.2 [it might have changed earlier than that, but that's when I first noticed]. To keep cluster users, supervisors and systems-administrators happy, I ended up writing these two wrappers:


General Wrapper for qstat

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#! /bin/bash
 
# A wrapper for 'qstat' command to show 
# the total number processors used/in use.
# ROCKS 5.2.2
 
# Thu, 10 Dec 2009 12:27:59 -0500
 
export PROC_TOTAL="142"
 
export USERS_LIST="
  user0
  user1
  user2
  user3
  user4"
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat -u $USERS_LIST | \
                  awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
                  tail -1 | awk '{print $NF}'`
 
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
echo '-----------------------------------------------------------------------------------------------------------------' 
/opt/gridengine/bin/lx26-amd64/qstat -u $USERS_LIST
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "   $PROC_TOTAL total processors :: $PROC_INUSE is/are in use :: $PROC_AVAIL is/are available"
echo '-----------------------------------------------------------------------------------------------------------------'
echo


User Specific Wrapper for qstat

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

#! /bin/bash
 
# A wrapper for 'qstat' command to show only the jobs
# submitted by a given user but show the total number
# processors used/in use.
# ROCKS 5.2.2
#
# Usage:
# qu [USERNAME]
# In the absence of any jobs submitted by USERNAME, or if
# USERNAME is not specified at all, it just displays the jobs, 
# if any, by the currently logged in user.
#
# Thu, 10 Dec 2009 12:27:59 -0500
 
export EXPECTED_ARGS="1"
export PROC_TOTAL="142"
 
export USERS_LIST="
  user0
  user1
  user2
  user3
  user4"
 
if [ $# -ne $EXPECTED_ARGS ];
then 
  export MYUSER="$USER"
else
  export MYUSER="$1"
fi
 
export PROC_USER=`/opt/gridengine/bin/lx26-amd64/qstat -u $MYUSER | \
                 awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
                 tail -1 | awk '{print $NF}'`
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat -u $USERS_LIST | \
                  awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
                  tail -1 | awk '{print $NF}'`
 
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
if [[ $PROC_USER -gt 0 ]] ;
then
  echo '-----------------------------------------------------------------------------------------------------------------' 
  /opt/gridengine/bin/lx26-amd64/qstat -u $MYUSER
  echo '-----------------------------------------------------------------------------------------------------------------' 
  echo " $PROC_TOTAL total processors :: $PROC_INUSE is/are in use :: $PROC_USER is/are use by $MYUSER :: $PROC_AVAIL is/are available "
  echo '-----------------------------------------------------------------------------------------------------------------'
  echo
else
  echo
  echo "  `hostname` @ `date -R`"
  echo
  echo "  $PROC_TOTAL total processors"
  echo "  $PROC_INUSE is/are in use "
  echo "  0 is/are use by $MYUSER"
  echo "  $PROC_AVAIL is/are available "
  echo
fi

Share

]]> http://sgowtham.net/blog/2009/12/10/bash-wrappers-for-qstat-in-npaci-rocks-5-2-2/feed/ 2 BASH – Wrappers For qstat In NPACI ROCKS http://sgowtham.net/blog/2008/10/13/bash-wrappers-for-qstat-in-npaci-rocks/ http://sgowtham.net/blog/2008/10/13/bash-wrappers-for-qstat-in-npaci-rocks/#comments Mon, 13 Oct 2008 22:34:07 +0000 Gowtham http://sgowtham.net/blog/?p=975 BASH is a free software UNIX shell written for the GNU Project. Its name is an acronym which stands for Bourne-again shell. The name is a pun on the name of the Bourne shell (sh), an early and important UNIX shell written by Stephen Bourne and distributed with Version 7 UNIX circa 1978, and the concept of being born again. BASH was created in 1987 by Brian Fox. In 1990 Chet Ramey became the primary maintainer. BASH is the default shell on most GNU/Linux systems as well as on Mac OS X and it can be run on most UNIX-like operating systems. It has also been ported to Microsoft Windows using the POSIX emulation provided by Cygwin, to MS-DOS by the DJGPP project and to Novell NetWare.

AWK is a general purpose programming language that is designed for processing text-based data, either in files or data streams, and was created at Bell Labs in the 1970s. The name AWK is derived from the family names of its authors — Alfred Aho, Peter Weinberger, and Brian Kernighan; however, it is not commonly pronounced as a string of separate letters but rather to sound the same as the name of the bird, auk. awk, when written in all lowercase letters, refers to the UNIX or Plan 9 program that runs other programs written in the AWK programming language. AWK is an example of a programming language that extensively uses the string data type, associative arrays (that is, arrays indexed by key strings), and regular expressions. The power, terseness, and limitations of AWK programs and sed scripts inspired Larry Wall to write PERL. Because of their dense notation, all these languages are often used for writing one-liner programs. AWK is one of the early tools to appear in Version 7 UNIX and gained popularity as a way to add computational features to a UNIX pipeline. A version of the AWK language is a standard feature of nearly every modern UNIX-like operating system.


What is ROCKS?

Rocks Cluster Distribution (originally called NPACI Rocks) is a Linux distribution intended for high-performance computing clusters. It was started by NPACI and the SDSC in 2000, and was initially funded in part by an NSF grant (2000-2007) but is currently funded by the followup NSF grant. Rocks was initially based on the Red Hat Linux distribution, however modern versions of Rocks are now based on CentOS, with a modified Anaconda installer that simplifies mass installation onto many computers. Rocks includes many tools (such as MPI) which are not part of CentOS but are integral components that make a group of computers into a cluster. Installations can be customized with additional software packages at install-time by using special user-supplied CDs (called Roll CDs). The Rolls extend the system by integrating seamlessly and automatically into the management and packaging mechanisms used by base software, greatly simplifying installation and configuration of large numbers of computers. Over a dozen Rolls have been created, including the SGE roll, the Condor roll, the Lustre roll, the Java roll, and the ganglia roll.


General Wrapper for qstat

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#! /bin/bash
 
# Save this as /home/local/bin/qstat
# A wrapper for 'qstat' command to show all currently submitted jobs along
# with the total number processors used/in use
#
# Check the path to 'qstat'
#
# Thu, 02 Oct 2008 21:06:05 -0400
 
# Modify this variable to meet the cluster's requirements
export PROC_TOTAL="128"
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat | \
       awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
       tail -1 | \
       awk '{print $NF}'`
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
export JOBS_LIST=`/opt/gridengine/bin/lx26-amd64/qstat`
 
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "$JOBS_LIST"
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "   $PROC_TOTAL total processors :: $PROC_INUSE are in use :: $PROC_AVAIL are available"
echo '-----------------------------------------------------------------------------------------------------------------'
echo



qstat General Wrapper


User Specific Wrapper for qstat

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

#! /bin/bash
 
# Save this as /home/local/bin/qu
# A wrapper for 'qstat' command to show all currently submitted jobs by a given 
# user ($USER) along with the total number processors used/in use
# If the $USER has no jobs, displays all currently submitted jobs.
#
# Check the path to 'qstat'
#
# Thu, 02 Oct 2008 21:35:04 -0400
 
 
# Modify this variable to meet the cluster's requirements
export PROC_TOTAL="128"
 
export PROC_INUSE=`/opt/gridengine/bin/lx26-amd64/qstat | \
       awk '{print} (NR>2) {used_procs+=\$9} END {print used_procs}' | \
       tail -1 | \
       awk '{print $NF}'`
export PROC_AVAIL=`expr $PROC_TOTAL - $PROC_INUSE`
 
export JOBS_LIST=`/opt/gridengine/bin/lx26-amd64/qstat -u $USER`
 
if [ "$JOBS_LIST" == "" ];
then
  export JOBS_LIST=`/opt/gridengine/bin/lx26-amd64/qstat`
fi
 
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "$JOBS_LIST"
echo '-----------------------------------------------------------------------------------------------------------------' 
echo "   $PROC_TOTAL total processors :: $PROC_INUSE are in use :: $PROC_AVAIL are available"
echo '-----------------------------------------------------------------------------------------------------------------'
echo



qstat User Specific Wrapper

Share

]]> http://sgowtham.net/blog/2008/10/13/bash-wrappers-for-qstat-in-npaci-rocks/feed/ 0 VASP, MPICH, Intel Compilers And NPACI Rocks http://sgowtham.net/blog/2005/12/22/vasp-mpich-intel-compilers-and-npaci-rocks/ http://sgowtham.net/blog/2005/12/22/vasp-mpich-intel-compilers-and-npaci-rocks/#comments Thu, 22 Dec 2005 18:44:41 +0000 Gowtham http://sgowtham.net/blog/?p=6 Much of the weekend was spent in some relaxation, shoveling snow (I have come to realize that this is a very good exercise – it can make one sweat even when it’s 10 F outside!) and trying to debug the errors associated with execution of parallel version of VASP 4.6.28. However, same errors persisted and I didn’t go too far ahead with it.

I got in touch with few of Intel’s authorized resellers (one in Bloomington, MN – who transferred the call to someone else in Boston, MA) to see if I can get a copy of version 8.x of FORTRAN and C compilers. Though the latter person promised to get back to me with some favorable information soon, I got directly in touch with an Intel Support Technician – hoping that he/she wouldn’t be from a call center in some other country. Fortunately, it was somebody in California and unfortunately, he redirected me back to their re-sellers. Last call, pretty much out of desperation, did what I wanted : I just had to register the non-commercially downloaded products for Premier Support and I would be entitled for previous versions too. If only Intel explicitly mentioned what Premier Support actually is, my worries would have ended a long time ago. By the way, if you are now wondering what the error message was (running on 2 processors), here it is:


1
2
3
4
5
6
7
8
9
10
11
12
13

running on    2 nodes
distr:  one band on    1 nodes,    2 groups
vasp.4.6.28 25Jul05 complex
POSCAR found :  4 types and   18 ions
LDA part: xc-table for Ceperly-Alder, standard interpolation
found WAVECAR, reading the header
POSCAR, INCAR and KPOINTS ok, starting setup
WARNING: wrap around errors must be expected
FFT: planning ...           1
reading WAVECAR
the WAVECAR file was read sucessfully
LAPACK: Routine ZPOTRF failed!           8
LAPACK: Routine ZPOTRF failed!           8

Having managed to get version 8.x and 7.x of Intel compilers, situation only got worse as the error message remained the same ( At this point, I must thank the help offered by VASP Tech Support and Andri Arnaldsson (from University of Washington) – they have been pretty quick in their responses, sent their copies of Makefiles along with several tips and tricks. Changing compiler versions, using a previous version of MPICH (1.2.7p1 to be precise), repeating compilation many times with different BLAS, LAPACK libraries … nothing helped.

Taking a break for an hour and watching an episode of South Park seemed to have helped. A modification in the key words used for Google! search and reading some discussion forum lot more carefully, I found that adding three lines at the end of VASP Makefile (what this does is to reduce the level of optimization for mpi.F), the error vanished and the calculations started running smoothly. I repeated the same calculation using 2, 4, 6 and 8 processors and noticed a slightly strange behavior – when the number of processors was 2, 4 or 8, energy optimization is exactly same as in a serial calculation but when the number of processors is 2*N (N=3 in this case), energy optimization route is different – final result is still exactly the same. Though I have to do more trials (say 3, 5, 7, 9, 10 processors) to completely convince myself, it appears to me that using 2N processors does the trick. Like Dave Kraus mentioned once before – knowing what trick works is certainly important, but knowing why that trick works is even more important.

Share

]]> http://sgowtham.net/blog/2005/12/22/vasp-mpich-intel-compilers-and-npaci-rocks/feed/ 1