HPCC 1.4.1 Benchmark With GCC 4.1.2 On Rocks 5.4.2
February 28th, 2012 @ 13:27:14 | BASH, Clusters, HPCC, HPL, Linux, MPI, Rocks, Scripts, Technology
Disclaimer
The instructions/steps given below worked for me (and Michigan Technological University) running NPACI Rocks 5.4.2 (with CentOS 5.5) – as has been a common practice for several years now, a full version of Operating System was installed. These instructions may very well work for you (or your institution), on Rocks-like or other linux clusters. Please note that if you decide to use these instructions on your machine, you are doing so entirely at your very own discretion and that neither this site, sgowtham.net, nor its author (or Michigan Technological University) is responsible for any/all damage – intellectual and/or otherwise.
A Bit About HPCC
Citing HPCC website,
HPCC benchmark suite has been released by DARPA High Productivity Computing Systems (HPCS) program to help define the performance boundaries of future Petascale computing systems. HPCC is a suite of tests that examine the performance of HPC architectures using kernels with memory access patterns more challenging than those of the HPL benchmark used in Top500. This suite is designed to augment the list, providing benchmarks that bound the performance of many real applications as a function of memory access characteristics. For e.g., spatial and temporal locality, and providing a framework for including additional tests. HPCC consists of seven tests that attempt to span high and low spatial and temporal locality space.
By design, the HPCC tests are scalable with the size of data sets being a function of the largest HPL matrix tested for the system. Since HPCC kernels consist of simple mathematical operations, it provides a unique opportunity to look at language and parallel programming issues. To characterize the architecture of the system, the following three scenarios are considered:
- Local – only a single processor is performing the computations
- Embarrassingly Parallel – each processor in the entire system is performing the computations but without explicit communication with each other
- Global – all processors in the system are performing the computations with explicit communication with each other
The seven tests that make up HPCC benchmark are as follows:
- HPL – the Linpack TPP benchmark which measures the floating point rate of execution for solving a linear system of equations
- DGEMM – measures the floating point rate of execution of double precision real matrix-matrix multiplication
- STREAM – a simple synthetic benchmark program that measures sustainable memory bandwidth (in GB/s) and the corresponding computation rate for simple vector kernel
- PTRANS (parallel matrix transpose) – exercises the communications where pairs of processors communicate with each other simultaneously. It is a useful test of the total communications capacity of the network
- RandomAccess – measures the rate of integer random updates of memory (GUPS)
- FFT – measures the floating point rate of execution of double precision complex one-dimensional Discrete Fourier Transform (DFT)
- Communication bandwidth and latency – a set of tests to measure latency and bandwidth of a number of simultaneous communication patterns; based on beff (effective bandwidth benchmark)
Pre-requisites: MPI & Goto BLAS
Please refer to the corresponding sections in HPL 2.0 Benchmark With GCC 4.1.2 On Rocks 5.4.2 to learn more about required pre-requisites as well as instructions for installing/compiling them. Following Rocks recommendations, pre-requisites will be installed under /share/apps/; software installed by me, in clusters at Michigan Tech, have the following template for their folder structure:
/share/apps/Software/Software_Version/Compiler/Compiler_Version
HPCC Installation/Compilation
With MPICH2 and Goto BLAS2 in place, HPCC 1.4.1 will be installed under
/share/apps/hpcc/1.4.1/mpich2/1.4.1p1/gcc/4.1.2
The following script, used in installation, will assume that one has downloaded HPCC 1.4.1 from here & the necessary Make.MPICH2141p1_GCC412.HPCC (listed below; it does look identical to the one used when compiling HPL 2.0 – the only difference is in the definition of TOPdir variable) and placed them in /share/apps/tmp/
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 | # # Makefile (Make.MPICH2141p1_GCC412) used to compile HPCC # on HPC clusters running NPACI Rocks (5.4.2) with CentOS (5.5) # at Michigan Technological University. # # Disclaimer: # Please note that you are using these instructions # at your very own risk and that Michigan Technological # University is not responsible for any/all damage caused to # your property, intellectual or otherwise. # # For additional help and/or comments, questions, suggestions, # please contact # # Gowtham # Information Technology Services & Security # Michigan Technological University # g@mtu.edu # # * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * # # # -- High Performance Computing Linpack Benchmark (HPL) # HPL - 2.0 - September 10, 2008 # Antoine P. Petitet # University of Tennessee, Knoxville # Innovative Computing Laboratory # (C) Copyright 2000-2008 All Rights Reserved # # -- Copyright notice and Licensing terms: # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions, and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # 3. All advertising materials mentioning features or use of this # software must display the following acknowledgement: # This product includes software developed at the University of # Tennessee, Knoxville, Innovative Computing Laboratory. # # 4. The name of the University, the name of the Laboratory, or the # names of its contributors may not be used to endorse or promote # products derived from this software without specific written # permission. # # -- Disclaimer: # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS # ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY # OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # # Shell Details SHELL = /bin/sh CD = cd CP = cp LN_S = ln -s MKDIR = mkdir RM = /bin/rm -f TOUCH = touch # # Platform Identifier ARCH = MPICH2141p1_GCC412 # # HPL Directory Structure / HPL library TOPdir = ../../.. INCdir = $(TOPdir)/include BINdir = $(TOPdir)/bin/$(ARCH) LIBdir = $(TOPdir)/lib/$(ARCH) HPLlib = $(LIBdir)/libhpl.a # # Message Passing library (MPI) MPdir = /share/apps/mpich2/1.4.1p1/gcc/4.1.2 MPinc = -I$(MPdir)/include MPlib = $(MPdir)/lib/libmpich.a # # Linear Algebra library (BLAS) LAlib = /share/apps/gotoblas2/1.13/gcc/4.1.2/libgoto2.a # # F77 / C Interface # You can skip this section if and only if you are not planning to use # a BLAS library featuring a Fortran 77 interface. Otherwise, it is # necessary to fill out the F2CDEFS variable with the appropriate # options. **One and only one** option should be chosen in **each** of # the 3 following categories: # # 1) name space (How C calls a Fortran 77 routine) # # -DAdd_ : all lower case and a suffixed underscore (Suns, # Intel, ...) [default] # -DNoChange : all lower case (IBM RS6000) # -DUpCase : all upper case (Cray) # -DAdd__ : the FORTRAN compiler in use is f2c # # 2) C and Fortran 77 integer mapping # # -DF77_INTEGER=int : Fortran 77 INTEGER is a C int [default] # -DF77_INTEGER=long : Fortran 77 INTEGER is a C long # -DF77_INTEGER=short : Fortran 77 INTEGER is a C short # # 3) Fortran 77 string handling # # -DStringSunStyle : The string address is passed at the string location # on the stack, and the string length is then # passed as an F77_INTEGER after all explicit # stack arguments [default] # -DStringStructPtr : The address of a structure is passed by a # Fortran 77 string, and the structure is of the # form: struct {char *cp; F77_INTEGER len;} # -DStringStructVal : A structure is passed by value for each Fortran # 77 string, and the structure is of the form: # struct {char *cp; F77_INTEGER len;} # -DStringCrayStyle : Special option for Cray machines, which uses # Cray fcd (fortran character descriptor) for # interoperation F2CDEFS = -DAdd_ # # HPL Includes / Libraries / Specifics HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc) HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib) # # HPL Compile Time Options # -DHPL_COPY_L force the copy of the panel L before bcast # -DHPL_CALL_CBLAS call the cblas interface # -DHPL_CALL_VSIPL call the vsip library # -DHPL_DETAILED_TIMING enable detailed timers # # By default HPL will: # *) not copy L before broadcast # *) call the BLAS Fortran 77 interface # *) not display detailed timing information HPL_OPTS = -DHPL_COPY_L -DHPL_CALL_CBLAS -DHPL_DETAILED_TIMING # # HPL Definitions HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES) # # Compilers / linkers - Optimization Flags CC = mpicc CCNOOPT = $(HPL_DEFS) CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall # # On some platforms, it is necessary to use the Fortran linker # to find the Fortran internals used in the BLAS library LINKER = mpif77 LINKFLAGS = $(CCFLAGS) ARCHIVER = ar ARFLAGS = r RANLIB = echo # |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | #! /bin/bash # # install_hpcc.sh # BASH script to install HPCC 1.4.1 (compiled against MPICH2 1.4.1p1) on a # Rocks 5.4.2 cluster's front end # Must be root (or at least have sudo privilege) to run this script # Begin root-check IF if [ $UID != 0 ]; then clear echo echo " You must be logged in as root!" echo " Exiting..." echo exit else # Set necessary variables export HPCC_VERSION="1.4.1" export MPICH2_VERSION="1.4.1p1" export GCC_VERSION="4.1.2" export HPCC_INSTALL="/share/apps/hpcc/${HPCC_VERSION}/mpich2/${MPICH2_VERSION}/gcc/${GCC_VERSION}" export MAKEFILE_ARCH="MPICH2141p1_GCC412" # Begin HPCC-${HPCC_VERSION}.tar.gz check IF if [ -e "/share/apps/tmp/hpcc-${HPCC_VERSION}.tar.gz" ] then mkdir /share/apps/hpcc/${HPCC_VERSION}/mpich2/${MPICH2_VERSION}/gcc/ cd /share/apps/hpcc/${HPCC_VERSION}/mpich2/${MPICH2_VERSION}/gcc/ tar -zxvf /share/apps/tmp/hpcc-${HPCC_VERSION}.tar.gz mv hpcc-${HPCC_VERSION} ${GCC_VERSION} echo echo " Step #0: copy the makefile, make clean and make" cd ${HPCC_INSTALL} cp /share/apps/tmp/Make.${MAKEFILE_ARCH}.HPCC ./hpl/Make.${MAKEFILE_ARCH} make clean arch=${MAKEFILE_ARCH} make arch=${MAKEFILE_ARCH} echo echo " Step #1: Update $HOME/.bashrc" cat <<EOF Add the following lines to $HOME/.bashrc and remember to source it # HPCC (${HPCC_VERSION}) settings export HPCC="${HPCC_INSTALL}" export PATH="\${PATH}:\${HPCC}/bin/${MAKEFILE_ARCH}" export LD_LIBRARY_PATH="\${LD_LIBRARY_PATH}:\${HPCC}/${MAKEFILE_ARCH}" EOF fi # End hpcc-${HPCC_VERSION}.tar.gz check IF echo fi # End root-check IF |
Running HPCC Benchmark
Please refer to the corresponding sections in HPL 2.0 Benchmark With GCC 4.1.2 On Rocks 5.4.2 to learn how to get a fix on N, NB, etc., given the hardware configuration of compute nodes. Running HPCC benchmark is very similar to running HPL 2.0 benchmark.
