/
VASP

VASP

1 Overview | 1.1 Using | 1.2 Multicore | 2 Installation Guidelines | 2.1 MedicineBow: Version 6.3.2 | 2.1.1 Suggested Module File Creation | 2.1.2 Installation: Potential Issues | 2.1.2.1 Bin Folder Executables: Permission Denied | 2.2 MedicineBow: Version 6.3.2: GPU

Overview

The Vienna Ab initio Simulation Package (VASP), is a package for performing ab initio quantum mechanical calculations using either Vanderbilt pseudopotentials, or the projector augmented wave method and a plane wave basis set.

  • VASP Homepage: Access to VASP News, Wiki, Support Forum and Portal

    • VASP Wiki: Menu

    • Please consult the VASP FAQ for more details regards licensing.

      • For Example: To whom are VASP licenses given?

        • VASP licenses are given to well-defined research groups only. They are not department-or institution-wide, also they are not personal licenses. All members of a VASP-group have to work in the same organizational unit (Department, Institute) at the same location.

        • VASP-licenses are not transferable from one research institution to another one. Any transfer of a license to another group at the same research institution has to be approved by the VASP Software GmbH explicitly.

Using

  • NOTE: VASP is NOT publicly available to users across the clusters. Instead, there are local installations per project with each project having its own license agreement. If ARCC comes across users abusing the licensing agreement we will have to act on it.

Multicore

Note: Although ARCC can assist in the functional setup and running of VASP on the cluster, we do leave the science up to the researchers.

With this in mind, there are a number of pages on the VASP Wiki that make suggestions on how best to configure simulation inputs with respect to nodes/cores, as well as memory considerations.

Installation Guidelines

The basic process is to unpack the provided source tar file. Copy and update an appropriate arch/makefile.include.<version> file, and then build.

Below is an example using a <version> that ARCC believes is suitable. By all means select and try a different <version> but you will need to select appropriate compilers and related libraries, as well as updating the makefile.include.

With the example below, you will need to update <project-name> appropriately.

MedicineBow: Version 6.3.2

# We suggest building this software underneath your related project, so it can be shared across that project, # specifically under /project/<project-name>/software/. []$ tar -xf vasp.6.3.2.tar []$ cd vasp.6.3.2/ []$ cp arch/makefile.include.gnu_omp . []$ mv makefile.include.gnu_omp makefile.include # Update the makefile.include file. []$ module load gcc/14.2.0 openmpi/5.0.5 fftw/3.3.10-ompi openblas/0.3.24 netlib-scalapack/2.2.0-ompi wannier90/3.1.0 hdf5/1.14.3__hl_True__fortran_True-ompi []$ make all []$ ls bin/ vasp_gam vasp_ncl vasp_std

You need to update the makefile.include file. The following modifications have been made:

  • Commented out native compiling.

  • Have explicitly added the paths to openblas, scalapack and fftw.

  • Building with, and thus adding paths to, hdf5 and wannier90.

    • The wannier90 library typically installs the actual library file in its parent folder, and not a sub lib/ folder. This modifies the makefile.include to: LLIBS += -L$(WANNIER90_ROOT)/ -lwannier i.e the lib folder is not included.

  • Compiling fftlib.

# Default precompiler options CPP_OPTIONS = -DHOST=\"LinuxGNU\" \ -DMPI -DMPI_BLOCK=8000 -Duse_collective \ -DscaLAPACK \ -DCACHE_SIZE=4000 \ -Davoidalloc \ -Dvasp6 \ -Duse_bse_te \ -Dtbdyn \ -Dfock_dblbuf \ -D_OPENMP CPP = gcc -E -C -w $*$(FUFFIX) >$*$(SUFFIX) $(CPP_OPTIONS) FC = mpif90 -fopenmp FCL = mpif90 -fopenmp FREE = -ffree-form -ffree-line-length-none FFLAGS = -w -ffpe-summary=none OFLAG = -O2 OFLAG_IN = $(OFLAG) DEBUG = -O0 OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o OBJECTS_O2 += fft3dlib.o # For what used to be vasp.5.lib CPP_LIB = $(CPP) FC_LIB = $(FC) CC_LIB = gcc CFLAGS_LIB = -O FFLAGS_LIB = -O1 FREE_LIB = $(FREE) OBJECTS_LIB = linpack_double.o # For the parser library CXX_PARS = g++ LLIBS = -lstdc++ ## ## Customize as of this point! Of course you may change the preceding ## part of this file as well if you like, but it should rarely be ## necessary ... ## # When compiling on the target machine itself, change this to the # relevant target when cross-compiling for another architecture # VASP_TARGET_CPU ?= -march=native # FFLAGS += $(VASP_TARGET_CPU) # For gcc-10 and higher (comment out for older versions) FFLAGS += -fallow-argument-mismatch # BLAS and LAPACK (mandatory) OPENBLAS_ROOT ?= /apps/u/spack/gcc/14.2.0/openblas/0.3.24-52ip5la/ BLASPACK = -L$(OPENBLAS_ROOT)/lib -lopenblas # scaLAPACK (mandatory) SCALAPACK_ROOT ?= /apps/u/spack/gcc/14.2.0/netlib-scalapack/2.2.0-dbchdg6/ SCALAPACK = -L$(SCALAPACK_ROOT)/lib -lscalapack LLIBS += $(SCALAPACK) $(BLASPACK) # FFTW (mandatory) FFTW_ROOT ?= /apps/u/spack/gcc/14.2.0/fftw/3.3.10-mk3sml6/ LLIBS += -L$(FFTW_ROOT)/lib -lfftw3 -lfftw3_omp INCS += -I$(FFTW_ROOT)/include # HDF5-support (optional but strongly recommended) CPP_OPTIONS+= -DVASP_HDF5 HDF5_ROOT ?= /apps/u/spack/gcc/14.2.0/hdf5/1.14.3-2frri7c/ LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran INCS += -I$(HDF5_ROOT)/include # For the VASP-2-Wannier90 interface (optional) CPP_OPTIONS += -DVASP2WANNIER90 WANNIER90_ROOT ?= /apps/u/opt/gcc/14.2.0/wannier90/3.1.0/ LLIBS += -L$(WANNIER90_ROOT)/ -lwannier # For the fftlib library (recommended) CPP_OPTIONS+= -Dsysv FCL += fftlib.o CXX_FFTLIB = g++ -fopenmp -std=c++11 -DFFTLIB_THREADSAFE INCS_FFTLIB = -I./include -I$(FFTW_ROOT)/include LIBS += fftlib LLIBS += -ldl

Note: If you wish to use a different make.include.your_choice or try another process, then you’ll need to module load potentially a different compiler, and build core libraries if not built for that chosen compiler.

Suggested Module File Creation

If you have installed under say: /project/<project-name>/software/vasp.6.3.2/ you could create the following:

[]$ cd /project/<project-name>/software/ []$ mkdir modules []$ cd modules []$ mkdir vasp []$ cd vasp []$ vim 6.3.2.lua
whatis([[Name : vasp]]) whatis([[Version : 6.3.2]]) whatis([[Short description : The Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles. ]]) help([[ vasp : 6.3.2 The Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles. https://vasp.at/ ]]) if not isloaded("gcc/14.2.0") then load("gcc/14.2.0") end if not isloaded("openmpi/5.0.5") then load("openmpi/5.0.5") end if not isloaded("fftw/3.3.10-ompi") then load("fftw/3.3.10-ompi") end if not isloaded("openblas/0.3.24") then load("openblas/0.3.24") end if not isloaded("netlib-scalapack/2.2.0-ompi") then load("netlib-scalapack/2.2.0-ompi") end if not isloaded("wannier90/3.1.0") then load("wannier90/3.1.0") end if not isloaded("hdf5/1.14.3__hl_True__fortran_True-ompi") then load("hdf5/1.14.3__hl_True__fortran_True-ompi") end prepend_path("PATH", "/project/<project-name>/software/vasp.6.3.2/bin")
[]$ module purge []$ module use /project/<project-name>/software/modules []$ module avail -------------------- /project/<project-name>/software/modules --------------------- vasp/6.3.2 ... []$ module load vasp/6.3.2

Installation: Potential Issues

Bin Folder Executables: Permission Denied

If you get an error of the form:

... /project/<project-name>/software/vasp.6.3.2/bin/vasp_std: Permission denied

Once built, the files in the bin folder should be executable. Check they have the x permission defined for the groups:

[bin]$ ls -al -rwxrwxr-x 1 <username> <project-name> 19190336 Oct 7 14:32 vasp_gam -rwxrwxr-x 1 <username> <project-name> 19547024 Oct 7 14:39 vasp_ncl -rwxrwxr-x 1 <username> <project-name> 19481464 Oct 7 14:25 vasp_std

If you don’t then use chmod ugo+x <filename> to update.

 

MedicineBow: Version 6.3.2: GPU

Here is the process to build a version that utilizes GPUs.

This example uses the nvidia hpc-sdk to build, and subsequently should use its provided version of mpiexec (not srun).

[~]$ cd /project/<project>/software/ [software]$ mkdir vasp [software]$ cd vasp [vasp]$ mkdir vasp.6.3.2.gpu [vasp]$ cp vasp.6.3.2.tar vasp.6.3.2.gpu/ [vasp]$ cd vasp.6.3.2.gpu/ [vasp.6.3.2.gpu]$ cd vasp.6.3.2/ [vasp.6.3.2]$ mv * .. [vasp.6.3.2]$ cd .. [vasp.6.3.2.gpu]$ rmdir vasp.6.3.2/ [vasp.6.3.2.gpu]$ rm vasp.6.3.2.tar # Setup makefile.include # Based on arch/makefile.include.nvhpc_ompi_mkl_omp_acc [vasp.6.3.2.gpu]$ vim makefile.include # Default precompiler options CPP_OPTIONS = -DHOST=\"LinuxNV\" \ -DMPI -DMPI_BLOCK=8000 -Duse_collective \ -DscaLAPACK \ -DCACHE_SIZE=4000 \ -Davoidalloc \ -Dvasp6 \ -Duse_bse_te \ -Dtbdyn \ -Dqd_emulate \ -Dfock_dblbuf \ -D_OPENMP \ -D_OPENACC \ -DUSENCCL -DUSENCCLP2P CPP = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX) > $*$(SUFFIX) # N.B.: you might need to change the cuda-version here # to one that comes with your NVIDIA-HPC SDK # FC = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.0 -mp # FCL = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.0 -mp -c++libs FC = mpif90 -acc -gpu=cc70,cc80,cuda12.3 -mp FCL = mpif90 -acc -gpu=cc70,cc80,cuda12.3 -mp -c++libs FREE = -Mfree FFLAGS = -Mbackslash -Mlarge_arrays OFLAG = -fast DEBUG = -Mfree -O0 -traceback OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o LLIBS = -cudalib=cublas,cusolver,cufft,nccl -cuda # Redefine the standard list of O1 and O2 objects SOURCE_O1 := pade_fit.o SOURCE_O2 := pead.o # For what used to be vasp.5.lib CPP_LIB = $(CPP) FC_LIB = nvfortran CC_LIB = nvc -w CFLAGS_LIB = -O FFLAGS_LIB = -O1 -Mfixed FREE_LIB = $(FREE) OBJECTS_LIB = linpack_double.o # For the parser library CXX_PARS = nvc++ --no_warnings ## ## Customize as of this point! Of course you may change the preceding ## part of this file as well if you like, but it should rarely be ## necessary ... ## # When compiling on the target machine itself , change this to the # relevant target when cross-compiling for another architecture # VASP_TARGET_CPU ?= -tp host # FFLAGS += $(VASP_TARGET_CPU) # Specify your NV HPC-SDK installation (mandatory) #... first try to set it automatically NVROOT =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }') # If the above fails, then NVROOT needs to be set manually #NVHPC ?= /opt/nvidia/hpc_sdk #NVVERSION = 21.11 #NVROOT = $(NVHPC)/Linux_x86_64/$(NVVERSION) ## Improves performance when using NV HPC-SDK >=21.11 and CUDA >11.2 # OFLAG_IN = -fast -Mwarperf # SOURCE_IN := nonlr.o # Software emulation of quadruple precsion (mandatory) QD ?= $(NVROOT)/compilers/extras/qd LLIBS += -L$(QD)/lib -lqdmod -lqd INCS += -I$(QD)/include/qd # Intel MKL for FFTW, BLAS, LAPACK, and scaLAPACK # MKLROOT ?= /path/to/your/mkl/installation MKLROOT ?= /apps/u/opt/compilers/oneapi/2024.2.1/mkl/2024.2 # LLIBS_MKL = -Mmkl -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 # MKLLIBS = -Mmkl MKLLIBS = -lmkl_intel_lp64 -lmkl_pgi_thread -lmkl_core -pgf90libs -mp -lpthread -lm -ldl LLIBS_MKL = -L$(MKLROOT)/lib -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 $(MKLLIBS) INCS += -I$(MKLROOT)/include/fftw # Use a separate scaLAPACK installation (optional but recommended in combination with OpenMPI) # Comment out the two lines below if you want to use scaLAPACK from MKL instead # SCALAPACK_ROOT ?= /path/to/your/scalapack/installation # LLIBS_MKL = -L$(SCALAPACK_ROOT)/lib -lscalapack -Mmkl LLIBS += $(LLIBS_MKL) # HDF5-support (optional but strongly recommended) #CPP_OPTIONS+= -DVASP_HDF5 #HDF5_ROOT ?= /path/to/your/hdf5/installation #LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran #INCS += -I$(HDF5_ROOT)/include # For the VASP-2-Wannier90 interface (optional) #CPP_OPTIONS += -DVASP2WANNIER90 #WANNIER90_ROOT ?= /path/to/your/wannier90/installation #LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier # For the fftlib library (hardly any benefit for the OpenACC GPU port, especially in combination with MKL's FFTs) #CPP_OPTIONS+= -Dsysv #FCL += fftlib.o #CXX_FFTLIB = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE #INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw #LIBS += fftlib #LLIBS += -ldl #- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - # Build VASP: [vasp.6.3.2.gpu]$ module load nvhpc-sdk/24.3 [vasp.6.3.2.gpu]$ module load nvhpc-openmpi3/24.3 [vasp.6.3.2.gpu]$ export MKLROOT=/apps/u/opt/compilers/oneapi/2024.2.1/mkl/2024.2 [vasp.6.3.2.gpu]$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MKLROOT/lib:/apps/u/spack/gcc/11.4.1/gcc/11.4.0-k3wm6wb/lib64/ [vasp.6.3.2.gpu]$ make all if [ ! -d build/std ] ; then mkdir -p build/std ; fi cp src/makefile src/.objects src/makedeps.awk makefile.include build/std make -C build/std VERSION=std check ... make[2]: Leaving directory '/cluster/medbow/project/design-lab/software/VASP/vasp.6.3.2.gpu/build/ncl' make[1]: Leaving directory '/cluster/medbow/project/design-lab/software/VASP/vasp.6.3.2.gpu/build/ncl' [vasp.6.3.2.gpu]$ ls bin/ vasp_gam vasp_ncl vasp_std # Change permissions to allow rest of project to use: [vasp.6.3.2.gpu]$ cd .. [vasp]$ chown -R <username>:<project> vasp.6.3.2.gpu

Example submission scripts: You will need to set <project> and various options such as time/memory/cpus-per-task as required.

#SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=8 #SBATCH --time=8:00:00 #SBATCH --account=<project> #SBATCH --partition=mb-a30 #SBATCH -G 1 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK module load nvhpc-sdk/24.3 module load nvhpc-openmpi3/24.3 export MKLROOT=/apps/u/opt/compilers/oneapi/2024.2.1/mkl/2024.2 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MKLROOT/lib:/apps/u/spack/gcc/11.4.1/gcc/11.4.0-k3wm6wb/lib64/ mpiexec -np 1 /project/<project>/software/VASP/vasp.6.3.2.gpu/bin/vasp_std
#SBATCH --nodes=2 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=8 #SBATCH --time=8:00:00 #SBATCH --account=<project> #SBATCH --partition=mb-a30 #SBATCH -G 2 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK module load nvhpc-sdk/24.3 module load nvhpc-openmpi3/24.3 export MKLROOT=/apps/u/opt/compilers/oneapi/2024.2.1/mkl/2024.2 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MKLROOT/lib:/apps/u/spack/gcc/11.4.1/gcc/11.4.0-k3wm6wb/lib64/ mpiexec -np 2 /project/<project>/software/VASP/vasp.6.3.2.gpu/bin/vasp_std