Sgstrohkorb (talk | contribs) (This is an issue that I ran into and I want to post it on the wiki so others can learn from my mistake. If there's a better way to fix this issue, please let me know.) |
(Converting to modules and slurm/centos) |
||
Line 2: | Line 2: | ||
For a complete list of all installed software, see [[NodePackageList]] | For a complete list of all installed software, see [[NodePackageList]] | ||
== Toolchains == | |||
A toolchain is a set of compilers, libraries and applications that are needed to build software. Some software functions better when using specific toolchains. | |||
We provide a good number of toolchains and versions of toolchains make sure your applications will compile and/or run correctly. | |||
These toolchains include (you can run 'module keyword keychain compiler'): | |||
; GCC: The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...). | |||
; GCCcore: The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...). | |||
; foss: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK. | |||
; gcccuda: GNU Compiler Collection (GCC) based compiler toolchain, along with CUDA toolkit. | |||
; gmvapich2: GNU Compiler Collection (GCC) based compiler toolchain, including MVAPICH2 for MPI support. | |||
; gompi: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support. | |||
; gompic: GNU Compiler Collection (GCC) based compiler toolchain along with CUDA toolkit, including OpenMPI for MPI support with CUDA features enabled. | |||
; goolfc: GCC based compiler toolchain __with CUDA support__, and including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK. | |||
; icc: C and C++ compiler from Intel | |||
; iccifort: Intel Cluster Toolkit Compiler Edition provides Intel C,C++ and fortran compilers, Intel MPI and Intel MKL | |||
; ifort: Fortran compiler from Intel | |||
; iomkl: Intel Cluster Toolchain Compiler Edition provides Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI. | |||
; iompi: Intel C/C++ and Fortran compilers, alongside Open MPI. | |||
You can run 'module spider $toolchain' to see the versions we have: | |||
$ module spider iomkl | |||
* iomkl/2017a | |||
* iomkl/2017b | |||
* iomkl/2017beocatb | |||
If you load one of those (module load iomkl/2017b), you can see the other modules and versions of software that it loaded with the 'module list': | |||
$ module list | |||
Currently Loaded Modules: | |||
1) icc/2017.4.196-GCC-6.4.0-2.28 | |||
2) binutils/2.28-GCCcore-6.4.0 | |||
3) ifort/2017.4.196-GCC-6.4.0-2.28 | |||
4) iccifort/2017.4.196-GCC-6.4.0-2.28 | |||
5) GCCcore/6.4.0 | |||
6) numactl/2.0.11-GCCcore-6.4.0 | |||
7) hwloc/1.11.7-GCCcore-6.4.0 | |||
8) OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28 | |||
9) iompi/2017b | |||
10) imkl/2017.3.196-iompi-2017b | |||
11) iomkl/2017b | |||
As you can see, toolchains can depend on each other. For instance, the iomkl toolchain, depends on iompi, which depends on iccifort, which depend on icc and ifort, which depend on GCCcore which depend on GCC. Hence it is very important that the correct versions of all related software are loaded. | |||
With software we provide, the toolchain used to compile is always specified in the "version" of the software that you want to load. | |||
== Most Commonly Used Software == | == Most Commonly Used Software == | ||
=== [http://www.open-mpi.org/ OpenMPI] === | === [http://www.open-mpi.org/ OpenMPI] === | ||
We provide lots of versions, you are most likely better off directly loading a toolchain or application to make sure you get the right version, but you can see the versions we have with 'module spider OpenMPI': | |||
* OpenMPI/2.0.2-GCC-6.3.0-2.27 | |||
* OpenMPI/2.0.2-iccifort-2017.1.132-GCC-6.3.0-2.27 | |||
* OpenMPI/2.1.1-GCC-6.4.0-2.28 | |||
* OpenMPI/2.1.1-GCC-7.2.0-2.29 | |||
* OpenMPI/2.1.1-gcccuda-2017b | |||
* OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28 | |||
* OpenMPI/2.1.1-iccifort-2018.0.128-GCC-7.2.0-2.29 | |||
=== [http://www.r-project.org/ R] === | === [http://www.r-project.org/ R] === | ||
We currently provide (module -r spider '^R$'): | |||
* R/3.4.0-foss-2017beocatb-X11-20170314 | |||
==== Modules ==== | ==== Modules ==== | ||
Line 18: | Line 69: | ||
To install your own module, login to Beocat and start R interactively | To install your own module, login to Beocat and start R interactively | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
module load R | |||
R | R | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Line 32: | Line 84: | ||
==== Running R Jobs ==== | ==== Running R Jobs ==== | ||
You cannot submit an R script directly. '<tt> | You cannot submit an R script directly. '<tt>sbatch myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSGE#Running_from_a_qsub_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.sbatch | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
#!/bin/bash | #!/bin/bash | ||
# | #SBATCH --mem-per-cpu=1G | ||
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS) | # Now we tell qsub how long we expect our work to take: 15 minutes (D-H:MM:SS) | ||
# | #SBATCH --time=0-0:15:00 | ||
# Now lets do some actual work. This starts R and loads the file myscript.R | # Now lets do some actual work. This starts R and loads the file myscript.R | ||
R --no-save -q < myscript.R | module load R | ||
R --no-save -q < myscript.R | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Now, to submit your R job, you would type | Now, to submit your R job, you would type | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
sbatch submit-R.sbatch | |||
</syntaxhighlight> | </syntaxhighlight> | ||
=== [http://www.java.com/ Java] === | === [http://www.java.com/ Java] === | ||
We | We currently provide (module spider Java): | ||
* Java/1.8.0_131 | |||
* Java/1.8.0_144 | |||
=== [http://www.python.org/about/ Python] === | === [http://www.python.org/about/ Python] === | ||
We currently provide (module spider Python) | |||
* Python/2.7.13-foss-2017beocatb | |||
* Python/2.7.13-GCCcore-7.2.0-bare | |||
* Python/2.7.13-iomkl-2017a | |||
* Python/2.7.13-iomkl-2017beocatb | |||
* Python/3.6.3-foss-2017b | |||
* Python/3.6.3-foss-2017beocatb | |||
* Python/3.6.3-iomkl-2017beocatb | |||
If you need modules that we do not have installed, you should use [https://virtualenv.pypa.io/en/stable/userguide/ virtualenv] to setup a virtual python environment in your home directory. This will let you install python modules as you please. | |||
If | |||
==== Setting up your virtual environment ==== | ==== Setting up your virtual environment ==== | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
# Load Python | |||
module load Python/3.6.3-iomkl-2017beocatb | |||
</syntaxhighlight> | </syntaxhighlight> | ||
* | * Create a location for your virtual environments (optional, but helps keep things organized) | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
mkdir ~/virtualenvs | |||
cd ~/virtualenvs | |||
</syntaxhighlight> | </syntaxhighlight> | ||
* '' | * Create a virtual environment. Here I will create a default virtual environment called 'test'. Note that <code>virtualenv --help</code> has many more useful options. | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
virtualenv test | |||
</syntaxhighlight> | </syntaxhighlight> | ||
* Lets look at our virtual environments | * Lets look at our virtual environments | ||
<pre> | <pre> | ||
% | % ls ~/virtualenvs | ||
test | |||
</pre> | </pre> | ||
* Activate one of these | * Activate one of these | ||
<pre> | <pre> | ||
% | %source ~/virtualenvs/test/bin/activate | ||
</pre> | </pre> | ||
* You can now install the python modules you want. This can be done using <tt>pip</tt>. | * You can now install the python modules you want. This can be done using <tt>pip</tt>. | ||
Line 140: | Line 151: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
#!/bin/bash | #!/bin/bash | ||
module load Python/3.6.3-iomkl-2017beocatb | |||
source ~/virtualenvs/test/bin/activate | |||
~/ | python ~/path/to/your/python/script.py | ||
</syntaxhighlight> | </syntaxhighlight> | ||
=== [http://www.perl.org/ Perl] === | |||
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads. | |||
If you need a newer version (or threads), just load one we provide in our modules (module spider Perl): | |||
* Perl/5.26.0-foss-2017beocatb | |||
* Perl/5.26.0-iompi-2017beocatb | |||
==== Submitting a job with Perl ==== | ==== Submitting a job with Perl ==== | ||
Much like R (above), you cannot simply '<tt> | Much like R (above), you cannot simply '<tt>sbatch myProgram.pl</tt>', but you must create a [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|submit script]] which will call perl. Here is an example: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
#!/bin/bash | #!/bin/bash | ||
# | #SBATCH --mem-per-cpu=1G | ||
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS) | # Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS) | ||
# | #SBATCH --time=0-0:15:00 | ||
# Now lets do some actual work. | # Now lets do some actual work. | ||
module load Perl | |||
perl /path/to/myProgram.pl | perl /path/to/myProgram.pl | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Revision as of 10:14, 10 January 2018
Drinking from the Firehose
For a complete list of all installed software, see NodePackageList
Toolchains
A toolchain is a set of compilers, libraries and applications that are needed to build software. Some software functions better when using specific toolchains.
We provide a good number of toolchains and versions of toolchains make sure your applications will compile and/or run correctly.
These toolchains include (you can run 'module keyword keychain compiler'):
- GCC
- The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...).
- GCCcore
- The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...).
- foss
- GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
- gcccuda
- GNU Compiler Collection (GCC) based compiler toolchain, along with CUDA toolkit.
- gmvapich2
- GNU Compiler Collection (GCC) based compiler toolchain, including MVAPICH2 for MPI support.
- gompi
- GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.
- gompic
- GNU Compiler Collection (GCC) based compiler toolchain along with CUDA toolkit, including OpenMPI for MPI support with CUDA features enabled.
- goolfc
- GCC based compiler toolchain __with CUDA support__, and including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
- icc
- C and C++ compiler from Intel
- iccifort
- Intel Cluster Toolkit Compiler Edition provides Intel C,C++ and fortran compilers, Intel MPI and Intel MKL
- ifort
- Fortran compiler from Intel
- iomkl
- Intel Cluster Toolchain Compiler Edition provides Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.
- iompi
- Intel C/C++ and Fortran compilers, alongside Open MPI.
You can run 'module spider $toolchain' to see the versions we have:
$ module spider iomkl
- iomkl/2017a
- iomkl/2017b
- iomkl/2017beocatb
If you load one of those (module load iomkl/2017b), you can see the other modules and versions of software that it loaded with the 'module list':
$ module list Currently Loaded Modules: 1) icc/2017.4.196-GCC-6.4.0-2.28 2) binutils/2.28-GCCcore-6.4.0 3) ifort/2017.4.196-GCC-6.4.0-2.28 4) iccifort/2017.4.196-GCC-6.4.0-2.28 5) GCCcore/6.4.0 6) numactl/2.0.11-GCCcore-6.4.0 7) hwloc/1.11.7-GCCcore-6.4.0 8) OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28 9) iompi/2017b 10) imkl/2017.3.196-iompi-2017b 11) iomkl/2017b
As you can see, toolchains can depend on each other. For instance, the iomkl toolchain, depends on iompi, which depends on iccifort, which depend on icc and ifort, which depend on GCCcore which depend on GCC. Hence it is very important that the correct versions of all related software are loaded.
With software we provide, the toolchain used to compile is always specified in the "version" of the software that you want to load.
Most Commonly Used Software
OpenMPI
We provide lots of versions, you are most likely better off directly loading a toolchain or application to make sure you get the right version, but you can see the versions we have with 'module spider OpenMPI':
- OpenMPI/2.0.2-GCC-6.3.0-2.27
- OpenMPI/2.0.2-iccifort-2017.1.132-GCC-6.3.0-2.27
- OpenMPI/2.1.1-GCC-6.4.0-2.28
- OpenMPI/2.1.1-GCC-7.2.0-2.29
- OpenMPI/2.1.1-gcccuda-2017b
- OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
- OpenMPI/2.1.1-iccifort-2018.0.128-GCC-7.2.0-2.29
R
We currently provide (module -r spider '^R$'):
- R/3.4.0-foss-2017beocatb-X11-20170314
Modules
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.
Installing your own modules
To install your own module, login to Beocat and start R interactively
module load R
R
Then install the package using
install.packages("PACKAGENAME")
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".
After installing you can test before leaving interactive mode by issuing the command
library("PACKAGENAME")
Running R Jobs
You cannot submit an R script directly. 'sbatch myscript.R' will result in an error. Instead, you need to make a bash script that will call R appropriately. Here is a minimal example. We'll save this as submit-R.sbatch
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (D-H:MM:SS)
#SBATCH --time=0-0:15:00
# Now lets do some actual work. This starts R and loads the file myscript.R
module load R
R --no-save -q < myscript.R
Now, to submit your R job, you would type
sbatch submit-R.sbatch
Java
We currently provide (module spider Java):
- Java/1.8.0_131
- Java/1.8.0_144
Python
We currently provide (module spider Python)
- Python/2.7.13-foss-2017beocatb
- Python/2.7.13-GCCcore-7.2.0-bare
- Python/2.7.13-iomkl-2017a
- Python/2.7.13-iomkl-2017beocatb
- Python/3.6.3-foss-2017b
- Python/3.6.3-foss-2017beocatb
- Python/3.6.3-iomkl-2017beocatb
If you need modules that we do not have installed, you should use virtualenv to setup a virtual python environment in your home directory. This will let you install python modules as you please.
Setting up your virtual environment
# Load Python
module load Python/3.6.3-iomkl-2017beocatb
- Create a location for your virtual environments (optional, but helps keep things organized)
mkdir ~/virtualenvs
cd ~/virtualenvs
- Create a virtual environment. Here I will create a default virtual environment called 'test'. Note that
virtualenv --help
has many more useful options.
virtualenv test
- Lets look at our virtual environments
% ls ~/virtualenvs test
- Activate one of these
%source ~/virtualenvs/test/bin/activate
- You can now install the python modules you want. This can be done using pip.
pip install numpy biopython
Using your virtual environment within a job
Here is a simple job script using the virtual environment testp2
#!/bin/bash
module load Python/3.6.3-iomkl-2017beocatb
source ~/virtualenvs/test/bin/activate
python ~/path/to/your/python/script.py
Perl
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.
If you need a newer version (or threads), just load one we provide in our modules (module spider Perl):
- Perl/5.26.0-foss-2017beocatb
- Perl/5.26.0-iompi-2017beocatb
Submitting a job with Perl
Much like R (above), you cannot simply 'sbatch myProgram.pl', but you must create a submit script which will call perl. Here is an example:
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS)
#SBATCH --time=0-0:15:00
# Now lets do some actual work.
module load Perl
perl /path/to/myProgram.pl
Installing my own software
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.
In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.
As a quick example of installing software in your home directory, we have a sample video on our Training Videos page. If you're still having problems or questions, please contact support as mentioned on our Main Page.