From Beocat
Jump to: navigation, search
(syntaxhighlighting and prettyfying)
Line 17: Line 17:
 
==== Installing your own modules ====
 
==== Installing your own modules ====
 
To install your own module, login to Beocat and start R interactively
 
To install your own module, login to Beocat and start R interactively
 
+
<syntaxhighlight lang="bash" line>
R
+
R
 
+
</syntaxhighlight>
 
Then install the package using
 
Then install the package using
 
+
<syntaxhighlight lang="rsplus" line>
install.packages("PACKAGENAME")
+
install.packages("PACKAGENAME")
 
+
</syntaxhighlight>
 
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".
 
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".
  
 
After installing you can test before leaving interactive mode by issuing the command
 
After installing you can test before leaving interactive mode by issuing the command
 
+
<syntaxhighlight lang="rsplus" line>
library("PACKAGENAME")
+
library("PACKAGENAME")
 
+
</syntaxhighlight>
 
==== Running R Jobs ====
 
==== Running R Jobs ====
  
 
You cannot submit an R script directly. 'qsub myscript.R' will result in an error. Instead, you need to make a bash script that will call R appropriately. Here is a minimal example. We'll save this as submit-R.qsub
 
You cannot submit an R script directly. 'qsub myscript.R' will result in an error. Instead, you need to make a bash script that will call R appropriately. Here is a minimal example. We'll save this as submit-R.qsub
  
<code>
+
<syntaxhighlight lang="bash" line>
 
  #!/bin/bash
 
  #!/bin/bash
 
  # First, lets tell the qsub command which resources we need
 
  # First, lets tell the qsub command which resources we need
Line 56: Line 56:
 
  echo -n "Ending the job at: "
 
  echo -n "Ending the job at: "
 
  date
 
  date
</code>
+
</syntaxhighlight>
  
 
Now, to submit your R job, you would type
 
Now, to submit your R job, you would type
qsub submit-R.qsub
+
<syntaxhighlight lang="bash" line>
 
+
qsub submit-R.qsub
 +
</syntaxhighlight>
 
=== [http://www.java.com/ Java] ===
 
=== [http://www.java.com/ Java] ===
 
Versions 1.6 and 1.7
 
Versions 1.6 and 1.7
  
We support 4 versions of the Java VM on Beocat. [http://en.wikipedia.org/wiki/IcedTea IcedTea] 6 and 7 (based on [http://en.wikipedia.org/wiki/OpenJDK OpenJDK]), Sun JDK 1.6 (Java 6), and Oracle JDK 1.7 (Java 7).
+
We support 4 versions of the Java VM on Beocat. [[wikipedia:IcedTea|IcedTea]] 6 and 7 (based on [[wikipedia:OpenJDK|OpenJDK]]), Sun JDK 1.6 (Java 6), and Oracle JDK 1.7 (Java 7).
  
 
We allow each user to select his or her Java version individually. If you do not select one, we default to Sun JDK 1.7.
 
We allow each user to select his or her Java version individually. If you do not select one, we default to Sun JDK 1.7.
Line 70: Line 71:
 
==== Selecting your Java version ====
 
==== Selecting your Java version ====
 
First, lets list the available versions. This can be done with the command <code>eselect java-vm list</code>
 
First, lets list the available versions. This can be done with the command <code>eselect java-vm list</code>
 
+
<pre>
% eselect java-vm list
+
% eselect java-vm list
Available Java Virtual Machines:
+
Available Java Virtual Machines:
  [1]  icedtea-bin-6
+
  [1]  icedtea-bin-6
  [2]  icedtea-bin-7
+
  [2]  icedtea-bin-7
  [3]  oracle-jdk-bin-1.7  system-vm
+
  [3]  oracle-jdk-bin-1.7  system-vm
  [4]  sun-jdk-1.6
+
  [4]  sun-jdk-1.6
 
+
</pre>
 
If you'll note,  oracle-jdk-bin-1.7 (marked "system-vm") is the default for all users. If you have a custom version set, it will be marked with "user-vm". Now if you wanted to use icedtea-6, you could run the following:
 
If you'll note,  oracle-jdk-bin-1.7 (marked "system-vm") is the default for all users. If you have a custom version set, it will be marked with "user-vm". Now if you wanted to use icedtea-6, you could run the following:
 
+
<syntaxhighlight lang="bash" line>
eselect java-vm set user 1
+
eselect java-vm set user 1
 
+
</syntaxhighlight>
 
Now, we see the difference when running the above command
 
Now, we see the difference when running the above command
 
+
<pre>
% eselect java-vm list
+
% eselect java-vm list
Available Java Virtual Machines:
+
Available Java Virtual Machines:
  [1]  icedtea-bin-6  user-vm
+
  [1]  icedtea-bin-6  user-vm
  [2]  icedtea-bin-7
+
  [2]  icedtea-bin-7
  [3]  oracle-jdk-bin-1.7  system-vm
+
  [3]  oracle-jdk-bin-1.7  system-vm
  [4]  sun-jdk-1.6
+
  [4]  sun-jdk-1.6
 
+
</pre>
 
To verify you are seeing the correct java, you can run <code>java -version</code>
 
To verify you are seeing the correct java, you can run <code>java -version</code>
 
+
<pre>
% java -version
+
% java -version
java version "1.6.0_27"
+
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.7) (Gentoo build 1.6.0_27-b27)
+
OpenJDK Runtime Environment (IcedTea6 1.12.7) (Gentoo build 1.6.0_27-b27)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
+
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
 
+
</pre>
 
=== [http://www.python.org/about/ Python] ===
 
=== [http://www.python.org/about/ Python] ===
  
Line 114: Line 115:
 
* [[Changing_Shells|Change your shell]] to bash
 
* [[Changing_Shells|Change your shell]] to bash
 
* Make sure ~/.bash_profile exists
 
* Make sure ~/.bash_profile exists
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
+
<syntaxhighlight lang="bash" line>
 +
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
 +
</syntaxhighlight>
 
* Add a line like <code>source /usr/bin/virtualenvwrapper.sh</code> to your .bash_profile.
 
* Add a line like <code>source /usr/bin/virtualenvwrapper.sh</code> to your .bash_profile.
echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.bash_profile
+
<syntaxhighlight lang="bash" line>
 +
echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.bash_profile
 +
</syntaxhighlight>
 
* Show your existing environments
 
* Show your existing environments
workon
+
<syntaxhighlight lang="bash" line>
 +
workon
 +
</syntaxhighlight>
 
* Create a virtual environment. Here I will create a default virtual environment called 'test', a python2 virtual environment called 'testp2', a python3 virtual environment called 'testp3', and a pypy environment called testpypy. Note that <code>mkvirtualenv --help</code> has many more useful options.
 
* Create a virtual environment. Here I will create a default virtual environment called 'test', a python2 virtual environment called 'testp2', a python3 virtual environment called 'testp3', and a pypy environment called testpypy. Note that <code>mkvirtualenv --help</code> has many more useful options.
 +
<syntaxhighlight lang="bash" line>
 
  mkvirtualenv -p $(which python2) testp2
 
  mkvirtualenv -p $(which python2) testp2
 
  mkvirtualenv -p $(which python3) testp3
 
  mkvirtualenv -p $(which python3) testp3
 
  mkvirtualenv -p $(which pypy-c2.0) testpypy
 
  mkvirtualenv -p $(which pypy-c2.0) testpypy
 +
</syntaxhighlight>
 
* Lets look at our virtual environments
 
* Lets look at our virtual environments
%workon
+
<pre>
testp2
+
%workon
testp3
+
testp2
testpypy
+
testp3
 +
testpypy
 +
</pre>
 
* Activate one of these
 
* Activate one of these
%workon testp2
+
<pre>
* You can now install the python modules you want. This can be done using pip.
+
%workon testp2
pip install numpy biopython
+
</pre>
 
+
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
 +
<syntaxhighlight lang="bash" line>
 +
pip install numpy biopython
 +
</syntaxhighlight>
 
==== Using your virtual environment within a job ====
 
==== Using your virtual environment within a job ====
 
Here is a simple job script using the virtual environment testp2
 
Here is a simple job script using the virtual environment testp2
#!/bin/bash
+
<syntaxhighlight lang="bash" line>
source /usr/bin/virtualenvwrapper.sh
+
#!/bin/bash
workon testp2
+
source /usr/bin/virtualenvwrapper.sh
~/path/to/your/python/script.py
+
workon testp2
 
+
~/path/to/your/python/script.py
 +
</syntaxhighlight>
 
==== A note on [http://www.numpy.org/ NumPy] ====
 
==== A note on [http://www.numpy.org/ NumPy] ====
 
NumPy is a commonly-used Python package.
 
NumPy is a commonly-used Python package.
  
 
Make sure the following is executed before running <code>pip install numpy</code>
 
Make sure the following is executed before running <code>pip install numpy</code>
 
+
<syntaxhighlight lang="bash" line>
cp /opt/beocat/numpy/.numpy-site.cfg ~/.numpy-site.cfg
+
cp /opt/beocat/numpy/.numpy-site.cfg ~/.numpy-site.cfg
 
+
</syntaxhighlight>
 
=== [http://www.perl.org/ Perl] ===
 
=== [http://www.perl.org/ Perl] ===
 
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.
 
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.
Line 152: Line 167:
 
==== Getting Perl with threads ====
 
==== Getting Perl with threads ====
 
* Setup perlbrew
 
* Setup perlbrew
** [[Changing_Shells|Change your shell]] to bash
+
** [[Changing Shells|Change your shell]] to bash
 
** Install perlbrew
 
** Install perlbrew
curl -L http://install.perlbrew.pl | bash
+
<syntaxhighlight lang="bash" line>
 +
curl -L http://install.perlbrew.pl | bash
 +
</syntaxhighlight>
 
** Make sure that ~/.bash_profile exists
 
** Make sure that ~/.bash_profile exists
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
+
<syntaxhighlight lang="bash" line>
 +
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
 +
</syntaxhighlight>
 
** Add <code>source ~/perl5/perlbrew/etc/bashrc</code> to ~/.bash_profile
 
** Add <code>source ~/perl5/perlbrew/etc/bashrc</code> to ~/.bash_profile
echo "source ~/perl5/perlbrew/etc/bashrc" >> ~/.bash_profile
+
<syntaxhighlight lang="bash" line>
 +
echo "source ~/perl5/perlbrew/etc/bashrc" >> ~/.bash_profile
 +
</syntaxhighlight>
 
** Then source your bash profile
 
** Then source your bash profile
source ~/.bash_profile
+
<syntaxhighlight lang="bash" line>
 +
source ~/.bash_profile
 +
</syntaxhighlight>
 
* Now, install perl with threads within perlbrew
 
* Now, install perl with threads within perlbrew
 
** Find the current Perl version.
 
** Find the current Perl version.
% perl -version
+
<pre>
+
% perl -version
This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux
+
 
(with 22 registered patches, see perl -V for more detail)
+
This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux
(...several more lines deleted)
+
(with 22 registered patches, see perl -V for more detail)
 +
(...several more lines deleted)
 +
</pre>
 
** In this case the version is 5.16.3, so we run
 
** In this case the version is 5.16.3, so we run
perlbrew install -f -n -D usethreads perl-5.16.3
+
<syntaxhighlight lang="bash" line>
 +
perlbrew install -f -n -D usethreads perl-5.16.3
 +
</syntaxhighlight>
 
** To temporarily use the new version of perl in the current shell, we now run
 
** To temporarily use the new version of perl in the current shell, we now run
perlbrew use perl-5.16.3
+
<syntaxhighlight lang="bash" line>
 +
perlbrew use perl-5.16.3
 +
</syntaxhighlight>
 
** To switch versions of perl for every new login or job, run
 
** To switch versions of perl for every new login or job, run
perlbrew switch perl-5.16.3
+
<syntaxhighlight lang="bash" line>
 +
perlbrew switch perl-5.16.3
 +
</syntaxhighlight>
 
** You can reverse this switch with
 
** You can reverse this switch with
perlbrew switch-off
+
<syntaxhighlight lang="bash" line>
 
+
perlbrew switch-off
 +
</syntaxhighlight>
 
== Installing my own software ==
 
== Installing my own software ==
 
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.
 
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.
Line 182: Line 214:
 
In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.
 
In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.
  
As a quick example of installing software in your home directory, we have a sample video on our [[Training_Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main_Page]].
+
As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

Revision as of 19:57, 23 April 2014

Drinking from the Firehose

For a complete list of all installed software, see NodePackageList

Most Commonly Used Software

OpenMPI

Version 1.4.3

Scilab

Version 5.4.0

R

Version 3.0.3

Modules

We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

Installing your own modules

To install your own module, login to Beocat and start R interactively

1 R

Then install the package using

1 install.packages("PACKAGENAME")

Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command

1 library("PACKAGENAME")

Running R Jobs

You cannot submit an R script directly. 'qsub myscript.R' will result in an error. Instead, you need to make a bash script that will call R appropriately. Here is a minimal example. We'll save this as submit-R.qsub

 1  #!/bin/bash
 2  # First, lets tell the qsub command which resources we need
 3  # lets start with memory (in this case I ask for 1 gigabyte).
 4  # For help on these, see [[SGEBasics]]
 5  
 6  #$ -l mem=1G
 7  # Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS)
 8  
 9  #$ -l h_rt=0:15:00
10  
11  # Lets output a little useful information This will put something like "Starting the job at: Thu Jan 26 10:43:26 CST 2012" in your output file
12  echo -n "Starting the job at: "
13  date
14  
15  # Now lets do some actual work. A lot of our users use R, so we'll go over that
16  # This starts R and loads the file myscript.R
17  R --no-save -q < myscript.R
18  
19  # like before, this is just useful information
20  echo -n "Ending the job at: "
21  date

Now, to submit your R job, you would type

1 qsub submit-R.qsub

Java

Versions 1.6 and 1.7

We support 4 versions of the Java VM on Beocat. IcedTea 6 and 7 (based on OpenJDK), Sun JDK 1.6 (Java 6), and Oracle JDK 1.7 (Java 7).

We allow each user to select his or her Java version individually. If you do not select one, we default to Sun JDK 1.7.

Selecting your Java version

First, lets list the available versions. This can be done with the command eselect java-vm list

% eselect java-vm list
Available Java Virtual Machines:
  [1]   icedtea-bin-6
  [2]   icedtea-bin-7
  [3]   oracle-jdk-bin-1.7  system-vm
  [4]   sun-jdk-1.6

If you'll note, oracle-jdk-bin-1.7 (marked "system-vm") is the default for all users. If you have a custom version set, it will be marked with "user-vm". Now if you wanted to use icedtea-6, you could run the following:

1 eselect java-vm set user 1

Now, we see the difference when running the above command

% eselect java-vm list
Available Java Virtual Machines:
  [1]   icedtea-bin-6  user-vm
  [2]   icedtea-bin-7
  [3]   oracle-jdk-bin-1.7  system-vm
  [4]   sun-jdk-1.6

To verify you are seeing the correct java, you can run java -version

% java -version
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.7) (Gentoo build 1.6.0_27-b27)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

Python

We have several versions of Python available:

For the uninitiated PyPy provides just-in-time compilation for python code. While it doesn't support all modules, code which does run under PyPy can see a significant performance increase.

If you just need python and its default modules, you can use python2 python3 pypy-c1.9 or pypy-c2.0 as you would any other application.

If, however, you need modules that we do not have installed, you should use virtualenvwrapper to setup a virtual python environment in your home directory. This will let you install python modules as you please.

Setting up your virtual environment

1 if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
  • Add a line like source /usr/bin/virtualenvwrapper.sh to your .bash_profile.
1 echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.bash_profile
  • Show your existing environments
1 workon
  • Create a virtual environment. Here I will create a default virtual environment called 'test', a python2 virtual environment called 'testp2', a python3 virtual environment called 'testp3', and a pypy environment called testpypy. Note that mkvirtualenv --help has many more useful options.
1  mkvirtualenv -p $(which python2) testp2
2  mkvirtualenv -p $(which python3) testp3
3  mkvirtualenv -p $(which pypy-c2.0) testpypy
  • Lets look at our virtual environments
%workon
testp2
testp3
testpypy
  • Activate one of these
%workon testp2
  • You can now install the python modules you want. This can be done using pip.
1 pip install numpy biopython

Using your virtual environment within a job

Here is a simple job script using the virtual environment testp2

1 #!/bin/bash
2 source /usr/bin/virtualenvwrapper.sh
3 workon testp2
4 ~/path/to/your/python/script.py

A note on NumPy

NumPy is a commonly-used Python package.

Make sure the following is executed before running pip install numpy

1 cp /opt/beocat/numpy/.numpy-site.cfg ~/.numpy-site.cfg

Perl

The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.

Getting Perl with threads

1 curl -L http://install.perlbrew.pl | bash
    • Make sure that ~/.bash_profile exists
1 if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
    • Add source ~/perl5/perlbrew/etc/bashrc to ~/.bash_profile
1 echo "source ~/perl5/perlbrew/etc/bashrc" >> ~/.bash_profile
    • Then source your bash profile
1 source ~/.bash_profile
  • Now, install perl with threads within perlbrew
    • Find the current Perl version.
% perl -version

This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux
(with 22 registered patches, see perl -V for more detail)
(...several more lines deleted)
    • In this case the version is 5.16.3, so we run
1 perlbrew install -f -n -D usethreads perl-5.16.3
    • To temporarily use the new version of perl in the current shell, we now run
1 perlbrew use perl-5.16.3
    • To switch versions of perl for every new login or job, run
1 perlbrew switch perl-5.16.3
    • You can reverse this switch with
1 perlbrew switch-off

Installing my own software

Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our Training Videos page. If you're still having problems or questions, please contact support as mentioned on our Main Page.