Beocat - User contributions [en]

Globus

2022-08-25T02:24:46Z

Kylehutson: /* Transferring Data using Globus */

FAQ

2022-08-10T19:20:10Z

Kylehutson: Added PuTTY instructions for automating Duo

== How do I connect to Beocat ==
{| class="wikitable"
|-
! colspan="2" | Connection Settings
|-
! Hostname
| style="text-align:right" | headnode.beocat.ksu.edu
|-
! Port
| style="text-align:right" | 22
|-
! Username
| style="text-align:right" | <tt>eID</tt>
|-
! Password
| style="text-align:right" | <tt>eID Password</tt>
|}

{| class="wikitable"
!colspan="2" | Supported Connection Software (Latest Versions of Each)
|-
!rowspan="3" | Shell
|-
| [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]
|-
| ssh from openssh
|-
!rowspan="4" | File Transfer Utilities
|-
| [https://filezilla-project.org/ Filezilla]
|-
| [http://winscp.net/ WinSCP]
|-
| scp and sftp from openssh
|-
!rowspan="2" | Combination
|-
| [http://mobaxterm.mobatek.net/ MobaXterm]
|}

===Duo===
If you're account is Duo Enabled, you will be asked to approve ''each'' connection through Duo's push system to your smart device by default for any non-interactive protocols. If you don't have a smart device, or your smart device is not currently able to be contacted by Duo, there are options.

====Automating Duo Method====
You would need to configure your connection client to send an ''Environment'' variable called <tt>DUO_PASSCODE</tt>. Its value could be the currently valid passcode from Duo, <tt>push</tt> or it could be set to <tt>phone</tt>. <tt>push</tt> will push the prompt to your smart device. <tt>phone</tt> will have duo call your phone number to approve.

With OpenSSH (Linux or Mac command-line), to automatically set the Duo method to "push", use the command
<syntaxhighlight lang="bash">DUO_PASSCODE=push ssh -o SendEnv=DUO_PASSCODE headnode.beocat.ksu.edu</syntaxhighlight>
In MobaXTerm, to automatically set the Duo method to "push", edit your SSH session and on the "Advanced SSH Settings" tab, change the "Execute command" to <syntaxhighlight lang="bash">DUO_PASSCODE=push bash</syntaxhighlight>
In PuTTY to automatically set the Duo method to "push", expand "Connection" (if it isn't already), then click "Data". Under Environment variables, enter "DUO_PASSCODE" beside "Variable" and "push" beside "Value". Click the "Add" button and it will show up underneath. Be sure to go back to "Session" to save this change for PuTTY to remember this change.

== How do I compile my programs? ==
=== Serial programs ===
==== Fortran ====
<tt>ifort</tt> or <tt>gfortran</tt>
==== C/C++ ====
<tt>icc</tt>, <tt>gcc</tt> and <tt>g++</tt>
=== Parallel programs ===
==== Fortran ====
<tt>mpif77</tt> or <tt>mpif90</tt>
==== C/C++ ====
<tt>mpicc</tt> or <tt>mpic++</tt>
== Do Beocat jobs have a maximum Time Limit ==
Yes, there is a time limit, the scheduler will reject jobs longer than 28 days. The other side of that is that we reserve the right to a maintenance period every 14 days. Unless it is an emergency, we will give at least 2 weeks notice before these maintenance periods actually occur. Jobs 14 days or less that have started when we announce a maintenance period should be able to complete before it begins.

With that being said, there is no guarantee that any physical piece of hardware and the software that runs on it will behave for any significant length of time. Memory, processors, disk drives can all fail with little to no warning. Software may have bugs. We have had issues with the shared filesystem that resulted in several nodes losing connectivity and forced reboots. If you can, we always recommend that you write your jobs so that they can be resumed if they get interrupted.

{{Note|The 28 day limit can be overridden on a temporary and per-user basis provided there is enough justification|reminder|inline=1}}

== How are the filesystems on Beocat set up? ==

{| class="wikitable"
|-
! Mountpoint !! Local / Shared !! Size !! Filesystem !! Advice
|-
| /bulk || Shared || 3.1PB shared with /homes and /scratch || cephfs || Slower than /homes; costs $45/TB/year
|-
| /homes || Shared || 3.1PB shared with /bulk and /scratch || cephfs || Good enough for most jobs; limited to 1TB per home directory
|-
| /scratch || Shared || 3.1PB shared with /bulk and /homes || cephfs || Fast shared tmp space; files not used in 30 days are automatically culled
|-
| /fastscratch || Shared || 280TB || nfs on top of ZFS || Faster than /scratch, built with all NVME disks; files not used in 30 days are automatically culled.
|-
| /tmp || Local || >100GB (varies per node) || ext4 || Good for I/O intensive jobs
|-
|}
=== Usage Advice ===
For most jobs you shouldn't need to worry, your default working directory
is your homedir and it will be fast enough for most tasks.
I/O intensive work should use /tmp, but you will need to remember to copy
your files to and from this partition as part of your job script. This is made
easier through the <tt>$TMPDIR</tt> environment variable in your jobs.

Example usage of <tt>$TMPDIR</tt> in a job script
<syntaxhighlight lang="bash" line>
#!/bin/bash

#copy our input file to $TMPDIR to make processing faster
cp ~/experiments/input.data $TMPDIR

#use the input file we copied over to the local system
#generate the output file in $TMPDIR as well
~/bin/my_program --input-file=$TMPDIR/input.data --output-file=$TMPDIR/output.data

#copy the results back from $TMPDIR
cp $TMPDIR/output.data ~/experiments/results.$SLURM_JOB_ID
</syntaxhighlight>

You need to remember to copy over your data from <tt>$TMPDIR</tt> as part of your job.
That directory and its contents are deleted when the job is complete.

== What is "killable:1" or "killable:0" ==
On Beocat, some of the machines have been purchased by specific users and/or groups. These users and/or groups get guaranteed access to their machines at any point in time. Often, these machines are sitting idle because the owners have no need for it at the time. This would be a significant waste of computational power if there were no other way to make use of the computing cycles.

If you're wondering why a job may have the exit status of <tt>PREEMPTED</tt> from kstat or sacct, this is the reason.

=== Enter the "killable" resource ===
Killable (--gres=killable:1) jobs are jobs that can be scheduled to these "owned" machines by users outside of the true group of owners. If a "killable" job starts on one of these owned machines and the owner of said machine comes along and submits a job, the "killable" job will be returned to the queue, (killed off as it were), and restarted at some future point in time. The job will still complete at some future point, and if the job makes use of a checkpointing algorithm it may complete even faster. The trade off between marking a job "killable" and not, is that sometimes applications need a significant amount of runtime, and cannot resume running from a partial output, meaning that it may get restarted over and over again, never reaching the finish line. As such, we only auto-enable "killable" for relatively short jobs (<=168:00:00). Some users still feel this is a hindrance, so we created a way to tell us not to automatically mark short jobs "killable"

=== Disabling killable ===
Specifying --gres=killable:0 will tell us to not mark your job as killable.

=== The trade-off ===
If a job is marked killable, there are a non-trivial amount of additional nodes that the job can run on. If your job checkpoints itself, or is relatively short, there should be no downside to marking the job killable, as the job will probably start sooner. If your job is long-running and doesn't checkpoint (save its state to restart a previous session) itself, it could cause your job to take longer to complete.

== Help! When I submit my jobs I get "Warning To stay compliant with standard unix behavior, there should be a valid #! line in your script i.e. #!/bin/tcsh" ==
Job submission scripts are supposed to have a line similar to '<code>#!/bin/bash</code>' in them to start. We have had problems with people submitting jobs with invalid #! lines, so we enforce that rule. When this happens the job fails and we have to manually clean it up. The warning message is there just to inform you that the job script should have a line in it, in most cases #!/bin/tcsh or #!/bin/bash, to indicate what program should be used to run the script. When the line is missing from a script, by default your default shell is used to execute the script (in your case /usr/local/bin/tcsh). This works in most cases, but may not be what you are wanting.

== Help! When I submit my jobs I get "A #! line exists, but it is not pointing to an executable. Please fix. Job not submitted." ==
Like the above, error says you need a #!/bin/bash or similar line in your job script. This error says that while the line exists, the #! line isn't mentioning an executable file, thus the script will not be able to run. Most likely you wanted #!/bin/bash instead of something else.

== Help! My jobs keep dying after 1 hour and I don't know why ==
Beocat has default runtime limit of 1 hour. If you need more than that, or need more than 1 GB of memory per core, you'll want to look at the documentation [[SlurmBasics|here]] to see how to request it.

In short, when you run sbatch for your job, you'll want to put something along the lines of '<code>--time=0-10:00:00</code>' before the job script if you want your job to run for 10 hours.

== Help my error file has "Warning: no access to tty" ==
The warning message "Warning: no access to tty (Bad file descriptor)" is safe to ignore. It typically happens with the tcsh shell.

== Help! My job isn't going to finish in the time I specified. Can I change the time requirement? ==
Generally speaking, no.

Jobs are scheduled based on execution times (among other things). If it were easy to change your time requirement, one could submit a job with a 15-minute run-time, get it scheduled quickly, and then say "whoops - I meant 15 weeks", effectively gaming the job scheduler. In extreme circumstances and depending on the job requirements, we '''may''' be able to manually intervene. This process prevents other users from using the node(s) you are currently using, so are not routinely approved. Contact Beocat support (below) if you feel your circumstances warrant special consideration.

== Help! My perl job runs fine on the head node, but only runs for a few seconds and then quits when submitted to the queue. ==
Take a look at our documentation on [[Installed_software#Perl|Perl]]

== Help! When using mpi I get 'CMA: no RDMA devices found' or 'A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces' ==
This message simply means that some but not all nodes the job is running on have infiniband cards. The job will still run, but will not use the fastest interconnect we have available. This may or may not be an issue, depending on how message heavy your job is. If you would like to not see this warning, you may request infiniband as a resource when submitting your job. <code>--gres=fabric:ib:1</code>

== Common Storage For Projects ==
Sometimes it is useful for groups of people to have a common storage area.

If you do not have a project, send a request via email to beocat@cs.ksu.edu. Note that these projects are generally reserved for tenure-track faculty and with a single project per eID.

If you already have a project you can do the following:

'''Note:''' The <tt>$group_name</tt> variable in the commands below needs to be replaced with the lower case name of your project. Membership of the projects can be done [https://account.beocat.ksu.edu/project here]
* Create a directory in one of the home directories of someone in your group, ideally the project owner's.
** <tt>mkdir $directory</tt>
* Set the default permissions for new files and directories created in the directory:
** <tt>setfacl -d -m g:$group_name:rX -R $directory</tt>
* Set the permissions for the existing files and directories:
** <tt>setfacl -m g:$group_name:rX -R $directory</tt>

This will give people in your group the ability to read files in the 'share' directory. If you also want them to be able to write or modify files in that directory then use change the ':rX' to ':rwX' instead. e.g. 'setfacl -d -m g:$group_name:rwX -R $directory' for both setfacl commands.

== How do I get more help? ==
There are many sources of help for most Linux systems.

=== Unix man pages ===
Linux provides man pages (short for manual pages). These are simple enough to call, for example: if you need information on submitting jobs to Beocat, you can type '<code>man sbatch</code>'. This will bring up the manual for sbatch.

=== GNU info system ===
Not all applications have "man pages." Most of the rest have what they call info pages. For example, if you needed information on finding a file you could use '<code>info find</code>'.

=== This documentation ===
This documentation is very thoroughly researched, and has been painstakingly assembled for your benefit. Please use it.

=== Contact support ===
Support can be contacted [mailto:beocat@cis.ksu.edu here]. Please include detailed information about your problem, including the job number, applications you are trying to run, and the current directory that you are in.

FAQ

2022-08-09T21:03:44Z

Kylehutson: Tips on automating Duo

== How do I connect to Beocat ==
{| class="wikitable"
|-
! colspan="2" | Connection Settings
|-
! Hostname
| style="text-align:right" | headnode.beocat.ksu.edu
|-
! Port
| style="text-align:right" | 22
|-
! Username
| style="text-align:right" | <tt>eID</tt>
|-
! Password
| style="text-align:right" | <tt>eID Password</tt>
|}

{| class="wikitable"
!colspan="2" | Supported Connection Software (Latest Versions of Each)
|-
!rowspan="3" | Shell
|-
| [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]
|-
| ssh from openssh
|-
!rowspan="4" | File Transfer Utilities
|-
| [https://filezilla-project.org/ Filezilla]
|-
| [http://winscp.net/ WinSCP]
|-
| scp and sftp from openssh
|-
!rowspan="2" | Combination
|-
| [http://mobaxterm.mobatek.net/ MobaXterm]
|}

===Duo===
If you're account is Duo Enabled, you will be asked to approve ''each'' connection through Duo's push system to your smart device by default for any non-interactive protocols. If you don't have a smart device, or your smart device is not currently able to be contacted by Duo, there are options.

====Automating Duo Method====
You would need to configure your connection client to send an ''Environment'' variable called <tt>DUO_PASSCODE</tt>. Its value could be the currently valid passcode from Duo, <tt>push</tt> or it could be set to <tt>phone</tt>. <tt>push</tt> will push the prompt to your smart device. <tt>phone</tt> will have duo call your phone number to approve.

With OpenSSH (Linux or Mac command-line), to automatically set the Duo method to "push", use the command
<syntaxhighlight lang="bash">DUO_PASSCODE=push ssh -o SendEnv=DUO_PASSCODE headnode.beocat.ksu.edu</syntaxhighlight>
In MobaXTerm, to automatically set the Duo method to "push", edit your SSH session and on the "Advanced SSH Settings" tab, change the "Execute command" to <syntaxhighlight lang="bash">DUO_PASSCODE=push bash</syntaxhighlight>

== How do I compile my programs? ==
=== Serial programs ===
==== Fortran ====
<tt>ifort</tt> or <tt>gfortran</tt>
==== C/C++ ====
<tt>icc</tt>, <tt>gcc</tt> and <tt>g++</tt>
=== Parallel programs ===
==== Fortran ====
<tt>mpif77</tt> or <tt>mpif90</tt>
==== C/C++ ====
<tt>mpicc</tt> or <tt>mpic++</tt>
== Do Beocat jobs have a maximum Time Limit ==
Yes, there is a time limit, the scheduler will reject jobs longer than 28 days. The other side of that is that we reserve the right to a maintenance period every 14 days. Unless it is an emergency, we will give at least 2 weeks notice before these maintenance periods actually occur. Jobs 14 days or less that have started when we announce a maintenance period should be able to complete before it begins.

With that being said, there is no guarantee that any physical piece of hardware and the software that runs on it will behave for any significant length of time. Memory, processors, disk drives can all fail with little to no warning. Software may have bugs. We have had issues with the shared filesystem that resulted in several nodes losing connectivity and forced reboots. If you can, we always recommend that you write your jobs so that they can be resumed if they get interrupted.

{{Note|The 28 day limit can be overridden on a temporary and per-user basis provided there is enough justification|reminder|inline=1}}

== How are the filesystems on Beocat set up? ==

{| class="wikitable"
|-
! Mountpoint !! Local / Shared !! Size !! Filesystem !! Advice
|-
| /bulk || Shared || 3.1PB shared with /homes and /scratch || cephfs || Slower than /homes; costs $45/TB/year
|-
| /homes || Shared || 3.1PB shared with /bulk and /scratch || cephfs || Good enough for most jobs; limited to 1TB per home directory
|-
| /scratch || Shared || 3.1PB shared with /bulk and /homes || cephfs || Fast shared tmp space; files not used in 30 days are automatically culled
|-
| /fastscratch || Shared || 280TB || nfs on top of ZFS || Faster than /scratch, built with all NVME disks; files not used in 30 days are automatically culled.
|-
| /tmp || Local || >100GB (varies per node) || ext4 || Good for I/O intensive jobs
|-
|}
=== Usage Advice ===
For most jobs you shouldn't need to worry, your default working directory
is your homedir and it will be fast enough for most tasks.
I/O intensive work should use /tmp, but you will need to remember to copy
your files to and from this partition as part of your job script. This is made
easier through the <tt>$TMPDIR</tt> environment variable in your jobs.

Example usage of <tt>$TMPDIR</tt> in a job script
<syntaxhighlight lang="bash" line>
#!/bin/bash

#copy our input file to $TMPDIR to make processing faster
cp ~/experiments/input.data $TMPDIR

#use the input file we copied over to the local system
#generate the output file in $TMPDIR as well
~/bin/my_program --input-file=$TMPDIR/input.data --output-file=$TMPDIR/output.data

#copy the results back from $TMPDIR
cp $TMPDIR/output.data ~/experiments/results.$SLURM_JOB_ID
</syntaxhighlight>

You need to remember to copy over your data from <tt>$TMPDIR</tt> as part of your job.
That directory and its contents are deleted when the job is complete.

== What is "killable:1" or "killable:0" ==
On Beocat, some of the machines have been purchased by specific users and/or groups. These users and/or groups get guaranteed access to their machines at any point in time. Often, these machines are sitting idle because the owners have no need for it at the time. This would be a significant waste of computational power if there were no other way to make use of the computing cycles.

If you're wondering why a job may have the exit status of <tt>PREEMPTED</tt> from kstat or sacct, this is the reason.

=== Enter the "killable" resource ===
Killable (--gres=killable:1) jobs are jobs that can be scheduled to these "owned" machines by users outside of the true group of owners. If a "killable" job starts on one of these owned machines and the owner of said machine comes along and submits a job, the "killable" job will be returned to the queue, (killed off as it were), and restarted at some future point in time. The job will still complete at some future point, and if the job makes use of a checkpointing algorithm it may complete even faster. The trade off between marking a job "killable" and not, is that sometimes applications need a significant amount of runtime, and cannot resume running from a partial output, meaning that it may get restarted over and over again, never reaching the finish line. As such, we only auto-enable "killable" for relatively short jobs (<=168:00:00). Some users still feel this is a hindrance, so we created a way to tell us not to automatically mark short jobs "killable"

=== Disabling killable ===
Specifying --gres=killable:0 will tell us to not mark your job as killable.

=== The trade-off ===
If a job is marked killable, there are a non-trivial amount of additional nodes that the job can run on. If your job checkpoints itself, or is relatively short, there should be no downside to marking the job killable, as the job will probably start sooner. If your job is long-running and doesn't checkpoint (save its state to restart a previous session) itself, it could cause your job to take longer to complete.

== Help! When I submit my jobs I get "Warning To stay compliant with standard unix behavior, there should be a valid #! line in your script i.e. #!/bin/tcsh" ==
Job submission scripts are supposed to have a line similar to '<code>#!/bin/bash</code>' in them to start. We have had problems with people submitting jobs with invalid #! lines, so we enforce that rule. When this happens the job fails and we have to manually clean it up. The warning message is there just to inform you that the job script should have a line in it, in most cases #!/bin/tcsh or #!/bin/bash, to indicate what program should be used to run the script. When the line is missing from a script, by default your default shell is used to execute the script (in your case /usr/local/bin/tcsh). This works in most cases, but may not be what you are wanting.

== Help! When I submit my jobs I get "A #! line exists, but it is not pointing to an executable. Please fix. Job not submitted." ==
Like the above, error says you need a #!/bin/bash or similar line in your job script. This error says that while the line exists, the #! line isn't mentioning an executable file, thus the script will not be able to run. Most likely you wanted #!/bin/bash instead of something else.

== Help! My jobs keep dying after 1 hour and I don't know why ==
Beocat has default runtime limit of 1 hour. If you need more than that, or need more than 1 GB of memory per core, you'll want to look at the documentation [[SlurmBasics|here]] to see how to request it.

In short, when you run sbatch for your job, you'll want to put something along the lines of '<code>--time=0-10:00:00</code>' before the job script if you want your job to run for 10 hours.

== Help my error file has "Warning: no access to tty" ==
The warning message "Warning: no access to tty (Bad file descriptor)" is safe to ignore. It typically happens with the tcsh shell.

== Help! My job isn't going to finish in the time I specified. Can I change the time requirement? ==
Generally speaking, no.

Jobs are scheduled based on execution times (among other things). If it were easy to change your time requirement, one could submit a job with a 15-minute run-time, get it scheduled quickly, and then say "whoops - I meant 15 weeks", effectively gaming the job scheduler. In extreme circumstances and depending on the job requirements, we '''may''' be able to manually intervene. This process prevents other users from using the node(s) you are currently using, so are not routinely approved. Contact Beocat support (below) if you feel your circumstances warrant special consideration.

== Help! My perl job runs fine on the head node, but only runs for a few seconds and then quits when submitted to the queue. ==
Take a look at our documentation on [[Installed_software#Perl|Perl]]

== Help! When using mpi I get 'CMA: no RDMA devices found' or 'A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces' ==
This message simply means that some but not all nodes the job is running on have infiniband cards. The job will still run, but will not use the fastest interconnect we have available. This may or may not be an issue, depending on how message heavy your job is. If you would like to not see this warning, you may request infiniband as a resource when submitting your job. <code>--gres=fabric:ib:1</code>

== Common Storage For Projects ==
Sometimes it is useful for groups of people to have a common storage area.

If you do not have a project, send a request via email to beocat@cs.ksu.edu. Note that these projects are generally reserved for tenure-track faculty and with a single project per eID.

If you already have a project you can do the following:

'''Note:''' The <tt>$group_name</tt> variable in the commands below needs to be replaced with the lower case name of your project. Membership of the projects can be done [https://account.beocat.ksu.edu/project here]
* Create a directory in one of the home directories of someone in your group, ideally the project owner's.
** <tt>mkdir $directory</tt>
* Set the default permissions for new files and directories created in the directory:
** <tt>setfacl -d -m g:$group_name:rX -R $directory</tt>
* Set the permissions for the existing files and directories:
** <tt>setfacl -m g:$group_name:rX -R $directory</tt>

This will give people in your group the ability to read files in the 'share' directory. If you also want them to be able to write or modify files in that directory then use change the ':rX' to ':rwX' instead. e.g. 'setfacl -d -m g:$group_name:rwX -R $directory' for both setfacl commands.

== How do I get more help? ==
There are many sources of help for most Linux systems.

=== Unix man pages ===
Linux provides man pages (short for manual pages). These are simple enough to call, for example: if you need information on submitting jobs to Beocat, you can type '<code>man sbatch</code>'. This will bring up the manual for sbatch.

=== GNU info system ===
Not all applications have "man pages." Most of the rest have what they call info pages. For example, if you needed information on finding a file you could use '<code>info find</code>'.

=== This documentation ===
This documentation is very thoroughly researched, and has been painstakingly assembled for your benefit. Please use it.

=== Contact support ===
Support can be contacted [mailto:beocat@cis.ksu.edu here]. Please include detailed information about your problem, including the job number, applications you are trying to run, and the current directory that you are in.

Main Page

2021-05-26T15:09:52Z

Kylehutson: Updated Freenode to Libera.chat

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of CentOS Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''beocat#beocat''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

==== Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] ====
==== Read about [[Installed software]] and languages ====
==== Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] ====
==== Run Interactive Jobs! [[OpenOnDemand]] ====

==== Big Data course on Beocat! [[BigDataOnBeocat]] ====

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

We are also available on IRC on the [https://libera.chat/guides/connect Libera chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''m0zes''' and/or '''kylehutson''' in your message, and it should grab our attention. [[Special:WebChat|Available from a web browser here.]]

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<H4>
Again, when you email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu] please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</H4>

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.
{{#widget:Twitter timeline|id=KSUBeocat|count=6}}

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the XSEDE (eXtreme Science and Engineering Discovery Environment) portal.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[XSEDE|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our XSEDE resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

LinuxBasics

2020-01-31T01:01:04Z

Kylehutson: Typo

'''Disclaimer:''' This is a ''very'' large topic, and much too broad to be covered on a single support page. There are many other sites (yes, entire sites) which cover the topic in more detail. We'll link so some of them below. This page is meant to be just the essentials.

== Logging in for the first time ==
To login to Beocat, you first need an "SSH Client". [[wikipedia:Secure_Shell|SSH]] (short for "secure shell") is a protocol that allows secure communication between two computers. We recommend the following.
* Windows
** [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] is by far the most common SSH client, both for Beocat and in the world.
** [http://mobaxterm.mobatek.net/ MobaXterm] is a fairly new client with some nice features, such as being able to SCP/SFTP (see below), and running X (which isn't terribly useful on Beocat, but might be if you connect to other Linux hosts).
** [http://www.cygwin.com/ Cygwin] is for those that would rather be running Linux but are stuck on Windows. It's purely a text interface.
* Macintosh
** OS-X has SSH a built-in application called "Terminal". It's not great, but it will work for most Beocat users.
** [http://www.iterm2.com/#/section/home iTerm2] is the terminal application we prefer.
* Others
** There are [[wikipedia:Comparison_of_SSH_clients|many SSH clients]] for many different platforms available. While we don't have experience with many of these, any should be sufficient for access to Beocat.

You'll need to connect your client (via the SSH protocol, if your client allows multiple protocols) to headnode.beocat.ksu.edu.

For command-line tools, the command to connect is
ssh ''username''@headnode.beocat.ksu.edu

Your username is your [http://eid.ksu.edu K-State eID] name and the password is your eID password.

'''Note:''' When you type your password, nothing shows up on the screen, not even asterisks.

You'll know you are successfully logged in when you see a prompt that says
(''machinename'':~) ''username''%
where ''machinename'' is the name of the machine you've logged into (currently either 'athena' or 'minerva') and ''username'' is your eID username

== Transferring files (SCP or SFTP) ==
Usually, one of the first things people want to do is to transfer files into or out of Beocat. To do so, you need to use [[wikipedia:Secure_copy|SCP]] (secure copy) or [[wikipedia:SSH_File_Transfer_Protocol|SFTP]] (SSH FTP or Secure FTP). Again, there are multiple programs that do this.
* Windows
** Putty (see above) has PSCP and PSFTP programs (both are included if you run the installer). It is a command-line interface (CLI) rather than a graphical user interface (GUI).
** MobaXterm (see above) has a built-in GUI SFTP client that automatically changes the directories as you change them in your SSH session.
** [https://filezilla-project.org/ FileZilla] (client) has an easy-to-use GUI. Be sure to use 'SFTP' mode rather than 'FTP' mode.
** [http://winscp.net/eng/index.php WinSCP] is another easy-to-use GUI.
** Cygwin (see above) has CLI scp and sftp programs.
* Macintosh
** [https://filezilla-project.org/ FileZilla] is also available for OS-X.
** Within terminal or iTerm, you can use the 'scp' or 'sftp' programs.
* Linux
** FileZilla also has a GUI linux version, in addition to the CLI tools.

=== Using a Command-Line Interface (CLI) ===
You can safely ignore this section if you're using a graphical interface (GUI). We highly recommend using a GUI when first learning how to use Beocat.

First test case: transfer a file called myfile.txt in your current folder to your home directory on Beocat. For these examples, I use bold text to show what you type and plain text to show Beocat's response

Using SCP:
'''scp myfile.txt ''username''@headnode.beocat.ksu.edu:'''
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

Note the colon at the end of the 'scp' line.

Using SFTP
'''sftp ''username''@headnode.beocat.ksu.edu'''
Password: '''(type your password here, it will not show any response on the screen)'''
Connected to headnode.beocat.ksu.edu.
sftp> '''put myfile.txt'''
Uploading myfile.txt to /homes/kylehutson/myfile.txt
myfile.txt 100% 0 0.0KB/s 00:00
sftp> '''exit'''

SFTP is interactive, so this is a two-step process. First, you connect to Beocat, then you transfer the file. As long as the system gives the <code>sftp> </code> prompt, you are in the sftp program, and you will remain there until you type 'exit'.

Second test case: transfer a file called myfile.txt in your current folder to a diretory named 'mydirectory' under your home directory on Beocat.

Here we run into one of the problems with scp - there is no easy way of creating 'mydirectory' if it doesn't already exist. If it does not already exist, you must login via ssh (as seen above) and create the directory using the 'mkdir' command (see Common Linux Commands) below.

'''scp myfile.txt ''username''@headnode.beocat.ksu.edu:mydirectory'''
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

An alternative version. If the colon is immediately followed by a slash, the directory name is taken from the root, rather than your home directory. So, given that your home directory on Beocat is /homes/''username'', we could instead type
'''scp myfile.txt ''username''@headnode.beocat.ksu.edu:/homes/''username''/mydirectory'''
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

Using SFTP:
sftp ''username''@headnode.beocat.ksu.edu
Password: '''(type your password here, it will not show any response on the screen)'''
Connected to headnode.beocat.ksu.edu.
sftp> '''mkdir mydirectory'''
[Note, if this directory already exists, you will get the response "Couldn't create directory: Failure"]
sftp> '''cd mydirectory'''
sftp> '''put myfile.txt'''
Uploading myfile.txt to /homes/''username''/mydirectory/myfile.txt
myfile.txt 100% 0 0.0KB/s 00:00
sftp> '''quit'''

Third test case: copy myfile.txt from your home directory on Beocat to your current folder.

Using scp:
scp ''username''@headnode.beocat.ksu.edu:myfile.txt .
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

Using SFTP:
'''sftp ''username''@headnode.beocat.ksu.edu'''
Password: '''(type your password here, it will not show any response on the screen)'''
Connected toheadnode.beocat.ksu.edu.
sftp> '''get myfile.txt'''
Fetching /homes/''username''/myfile.txt to myfile.txt
myfile.txt 100% 0 0.0KB/s 00:00
sftp> '''exit'''

== Basic Linux Commands ==
Again, this guide is very limited, mostly limited to directory navigation and basic file commands. [http://www.ee.surrey.ac.uk/Teaching/Unix/ Here] is a pretty decent tutorial if you want to dig deeper. If you want more, entire books have been written on the subject.

=== The Lingo ===
{| class="wikitable"
!''Term''
!''Definition''
|-
|Directory
|A "Folder" in Windows or OS-X terms. A location where files or other directories are stored. The current directory is sometimes represented as `.` and the parent directory can be referenced as `..`
|-
|Shell
|The interface or environment under which you can run commands. There is a section below on shells
|-
|SSH
|Secure Shell. A protocol that encrypts data and can give access to another system, usually by a username and password
|-
|SCP
|Secure Copy. Copying to or from a remote system using part of SSH
|-
|path
|The list of directories which are searched when you type the name of a program. There is a section below on this
|-
|ownership
|Every file and directory has an user and a group attached to it, called its owners. These affect permissions.
|-
|permissions
|The ability to read, write, and/or execute a file. Permissions are based on ownership
|-
|switches
|Modifiers to a command-line program, usually in the form of -(letter) or --``(word). Several examples are given below, such as the '-a' on the 'ls' command
|-
|pipes and redirects
|Changes the input (often called 'stdin') and/or output (often called stdout) to a program or a file
|}

=== Linux Command Line Cheat Sheet ===
{| class="wikitable"
|+File System Navigation
|-
!''Command''
!''What it does''
!''Example Usage''
!''Example Output''
|-
|pwd
|"Print working directory", Where am I now?
|<code>pwd</code>
|<code>/homes/mozes</code>
|-
|ls
|Lists files and folders
|<code>ls ~/</code>
|<code>NewFile NewFolder</code>
|-
|ls -lh
|Lists files and folders with perms size and ownership
|<code>ls -lh ~/</code>
|<code>-rw-r--r-- 1 mozes mozes_users 1 Jul 13 2011 NewFile
drwxr-xr-x 9 mozes mozes_users 9.0K Apr 12 2010 NewFolder</code>
|-
|ls -a
|Lists all files and folders
|<code>ls -a ~/</code>
|<code>. .. .bashrc .bash_profile .tcshrc NewFile NewFolder</code>
|-
|cd
|Changes directory
|<code>cd NewFolder</code>
|
|-
|cd ..
|Changes to parent directory
|<code>cd ..</code>
|
|-
|cd -
|Changes to the previous directory you were in
|<code>cd -</code>
|
|-
|cd ~
|Changes to your home directory
|<code>cd ~</code>
|
|}

{| class="wikitable"
|+Working with files
|-
!Command'
!What it does
!Example Usage'
!Example Output''
|-
|file
|Identifies the type of object a file is
|<code>file NewFile</code>
|<code>NewFile: a /usr/bin/python script, ASCII text executable</code>
|-
|cat
|Prints the contents of one or more files
|<code>cat NewFile</code>
|<code>This is line one
This is line two</code>
|-
|cp
|copy a file
|<code>cp OldFile NewFile</code>
|
|-
|cp -i
|copy a file, ask to overwrite
|<code>cp -i OldFile NewFile}</code>
|<code>overwrite NewFile? (y/n [n])</code>
|-
|cp -r
|copy a directory, including contents
|<code>cp -r OldFolder NewFolder</code>
|
|-
|mv
|move, or rename, a file
|<code>mv OldFile NewFile</code>
|
|-
|mv -i
|move, or rename, a file, ask to overwrite
|<code>mv -i OldFile NewFile</code>
|<code>overwrite NewFile? (y/n [n])</code>
|-
|rm
|remove a file
|<code>rm NewFile</code>
|
|-
|rm -i
|remove a file, ask to be sure (useful with -r)
|<code>rm -i NewFile</code>
|<code>remove NewFile? (y/n [n])</code>
|-
|rm -r
|remove a direcory and its contents
|<code>rm -r NewFolder</code>
|
|-
|mkdir
|creates a directory
|<code>mkdir TempFolder</code>
|
|-
|rmdir
|removes an empty directory
|<code>rmdir TempFolder</code>
|
|-
|touch
|creates an empty file
|<code>touch TempFile</code>
|
|}

{| class="wikitable"
|+Finding files and directories with [http://linux.die.net/man/1/find find]
|-
!''Command''
!''What it does''
!''Example Usage''
|-
| find <directory>
| finds all files and folders within <directory>
| find ~/
|-
| find <directory> -iname '<filename>'
| finds all files and directories within <directory> that match <filename>
| find ~/ -iname 'hello.qsub'
|-
| find <directory> -iname '*<partialmatch>*'
| finds all files and directories within <directory> that partially match <partialmatch>
| find ~/ -iname '*.qsub*'
|}

Other useful commands include <code>htop</code>, <code>less</code>, and <code>man</code>. <code>man</code> followed by a command name above will give you the manual page for the specified command full of many other useful options for the command. <code>htop</code> will give you an overview of the processes currently being run on the host you are connected to. <code>less</code> allows you to page through files and see their contents using <PgUp> and <PgDn>.

=== Editing Text Files ===
If you're new to Linux, the editor you will probably want to use is 'nano'. It works much the same as 'Notepad' in Windows or 'textedit' on OS-X. Note that you cannot use your mouse to change position within the document as you can with your local computer. You must use the arrow keys, instead.

So, if I wanted to edit my .bashrc (as shown below), and I was already in my home directory (see above), I would type
nano .bashrc

While in nano, there is a list of actions you can take at the bottom of the screen. <Ctrl> is represented by a caret (`^`), so to exit (labeled as `^`X at the bottom of the screen), I would type <ctrl>-x. This action prompts you whether you want to save and exit (Y), lose changes and exit (N), or cancel and go back to editing (<ctrl>-c).

If you do a significant amount of text editing in Linux, you'll probably want to switch to a more powerful editor, such as vim. The usage of vim is beyond the scope of this document. It is not at all intuitive to the beginning user, but with a little practice it becomes a much faster way of editing text files. If you're interested in using vim, [http://www.openvim.com/tutorial.html there is a nice tutorial here].

=== Shells ===
==== What is a Shell? ====
In this case, I don't believe I can do a better job explaining shells than [[wikipedia:Shell_(computing)|this]].
==== tcsh ====
Elsewhere at Kansas State University, the default Shell is set to tcsh. tcsh stands for "TENEX C SHell." It is considered a replacement for csh and uses many of the same features. If you have experience with either csh or tcsh you'll probably feel right at home. This was the default shell until July of 2013. If you had an account before then, it is probably still tcsh.

But what if you don't want or like tcsh, what can you do? Well, we have other shells available of Beocat as well.
==== bash ====
[http://www.gnu.org/software/bash/ Bash] seems to be the defacto standard shell in most Linux installs today. Bash is common and probably what most of you are used to. As of July 2013, bash is our new default shell. All new users will be set to bash initially. [https://software-carpentry.org/ Software Carpentry] teaches classes on several subjects specifically targeting researchers, including the bash shell. Their documentation is all freely available. [http://swcarpentry.github.io/shell-novice/ Here is a link to their excellent tutorial on using BASH.] Most of our documentation assumes you are using BASH.

bash configuration files:

This section gets into some minutiae with the way our job scheduler interacts with bash. If you're trying to solve a problem, read on, otherwise you can probably skip this section.

Bash has 3 user configurable configuration files, <code>~/.bashrc ~/.bash_profile</code> and <code>~/.bash_logout</code>. We'll look at the two more relevant ones <code>~/.bashrc</code> and <code>~/.bash_profile</code>

Bash has 3 ways of looking at things, '''login''', '''interactive''', or '''none'''.

Normally what happens is that shells that are '''interactive''' read <code>~/.bash_profile</code>, shells that are '''login''' read <code>~/.bashrc</code>. '''none''' shells read neither.

sbatch jobs are '''login''', srun jobs are '''login+interactive''', logging into Beocat in a way that you can enter commands is '''login+interactive'''. There are very few cases that you will get '''none'''. For any session that isn't '''interactive''', your sourced files cannot output anything to the screen, or else it can break scp or sftp file transfers.

If they are ''quiet'' statements, and you want them in all shells, you can put them in your <code>~/.bashrc</code>. If they are not ''quiet'' or they output ''anything'' to the screen, you must put them in your <code>~/.bash_profile</code>.

==== zsh ====
[http://zsh.sourceforge.net/ zsh] is an alternative to bash and tcsh. It tends to support more complex features than either of the other two while using a syntax remarkably similar to bash. Unless specifically noted, when we specify '''Change your shell to bash''', <tt>zsh</tt> should work as well.

==== Changing Shells ====
Previously, we gave you the option of using a <code>~/.login</code> to modify your shell. This is no longer supported, if you have issues with your shell/paths/environment variables we will ask you to delete your <code>~/.login</code> file and change your shell via the method below.

You can change your shell is via <code>chsh</code> on either of the headnodes (athena/minerva). This does not need to be re-done if you've changed to it to your preferred shell in the past.

Use the appropriate of the following three lines:
<syntaxhighlight lang="bash" line>
/usr/local/bin/chsh -s bash && bash -l
/usr/local/bin/chsh -s tcsh && tcsh -l
/usr/local/bin/chsh -s zsh && zsh -l
</syntaxhighlight>

=== Changing your PATH ===
Typically, you don't have to change your PATH, but it is useful to know what your PATH is and what it does. The PATH is the list of directories which are searched when you type the name of a program. Note that by default the current directory is NOT included in the path, so if you were wanting to run a program called MyProgram in the current directory, you could NOT simply type 'MyProgram', you would instead type <code>'./MyProgram'</code> (where the '.' represents the current directory).

To find your PATH, we need to identify which shell you are using. If you do not know, run the following:
<syntaxhighlight lang="bash" line>
ps | awk '/sh/ {print $4}'
</syntaxhighlight>

==== tcsh ====
You'll need to edit a file in your home directory called .tcshrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.
<syntaxhighlight lang="bash" line>
setenv PATH /usr/local/bin:$PATH
</syntaxhighlight>
==== bash ====
You'll need to edit a file in your home directory called .bashrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.
<syntaxhighlight lang="bash" line>
export PATH=/usr/local/bin:$PATH
</syntaxhighlight>

==== zsh ====
You'll need to edit a file in your home directory called .zshrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.
<syntaxhighlight lang="bash" line>
export PATH="/usr/local/bin:$PATH"
</syntaxhighlight>

=== Ownership and Permissions ===
Every file and directory has a user and group associated with it. You can view ownership information by using the '-l' switch on ls. By default on Beocat, files you create have a user ownership of your username (i.e., your eID) and a group ownership of your username_users. So, if I were logged in as 'myusername' and I had a single file in my home directory called MyProgram, the result of typing 'ls -l' would be something like this:
total 0
-rwxr-x--- 1 myusername myusername_users 79 May 31 2011 MyProgram
This tells us several things.
* The first column ('-rwxr-x---') is permissions (covered below)
* The second column ('1') is the number of links to this file. You can safely ignore this (unless you're both masochistic and interested in filesystem details)
* The third column ('myusername') shows the user ownership
* The fourth column ('myusername_users') shows the group ownership
* The fifth column ('79') gives the size of the file in bytes
* The next columns ('May 31 2011'), as you have probably guessed, gives the date the file was last changed
* The final column ('MyProgram') is the name of the file

So why is this interesting to us? Because whenever things ''don't'' work, it's usually because of file ownership or permissions. Looking at these often gives us some useful diagnostic information.

The permissions field shows us who has permissions to do what with this file. It is always 10 characters. The first character (-) is usually either a '-' for a regular file or a 'd' for a directory. The next 9 characters are broken into three groups of three, with each group showing read (r), write (w), and execute (x) permissions for the owner, group, and world, in that order.
* The first group (rwx) shows permissions for the owner (myusername). The owner here has read, write, and execute permissions
* The next group (r-x) shows permissions for the group (myusername_users). The group here has read and execute permissions, but cannot write.
* The last group (---) shows permissions for the rest of the world. The world has no permissions to read, write, or execute.

When you create a shell script with a text editor, and sometimes when you copy programs to Beocat via SCP, the execute flag is not set. The permissions string may look more like (-rw-r--r--). To change this, you need to give yourself permission to execute this program. This is done with the 'chmod' (change mode) command. 'chmod' can have a long and confusing syntax, but since by far the most common problem is to give yourself execute permissions, here is the command to change that:
chmod u+x MyProgram
This changes the permissions so that the user ('u', i.e., the owner) adds ('+') execute permission ('x').

For more complex ownership or permissions changes, please feel free to contact the Beocat staff.

=== Manual (man) pages ===
Most commands have a complex set of switches that will modify the amount or type of information they display. To find out what switches are available, or how a program expects data, you can use the manual pages by typing "`man` ''command''". Using one of the most common Linux commands, take a look the output of 'man ls'. It shows that it has over 50 switches available, ranging from which files to include, to how to display file sizes, to sort order and more. (I'm not pasting it here, because it's over 200 lines long!) To navigate a 'manpage', use the up-arrow and down-arrow keys. Press 'q' to quit.

=== Pipes and Redirects ===
Typically a Linux program takes data from the keyboard and outputs data to the screen. In Unix and Linux terminology, the keyboard is the default 'stdin' (pronounced "standard in") and the screen is the default 'stdout' (pronounced "standard out"). Many times, we want to take data from somewhere else (like a file, or the output of another program) and send it to yet another location. These redirectors are:
{|
|cmd > filename
|Redirect output from cmd to filename ||
|-
|cmd >> filename
|Redirect output from cmd and append to filename
|-
|cmd < filename
|Redirect input from cmd to filename
|-
| cmd1 | cmd2
| Use the output from cmd1 as the input to cmd2
|}
Here is a quick example. Let's say I have a thousands of files in a directory, and I want a list of those that end in '.sh'
'ls' by itself scrolls so far I can't see even a fraction of them. So, I redirect the output to a file
ls > ~/filelist.txt
That gives me all the files in the current folder and saves them in my home directory in 'filelist.txt'.
A quick look through the file in my favorite editor tells me this is still going to take too long, so I need another step. The 'grep' program is a commonly-used program to perform pattern matching. The syntax of 'grep' is beyond the scope of this document, but take my word for it that
grep '\.sh$'
will return all lines that end in .sh.

We can now redirect the input from grep to the file we just created:
grep '\.sh$' < ~/filelist.txt
Great! We now have our list. However, we wanted to save this as filelist.txt, and instead we have another list that we have to copy-and-paste. Instead of redirecting to a file, we'll use the vertical bar '|' (which we often term a "pipe") to send the output of one command to another.
ls | grep '\.sh$' > ~/filelist.txt
This time the output of 'ls' is ''not'' redirected to a file, but is redirected to the next command (grep). The output of grep (which is all our .sh files) instead of being sent to the screen is redirected to the file ~/filelist.txt.

This example is a very simple demonstration of how pipes and redirects work. Many more examples with complex structures can be found at http://www.ibm.com/developerworks/linux/library/l-lpic1-v3-103-4/index.html

FAQ

2019-12-10T03:16:04Z

Kylehutson: /* Help! My job isn't going to finish in the time I specified. Can I change the time requirement? */

== How do I connect to Beocat ==
{| class="wikitable"
|-
! colspan="2" | Connection Settings
|-
! Hostname
| style="text-align:right" | headnode.beocat.ksu.edu
|-
! Port
| style="text-align:right" | 22
|-
! Username
| style="text-align:right" | <tt>eID</tt>
|-
! Password
| style="text-align:right" | <tt>eID Password</tt>
|}

{| class="wikitable"
!colspan="2" | Supported Connection Software (Latest Versions of Each)
|-
!rowspan="3" | Shell
|-
| [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]
|-
| ssh from openssh
|-
!rowspan="4" | File Transfer Utilities
|-
| [https://filezilla-project.org/ Filezilla]
|-
| [http://winscp.net/ WinSCP]
|-
| scp and sftp from openssh
|-
!rowspan="2" | Combination
|-
| [http://mobaxterm.mobatek.net/ MobaXterm]
|}

== How do I compile my programs? ==
=== Serial programs ===
==== Fortran ====
<tt>ifort</tt> or <tt>gfortran</tt>
==== C/C++ ====
<tt>icc</tt>, <tt>gcc</tt> and <tt>g++</tt>
=== Parallel programs ===
==== Fortran ====
<tt>mpif77</tt> or <tt>mpif90</tt>
==== C/C++ ====
<tt>mpicc</tt> or <tt>mpic++</tt>
== How are the filesystems on Beocat set up? ==

{| class="wikitable"
|-
! Mountpoint !! Local / Shared !! Size !! Filesystem !! Advice
|-
| /bulk || Shared || 2.1PB shared with /bulk and /scratch || cephfs || Slower than /homes; very old files are culled automatically
|-
| /homes || Shared || 2.1PB shared with /bulk and /scratch || cephfs || Good enough for most jobs; limited to 1TB per home directory
|-
| /scratch || Shared || 2.1PB shared with /bulk and /homes || cephfs || Fast shared tmp space; files not used in 30 days are automatically culled
|-
| /tmp || Local || >100GB (varies per node) || ext4 || Good for I/O intensive jobs
|-
|}
=== Usage Advice ===
For most jobs you shouldn't need to worry, your default working directory
is your homedir and it will be fast enough for most tasks.
I/O intensive work should use /tmp, but you will need to remember to copy
your files to and from this partition as part of your job script. This is made
easier through the <tt>$TMPDIR</tt> environment variable in your jobs.

Example usage of <tt>$TMPDIR</tt> in a job script
<syntaxhighlight lang="bash" line>
#!/bin/bash

#copy our input file to $TMPDIR to make processing faster
cp ~/experiments/input.data $TMPDIR

#use the input file we copied over to the local system
#generate the output file in $TMPDIR as well
~/bin/my_program --input-file=$TMPDIR/input.data --output-file=$TMPDIR/output.data

#copy the results back from $TMPDIR
cp $TMPDIR/output.data ~/experiments/results.$SLURM_JOB_ID
</syntaxhighlight>

You need to remember to copy over your data from <tt>$TMPDIR</tt> as part of your job.
That directory and its contents are deleted when the job is complete.

== What is "killable:1" or "killable:0" ==
On Beocat, some of the machines have been purchased by specific users and/or groups. These users and/or groups get guaranteed access to their machines at any point in time. Often, these machines are sitting idle because the owners have no need for it at the time. This would be a significant waste of computational power if there were no other way to make use of the computing cycles.

=== Enter the "killable" resource ===
Killable (--gres=killable:1) jobs are jobs that can be scheduled to these "owned" machines by users outside of the true group of owners. If a "killable" job starts on one of these owned machines and the owner of said machine comes along and submits a job, the "killable" job will be returned to the queue, (killed off as it were), and restarted at some future point in time. The job will still complete at some future point, and if the job makes use of a checkpointing algorithm it may complete even faster. The trade off between marking a job "killable" and not, is that sometimes applications need a significant amount of runtime, and cannot resume running from a partial output, meaning that it may get restarted over and over again, never reaching the finish line. As such, we only auto-enable "killable" for relatively short jobs (<=24:00:00). Some users still feel this is a hinderance, so we created a way to tell us not to automatically mark short jobs "killable"

=== Disabling killable ===
Specifying --gres=killable:0 will tell us to not mark your job as killable.

=== The trade-off ===
If a job is marked killable, there are a non-trivial amount of additional nodes that the job can run on. If your job checkpoints itself, or is relatively short, there should be no downside to marking the job killable, as the job will probably start sooner. If your job is long-running and doesn't checkpoint (save its state to restart a previous session) itself, it could cause your job to take longer to complete.

== Help! When I submit my jobs I get "Warning To stay compliant with standard unix behavior, there should be a valid #! line in your script i.e. #!/bin/tcsh" ==
Job submission scripts are supposed to have a line similar to '<code>#!/bin/bash</code>' in them to start. We have had problems with people submitting jobs with invalid #! lines, so we enforce that rule. When this happens the job fails and we have to manually clean it up. The warning message is there just to inform you that the job script should have a line in it, in most cases #!/bin/tcsh or #!/bin/bash, to indicate what program should be used to run the script. When the line is missing from a script, by default your default shell is used to execute the script (in your case /usr/local/bin/tcsh). This works in most cases, but may not be what you are wanting.

== Help! When I submit my jobs I get "A #! line exists, but it is not pointing to an executable. Please fix. Job not submitted." ==
Like the above, error says you need a #!/bin/bash or similar line in your job script. This error says that while the line exists, the #! line isn't mentioning an executable file, thus the script will not be able to run. Most likely you wanted #!/bin/bash instead of something else.

== Help! My jobs keep dying after 1 hour and I don't know why ==
Beocat has default runtime limit of 1 hour. If you need more than that, or need more than 1 GB of memory per core, you'll want to look at the documentation [[SlurmBasics|here]] to see how to request it.

In short, when you run sbatch for your job, you'll want to put something along the lines of '<code>--time=0-10:00:00</code>' before the job script if you want your job to run for 10 hours.

== Help my error file has "Warning: no access to tty" ==
The warning message "Warning: no access to tty (Bad file descriptor)" is safe to ignore. It typically happens with the tcsh shell.

== Help! My job isn't going to finish in the time I specified. Can I change the time requirement? ==
Generally speaking, no.

Jobs are scheduled based on execution times (among other things). If it were easy to change your time requirement, one could submit a job with a 15-minute run-time, get it scheduled quickly, and then say "whoops - I meant 15 weeks", effectively gaming the job scheduler. In extreme circumstances and depending on the job requirements, we '''may''' be able to manually intervene. This process prevents other users from using the node(s) you are currently using, so are not routinely approved. Contact Beocat support (below) if you feel your circumstances warrant special consideration.

== Help! My perl job runs fine on the head node, but only runs for a few seconds and then quits when submitted to the queue. ==
Take a look at our documentation on [[Installed_software#Perl|Perl]]

== Help! When using mpi I get 'CMA: no RDMA devices found' or 'A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces' ==
This message simply means that some but not all nodes the job is running on have infiniband cards. The job will still run, but will not use the fastest interconnect we have available. This may or may not be an issue, depending on how message heavy your job is. If you would like to not see this warning, you may request infiniband as a resource when submitting your job. <code>--gres=fabric:ib:1</code>

== What happens to my data when I leave K-State? ==
First of all, although we use eid credentials, we are not tied in with K-State's central IT policies which apply to employees or students leaving the university. As long as you keep your eid password current, you still have access to Beocat. Once we deem your data to be "stale", we will archive your data and disable your account. We have no written policy on when we do this, because we only do so as necessity dictates, but generally speaking if you have any data which is modified for less than two years will not be marked as stale. If your account is disabled for this reason, you will have to apply for a new account and un-archive your data.

== Common Storage For Projects ==
Sometimes it is useful for groups of people to have a common storage area.

If you do not have a project, send a request via email to beocat@cs.ksu.edu. Note that these projects are generally reserved for tenure-track faculty and with a single project per eID.

If you already have a project you can do the following:

'''Note:''' The <tt>$group_name</tt> variable in the commands below needs to be replaced with the lower case name of your project. Membership of the projects can be done [https://account.beocat.ksu.edu/project here]
* Create a directory in one of the home directories of someone in your group, ideally the project owner's.
** <tt>mkdir $directory</tt>
* Change the group to the name assigned by the Beocat admins
** <tt>chgrp -R $group_name $directory</tt>
* Set the directory writeable and sticky for the group
** <tt>chmod g+ws $directory</tt>

* Change your umask to 002 (there will probably be a setting for it in your file transfer utilities, also). This step needs to be done by all group members.
** <syntaxhighlight lang=bash>
[ "$GROUPS" == "$(getent group $group_name | cut -d: -f3)" ] || newgrp $group_name
umask 002
</syntaxhighlight> It needs to go at the very end of your <tt>.bashrc</tt> file.

* Finally logout and log back in

== How do I get more help? ==
There are many sources of help for most Linux systems.

=== Unix man pages ===
Linux provides man pages (short for manual pages). These are simple enough to call, for example: if you need information on submitting jobs to Beocat, you can type '<code>man sbatch</code>'. This will bring up the manual for sbatch.

=== GNU info system ===
Not all applications have "man pages." Most of the rest have what they call info pages. For example, if you needed information on finding a file you could use '<code>info find</code>'.

=== This documentation ===
This documentation is very thoroughly researched, and has been painstakingly assembled for your benefit. Please use it.

=== Contact support ===
Support can be contacted [mailto:beocat@cis.ksu.edu here]. Please include detailed information about your problem, including the job number, applications you are trying to run, and the current directory that you are in.

FAQ

2019-12-04T19:46:03Z

Kylehutson: Clarified how to create a new project.

== How do I connect to Beocat ==
{| class="wikitable"
|-
! colspan="2" | Connection Settings
|-
! Hostname
| style="text-align:right" | headnode.beocat.ksu.edu
|-
! Port
| style="text-align:right" | 22
|-
! Username
| style="text-align:right" | <tt>eID</tt>
|-
! Password
| style="text-align:right" | <tt>eID Password</tt>
|}

{| class="wikitable"
!colspan="2" | Supported Connection Software (Latest Versions of Each)
|-
!rowspan="3" | Shell
|-
| [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]
|-
| ssh from openssh
|-
!rowspan="4" | File Transfer Utilities
|-
| [https://filezilla-project.org/ Filezilla]
|-
| [http://winscp.net/ WinSCP]
|-
| scp and sftp from openssh
|-
!rowspan="2" | Combination
|-
| [http://mobaxterm.mobatek.net/ MobaXterm]
|}

== How do I compile my programs? ==
=== Serial programs ===
==== Fortran ====
<tt>ifort</tt> or <tt>gfortran</tt>
==== C/C++ ====
<tt>icc</tt>, <tt>gcc</tt> and <tt>g++</tt>
=== Parallel programs ===
==== Fortran ====
<tt>mpif77</tt> or <tt>mpif90</tt>
==== C/C++ ====
<tt>mpicc</tt> or <tt>mpic++</tt>
== How are the filesystems on Beocat set up? ==

{| class="wikitable"
|-
! Mountpoint !! Local / Shared !! Size !! Filesystem !! Advice
|-
| /bulk || Shared || 2.1PB shared with /bulk and /scratch || cephfs || Slower than /homes; very old files are culled automatically
|-
| /homes || Shared || 2.1PB shared with /bulk and /scratch || cephfs || Good enough for most jobs; limited to 1TB per home directory
|-
| /scratch || Shared || 2.1PB shared with /bulk and /homes || cephfs || Fast shared tmp space; files not used in 30 days are automatically culled
|-
| /tmp || Local || >100GB (varies per node) || ext4 || Good for I/O intensive jobs
|-
|}
=== Usage Advice ===
For most jobs you shouldn't need to worry, your default working directory
is your homedir and it will be fast enough for most tasks.
I/O intensive work should use /tmp, but you will need to remember to copy
your files to and from this partition as part of your job script. This is made
easier through the <tt>$TMPDIR</tt> environment variable in your jobs.

Example usage of <tt>$TMPDIR</tt> in a job script
<syntaxhighlight lang="bash" line>
#!/bin/bash

#copy our input file to $TMPDIR to make processing faster
cp ~/experiments/input.data $TMPDIR

#use the input file we copied over to the local system
#generate the output file in $TMPDIR as well
~/bin/my_program --input-file=$TMPDIR/input.data --output-file=$TMPDIR/output.data

#copy the results back from $TMPDIR
cp $TMPDIR/output.data ~/experiments/results.$SLURM_JOB_ID
</syntaxhighlight>

You need to remember to copy over your data from <tt>$TMPDIR</tt> as part of your job.
That directory and its contents are deleted when the job is complete.

== What is "killable:1" or "killable:0" ==
On Beocat, some of the machines have been purchased by specific users and/or groups. These users and/or groups get guaranteed access to their machines at any point in time. Often, these machines are sitting idle because the owners have no need for it at the time. This would be a significant waste of computational power if there were no other way to make use of the computing cycles.

=== Enter the "killable" resource ===
Killable (--gres=killable:1) jobs are jobs that can be scheduled to these "owned" machines by users outside of the true group of owners. If a "killable" job starts on one of these owned machines and the owner of said machine comes along and submits a job, the "killable" job will be returned to the queue, (killed off as it were), and restarted at some future point in time. The job will still complete at some future point, and if the job makes use of a checkpointing algorithm it may complete even faster. The trade off between marking a job "killable" and not, is that sometimes applications need a significant amount of runtime, and cannot resume running from a partial output, meaning that it may get restarted over and over again, never reaching the finish line. As such, we only auto-enable "killable" for relatively short jobs (<=24:00:00). Some users still feel this is a hinderance, so we created a way to tell us not to automatically mark short jobs "killable"

=== Disabling killable ===
Specifying --gres=killable:0 will tell us to not mark your job as killable.

=== The trade-off ===
If a job is marked killable, there are a non-trivial amount of additional nodes that the job can run on. If your job checkpoints itself, or is relatively short, there should be no downside to marking the job killable, as the job will probably start sooner. If your job is long-running and doesn't checkpoint (save its state to restart a previous session) itself, it could cause your job to take longer to complete.

== Help! When I submit my jobs I get "Warning To stay compliant with standard unix behavior, there should be a valid #! line in your script i.e. #!/bin/tcsh" ==
Job submission scripts are supposed to have a line similar to '<code>#!/bin/bash</code>' in them to start. We have had problems with people submitting jobs with invalid #! lines, so we enforce that rule. When this happens the job fails and we have to manually clean it up. The warning message is there just to inform you that the job script should have a line in it, in most cases #!/bin/tcsh or #!/bin/bash, to indicate what program should be used to run the script. When the line is missing from a script, by default your default shell is used to execute the script (in your case /usr/local/bin/tcsh). This works in most cases, but may not be what you are wanting.

== Help! When I submit my jobs I get "A #! line exists, but it is not pointing to an executable. Please fix. Job not submitted." ==
Like the above, error says you need a #!/bin/bash or similar line in your job script. This error says that while the line exists, the #! line isn't mentioning an executable file, thus the script will not be able to run. Most likely you wanted #!/bin/bash instead of something else.

== Help! My jobs keep dying after 1 hour and I don't know why ==
Beocat has default runtime limit of 1 hour. If you need more than that, or need more than 1 GB of memory per core, you'll want to look at the documentation [[SlurmBasics|here]] to see how to request it.

In short, when you run sbatch for your job, you'll want to put something along the lines of '<code>--time=0-10:00:00</code>' before the job script if you want your job to run for 10 hours.

== Help my error file has "Warning: no access to tty" ==
The warning message "Warning: no access to tty (Bad file descriptor)" is safe to ignore. It typically happens with the tcsh shell.

== Help! My job isn't going to finish in the time I specified. Can I change the time requirement? ==
Generally speaking, no.

Jobs are scheduled based on execution times (among other things). If it were easy to change your time requirement, one could submit a job with a 15-minute run-time, get it scheduled quickly, and then say "whoops - I meant 15 weeks", effectively gaming the job scheduler. In fact, even the administrators cannot change the run-time requirement of a particular job. In extreme circumstances and depending on the job requirements, we '''may''' be able to manually intervene. This process prevents other users from using the node(s) you are currently using, so are not routinely approved. Contact Beocat support (below) if you feel your circumstances warrant special consideration.

== Help! My perl job runs fine on the head node, but only runs for a few seconds and then quits when submitted to the queue. ==
Take a look at our documentation on [[Installed_software#Perl|Perl]]

== Help! When using mpi I get 'CMA: no RDMA devices found' or 'A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces' ==
This message simply means that some but not all nodes the job is running on have infiniband cards. The job will still run, but will not use the fastest interconnect we have available. This may or may not be an issue, depending on how message heavy your job is. If you would like to not see this warning, you may request infiniband as a resource when submitting your job. <code>--gres=fabric:ib:1</code>

== What happens to my data when I leave K-State? ==
First of all, although we use eid credentials, we are not tied in with K-State's central IT policies which apply to employees or students leaving the university. As long as you keep your eid password current, you still have access to Beocat. Once we deem your data to be "stale", we will archive your data and disable your account. We have no written policy on when we do this, because we only do so as necessity dictates, but generally speaking if you have any data which is modified for less than two years will not be marked as stale. If your account is disabled for this reason, you will have to apply for a new account and un-archive your data.

== Common Storage For Projects ==
Sometimes it is useful for groups of people to have a common storage area.

If you do not have a project, send a request via email to beocat@cs.ksu.edu. Note that these projects are generally reserved for tenure-track faculty and with a single project per eID.

If you already have a project you can do the following:

'''Note:''' The <tt>$group_name</tt> variable in the commands below needs to be replaced with the lower case name of your project. Membership of the projects can be done [https://account.beocat.ksu.edu/project here]
* Create a directory in one of the home directories of someone in your group, ideally the project owner's.
** <tt>mkdir $directory</tt>
* Change the group to the name assigned by the Beocat admins
** <tt>chgrp -R $group_name $directory</tt>
* Set the directory writeable and sticky for the group
** <tt>chmod g+ws $directory</tt>

* Change your umask to 002 (there will probably be a setting for it in your file transfer utilities, also). This step needs to be done by all group members.
** <syntaxhighlight lang=bash>
[ "$GROUPS" == "$(getent group $group_name | cut -d: -f3)" ] || newgrp $group_name
umask 002
</syntaxhighlight> It needs to go at the very end of your <tt>.bashrc</tt> file.

* Finally logout and log back in

== How do I get more help? ==
There are many sources of help for most Linux systems.

=== Unix man pages ===
Linux provides man pages (short for manual pages). These are simple enough to call, for example: if you need information on submitting jobs to Beocat, you can type '<code>man sbatch</code>'. This will bring up the manual for sbatch.

=== GNU info system ===
Not all applications have "man pages." Most of the rest have what they call info pages. For example, if you needed information on finding a file you could use '<code>info find</code>'.

=== This documentation ===
This documentation is very thoroughly researched, and has been painstakingly assembled for your benefit. Please use it.

=== Contact support ===
Support can be contacted [mailto:beocat@cis.ksu.edu here]. Please include detailed information about your problem, including the job number, applications you are trying to run, and the current directory that you are in.

LinuxBasics

2019-09-09T22:07:27Z

Kylehutson: Added link to SW carpentry's tutorial

'''Disclaimer:''' This is a ''very'' large topic, and much too broad to be covered on a single support page. There are many other sites (yes, entire sites) which cover the topic in more detail. We'll link so some of them below. This page is meant to be just the essentials.

== Logging in for the first time ==
To login to Beocat, you first need an "SSH Client". [[wikipedia:Secure_Shell|SSH]] (short for "secure shell") is a protocol that allows secure communication between two computers. We recommend the following.
* Windows
** [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] is by far the most common SSH client, both for Beocat and in the world.
** [http://mobaxterm.mobatek.net/ MobaXterm] is a fairly new client with some nice features, such as being able to SCP/SFTP (see below), and running X (which isn't terribly useful on Beocat, but might be if you connect to other Linux hosts).
** [http://www.cygwin.com/ Cygwin] is for those that would rather be running Linux but are stuck on Windows. It's purely a text interface.
* Macintosh
** OS-X has SSH a built-in application called "Terminal". It's not great, but it will work for most Beocat users.
** [http://www.iterm2.com/#/section/home iTerm2] is the terminal application we prefer.
* Others
** There are [[wikipedia:Comparison_of_SSH_clients|many SSH clients]] for many different platforms available. While we don't have experience with many of these, any should be sufficient for access to Beocat.

You'll need to connect your client (via the SSH protocol, if your client allows multiple protocols) to headnode.beocat.ksu.edu.

For command-line tools, the command to connect is
ssh ''username''@headnode.beocat.ksu.edu

Your username is your [http://eid.ksu.edu K-State eID] name and the password is your eID password.

'''Note:''' When you type your password, nothing shows up on the screen, not even asterisks.

You'll know you are successfully logged in when you see a prompt that says
(''machinename'':~) ''username''%
where ''machinename'' is the name of the machine you've logged into (currently either 'athena' or 'minerva') and ''username'' is your eID username

== Transferring files (SCP or SFTP) ==
Usually, one of the first things people want to do is to transfer files into or out of Beocat. To do so, you need to use [[wikipedia:Secure_copy|SCP]] (secure copy) or [[wikipedia:SSH_File_Transfer_Protocol|SFTP]] (SSH FTP or Secure FTP). Again, there are multiple programs that do this.
* Windows
** Putty (see above) has PSCP and PSFTP programs (both are included if you run the installer). It is a command-line interface (CLI) rather than a graphical user interface (GUI).
** MobaXterm (see above) has a built-in GUI SFTP client that automatically changes the directories as you change them in your SSH session.
** [https://filezilla-project.org/ FileZilla] (client) has an easy-to-use GUI. Be sure to use 'SFTP' mode rather than 'FTP' mode.
** [http://winscp.net/eng/index.php WinSCP] is another easy-to-use GUI.
** Cygwin (see above) has CLI scp and sftp programs.
* Macintosh
** [https://filezilla-project.org/ FileZilla] is also available for OS-X.
** Within terminal or iTerm, you can use the 'scp' or 'sftp' programs.
* Linux
** FileZilla also has a GUI linux version, in additon to the CLI tools.

=== Using a Command-Line Interface (CLI) ===
You can safely ignore this section if you're using a graphical interface (GUI). We highly recommend using a GUI when first learning how to use Beocat.

First test case: transfer a file called myfile.txt in your current folder to your home directory on Beocat. For these examples, I use bold text to show what you type and plain text to show Beocat's response

Using SCP:
'''scp myfile.txt ''username''@headnode.beocat.ksu.edu:'''
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

Note the colon at the end of the 'scp' line.

Using SFTP
'''sftp ''username''@headnode.beocat.ksu.edu'''
Password: '''(type your password here, it will not show any response on the screen)'''
Connected to headnode.beocat.ksu.edu.
sftp> '''put myfile.txt'''
Uploading myfile.txt to /homes/kylehutson/myfile.txt
myfile.txt 100% 0 0.0KB/s 00:00
sftp> '''exit'''

SFTP is interactive, so this is a two-step process. First, you connect to Beocat, then you transfer the file. As long as the system gives the <code>sftp> </code> prompt, you are in the sftp program, and you will remain there until you type 'exit'.

Second test case: transfer a file called myfile.txt in your current folder to a diretory named 'mydirectory' under your home directory on Beocat.

Here we run into one of the problems with scp - there is no easy way of creating 'mydirectory' if it doesn't already exist. If it does not already exist, you must login via ssh (as seen above) and create the directory using the 'mkdir' command (see Common Linux Commands) below.

'''scp myfile.txt ''username''@headnode.beocat.ksu.edu:mydirectory'''
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

An alternative version. If the colon is immediately followed by a slash, the directory name is taken from the root, rather than your home directory. So, given that your home directory on Beocat is /homes/''username'', we could instead type
'''scp myfile.txt ''username''@headnode.beocat.ksu.edu:/homes/''username''/mydirectory'''
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

Using SFTP:
sftp ''username''@headnode.beocat.ksu.edu
Password: '''(type your password here, it will not show any response on the screen)'''
Connected to headnode.beocat.ksu.edu.
sftp> '''mkdir mydirectory'''
[Note, if this directory already exists, you will get the response "Couldn't create directory: Failure"]
sftp> '''cd mydirectory'''
sftp> '''put myfile.txt'''
Uploading myfile.txt to /homes/''username''/mydirectory/myfile.txt
myfile.txt 100% 0 0.0KB/s 00:00
sftp> '''quit'''

Third test case: copy myfile.txt from your home directory on Beocat to your current folder.

Using scp:
scp ''username''@headnode.beocat.ksu.edu:myfile.txt .
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

Using SFTP:
'''sftp ''username''@headnode.beocat.ksu.edu'''
Password: '''(type your password here, it will not show any response on the screen)'''
Connected toheadnode.beocat.ksu.edu.
sftp> '''get myfile.txt'''
Fetching /homes/''username''/myfile.txt to myfile.txt
myfile.txt 100% 0 0.0KB/s 00:00
sftp> '''exit'''

== Basic Linux Commands ==
Again, this guide is very limited, mostly limited to directory navigation and basic file commands. [http://www.ee.surrey.ac.uk/Teaching/Unix/ Here] is a pretty decent tutorial if you want to dig deeper. If you want more, entire books have been written on the subject.

=== The Lingo ===
{| class="wikitable"
!''Term''
!''Definition''
|-
|Directory
|A "Folder" in Windows or OS-X terms. A location where files or other directories are stored. The current directory is sometimes represented as `.` and the parent directory can be referenced as `..`
|-
|Shell
|The interface or environment under which you can run commands. There is a section below on shells
|-
|SSH
|Secure Shell. A protocol that encrypts data and can give access to another system, usually by a username and password
|-
|SCP
|Secure Copy. Copying to or from a remote system using part of SSH
|-
|path
|The list of directories which are searched when you type the name of a program. There is a section below on this
|-
|ownership
|Every file and directory has an user and a group attached to it, called its owners. These affect permissions.
|-
|permissions
|The ability to read, write, and/or execute a file. Permissions are based on ownership
|-
|switches
|Modifiers to a command-line program, usually in the form of -(letter) or --``(word). Several examples are given below, such as the '-a' on the 'ls' command
|-
|pipes and redirects
|Changes the input (often called 'stdin') and/or output (often called stdout) to a program or a file
|}

=== Linux Command Line Cheat Sheet ===
{| class="wikitable"
|+File System Navigation
|-
!''Command''
!''What it does''
!''Example Usage''
!''Example Output''
|-
|pwd
|"Print working directory", Where am I now?
|<code>pwd</code>
|<code>/homes/mozes</code>
|-
|ls
|Lists files and folders
|<code>ls ~/</code>
|<code>NewFile NewFolder</code>
|-
|ls -lh
|Lists files and folders with perms size and ownership
|<code>ls -lh ~/</code>
|<code>-rw-r--r-- 1 mozes mozes_users 1 Jul 13 2011 NewFile
drwxr-xr-x 9 mozes mozes_users 9.0K Apr 12 2010 NewFolder</code>
|-
|ls -a
|Lists all files and folders
|<code>ls -a ~/</code>
|<code>. .. .bashrc .bash_profile .tcshrc NewFile NewFolder</code>
|-
|cd
|Changes directory
|<code>cd NewFolder</code>
|
|-
|cd ..
|Changes to parent directory
|<code>cd ..</code>
|
|-
|cd -
|Changes to the previous directory you were in
|<code>cd -</code>
|
|-
|cd ~
|Changes to your home directory
|<code>cd ~</code>
|
|}

{| class="wikitable"
|+Working with files
|-
!Command'
!What it does
!Example Usage'
!Example Output''
|-
|file
|Identifies the type of object a file is
|<code>file NewFile</code>
|<code>NewFile: a /usr/bin/python script, ASCII text executable</code>
|-
|cat
|Prints the contents of one or more files
|<code>cat NewFile</code>
|<code>This is line one
This is line two</code>
|-
|cp
|copy a file
|<code>cp OldFile NewFile</code>
|
|-
|cp -i
|copy a file, ask to overwrite
|<code>cp -i OldFile NewFile}</code>
|<code>overwrite NewFile? (y/n [n])</code>
|-
|cp -r
|copy a directory, including contents
|<code>cp -r OldFolder NewFolder</code>
|
|-
|mv
|move, or rename, a file
|<code>mv OldFile NewFile</code>
|
|-
|mv -i
|move, or rename, a file, ask to overwrite
|<code>mv -i OldFile NewFile</code>
|<code>overwrite NewFile? (y/n [n])</code>
|-
|rm
|remove a file
|<code>rm NewFile</code>
|
|-
|rm -i
|remove a file, ask to be sure (useful with -r)
|<code>rm -i NewFile</code>
|<code>remove NewFile? (y/n [n])</code>
|-
|rm -r
|remove a direcory and its contents
|<code>rm -r NewFolder</code>
|
|-
|mkdir
|creates a directory
|<code>mkdir TempFolder</code>
|
|-
|rmdir
|removes an empty directory
|<code>rmdir TempFolder</code>
|
|-
|touch
|creates an empty file
|<code>touch TempFile</code>
|
|}

{| class="wikitable"
|+Finding files and directories with [http://linux.die.net/man/1/find find]
|-
!''Command''
!''What it does''
!''Example Usage''
|-
| find <directory>
| finds all files and folders within <directory>
| find ~/
|-
| find <directory> -iname '<filename>'
| finds all files and directories within <directory> that match <filename>
| find ~/ -iname 'hello.qsub'
|-
| find <directory> -iname '*<partialmatch>*'
| finds all files and directories within <directory> that partially match <partialmatch>
| find ~/ -iname '*.qsub*'
|}

Other useful commands include <code>htop</code>, <code>less</code>, and <code>man</code>. <code>man</code> followed by a command name above will give you the manual page for the specified command full of many other useful options for the command. <code>htop</code> will give you an overview of the processes currently being run on the host you are connected to. <code>less</code> allows you to page through files and see their contents using <PgUp> and <PgDn>.

=== Editing Text Files ===
If you're new to Linux, the editor you will probably want to use is 'nano'. It works much the same as 'Notepad' in Windows or 'textedit' on OS-X. Note that you cannot use your mouse to change position within the document as you can with your local computer. You must use the arrow keys, instead.

So, if I wanted to edit my .bashrc (as shown below), and I was already in my home directory (see above), I would type
nano .bashrc

While in nano, there is a list of actions you can take at the bottom of the screen. <Ctrl> is represented by a caret (`^`), so to exit (labeled as `^`X at the bottom of the screen), I would type <ctrl>-x. This action prompts you whether you want to save and exit (Y), lose changes and exit (N), or cancel and go back to editing (<ctrl>-c).

If you do a significant amount of text editing in Linux, you'll probably want to switch to a more powerful editor, such as vim. The usage of vim is beyond the scope of this document. It is not at all intuitive to the beginning user, but with a little practice it becomes a much faster way of editing text files. If you're interested in using vim, [http://www.openvim.com/tutorial.html|there is a nice tutorial here].

=== Shells ===
==== What is a Shell? ====
In this case, I don't believe I can do a better job explaining shells than [[wikipedia:Shell_(computing)|this]].
==== tcsh ====
Elsewhere at Kansas State University, the default Shell is set to tcsh. tcsh stands for "TENEX C SHell." It is considered a replacement for csh and uses many of the same features. If you have experience with either csh or tcsh you'll probably feel right at home. This was the default shell until July of 2013. If you had an account before then, it is probably still tcsh.

But what if you don't want or like tcsh, what can you do? Well, we have other shells available of Beocat as well.
==== bash ====
[http://www.gnu.org/software/bash/ Bash] seems to be the defacto standard shell in most Linux installs today. Bash is common and probably what most of you are used to. As of July 2013, bash is our new default shell. All new users will be set to bash initially. [https://software-carpentry.org/ Software Carpentry] teaches classes on several subjects specifically targeting researchers, including the bash shell. Their documentation is all freely available. [http://swcarpentry.github.io/shell-novice/ Here is a link to their excellent tutorial on using BASH.] Most of our documentation assumes you are using BASH.

bash configuration files:

This section gets into some minutiae with the way our job scheduler interacts with bash. If you're trying to solve a problem, read on, otherwise you can probably skip this section.

Bash has 3 user configurable configuration files, <code>~/.bashrc ~/.bash_profile</code> and <code>~/.bash_logout</code>. We'll look at the two more relevant ones <code>~/.bashrc</code> and <code>~/.bash_profile</code>

Bash has 3 ways of looking at things, '''login''', '''interactive''', or '''none'''.

Normally what happens is that shells that are '''interactive''' read <code>~/.bash_profile</code>, shells that are '''login''' read <code>~/.bashrc</code>. '''none''' shells read neither.

sbatch jobs are '''login''', srun jobs are '''login+interactive''', logging into Beocat in a way that you can enter commands is '''login+interactive'''. There are very few cases that you will get '''none'''. For any session that isn't '''interactive''', your sourced files cannot output anything to the screen, or else it can break scp or sftp file transfers.

If they are ''quiet'' statements, and you want them in all shells, you can put them in your <code>~/.bashrc</code>. If they are not ''quiet'' or they output ''anything'' to the screen, you must put them in your <code>~/.bash_profile</code>.

==== zsh ====
[http://zsh.sourceforge.net/ zsh] is an alternative to bash and tcsh. It tends to support more complex features than either of the other two while using a syntax remarkably similar to bash. Unless specifically noted, when we specify '''Change your shell to bash''', <tt>zsh</tt> should work as well.

==== Changing Shells ====
Previously, we gave you the option of using a <code>~/.login</code> to modify your shell. This is no longer supported, if you have issues with your shell/paths/environment variables we will ask you to delete your <code>~/.login</code> file and change your shell via the method below.

You can change your shell is via <code>chsh</code> on either of the headnodes (athena/minerva). This does not need to be re-done if you've changed to it to your preferred shell in the past.

Use the appropriate of the following three lines:
<syntaxhighlight lang="bash" line>
/usr/local/bin/chsh -s bash && bash -l
/usr/local/bin/chsh -s tcsh && tcsh -l
/usr/local/bin/chsh -s zsh && zsh -l
</syntaxhighlight>

=== Changing your PATH ===
Typically, you don't have to change your PATH, but it is useful to know what your PATH is and what it does. The PATH is the list of directories which are searched when you type the name of a program. Note that by default the current directory is NOT included in the path, so if you were wanting to run a program called MyProgram in the current directory, you could NOT simply type 'MyProgram', you would instead type <code>'./MyProgram'</code> (where the '.' represents the current directory).

To find your PATH, we need to identify which shell you are using. If you do not know, run the following:
<syntaxhighlight lang="bash" line>
ps | awk '/sh/ {print $4}'
</syntaxhighlight>

==== tcsh ====
You'll need to edit a file in your home directory called .tcshrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.
<syntaxhighlight lang="bash" line>
setenv PATH /usr/local/bin:$PATH
</syntaxhighlight>
==== bash ====
You'll need to edit a file in your home directory called .bashrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.
<syntaxhighlight lang="bash" line>
export PATH=/usr/local/bin:$PATH
</syntaxhighlight>

==== zsh ====
You'll need to edit a file in your home directory called .zshrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.
<syntaxhighlight lang="bash" line>
export PATH="/usr/local/bin:$PATH"
</syntaxhighlight>

=== Ownership and Permissions ===
Every file and directory has a user and group associated with it. You can view ownership information by using the '-l' switch on ls. By default on Beocat, files you create have a user ownership of your username (i.e., your eID) and a group ownership of your username_users. So, if I were logged in as 'myusername' and I had a single file in my home directory called MyProgram, the result of typing 'ls -l' would be something like this:
total 0
-rwxr-x--- 1 myusername myusername_users 79 May 31 2011 MyProgram
This tells us several things.
* The first column ('-rwxr-x---') is permissions (covered below)
* The second column ('1') is the number of links to this file. You can safely ignore this (unless you're both masochistic and interested in filesystem details)
* The third column ('myusername') shows the user ownership
* The fourth column ('myusername_users') shows the group ownership
* The fifth column ('79') gives the size of the file in bytes
* The next columns ('May 31 2011'), as you have probably guessed, gives the date the file was last changed
* The final column ('MyProgram') is the name of the file

So why is this interesting to us? Because whenever things ''don't'' work, it's usually because of file ownership or permissions. Looking at these often gives us some useful diagnostic information.

The permissions field shows us who has permissions to do what with this file. It is always 10 characters. The first character (-) is usually either a '-' for a regular file or a 'd' for a directory. The next 9 characters are broken into three groups of three, with each group showing read (r), write (w), and execute (x) permissions for the owner, group, and world, in that order.
* The first group (rwx) shows permissions for the owner (myusername). The owner here has read, write, and execute permissions
* The next group (r-x) shows permissions for the group (myusername_users). The group here has read and execute permissions, but cannot write.
* The last group (---) shows permissions for the rest of the world. The world has no permissions to read, write, or execute.

When you create a shell script with a text editor, and sometimes when you copy programs to Beocat via SCP, the execute flag is not set. The permissions string may look more like (-rw-r--r--). To change this, you need to give yourself permission to execute this program. This is done with the 'chmod' (change mode) command. 'chmod' can have a long and confusing syntax, but since by far the most common problem is to give yourself execute permissions, here is the command to change that:
chmod u+x MyProgram
This changes the permissions so that the user ('u', i.e., the owner) adds ('+') execute permission ('x').

For more complex ownership or permissions changes, please feel free to contact the Beocat staff.

=== Manual (man) pages ===
Most commands have a complex set of switches that will modify the amount or type of information they display. To find out what switches are available, or how a program expects data, you can use the manual pages by typing "`man` ''command''". Using one of the most common Linux commands, take a look the output of 'man ls'. It shows that it has over 50 switches available, ranging from which files to include, to how to display file sizes, to sort order and more. (I'm not pasting it here, because it's over 200 lines long!) To navigate a 'manpage', use the up-arrow and down-arrow keys. Press 'q' to quit.

=== Pipes and Redirects ===
Typically a Linux program takes data from the keyboard and outputs data to the screen. In Unix and Linux terminology, the keyboard is the default 'stdin' (pronounced "standard in") and the screen is the default 'stdout' (pronounced "standard out"). Many times, we want to take data from somewhere else (like a file, or the output of another program) and send it to yet another location. These redirectors are:
{|
|cmd > filename
|Redirect output from cmd to filename ||
|-
|cmd >> filename
|Redirect output from cmd and append to filename
|-
|cmd < filename
|Redirect input from cmd to filename
|-
| cmd1 | cmd2
| Use the output from cmd1 as the input to cmd2
|}
Here is a quick example. Let's say I have a thousands of files in a directory, and I want a list of those that end in '.sh'
'ls' by itself scrolls so far I can't see even a fraction of them. So, I redirect the output to a file
ls > ~/filelist.txt
That gives me all the files in the current folder and saves them in my home directory in 'filelist.txt'.
A quick look through the file in my favorite editor tells me this is still going to take too long, so I need another step. The 'grep' program is a commonly-used program to perform pattern matching. The syntax of 'grep' is beyond the scope of this document, but take my word for it that
grep '\.sh$'
will return all lines that end in .sh.

We can now redirect the input from grep to the file we just created:
grep '\.sh$' < ~/filelist.txt
Great! We now have our list. However, we wanted to save this as filelist.txt, and instead we have another list that we have to copy-and-paste. Instead of redirecting to a file, we'll use the vertical bar '|' (which we often term a "pipe") to send the output of one command to another.
ls | grep '\.sh$' > ~/filelist.txt
This time the output of 'ls' is ''not'' redirected to a file, but is redirected to the next command (grep). The output of grep (which is all our .sh files) instead of being sent to the screen is redirected to the file ~/filelist.txt.

This example is a very simple demonstration of how pipes and redirects work. Many more examples with complex structures can be found at http://www.ibm.com/developerworks/linux/library/l-lpic1-v3-103-4/index.html

AdvancedSlurm

2019-09-09T21:59:52Z

Kylehutson: /* Globus */

== Resource Requests ==
Aside from the time, RAM, and CPU requirements listed on the [[SlurmBasics]] page, we have a couple other requestable resources:
Valid gres options are:
gpu[[:type]:count]
fabric[[:type]:count]
Generally, if you don't know if you need a particular resource, you should use the default. These can be generated with the command
<tt>srun --gres=help</tt>
=== Fabric ===
We currently offer 3 "fabrics" as request-able resources in Slurm. The "count" specified is the line-rate (in Gigabits-per-second) of the connection on the node.
==== Infiniband ====
First of all, let me state that just because it sounds "cool" doesn't mean you need it or even want it. InfiniBand does absolutely no good if running on a single machine. InfiniBand is a high-speed host-to-host communication fabric. It is (most-often) used in conjunction with MPI jobs (discussed below). Several times we have had jobs which could run just fine, except that the submitter requested InfiniBand, and all the nodes with InfiniBand were currently busy. In fact, some of our fastest nodes do not have InfiniBand, so by requesting it when you don't need it, you are actually slowing down your job. To request Infiniband, add <tt>--gres=fabric:ib:1</tt> to your sbatch command-line.
==== ROCE ====
ROCE, like InfiniBand is a high-speed host-to-host communication layer. Again, used most often with MPI. Most of our nodes are ROCE enabled, but this will let you guarantee the nodes allocated to your job will be able to communicate with ROCE. To request ROCE, add <tt>--gres=fabric:roce:1</tt> to your sbatch command-line.

==== Ethernet ====
Ethernet is another communication fabric. All of our nodes are connected by ethernet, this is simply here to allow you to specify the interconnect speed. Speeds are selected in units of Gbps, with all nodes supporting 1Gbps or above. The currently available speeds for ethernet are: <tt>1, 10, 40, and 100</tt>. To select nodes with 40Gbps and above, you could specify <tt>--gres=fabric:eth:40</tt> on your sbatch command-line. Since ethernet is used to connect to the file server, this can be used to select nodes that have fast access for applications doing heavy IO. The Dwarves and Heroes have 40 Gbps ethernet and we measure single stream performance as high as 20 Gbps, but if your application
requires heavy IO then you'd want to avoid the Moles which are connected to the file server with only 1 Gbps ethernet.

=== CUDA ===
[[CUDA]] is the resource required for GPU computing. 'kstat -g' will show you the GPU nodes and the jobs running on them. To request a GPU node, add <tt>--gres=gpu:1</tt> for example to request 1 GPU for your job. You can also request a given type of GPU (kstat -g -l to show types) by using <tt>--gres=gpu:nvidia_geforce_gtx_1080_ti:1</tt> for a 1080Ti GPU on the Wizards or Dwarves, <tt>--gres=gpu:nvidia_quadro_gp100:1</tt> for the P100 GPUs on Wizard20-21 that are best for 64-bit codes like Vasp, or <tt>--gres=gpu:nvidia_geforce_gtx_980_ti:1</tt> for the older 980Ti GPUs on Dwarf38-39. Most of these GPU nodes are owned by various groups. If you want access to GPU nodes and your group does not own any, we can add you to the <tt>--partition=ksu-gen-gpu.q</tt> group that has priority on Dwarf38-39. For more information on compiling CUDA code click on this [[CUDA]] link.

== Parallel Jobs ==
There are two ways jobs can run in parallel, ''intra''node and ''inter''node. '''Note: Beocat will not automatically make a job run in parallel.''' Have I said that enough? It's a common misperception.
=== Intranode jobs ===
''Intra''node jobs run on many cores in the same node. These jobs can take advantage of many common libraries, such as [http://openmp.org/wp/ OpenMP], or any programming language that has the concept of ''threads''. Often, your program will need to know how many cores you want it to use, and many will use all available cores if not told explicitly otherwise. This can be a problem when you are sharing resources, as Beocat does. To request multiple cores, use the sbatch directives '<tt>--nodes=1 --cpus-per-task=n</tt>' or '<tt>--nodes=1 --ntasks-per-node=n</tt>', where ''n'' is the number of cores you wish to use. If your command can take an environment variable, you can use $SLURM_CPUS_ON_NODE to tell how many cores you've been allocated.

=== Internode (MPI) jobs ===
''Inter''node jobs can utilize many cores on one or more nodes. Communicating between nodes is trickier than talking between cores on the same node. The specification for doing so is called "[[wikipedia:Message_Passing_Interface|Message Passing Interface]]", or MPI. We have [http://www.open-mpi.org/ OpenMPI] installed on Beocat for this purpose. Most programs written to take advantage of large multi-node systems will use MPI, but MPI also allows an application to run on multiple cores within a node. You can tell if you have an MPI-enabled program because its directions will tell you to run '<tt>mpirun ''program''</tt>'. Requesting MPI resources is only mildly more difficult than requesting single-node jobs. Instead of using '<tt>--cpus-per-task=''n''</tt>', you would use '<tt>--nodes=''n'' --tasks-per-node=''m''</tt>' ''or'' '<tt>--nodes=''n'' --ntasks=''o''</tt>' for your sbatch request, where ''n'' is the number of nodes you want, ''m'' is the number of cores per node you need, and ''o'' is the total number of cores you need.

Some quick examples:

<tt>--nodes=6 --ntasks-per-node=4</tt> will give you 4 cores on each of 6 nodes for a total of 24 cores.

<tt>--ntasks=40</tt> will give you 40 cores spread across any number of nodes.

<tt>--nodes=10 --ntasks=100</tt> will give you a total of 100 cores across 10 nodes.

== Requesting memory for multi-core jobs ==
Memory requests are easiest when they are specified '''per core'''. For instance, if you specified the following: '<tt>--tasks=20 --mem-per-core=20G</tt>', your job would have access to 400GB of memory total.
== Other Handy Slurm Features ==
=== Email status changes ===
One of the most commonly used options when submitting jobs not related to resource requests is to have have Slurm email you when a job changes its status. This takes may need two directives to sbatch: <tt>--mail-user</tt> and <tt>--mail-type</tt>.
==== --mail-type ====
<tt>--mail-type</tt> is used to tell Slurm to notify you about certain conditions. Options are comma separated and include the following
{| class="wikitable"
!Option!!Explanation
|-
| NONE || This disables event-based mail
|-
| BEGIN || Sends a notification when the job begins
|-
| END || Sends a notification when the job ends
|-
| FAIL || Sends a notification when the job fails.
|-
| REQUEUE || Sends a notification if the job is put back into the queue from a running state
|-
| STAGE_OUT || Burst buffer stage out and teardown completed
|-
| ALL || Equivalent to BEGIN,END,FAIL,REQUEUE,STAGE_OUT
|-
| TIME_LIMIT || Notifies if the job ran out of time
|-
| TIME_LIMIT_90 || Notifies when the job has used 90% of its allocated time
|-
| TIME_LIMIT_80 || Notifies when the job has used 80% of its allocated time
|-
| TIME_LIMIT_50 || Notifies when the job has used 50% of its allocated time
|-
| ARRAY_TASKS || Modifies the BEGIN, END, and FAIL options to apply to each array task (instead of notifying for the entire job
|}

==== --mail-user ====
<tt>--mail-user</tt> is optional. It is only needed if you intend to send these job status updates to a different e-mail address than what you provided in the [https://acount.beocat.ksu.edu/user Account Request Page]. It is specified with the following arguments to sbatch: <tt>--mail-user=someone@somecompany.com</tt>

=== Job Naming ===
If you have several jobs in the queue, running the same script with different parameters, it's handy to have a different name for each job as it shows up in the queue. This is accomplished with the '<tt>-J ''JobName''</tt>' sbatch directive.

=== Separating Output Streams ===
Normally, Slurm will create one output file, containing both STDERR and STDOUT. If you want both of these to be separated into two files, you can use the sbatch directives '<tt>--output</tt>' and '<tt>--error</tt>'.

{| class="wikitable"
! option !! default !! example
|-
| --output || slurm-%j.out || slurm-206.out
|-
| --error || slurm-%j.out || slurm-206.out
|}
<tt>%j</tt> above indicates that it should be replaced with the job id.

=== Running from the Current Directory ===
By default, jobs run from your home directory. Many programs incorrectly assume that you are running the script from the current directory. You can use the '<tt>-cwd</tt>' directive to change to the "current working directory" you used when submitting the job.
=== Running in a specific class of machine ===
If you want to run on a specific class of machines, e.g., the Dwarves, you can add the flag "--constraint=dwarves" to select any of those machines.

=== Processor Constraints ===
Because Beocat is a heterogenous cluster (we have machines from many years in the cluster), not all of our processors support every new and fancy feature. You might have some applications that require some newer processor features, so we provide a mechanism to request those.

<tt>--contraint</tt> tells the cluster to apply constraints to the types of nodes that the job can run on. For instance, we know of several applications that must be run on chips that have "AVX" processor extensions. To do that, you would specify <tt>--constraint=avx</tt> on you ''<tt>sbatch</tt>'' '''or''' ''<tt>srun</tt>'' command lines.
Using <tt>--constraint=avx</tt> will prohibit your job from running on the Mages while <tt>--contraint=avx2</tt> will eliminate the Elves as well as the Mages.

=== Slurm Environment Variables ===
Within an actual job, sometimes you need to know specific things about the running environment to setup your scripts correctly. Here is a listing of environment variables that Slurm makes available to you. Of course the value of these variables will be different based on many different factors.
<syntaxhighlight lang="bash">
CUDA_VISIBLE_DEVICES=NoDevFiles
ENVIRONMENT=BATCH
GPU_DEVICE_ORDINAL=NoDevFiles
HOSTNAME=dwarf37
SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint
SLURM_CLUSTER_NAME=beocat
SLURM_CPUS_ON_NODE=1
SLURM_DISTRIBUTION=cyclic
SLURMD_NODENAME=dwarf37
SLURM_GTIDS=0
SLURM_JOB_CPUS_PER_NODE=1
SLURM_JOB_GID=163587
SLURM_JOB_ID=202
SLURM_JOBID=202
SLURM_JOB_NAME=slurm_simple.sh
SLURM_JOB_NODELIST=dwarf37
SLURM_JOB_NUM_NODES=1
SLURM_JOB_PARTITION=batch.q,killable.q
SLURM_JOB_QOS=normal
SLURM_JOB_UID=163587
SLURM_JOB_USER=mozes
SLURM_LAUNCH_NODE_IPADDR=10.5.16.37
SLURM_LOCALID=0
SLURM_MEM_PER_NODE=1024
SLURM_NNODES=1
SLURM_NODEID=0
SLURM_NODELIST=dwarf37
SLURM_NPROCS=1
SLURM_NTASKS=1
SLURM_PRIO_PROCESS=0
SLURM_PROCID=0
SLURM_SRUN_COMM_HOST=10.5.16.37
SLURM_SRUN_COMM_PORT=37975
SLURM_STEP_ID=0
SLURM_STEPID=0
SLURM_STEP_LAUNCHER_PORT=37975
SLURM_STEP_NODELIST=dwarf37
SLURM_STEP_NUM_NODES=1
SLURM_STEP_NUM_TASKS=1
SLURM_STEP_TASKS_PER_NODE=1
SLURM_SUBMIT_DIR=/homes/mozes
SLURM_SUBMIT_HOST=dwarf37
SLURM_TASK_PID=23408
SLURM_TASKS_PER_NODE=1
SLURM_TOPOLOGY_ADDR=due1121-prod-core-40g-a1,due1121-prod-core-40g-c1.due1121-prod-sw-100g-a9.dwarf37
SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node
SLURM_UMASK=0022
SRUN_DEBUG=3
TERM=screen-256color
TMPDIR=/tmp
USER=mozes
</syntaxhighlight>
Sometimes it is nice to know what hosts you have access to during a job. You would checkout the SLURM_JOB_NODELIST to know that. There are lots of useful Environment Variables there, I will leave it to you to identify the ones you want.

Some of the most commonly-used variables we see used are $SLURM_CPUS_ON_NODE, $HOSTNAME, and $SLURM_JOB_ID.

== Running from a sbatch Submit Script ==
No doubt after you've run a few jobs you get tired of typing something like 'sbatch -l mem=2G,h_rt=10:00 -pe single 8 -n MyJobTitle MyScript.sh'. How are you supposed to remember all of these every time? The answer is to create a 'submit script', which outlines all of these for you. Below is a sample submit script, which you can modify and use for your own purposes.
<syntaxhighlight lang="bash">
#!/bin/bash

## A Sample sbatch script created by Kyle Hutson
##
## Note: Usually a '#" at the beginning of the line is ignored. However, in
## the case of sbatch, lines beginning with #SBATCH are commands for sbatch
## itself, so I have taken the convention here of starting *every* line with a
## '#', just Delete the first one if you want to use that line, and then modify
## it to your own purposes. The only exception here is the first line, which
## *must* be #!/bin/bash (or another valid shell).

## There is one strict rule for guaranteeing Slurm reads all of your options:
## Do not put *any* lines above your resource requests that aren't either:
## 1) blank. (no other characters)
## 2) comments (lines must begin with '#')

## Specify the amount of RAM needed _per_core_. Default is 1G
##SBATCH --mem-per-cpu=1G

## Specify the maximum runtime in DD-HH:MM:SS form. Default is 1 hour (1:00:00)
##SBATCH --time=1:00:00

## Require the use of infiniband. If you don't know what this is, you probably
## don't need it.
##SBATCH --gres=fabric:ib:1

## GPU directive. If You don't know what this is, you probably don't need it
##SBATCH --gres:gpu:1

## number of cores/nodes:
## quick note here. Jobs requesting 16 or fewer cores tend to get scheduled
## fairly quickly. If you need a job that requires more than that, you might
## benefit from emailing us at beocat@cs.ksu.edu to see how we can assist in
## getting your job scheduled in a reasonable amount of time. Default is
##SBATCH --cpus-per-task=1
##SBATCH --cpus-per-task=12
##SBATCH --nodes=2 --tasks-per-node=1
##SBATCH --tasks=20

## Constraints for this job. Maybe you need to run on the elves
##SBATCH --constraint=elves
## or perhaps you just need avx processor extensions
##SBATCH --constraint=avx

## Output file name. Default is slurm-%j.out where %j is the job id.
##SBATCH --output=MyJobTitle.o%j

## Split the errors into a seperate file. Default is the same as output
##SBATCH --error=MyJobTitle.e%j

## Name my job, to make it easier to find in the queue
##SBATCH -J MyJobTitle

## Send email when certain criteria are met.
## Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to
## BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage
## out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent
## of time limit), TIME_LIMIT_80 (reached 80 percent of time limit),
## TIME_LIMIT_50 (reached 50 percent of time limit) and ARRAY_TASKS (send
## emails for each array task). Multiple type values may be specified in a
## comma separated list. Unless the ARRAY_TASKS option is specified, mail
## notifications on job BEGIN, END and FAIL apply to a job array as a whole
## rather than generating individual email messages for each task in the job
## array.
##SBATCH --mail-type=ALL

## Email address to send the email to based on the above line.
## Default is to send the mail to the e-mail address entered on the account
## request form.
##SBATCH --mail-user myemail@ksu.edu

## And finally, we run the job we came here to do.
## $HOME/ProgramDir/ProgramName ProgramArguments

## OR, for the case of MPI-capable jobs
## mpirun $HOME/path/MpiJobName
</syntaxhighlight>

== File Access ==
Beocat has a variety of options for storing and accessing your files.
Every user has a home directory for general use which is limited in size, has decent file access performance,
and will soon be backed up nightly. Larger files should be stored in the /bulk subdirectories which have the same decent performance
but are not backed up. The /scratch file system will soon be implemented on a Lustre file system that will provide very fast
temporary file access. When fast IO is critical to the application performance, access to the local disk on each node or to a
RAM disk are the best options.

===Home directory===

Every user has a <tt>/homes/''username''</tt> directory that they drop into when they log into Beocat.
The home directory is for general use and provides decent performance for most file IO.
Disk space in each home directory is limited to 1 TB, so larger files should be kept in the /bulk
directory, and there is a limit of 100,000 files in each subdirectory in your account.
This file system is fully redundant, so 3 specific hard disks would need to fail before any data was lost.
All files will soon be backed up nightly to a separate file server in Nichols Hall, so if you do accidentally
delete something it can be recovered.

===Bulk directory===

Each user also has a <tt>/bulk/''username''</tt> directory where large files should be stored.
File access is the same speed as for the home directories, and the same limit of 100,000 files
per subdirectory applies. There is no limit to the disk space you can use in your bulk directory,
but the files there will not be backed up. They are still redundantly stored so you don't need to
worry about losing data to hardware failures, just don't delete something by accident. Unused files will be automatically removed after two years.
If you need to back up large files in the bulk directory, talk to Dan Andresen (dan@ksu.edu) about
purchasing some hard disks for archival storage.

===Scratch file system===

The /scratch file system may be faster than /bulk or /homes since each file written will access fewer disks.
In order to use scratch, you first need to make a directory for yourself.
Scratch is meant as temporary space for prepositioning files and accessing them
during runs. Once runs are completed, any files that need to be kept should be moved to your home
or bulk directories since files on the scratch file system may get purged after 30 days.

<syntaxhighlight lang=bash>
mkdir /scratch/$USER
</syntaxhighlight>

===Local disk===

If you are running on a single node, it may also be faster to access your files from the local disk
on that node. Each job creates a subdirectory /tmp/job# where '#' is the job ID number on the
local disk of each node the job uses. This can be accessed simply by writing to /tmp rather than
needing to use /tmp/job#.

You may need to copy files to
local disk at the start of your script, or set the output directory for your application to point
to a file on the local disk, then you'll need to copy any files you want off the local disk before
the job finishes since Slurm will remove all files in your job's directory on /tmp on completion
of the job or when it aborts. When we get the scratch file system working with Lustre, it may
end up being faster than accessing local disk so we will post the access rates for each. Use 'kstat -l -h'
to see how much /tmp space is available on each node.

<syntaxhighlight lang=bash>
# Copy input files to the tmp directory if needed
cp $input_files /tmp

# Make an 'out' directory to pass to the app if needed
mkdir /tmp/out

# Example of running an app and passing the tmp directory in/out
app -input_directory /tmp -output_directory /tmp/out

# Copy the 'out' directory back to the current working directory after the run
cp -rp /tmp/out .
</syntaxhighlight>

===RAM disk===

If you need ultrafast access to files, you can use a RAM disk which is a file system set up in the
memory of the compute node you are running on. The RAM disk is limited to the requested memory on that node, so you should account for this usage when you request
memory for your job. Below is an example of how to use the RAM disk.

<syntaxhighlight lang=bash>
# Copy input files over if necessary
cp $any_input_files /dev/shm/

# Run the application, possibly giving it the path to the RAM disk to use for output files
app -output_directory /dev/shm/

# Copy files from the RAM disk to the current working directory and clean it up
cp /dev/shm/* .
</syntaxhighlight>

===When you leave KSU===

If you are done with your account and leaving KSU, please clean up your directory, move any files
to your supervisor's account that need to be kept after you leave, and notify us so that we can disable your
account. The easiest way to move your files to your supervisor's account is for them to set up
a subdirectory for you with the appropriate write permissions. The example below shows moving
just a user's 'data' subdirectory to their supervisor. The 'nohup' command is used so that the move will
continue even if the window you are doing the move from gets disconnected.

<syntaxhighlight lang=bash>
# Supervisor:
mkdir /bulk/$USER/$STUDENT_USERNAME
chmod ugo+w /bulk/$USER/$STUDENT_USERNAME

# Student:
nohup mv /homes/$USER/data /bulk/$SUPERVISOR_USERNAME/$USER &
</syntaxhighlight>

==File Sharing==

This section will cover methods of sharing files with other users within Beocat and on remote systems.

===Securing your home directory===

By default your home directory is accessible to other users on Beocat for reading but not writing. If you do not want others to have any
access to files in your home directory, you can set the permissions to restrict access to just yourself.

chmod go-rwx /homes/your_user_name

This removes read, write, and execute permission to everyone but yourself. Be aware that it may make it more difficult for us to help you out when
you run into problems.

===Sharing files within your group===

By default all your files and directories have a 'group' that is your user name followed by _users as 'ls -l' shows.
In my case they have the group of daveturner_users.
If your working group owns any nodes on Beocat, then you have a group name that can be used to securely share
files with others within your group. Below is an example of creating a directory called 'share', changing the group
to ksu-cis-hpc (my group is ksu-cis-hpc so I submit jobs to --partition=ksu-cis-hpc.q), then changing the permissions to restrict access to
just that group.

mkdir share
chgrp ksu-cis-hpc share
chmod g+rx share
chmod o-rwx share

This will give people in your group the ability to read files in the 'share' directory. If you also want
them to be able to write or modify files in that directory then use 'chmod g+rwx' instead.

If you want to know what groups you belong to use the line below.

groups

If your group does not own any nodes, you can still request a group name and manage the participants yourself.

===Openly sharing files on the web===

If you create a 'public_html' directory on your home directory, then any files put there will be shared
openly on the web. There is no way to restrict who has access to those files.

cd
mkdir public_html

Then access the data from a web browser using the URL:

http://people.beocat.ksu.edu/~your_user_name

This will show a list of the files you have in your public_html subdirectory.

===Globus===

We have a page here dedicated to [[Globus]]

== Array Jobs ==
One of Slurm's useful options is the ability to run "Array Jobs"

It can be used with the following option to sbatch.

--array=n[-m[:s]]
Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Slurm
almost like a series of jobs. The option argument to --arrat specifies the number of array job tasks and the index number which will be
associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SLURM_ARRAY_TASK_ID. The option
arguments n, and m will be available through the environment variables SLURM_ARRAY_TASK_MIN and SLURM_ARRAY_TASK_MAX.

The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size.
Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SLURM_ARRAY_TASK_ID containing one of the 5 index numbers.

Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The
number of tasks in a array job is unlimited.

STDOUT and STDERR of array job tasks follow a slightly different naming convention (which can be controlled in the same way as mentioned above).

slurm-%A_%a.out

%A is the SLURM_ARRAY_JOB_ID, and %a is the SLURM_ARRAY_TASK_ID

=== Examples ===
==== Change the Size of the Run ====
Array Jobs have a variety of uses, one of the easiest to comprehend is the following:

I have an application, app1 I need to run the exact same way, on the same data set, with only the size of the run changing.

My original script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
RUNSIZE=50
#RUNSIZE=100
#RUNSIZE=150
#RUNSIZE=200
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
For every run of that job I have to change the RUNSIZE variable, and submit each script. This gets tedious.

With Array Jobs the script can be written like so:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=50-200:50
RUNSIZE=$SLURM_ARRAY_TASK_ID
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
I then submit that job, and Slurm understands that it needs to run it 4 times, once for each task. It also knows that it can and should run these tasks in parallel.

==== Choosing a Dataset ====
A slightly more complex use of Array Jobs is the following:

I have an application, app2, that needs to be run against every line of my dataset. Every line changes how app2 runs slightly, but I need to compare the runs against each other.

Originally I had to take each line of my dataset and generate a new submit script and submit the job. This was done with yet another script:

<syntaxhighlight lang="bash">
#!/bin/bash
DATASET=dataset.txt
scriptnum=0
while read LINE
do
echo "app2 $LINE" > ${scriptnum}.sh
sbatch ${scriptnum}.sh
scriptnum=$(( $scriptnum + 1 ))
done < $DATASET
</syntaxhighlight>
Not only is this needlessly complex, it is also slow, as sbatch has to verify each job as it is submitted. This can be done easily with array jobs, as long as you know the number of lines in the dataset. This number can be obtained like so: wc -l dataset.txt in this case lets call it 5000.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=1:5000
app2 `sed -n "${SLURM_ARRAY_TASK_ID}p" dataset.txt`
</syntaxhighlight>
This uses a subshell via `, and has the sed command print out only the line number $SLURM_ARRAY_TASK_ID out of the file dataset.txt.

Not only is this a smaller script, it is also faster to submit because it is one job instead of 5000, so sbatch doesn't have to verify as many.

To give you an idea about time saved: submitting 1 job takes 1-2 seconds. by extension if you are submitting 5000, that is 5,000-10,000 seconds, or 1.5-3 hours.

== Checkpoint/Restart using DMTCP ==

DMTCP is Distributed Multi-Threaded CheckPoint software that will checkpoint your application without modification, and
can be set up to automatically restart your job from the last checkpoint if for example the node you are running on fails.
This has been tested successfully
on Beocat for some scalar and OpenMP codes, but has failed on all MPI tests so far. We would like to encourage users to
try DMTCP out if their non-MPI jobs run longer than 24 hours. If you want to try this, please contact us first since we are still
experimenting with DMTCP.

The sample job submission script below shows how dmtcp_launch is used to start the application, then dmtcp_restart is used to start from a checkpoint if the job has failed and been rescheduled.
If you are putting this in an array script, then add the Slurm array task ID to the end of the ckeckpoint directory name
like ckptdir=ckpt-$SLURM_ARRAY_TASK_ID.

#!/bin/bash -l
#SBATCH --job-name=gromacs
#SBATCH --mem=50G
#SBATCH --time=24:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module purge
module load GROMACS/2016.4-foss-2017beocatb-hybrid
module load DMTCP
module list

ckptdir=ckpt
mkdir -p $ckptdir
export DMTCP_CHECKPOINT_DIR=$ckptdir

if ! ls -1 $ckptdir | grep -c dmtcp_restart_script > /dev/null
then
echo "Using dmtcp_launch to start the app the first time"
dmtcp_launch --no-coordinator mpirun -np 1 -x OMP_NUM_THREADS=4 gmx_mpi mdrun -nsteps 50000 -ntomp 4 -v -deffnm 1ns -c 1ns.pdb -nice 0
else
echo "Using dmtcp_restart from $ckptdir to continue from a checkpoint"
dmtcp_restart $ckptdir/*.dmtcp
fi

You will need to run several tests to verify that DMTCP is working properly with your application.
First, run a short test without DMTCP and another with DMTCP with the checkpoint interval set to 5 minutes
by adding the line export DMTCP_CHECKPOINT_INTERVAL=300 to your script. Then use kstat -d 1 to
check that the memory in both runs is close to the same. Also use this information to calculate the time
that each checkpoint takes. In most cases I've seen times less than a minute for checkpointing that will normally
be done once each hour. If your application is taking more time, let us know. Sometimes this can be sped up
by simply turning off compression by adding the line export DMTCP_GZIP=0. Make sure to remove the
line where you set the checkpoint interval to 300 seconds so that the default time of once per hour will be used.

After verifying that your code completes using DMTCP and does not take significantly more time or memory, you
will need to start a run then scancel it after the first checkpoint, then resubmit the same script to make
sure that it restarts and runs to completion. If you are working with an array job script, the last is to try a few
array tasks at once to make sure there is no conflict between the jobs.

== Running jobs interactively ==
Some jobs just don't behave like we think they should, or need to be run with somebody sitting at the keyboard and typing in response to the output the computers are generating. Beocat has a facility for this, called 'srun'. srun uses the exact same command-line arguments as sbatch, but you need to add the following arguments at the end: <tt>--pty bash</tt>. If no node is available with your resource requirements, srun will tell you something like the following:
srun --pty bash
srun: Force Terminated job 217
srun: error: CPU count per node can not be satisfied
srun: error: Unable to allocate resources: Requested node configuration is not available
Note that, like sbatch, your interactive job will timeout after your allotted time has passed.

== Connecting to an existing job ==
You can connect to an existing job using srun in the same way that the MonitorNode command
allowed us to in the old cluster. This is essentially like using ssh to get into the node where your job is running which
can be very useful in allowing you to look at files in /tmp/job# or in running htop to view the
activity level for your job.

srun --jobid=# --pty bash where '#' is the job ID number

== Altering Job Requests ==
We generally do not support users to modify job parameters once the job has been submitted. It can be done, but there are numerous catches, and all of the variations can be a bit problematic; it is normally easier to simply delete the job (using '''scancel ''jobid''''') and resubmit it with the right parameters. '''If your job doesn't start after modifying such parameters (after a reasonable amount of time), delete the job and resubmit it.'''

As it is unsupported, this is an excercise left to the reader. A starting point is <tt>man scontrol</tt>
== Killable jobs ==
There are a growing number of machines within Beocat that are owned by a particular person or group. Normally jobs from users that aren't in the group designated by the owner of these machines cannot use them. This is because we have guaranteed that the nodes will be accessible and available to the owner at any given time. We will allow others to use these nodes if they designate their job as "killable." If your job is designated as killable, your job will be able to use these nodes, but can (and will) be killed off at any point in time to make way for the designated owner's jobs. Jobs that are marked killable will be re-queued and may restart on another node.

The way you would designate your job as killable is to add <tt>--gres=killable:1</tt> to the '''<tt>sbatch</tt> or <tt>srun</tt>''' arguments. This could be either on the command-line or in your script file.

''Note: This is a submit-time only request, it cannot be added by a normal user after the job has been submitted.'' If you would like jobs modified to be '''killable''' after the jobs have been submitted (and it is too much work to <tt>scancel</tt> the jobs and re-submit), send an e-mail to the administrators detailing the job ids and what you would like done.

== Scheduling Priority ==
Some users are members of projects that have contributed to Beocat. When those users have contributed nodes, the group gets access to a "partition" giving you priority on those nodes.

In most situations, the scheduler will automatically add those priority partitions to the jobs as submitted. You should not need to include a partition list in your job submission.

There are currently just a few exceptions that we will not automatically add:
* ksu-chem-mri.q
* ksu-gen-gpu.q
* ksu-gen-highmem.q

To determine the partitions you have access to, run <tt>sinfo -hso '%P'</tt>
That will return a list that looks something like this:
killable.q
batch.q
ksu-gen-highmem.q

If you have access to those any of the non-automatic partitions, and have need of the resources in that partition, you can then alter your <tt>#SBATCH</tt> lines to include your new partition:
#SBATCH --partition=ksu-gen-highmem.q

== Graphical Applications ==
Some applications are graphical and need to have some graphical input/output. We currently accomplish this with X11 forwarding
=== Connecting with an X11 client ===
==== Windows ====
If you are running Windows, we recommend MobaXTerm as your file/ssh manager, this is because it is one relatively simple tool to do everything. MobaXTerm also automatically connects with X11 forwarding enabled.
==== Linux/OSX ====
Both Linux and OSX can connect in an X11 forwarding mode. Linux will have all of the tools you need installed already, OSX will need [https://www.xquartz.org/ XQuartz] installed.

Then you will need to change your 'ssh' command slightly:

ssh -Y eid@headnode.beocat.ksu.edu

The '''-Y''' argument tells ssh to setup X11 forwarding.
=== Starting an Graphical job ===
All graphical jobs, by design, must be interactive, so we'll use the srun command. On a headnode, we run the following:
# load an X11 enabled application
module load Octave
# start an X11 job, sbatch arguments are accepted for srun as well, 1 node, 1 hour, 1 gb of memory
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 octave --gui

Because these jobs are interactive, they may not be able to run at all times, depending on how busy the scheduler is at any point in time. '''--pty --x11''' are required arguments setting up the job, and '''octave --gui''' is the command to run inside the job.
== Job Accounting ==
Some people may find it useful to know what their job did during its run. The sacct tool will read Slurm's accounting database and give you summarized or detailed views on jobs that have run within Beocat.
=== sacct ===
This data can usually be used to diagnose two very common job failures.
==== Job debugging ====
It is simplest if you know the job number of the job you are trying to get information on.
<syntaxhighlight lang="bash">
# if you know the jobid, put it here:
sacct -j 1122334455 -l
# if you don't know the job id, you can look at your jobs started since some day:
sacct -S 2017-01-01
</syntaxhighlight>

===== My job didn't do anything when it ran! =====
{{Scrolling table/top}}
{{Scrolling table/mid}}
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|218||218||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||12||00:00:00||FAILED||2:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=12,mem=1G,node=1||cpu=12,mem=1G,node=1
|-
|218.batch||218.batch||batch||||137940K||dwarf37||0||137940K||1576K||dwarf37||0||1576K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||1.36G||0||0||0||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|-
|218.0||218.0||qqqqstat||||204212K||dwarf37||0||204212K||1420K||dwarf37||0||1420K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||196.52M||Unknown||Unknown||Unknown||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
{{Scrolling table/end}}
If you look at the columns showing Elapsed and State, you can see that they show 00:00:00 and FAILED respectively. This means that the job started and then promptly ended. This points to something being wrong with your submission script. Perhaps there is a typo somewhere in it.

===== My job ran but didn't finish! =====
{{Scrolling table/top}}
{{Scrolling table/mid}}
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|220||220||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:01:27||TIMEOUT||0:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=1,mem=1G,node=1||cpu=1,mem=1G,node=1
|-
|220.batch||220.batch||batch||||370716K||dwarf37||0||370716K||7060K||dwarf37||0||7060K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:28||CANCELLED||0:15||1.23G||0||0||0||1Gn||0||0.16M||dwarf37||0||0.16M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|-
|220.0||220.0||sleep||||204212K||dwarf37||0||107916K||1000K||dwarf37||0||620K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:27||CANCELLED||0:15||1.54G||Unknown||Unknown||Unknown||1Gn||0||0.05M||dwarf37||0||0.05M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
{{Scrolling table/end}}
If you look at the column showing State, we can see some pointers to the issue. The job ran out of time (TIMEOUT) and then was killed (CANCELLED).
{{Scrolling table/top}}
{{Scrolling table/mid}}
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|221||221||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:00:00||CANCELLED by 0||0:0||||Unknown||Unknown||Unknown||1Mn||||||||||||||||||||||||cpu=1,mem=1M,node=1||cpu=1,mem=1M,node=1
|-
|221.batch||221.batch||batch||||137940K||dwarf37||0||137940K||1144K||dwarf37||0||1144K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:00:01||CANCELLED||0:15||2.62G||0||0||0||1Mn||0||0||dwarf37||65534||0||0||dwarf37||65534||0||||||||cpu=1,mem=1M,node=1
{{Scrolling table/end}}
If you look at the column showing State, we see it was "CANCELLED by 0", then we look at the AllocTRES column to see our allocated resources, and see that 1MB of memory was granted. Combine that with the column "MaxRSS" and we see that the memory granted was less than the memory we tried to use, thus the job was "CANCELLED".

Globus

2019-09-09T21:57:46Z

Kylehutson: Added video link

Globus

2019-08-23T21:51:15Z

Kylehutson: Finished using the on-campus DTN

== Transferring Data using Globus ==

[https://www.globus.org/ Globus] is a high-speed data transfer service. It is primarily used to transfer data between research institutions, but can also be used to transfer data between Beocat and a laptop or desktop. We suggest using Globus over other file transfer options if you are transferring large data sets. Globus also allows you to share data with those who do not have Beocat accounts.

Beocat has two Globus servers - one on the main campus network, and one directly connected to [https://www.kanren.net/ KanREN] (essentially for our purposes, the university's Internet Service Provider). To understand which one you should be using an overview of how Beocat connects to the Internet is useful:
[[File:CampusNetworkOverview.png|thumb|left|Campus Network Overview - Click for a larger view]]
As you can see, if you are ON campus, it's faster to use the "DTN" endpoint, but if you are OFF campus, it is faster to use the "FIONA" endpoint. That being said, due to software differences, those two endpoints behave differently, and either CAN be used either on- or off-campus.

=== Using either endpoint ===
# Go to [https://www.globus.org https://www.globus.org] and click the "Log In" button in the upper-right corner.
# Under "Use your existing organizational login", find "Kansas State University" and click "Continue"
# You will be redirected to a K-State page where you can login with your eID credentials
# You will then be logged into the Globus File Manager.
# Click on "Endpoints"
# In the top box, search for "Beocat" (this is case insensitive)
# This is where the two endpoints diverge in how they work, and where we pick up from below.

=== Using the on-campus DTN endpoint ===
# Click on "Kansas State University Beocat". You will see a screen that gives some information about the Endpoint (which you will most likely ignore).
# On the right side of the screen, click on "Open in File Manager"
## You will now see your files (your home directory) in Beocat in the left window. From here, you can perform some basic operations, such as renaming or deleting a file, but you cannot upload or download files. (NOTE: If you click the 3-lined icon in the middle of the page as circled in red in the following image, the descriptions of the icons will pop-out): [[File:GlobusToolExpansion.png|thumb|Globus Tool Expansion]]
## On the right side of the screen, click where it says "Transfer or sync to".
## Here you can select another research institution to which you have access, or you can click the link to "Install Globus Connect Personal", which will install software on your own computer.
## Once you install Globus Connect Personal, you can use that as your other Endpoint.
# Now you have two systems to view. Beocat and another. You may drag-and-drop files or folders from one location to another, or use the other features to synchronize files between the two. Once initiated, file transfers via Globus are done in the background. You can close your browser window, or log out of your computer - as long as both endpoints are connected, file transfers will continue. (Note, obviously if you're running Globus Connect Personal on your laptop and you shut it off to take it with you, it is no longer connected. However, as long as the transfer hasn't timed out (24 hours?) it should resume the transfer when you turn it back on.
# You can change directories on either side by clicking on them, or typing the name of the directory (/bulk/username tends to be one frequently used on Beocat).

==== Sharing data from the on-campus DTN endpoint ====
Now that you have logged into Beocat using Globus (above), you can share files or folders.
In my example here, I have a folder called "sharedfolder" in my home directory, which I want to make available to others. [[File:Globus Sharing.png|thumb|Screenshot of sharing my 'sharedfolder' directory]]
# Scroll down so you can see the "sharedfolder" directory, and click on it.
# Click the 'Share' button
# Click on 'Add a Shared Endpoint'
# Give any optional info that may make it easier for those with whom you are sharing to find. In my case, I entered 'Beocat Shared Folder Demo for my Display name and 'Shared Folder for Demonstration Purposes' for the description.
# Click 'Create Share'
# By default, only you have access to this share. Click the "Add Permissions - Share With" button to share with anybody you like.
## A user is typically designated by their email address.
## A group is a group of users you have defined. There is a link on the left side of the web app to create groups.
## All users means anybody who logs in with Globus. Note that if you use this option, anybody will be able to search for and use this collection
# The people you share with have the permission you give: read and/or write. I ''strongly'' suggest you do not give "all users" write access to your folder. In fact, be very careful of anybody to whom you give write permission. Remember, you are responsible for the content on your account.
# Complete the process by clicking 'Add Permission'
# Anybody who has permissions to your folder can now, once they login to Globus, search for the collection you just shared. The 'Sharing' tab for your collection (which is where you are after clicking 'Add Permission') has a "View link for Sharing", which you can send to any collaborators so they can more easily find the data which you have shared.

File:Globus Sharing.png

2019-08-23T21:31:27Z

Kylehutson:

A screenshot of preparing to share files in Globus

Globus

2019-07-12T16:04:33Z

Kylehutson: Still writing initial page

File:CampusNetworkOverview.png

2019-07-12T15:05:45Z

Kylehutson:

Globus

2019-07-11T21:58:33Z

Kylehutson: Not ready for publishing, but leaving for the day.

Main Page

2019-07-11T20:51:17Z

Kylehutson: /* What is Beocat? */

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of CentOS Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''beocat#beocat''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

<H4>Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] 
Read about [[Installed software]] and languages 
Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] 
</H4>

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

We are also available on IRC on the [http://freenode.net/using_the_network.shtml freenode chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''m0zes''' and/or '''kylehutson''' in your message, and it should grab our attention. Available from a web browser [[Special:WebChat|here.]]

For in person help, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<H4>
Again, when you email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu] please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</H4>

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.
Here are some recent tweets:
<ShoogleTweet limit="6">KSUBeocat</ShoogleTweet>

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat.

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Policy

2019-05-21T18:08:01Z

Kylehutson: /* Acknowledging Use of Beocat Resources and/or Personnel in Publications */

== K-State Information Technology Usage Policy ==
As Beocat is a K-State resource, the following usage policy also applies: http://www.k-state.edu/policies/ppm/3420.html

Please pay close attention to .050-2, and .050-4, as these are egregious violations.
== Classified, PII/HIPAA, and/or export controlled data ==
Beocat is not equipped to handle [[wikipedia:Classified_information|Classified]], [[wikipedia:Personally_identifiable_information|PII]], [[wikipedia:Health_Insurance_Portability_and_Accountability_Act|HIPAA]] or [[wikipedia:Export_Administration_Regulations|export controlled]] data.
== CUI Data ==
For those that need to store and/or compute on [[wikipedia:Controlled Unclassified Information|CUI Data]], we may be able to work something out. You must speak with us before storing any such data on Beocat.
== Maintenance ==
Beocat reserves the right to a 24 hour maintenance period every other week. However, this maintenance is not always necessary. Maintenance intentions and reservations will always be announced on the mailing list 2 weeks before an actual maintenance period is in effect.

== Head node computational tasks ==
The head node serves as a shell server and development environment for Beocat users. We wish to keep this machine running responsively to make work easier. We do not have a problem with running simple post-processing work on the head node directly. But, if your process seems too computation or memory intensive, it may have its priority severely reduced or may be killed completely. If in doubt, ask.

Due to abuses of the head node, there are now strict limits in place. If a process uses more than 4GB of RSS memory or 6GB of virtual memory, it will get killed automatically. RSS Memory is limited to 12GB across all users. CPU Usage is allocated with a fair-share algorithm, all users have equivalent access to CPU time.
== Backups ==
For those of you using our hosted virtual machines, no backups of said machines or data are made.

At this point in time, due to the size of our main storage, we are unable to provide backups of any data.

== Home Directory Quota ==
Each home directory has a quota of 1TB. If you use more that 1TB in your home directory, we will notify you and provide a window for resolving the issue. If no action is underwent, we will move data elsewhere.

== Bulk Usage ==
We have no quota for usage within our /bulk filesystem. To keep this from out of control growth, files that have not been read from or written to in 2 years will be automatically removed (unless prior arrangements are made with Beocat Staff).

== Account deactivation ==
If your account meets any of the following criteria:
* inactive for 2 years
* invalid e-mail address on file
* unsubscribed from our mailing list
we will mark the account for archival, and remove your ability to login. If you should need the account again, please fill out our [https://account.beocat.cis.ksu.edu/user account request form.]

== Acknowledging Use of Beocat Resources and/or Personnel in Publications ==
Click [[PapersAndGrants|here]] for a list of publications that used Beocat resources and/or personnel.

# A publication that is based in whole or in part on computations performed using Beocat systems, including but not limited to hardware, storage, networking and/or software, should incorporate the following text into the Acknowledgements section of the publication:
#* [Some of] The computing for this project was performed on the Beocat Research Cluster at Kansas State University, which is funded in part by NSF grants CNS-1006860, EPS-1006860, EPS-0919443, ACI-1440548, CHE-1726332, and NIH P20GM113109.
# If any Beocat staff member(s) assisted with the work in any way, then for each Beocat staff member that was involved in the work:
## If the publication includes a substantial amount of text about the work that the Beocat staff member contributed to, and if the Beocat staff member did a substantial amount of development or optimization of software, and/or they contributed significantly to the writing of the publication, then that staff member should be included as a co-author on that publication, with author order to be negotiated among the authors.
##; NOTE : This requirement can be waived for tenure track (but not yet tenured) faculty if the faculty member has a compelling tenure-related interest in, for example, producing single-author publications.
## If the conditions above don't apply, then the Beocat staff member should be acknowledged by name and job title in the Acknowledgements section of the paper.
##; For example : Beocat Director Daniel Andresen and Beocat Systems Administrator Adam Tygart provided valuable technical expertise.
# A citation for your publication should be added to our [[PapersAndGrants|papers and grants page]].

== IRB Statement ==
=== INFORMATION ===
If you are a Beocat user, whenever you submit a job, delete a job, or otherwise interact with the
scheduler, automatic information about this is logged and will be used in this study. This will include
information about the job including requested resources (memory, processors, duration, modules, etc.).
We may send you a followup request for more information if, for example, you delete a job. Your
participation is optional.
=== RISKS ===
There are no anticipated risks with participation in this study other than the time responding to a
followup information request.
=== BENEFITS ===
Your participation in our studies will help us learn how to optimize the performance of Beocat and other
HPC resources, which will help our users and our overall science and education efforts.
=== CONFIDENTIALITY ===
All information gathered in this study will be kept confidential. All information about your jobs will not
use real names or eIDs, and any publications will aggregate overall information.
=== CONTACT ===
If you have any questions at any time about the study or procedures, please contact Dr. Daniel Andresen
at Kansas State University, Department of Computer Science: Phone: (785) 532-7914 or Email:
dan@ksu.edu.
If you feel you have not been treated according to the description in this page, or your rights as a
participant in research have been violated during the course of this study, you may contact the office for
the Kansas State University Committee on Research Involving Human Subjects, 203 Fairchild Hall, Kansas
State University, Manhattan, KS 66506. (785) 532-3224.
=== PARTICIPATION ===

Your participation in this study is strictly voluntary; you may refuse to participate in any followup
surveys or withdraw your information from the study without penalty. If you decide to participate, you
may withdraw from the study at any time without penalty. To remove your data from use in the study,
contact Dr. Daniel Andresen as described above.

Installed software

2019-02-26T00:05:28Z

Kylehutson: Fixed initialization steps for interactive example

== Drinking from the Firehose ==
For a complete list of all installed modules, see [[ModuleList]]

== Toolchains ==
A toolchain is a set of compilers, libraries and applications that are needed to build software. Some software functions better when using specific toolchains.

We provide a good number of toolchains and versions of toolchains make sure your applications will compile and/or run correctly.

These toolchains include (you can run 'module keyword toolchain'):
; foss: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
; gcccuda: GNU Compiler Collection (GCC) based compiler toolchain, along with CUDA toolkit.
; gmvapich2: GNU Compiler Collection (GCC) based compiler toolchain, including MVAPICH2 for MPI support.
; gompi: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.
; gompic: GNU Compiler Collection (GCC) based compiler toolchain along with CUDA toolkit, including OpenMPI for MPI support with CUDA features enabled.
; goolfc: GCC based compiler toolchain __with CUDA support__, and including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
; iomkl: Intel Cluster Toolchain Compiler Edition provides Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.

You can run 'module spider $toolchain' to see the versions we have:
$ module spider iomkl
* iomkl/2017a
* iomkl/2017b
* iomkl/2017beocatb

If you load one of those (module load iomkl/2017b), you can see the other modules and versions of software that it loaded with the 'module list':
$ module list
Currently Loaded Modules:
1) icc/2017.4.196-GCC-6.4.0-2.28
2) binutils/2.28-GCCcore-6.4.0
3) ifort/2017.4.196-GCC-6.4.0-2.28
4) iccifort/2017.4.196-GCC-6.4.0-2.28
5) GCCcore/6.4.0
6) numactl/2.0.11-GCCcore-6.4.0
7) hwloc/1.11.7-GCCcore-6.4.0
8) OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
9) iompi/2017b
10) imkl/2017.3.196-iompi-2017b
11) iomkl/2017b

As you can see, toolchains can depend on each other. For instance, the iomkl toolchain, depends on iompi, which depends on iccifort, which depend on icc and ifort, which depend on GCCcore which depend on GCC. Hence it is very important that the correct versions of all related software are loaded.

With software we provide, the toolchain used to compile is always specified in the "version" of the software that you want to load.

If you mix toolchains, inconsistent things may happen.
== Most Commonly Used Software ==
=== [http://www.open-mpi.org/ OpenMPI] ===
We provide lots of versions, you are most likely better off directly loading a toolchain or application to make sure you get the right version, but you can see the versions we have with 'module spider OpenMPI':

* OpenMPI/2.0.2-GCC-6.3.0-2.27
* OpenMPI/2.0.2-iccifort-2017.1.132-GCC-6.3.0-2.27
* OpenMPI/2.1.1-GCC-6.4.0-2.28
* OpenMPI/2.1.1-GCC-7.2.0-2.29
* OpenMPI/2.1.1-gcccuda-2017b
* OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
* OpenMPI/2.1.1-iccifort-2018.0.128-GCC-7.2.0-2.29

=== [http://www.r-project.org/ R] ===
We currently provide (module -r spider '^R$'):
* R/3.4.0-foss-2017beocatb-X11-20170314

==== Packages ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own R Packages ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
module load R
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="R">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="R">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. '<tt>sbatch myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.sbatch

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (D-H:MM:SS)
#SBATCH --time=0-0:15:00

# Now lets do some actual work. This starts R and loads the file myscript.R
module load R
R --no-save -q < myscript.R
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
sbatch submit-R.sbatch
</syntaxhighlight>

=== [http://www.java.com/ Java] ===
We currently provide (module spider Java):
* Java/1.8.0_131
* Java/1.8.0_144

=== [http://www.python.org/about/ Python] ===
We currently provide (module spider Python)
* Python/2.7.13-foss-2017beocatb
* Python/2.7.13-GCCcore-7.2.0-bare
* Python/2.7.13-iomkl-2017a
* Python/2.7.13-iomkl-2017beocatb
* Python/3.6.3-foss-2017b
* Python/3.6.3-foss-2017beocatb
* Python/3.6.3-iomkl-2017beocatb

If you need modules that we do not have installed, you should use [https://virtualenv.pypa.io/en/stable/userguide/ virtualenv] to setup a virtual python environment in your home directory. This will let you install python modules as you please.

==== Setting up your virtual environment ====
<syntaxhighlight lang="bash">
# Load Python
module load Python/3.6.3-iomkl-2017beocatb
</syntaxhighlight>
(After running this command Python is loaded. After you logoff and then logon again Python will not be loaded so you must rerun this command every time you logon.)
* Create a location for your virtual environments (optional, but helps keep things organized)
<syntaxhighlight lang="bash">
mkdir ~/virtualenvs
cd ~/virtualenvs
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test'. Note that <code>virtualenv --help</code> has many more useful options.
<syntaxhighlight lang="bash">
virtualenv test
</syntaxhighlight>
* Lets look at our virtual environments (the virtual environment name should be in the output):
<syntaxhighlight lang="bash">
ls ~/virtualenvs
</syntaxhighlight>
* Activate one of these
<syntaxhighlight lang="bash">
source ~/virtualenvs/test/bin/activate
</syntaxhighlight>
(After running this command your virtual environment is activated. After you logoff and then logon again your virtual environment will not be loaded so you must rerun this command every time you logon.)
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>

==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment test
<syntaxhighlight lang="bash">
#!/bin/bash
module load Python/3.6.3-iomkl-2017beocatb
source ~/virtualenvs/test/bin/activate
export PYTHONDONTWRITEBYTECODE=1
python ~/path/to/your/python/script.py
</syntaxhighlight>

=== [http://spark.apache.org/ Spark] ===

Spark is a programming language for large scale data processing.
It can be used in conjunction with Python, R, Scala, Java, and SQL.
Spark can be run on Beocat interactively or through the Slurm queue.

To run interactively, you must first request a node or nodes from the Slurm queue.
The line below requests 1 node and 1 core for 24 hours and if available will drop
you into the bash shell on that node.

srun -J srun -N 1 -n 1 -t 24:00:00 --mem=10G --pty bash

We have some sample python based Spark code you can try out that came from the
exercises and homework from the PSC Spark workshop.

mkdir spark-test
cd spark-test
cp -rp /homes/daveturner/projects/PSC-BigData-Workshop/Shakespeare .

The sample code requires 'nltk' and 'numpy' packages, so the first time you run it, you need to create the virtualenv and install these packages.

module load Python
mkdir ~/virtualenvs
cd ~/virtualenvs
virtualenv spark-test
source ~/virtualenvs/spark-test/bin/activate
pip install nltk
pip install numpy

On any subsequent runs, you can then just enter that virtualenv without running all of the above commands:

module load Python
source ~/virtualenvs/spark-test/bin/activate

Then load the Spark module (Python should already be loaded from above), change to the sample directory, fire up pyspark, and run the sample code.

module load Spark
cd ~/spark-test/Shakespeare
pyspark
>>> exec(open("shakespeare.py").read())

You can work interactively from the pyspark prompt (>>>) in addition to running scripts as above.

The Shakespeare directory also contains a sample sbatch submit script that will run the
same shakespeare.py code through the Slurm batch queue.

#!/bin/bash -l
#SBATCH --job-name=shakespeare
#SBATCH --mem=10G
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

# Load Spark and Python (version 3 here)
module load Spark
module load Python

spark-submit shakespeare.py

When you run interactively, pyspark initializes your spark context sc.
You will need to do this manually as in the sample python code when you want
to submit jobs through the Slurm queue.

# If there is no Spark Context (not running interactive from pyspark), create it
try:
sc
except NameError:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("App")
sc = SparkContext(conf = conf)

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.

If you need a newer version (or threads), just load one we provide in our modules (module spider Perl):
* Perl/5.26.0-foss-2017beocatb
* Perl/5.26.0-iompi-2017beocatb

==== Submitting a job with Perl ====
Much like R (above), you cannot simply '<tt>sbatch myProgram.pl</tt>', but you must create a [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|submit script]] which will call perl. Here is an example:
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS)
#SBATCH --time=0-0:15:00
# Now lets do some actual work.
module load Perl
perl /path/to/myProgram.pl
</syntaxhighlight>

=== Octave for MatLab codes ===

module load Octave/4.2.1-foss-2017beocatb-enable64

The 64-bit version of Octave can be loaded using the command above. Octave can then be used
to work with MatLab codes on the head node and to submit jobs to the compute nodes through the
sbatch scheduler. Octave is made to run MatLab code, but it does have limitations and does not support
everything that MatLab itself does.

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=octave
#SBATCH --output=octave.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module purge
module load Octave/4.2.1-foss-2017beocatb-enable64

octave < matlab_code.m
</syntaxhighlight>

=== MatLab compiler ===

Beocat also has a single-user license for the MatLab compiler and the most common toolboxes
including the Parallel Computing Toolbox, Optimization Toolbox, Statistics and Machine Learning Toolbox,
Image Processing Toolbox, Curve Fitting Toolbox, Neural Network Toolbox, Sumbolic Math Toolbox,
Global Optimization Toolbox, and the Bioinformatics Toolbox.

Since we only have a single-user license, this means that you will be expected to develop your MatLab code
with Octave or elsewhere on a laptop or departmental server. Once you're ready to do large runs, then you
move your code to Beocat, compile the MatLab code into an executable, and you can submit as many jobs as
you want to the scheduler. To use the MatLab compiler, you need to load the MATLAB module to compile code and
load the mcr module to run the resulting MatLab executable.

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -o matlab_executable_name
</syntaxhighlight>

If you have addpath() commands in your code, you will need to wrap them in an "if ~deployed" block and tell the
compiler to include that path via the -I flag.

<syntaxhighlight lang="MATLAB">
% wrap addpath() calls like so:
if ~deployed
addpath('./another/folder/with/code/')
end
</syntaxhighlight>

NOTE: The license manager checks the mcc compiler out for a minimum of 30 minutes, so if another user compiles a code
you unfortunately may need to wait for up to 30 minutes to compile your own code.

Compiling with additional paths:

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -I ./another/folder/with/code/ -o matlab_executable_name
</syntaxhighlight>

Any directories added with addpath() will need to be added to the list of compile options as -I arguments. You
can have multiple -I arguments in your compile command.

Here is an example job submission script. Modify time, memory, tasks-per-node, and job name as you see fit:

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=matlab
#SBATCH --output=matlab.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module purge
module load mcr

./matlab_executable_name
</syntaxhighlight>

For those who make use of mex files - compiled C and C++ code with matlab bindings - you will need to add these
files to the compiled archive via the -a flag. See the behavior of this flag in the [https://www.mathworks.com/help/compiler/mcc.html compiler documentation]. You can either target specific .mex files or entire directories.

Because codes often require adding several directories to the Matlab path as well as mex files from several locations,
we recommend writing a script to preserve and help document the steps to compile your Matlab code. Here is an
abbreviated example from a current user:

<syntaxhighlight lang="bash">
#!/bin/bash -l

module load MATLAB

cd matlabPyrTools/MEX/

# compile mex files
mex upConv.c convolve.c wrap.c edges.c
mex corrDn.c convolve.c wrap.c edges.c
mex histo.c
mex innerProd.c

cd ../..

mcc -m mongrel_creation.m \
-I ./matlabPyrTools/MEX/ \
-I ./matlabPyrTools/ \
-I ./FastICA/ \
-a ./matlabPyrTools/MEX/ \
-a ./texturesynth/ \
-o mongrel_creation_binary
</syntaxhighlight>

Again, we only have a single-user license for MatLab so the model is to develop and debug your MatLab code
elsewhere or using Octave on Beocat, then you can compile the MatLab code into an executable and run it without
limits on Beocat.

For more info on the mcc compiler see: https://www.mathworks.com/help/compiler/mcc.html

=== COMSOL ===
Beocat has no license for COMSOL. If you want to use it, you must provide your own.

module spider COMSOL
----------------------------------------------------------------------------
COMSOL: COMSOL/5.3
----------------------------------------------------------------------------
Description:
COMSOL Multiphysics software, an interactive environment for modeling
and simulating scientific and engineering problems

This module can be loaded directly: module load COMSOL/5.3

Help:

Description
===========
COMSOL Multiphysics software, an interactive environment for modeling and
simulating scientific and engineering problems
You must provide your own license.
export LM_LICENSE_FILE=/the/path/to/your/license/file
*OR*
export LM_LICENSE_FILE=$LICENSE_SERVER_PORT@$LICENSE_SERVER_HOSTNAME
e.g. export LM_LICENSE_FILE=1719@some.flexlm.server.ksu.edu

More information
================
- Homepage: https://www.comsol.com/
==== Graphical COMSOL ====
Running COMSOL in graphical mode on a cluster is generally a bad idea. If you choose to run it in graphical mode on a compute node, you will need to do something like the following:
<syntaxhighlight lang="bash">
# Connect to the cluster with X11 forwarding (ssh -Y or mobaxterm)
# load the comsol module on the headnode
module load COMSOL
# export your comsol license as mentioned above, and tell the scheduler to run the software
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 comsol -3drend sw
</syntaxhighlight>

=== .NET Core ===
==== Load .NET ====
mozes@[eunomia] ~ $ module load dotNET-Core-SDK
==== create an application ====
Following instructions from [https://docs.microsoft.com/en-us/dotnet/core/tutorials/using-with-xplat-cli here], we'll create a simple 'Hello World' application
mozes@[eunomia] ~ $ mkdir Hello

mozes@[eunomia] ~ $ cd Hello

mozes@[eunomia] ~/Hello $ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true

mozes@[eunomia] ~/Hello $ dotnet new console
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /homes/mozes/Hello/Hello.csproj...
Restoring packages for /homes/mozes/Hello/Hello.csproj...
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.props.
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.targets.
Restore completed in 358.43 ms for /homes/mozes/Hello/Hello.csproj.

Restore succeeded.

==== Edit your program ====
mozes@[eunomia] ~/Hello $ vi Program.cs
==== Run your .NET application ====
mozes@[eunomia] ~/Hello $ dotnet run
Hello World!
==== Build and run the built application ====
mozes@[eunomia] ~/Hello $ dotnet build
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

Restore completed in 106.12 ms for /homes/mozes/Hello/Hello.csproj.
Hello -> /homes/mozes/Hello/bin/Debug/netcoreapp2.1/Hello.dll

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:02.86

mozes@[eunomia] ~/Hello $ dotnet bin/Debug/netcoreapp2.1/Hello.dll
Hello World!

== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

Installed software

2019-02-25T23:37:11Z

Kylehutson: Typos and prereqs for spark

== Drinking from the Firehose ==
For a complete list of all installed modules, see [[ModuleList]]

== Toolchains ==
A toolchain is a set of compilers, libraries and applications that are needed to build software. Some software functions better when using specific toolchains.

We provide a good number of toolchains and versions of toolchains make sure your applications will compile and/or run correctly.

These toolchains include (you can run 'module keyword toolchain'):
; foss: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
; gcccuda: GNU Compiler Collection (GCC) based compiler toolchain, along with CUDA toolkit.
; gmvapich2: GNU Compiler Collection (GCC) based compiler toolchain, including MVAPICH2 for MPI support.
; gompi: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.
; gompic: GNU Compiler Collection (GCC) based compiler toolchain along with CUDA toolkit, including OpenMPI for MPI support with CUDA features enabled.
; goolfc: GCC based compiler toolchain __with CUDA support__, and including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
; iomkl: Intel Cluster Toolchain Compiler Edition provides Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.

You can run 'module spider $toolchain' to see the versions we have:
$ module spider iomkl
* iomkl/2017a
* iomkl/2017b
* iomkl/2017beocatb

If you load one of those (module load iomkl/2017b), you can see the other modules and versions of software that it loaded with the 'module list':
$ module list
Currently Loaded Modules:
1) icc/2017.4.196-GCC-6.4.0-2.28
2) binutils/2.28-GCCcore-6.4.0
3) ifort/2017.4.196-GCC-6.4.0-2.28
4) iccifort/2017.4.196-GCC-6.4.0-2.28
5) GCCcore/6.4.0
6) numactl/2.0.11-GCCcore-6.4.0
7) hwloc/1.11.7-GCCcore-6.4.0
8) OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
9) iompi/2017b
10) imkl/2017.3.196-iompi-2017b
11) iomkl/2017b

As you can see, toolchains can depend on each other. For instance, the iomkl toolchain, depends on iompi, which depends on iccifort, which depend on icc and ifort, which depend on GCCcore which depend on GCC. Hence it is very important that the correct versions of all related software are loaded.

With software we provide, the toolchain used to compile is always specified in the "version" of the software that you want to load.

If you mix toolchains, inconsistent things may happen.
== Most Commonly Used Software ==
=== [http://www.open-mpi.org/ OpenMPI] ===
We provide lots of versions, you are most likely better off directly loading a toolchain or application to make sure you get the right version, but you can see the versions we have with 'module spider OpenMPI':

* OpenMPI/2.0.2-GCC-6.3.0-2.27
* OpenMPI/2.0.2-iccifort-2017.1.132-GCC-6.3.0-2.27
* OpenMPI/2.1.1-GCC-6.4.0-2.28
* OpenMPI/2.1.1-GCC-7.2.0-2.29
* OpenMPI/2.1.1-gcccuda-2017b
* OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
* OpenMPI/2.1.1-iccifort-2018.0.128-GCC-7.2.0-2.29

=== [http://www.r-project.org/ R] ===
We currently provide (module -r spider '^R$'):
* R/3.4.0-foss-2017beocatb-X11-20170314

==== Packages ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own R Packages ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
module load R
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="R">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="R">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. '<tt>sbatch myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.sbatch

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (D-H:MM:SS)
#SBATCH --time=0-0:15:00

# Now lets do some actual work. This starts R and loads the file myscript.R
module load R
R --no-save -q < myscript.R
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
sbatch submit-R.sbatch
</syntaxhighlight>

=== [http://www.java.com/ Java] ===
We currently provide (module spider Java):
* Java/1.8.0_131
* Java/1.8.0_144

=== [http://www.python.org/about/ Python] ===
We currently provide (module spider Python)
* Python/2.7.13-foss-2017beocatb
* Python/2.7.13-GCCcore-7.2.0-bare
* Python/2.7.13-iomkl-2017a
* Python/2.7.13-iomkl-2017beocatb
* Python/3.6.3-foss-2017b
* Python/3.6.3-foss-2017beocatb
* Python/3.6.3-iomkl-2017beocatb

If you need modules that we do not have installed, you should use [https://virtualenv.pypa.io/en/stable/userguide/ virtualenv] to setup a virtual python environment in your home directory. This will let you install python modules as you please.

==== Setting up your virtual environment ====
<syntaxhighlight lang="bash">
# Load Python
module load Python/3.6.3-iomkl-2017beocatb
</syntaxhighlight>
(After running this command Python is loaded. After you logoff and then logon again Python will not be loaded so you must rerun this command every time you logon.)
* Create a location for your virtual environments (optional, but helps keep things organized)
<syntaxhighlight lang="bash">
mkdir ~/virtualenvs
cd ~/virtualenvs
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test'. Note that <code>virtualenv --help</code> has many more useful options.
<syntaxhighlight lang="bash">
virtualenv test
</syntaxhighlight>
* Lets look at our virtual environments (the virtual environment name should be in the output):
<syntaxhighlight lang="bash">
ls ~/virtualenvs
</syntaxhighlight>
* Activate one of these
<syntaxhighlight lang="bash">
source ~/virtualenvs/test/bin/activate
</syntaxhighlight>
(After running this command your virtual environment is activated. After you logoff and then logon again your virtual environment will not be loaded so you must rerun this command every time you logon.)
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>

==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment test
<syntaxhighlight lang="bash">
#!/bin/bash
module load Python/3.6.3-iomkl-2017beocatb
source ~/virtualenvs/test/bin/activate
export PYTHONDONTWRITEBYTECODE=1
python ~/path/to/your/python/script.py
</syntaxhighlight>

=== [http://spark.apache.org/ Spark] ===

Spark is a programming language for large scale data processing.
It can be used in conjunction with Python, R, Scala, Java, and SQL.
Spark can be run on Beocat interactively or through the Slurm queue.

To run interactively, you must first request a node or nodes from the Slurm queue.
The line below requests 1 node and 1 core for 24 hours and if available will drop
you into the bash shell on that node.

srun -J srun -N 1 -n 1 -t 24:00:00 --mem=10G --pty bash

We have some sample python based Spark code you can try out that came from the
exercises and homework from the PSC Spark workshop.

mkdir spark-test
cd spark-test
cp -rp /homes/daveturner/projects/PSC-BigData-Workshop/Shakespeare .

Then load the Spark and Python modules, change to the sample directory, fire up pyspark, and run the sample code.

module load Spark Python
cd Shakespeare
pyspark
>>> exec(open("shakespeare.py").read())

You can work interactively from the pyspark prompt (>>>) in addition to running scripts as above.

The Shakespeare directory also contains a sample sbatch submit script that will run the
same shakespeare.py code through the Slurm batch queue.

#!/bin/bash -l
#SBATCH --job-name=shakespeare
#SBATCH --mem=10G
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

# Load Spark and Python (version 3 here)
module load Spark
module load Python

spark-submit shakespeare.py

When you run interactively, pyspark initializes your spark context sc.
You will need to do this manually as in the sample python code when you want
to submit jobs through the Slurm queue.

# If there is no Spark Context (not running interactive from pyspark), create it
try:
sc
except NameError:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("App")
sc = SparkContext(conf = conf)

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.

If you need a newer version (or threads), just load one we provide in our modules (module spider Perl):
* Perl/5.26.0-foss-2017beocatb
* Perl/5.26.0-iompi-2017beocatb

==== Submitting a job with Perl ====
Much like R (above), you cannot simply '<tt>sbatch myProgram.pl</tt>', but you must create a [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|submit script]] which will call perl. Here is an example:
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS)
#SBATCH --time=0-0:15:00
# Now lets do some actual work.
module load Perl
perl /path/to/myProgram.pl
</syntaxhighlight>

=== Octave for MatLab codes ===

module load Octave/4.2.1-foss-2017beocatb-enable64

The 64-bit version of Octave can be loaded using the command above. Octave can then be used
to work with MatLab codes on the head node and to submit jobs to the compute nodes through the
sbatch scheduler. Octave is made to run MatLab code, but it does have limitations and does not support
everything that MatLab itself does.

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=octave
#SBATCH --output=octave.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module purge
module load Octave/4.2.1-foss-2017beocatb-enable64

octave < matlab_code.m
</syntaxhighlight>

=== MatLab compiler ===

Beocat also has a single-user license for the MatLab compiler and the most common toolboxes
including the Parallel Computing Toolbox, Optimization Toolbox, Statistics and Machine Learning Toolbox,
Image Processing Toolbox, Curve Fitting Toolbox, Neural Network Toolbox, Sumbolic Math Toolbox,
Global Optimization Toolbox, and the Bioinformatics Toolbox.

Since we only have a single-user license, this means that you will be expected to develop your MatLab code
with Octave or elsewhere on a laptop or departmental server. Once you're ready to do large runs, then you
move your code to Beocat, compile the MatLab code into an executable, and you can submit as many jobs as
you want to the scheduler. To use the MatLab compiler, you need to load the MATLAB module to compile code and
load the mcr module to run the resulting MatLab executable.

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -o matlab_executable_name
</syntaxhighlight>

If you have addpath() commands in your code, you will need to wrap them in an "if ~deployed" block and tell the
compiler to include that path via the -I flag.

<syntaxhighlight lang="MATLAB">
% wrap addpath() calls like so:
if ~deployed
addpath('./another/folder/with/code/')
end
</syntaxhighlight>

NOTE: The license manager checks the mcc compiler out for a minimum of 30 minutes, so if another user compiles a code
you unfortunately may need to wait for up to 30 minutes to compile your own code.

Compiling with additional paths:

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -I ./another/folder/with/code/ -o matlab_executable_name
</syntaxhighlight>

Any directories added with addpath() will need to be added to the list of compile options as -I arguments. You
can have multiple -I arguments in your compile command.

Here is an example job submission script. Modify time, memory, tasks-per-node, and job name as you see fit:

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=matlab
#SBATCH --output=matlab.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module purge
module load mcr

./matlab_executable_name
</syntaxhighlight>

For those who make use of mex files - compiled C and C++ code with matlab bindings - you will need to add these
files to the compiled archive via the -a flag. See the behavior of this flag in the [https://www.mathworks.com/help/compiler/mcc.html compiler documentation]. You can either target specific .mex files or entire directories.

Because codes often require adding several directories to the Matlab path as well as mex files from several locations,
we recommend writing a script to preserve and help document the steps to compile your Matlab code. Here is an
abbreviated example from a current user:

<syntaxhighlight lang="bash">
#!/bin/bash -l

module load MATLAB

cd matlabPyrTools/MEX/

# compile mex files
mex upConv.c convolve.c wrap.c edges.c
mex corrDn.c convolve.c wrap.c edges.c
mex histo.c
mex innerProd.c

cd ../..

mcc -m mongrel_creation.m \
-I ./matlabPyrTools/MEX/ \
-I ./matlabPyrTools/ \
-I ./FastICA/ \
-a ./matlabPyrTools/MEX/ \
-a ./texturesynth/ \
-o mongrel_creation_binary
</syntaxhighlight>

Again, we only have a single-user license for MatLab so the model is to develop and debug your MatLab code
elsewhere or using Octave on Beocat, then you can compile the MatLab code into an executable and run it without
limits on Beocat.

For more info on the mcc compiler see: https://www.mathworks.com/help/compiler/mcc.html

=== COMSOL ===
Beocat has no license for COMSOL. If you want to use it, you must provide your own.

module spider COMSOL
----------------------------------------------------------------------------
COMSOL: COMSOL/5.3
----------------------------------------------------------------------------
Description:
COMSOL Multiphysics software, an interactive environment for modeling
and simulating scientific and engineering problems

This module can be loaded directly: module load COMSOL/5.3

Help:

Description
===========
COMSOL Multiphysics software, an interactive environment for modeling and
simulating scientific and engineering problems
You must provide your own license.
export LM_LICENSE_FILE=/the/path/to/your/license/file
*OR*
export LM_LICENSE_FILE=$LICENSE_SERVER_PORT@$LICENSE_SERVER_HOSTNAME
e.g. export LM_LICENSE_FILE=1719@some.flexlm.server.ksu.edu

More information
================
- Homepage: https://www.comsol.com/
==== Graphical COMSOL ====
Running COMSOL in graphical mode on a cluster is generally a bad idea. If you choose to run it in graphical mode on a compute node, you will need to do something like the following:
<syntaxhighlight lang="bash">
# Connect to the cluster with X11 forwarding (ssh -Y or mobaxterm)
# load the comsol module on the headnode
module load COMSOL
# export your comsol license as mentioned above, and tell the scheduler to run the software
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 comsol -3drend sw
</syntaxhighlight>

=== .NET Core ===
==== Load .NET ====
mozes@[eunomia] ~ $ module load dotNET-Core-SDK
==== create an application ====
Following instructions from [https://docs.microsoft.com/en-us/dotnet/core/tutorials/using-with-xplat-cli here], we'll create a simple 'Hello World' application
mozes@[eunomia] ~ $ mkdir Hello

mozes@[eunomia] ~ $ cd Hello

mozes@[eunomia] ~/Hello $ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true

mozes@[eunomia] ~/Hello $ dotnet new console
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /homes/mozes/Hello/Hello.csproj...
Restoring packages for /homes/mozes/Hello/Hello.csproj...
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.props.
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.targets.
Restore completed in 358.43 ms for /homes/mozes/Hello/Hello.csproj.

Restore succeeded.

==== Edit your program ====
mozes@[eunomia] ~/Hello $ vi Program.cs
==== Run your .NET application ====
mozes@[eunomia] ~/Hello $ dotnet run
Hello World!
==== Build and run the built application ====
mozes@[eunomia] ~/Hello $ dotnet build
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

Restore completed in 106.12 ms for /homes/mozes/Hello/Hello.csproj.
Hello -> /homes/mozes/Hello/bin/Debug/netcoreapp2.1/Hello.dll

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:02.86

mozes@[eunomia] ~/Hello $ dotnet bin/Debug/netcoreapp2.1/Hello.dll
Hello World!

== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

Main Page

2018-08-15T21:44:31Z

Kylehutson: Add Twitter

SlurmBasics

2018-05-10T18:36:39Z

Kylehutson: Fixed a typo in example

== The CentOS/Slurm nodes ==

We have converted Beocat from Gentoo Linux to CentOS Linux on December 26th of 2017. Any applications or libraries from the old system must be recompiled. We also converted Beocat to use the Slurm scheduler instead of SGE. You will therefore also need to convert all your old qsub scripts over to sbatch scripts. We have developed tools to make this process as easy as possible.

<H3>Using Modules</H3>

If you're using a common code that others may also be using, we may already have it compiled in a module. You can list the modules available and load an application as in the example below for Vasp.

eos> module avail 
eos> module load VASP 
eos> module list

When a module gets loaded, all the necessary libraries are also loaded and the paths to the libraries and executables are automatically set up. Loading Vasp for example also loads the OpenMPI library needed to run it and adds the path to the MPI commands and Vasp executables. To see how the path is set up, try executing which vasp_std. The module system allows you to easily switch between different version of applications, libraries, or languages as well.

If you are using a custom code or one that is not installed in a module, you'll need to recompile it yourself. This process is easier under CentOS as some of the work just involves loading the necessary set of modules. The first step is to decide whether to use the Intel compiler toolchain or the GNU toolchain, each of which includes the compilers and other math libraries. The module commands for each are below, and you can load these automatically when you log in by adding one of these module load statements to your .bashrc file. See /homes/daveturner/.bashrc as an example, where I put the module load statements .

To load the Intel compiler tool chain including the Intel Math Kernel Library: 
eos> module load iomkl 

To load the GNU compiler tool chain including OpenBLAS, FFTW, and ScalaPack load foss (free open source software): 
eos> module load foss 

Modules provide an easy way to set up the compilers and libraries you may need to compile your code. Beyond that there are many different ways to compile codes so you'll just need to follow the directions. If you need help you can always email us at beocat@cs.ksu.edu.

<H3>Converting your qsub script for sbatch using kstat.convert</H3>

If you already have a qsub script, I have created a new perl program called kstat.convert that will automatically convert your qsub script over to an sbatch script.

kstat.convert --sge qsub_script.sh --slurm slurm_script.sh

Below is an example of a simple qsub script and the resulting sbatch script after conversion.

<syntaxhighlight lang="bash">
#!/bin/bash
#$ -j y
#$ -cwd
#$ -N netpipe
#$ -P KSU-CIS-HPC

#$ -l mem=4G
#$ -l h_rt=100:00:00
#$ -pe single 32

#$ -M daveturner@ksu.edu
#$ -m ab

mpirun -np $NSLOTS NPmpi -o np.out
</syntaxhighlight>

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=netpipe

#SBATCH --mem-per-cpu=4G # Memory per core, use --mem= for memory per node
#SBATCH --time=4-04:00:00 # Use the form DD-HH:MM:SS
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=32

#SBATCH --mail-user=daveturner@ksu.edu
#SBATCH --mail-type=ALL # same as =BEGIN,FAIL,END

mpirun -np $SLURM_NPROCS NPmpi -o np.out
</syntaxhighlight>

The sbatch file uses #SBATCH to identify command options for the scheduler where the qsub file uses #$. Most options are similar but simply use a different syntax. The memory can still be defined on a per core basis as with SGE, or you can use --mem=128G to specify the total memory per node if you'd prefer. The --nodes= and --ntasks-per-node= provide an easy way to request the core configuration you want. If your code can be distributed across multiple nodes and you don't care what the arrangement is, you can instead just specify the number of cores using --ntasks=. For more in depth documentation on converting from SGE to Slurm follow the links below:

https://srcc.stanford.edu/sge-slurm-conversion 
https://slurm.schedmd.com/sbatch.html

<H3>Submitting jobs to Slurm</H3>

Once your qsub script has been converted to an sbatch script and you have an application compiled for CentOS, you can submit the job using the sbatch command.

eos> sbatch sbatch_script.sh 
eos> kstat --me

This will submit the script and show you a list of your jobs that are running and the jobs you have in the queue. By default the output for each job will go into a slurm-###.out file where ### is the job ID number. If you need to kill a job, you can use the scancel command with the job ID number.

== Submitting your first job ==
To submit a job to run under Slurm, we use the sbatch (submit batch) command. The scheduler finds the optimum place for your job to run. With over 300 nodes and 7500 cores to schedule, as well as differing priorities, hardware, and individual resources, the scheduler's job is not trivial and it can take some time for a job to start even when there are empty nodes available.

There are a few things you'll need to know before running sbatch.
* How many cores you need. Note that unless your program is created to use multiple cores (called "threading"), asking for more cores will not speed up your job. This is a common misperception. '''Beocat will not magically make your program use multiple cores!''' For this reason the default is 1 core.
* How much time you need. Many users when beginning to use Beocat neglect to specify a time requirement. The default is one hour, and we get asked why their job died after one hour. We usually point them to the [[FAQ]].
* How much memory you need. The default is 1 GB. If your job uses significantly more than you ask, your job will be killed off.
* Any advanced options. See the [[AdvancedSlurm]] page for these requests. For our basic examples here, we will ignore these.

So let's now create a small script to test our ability to submit jobs. Create the following file (either by copying it to Beocat or by editing a text file and we'll name it <code>myhost.sh</code>. Both of these methods are documented on our [[LinuxBasics]] page.
<syntaxhighlight lang="bash" line>
#!/bin/sh
srun hostname
</syntaxhighlight>

Be sure to make it executable
chmod u+x myhost.sh

So, now lets submit it as a job and see what happens. Here I'm going to use five options
* <code>--mem-per-cpu=</code> tells how much memory I need. In my example, I'm using our system minimum of 512 MB, which is more than enough. Note that your memory request is '''per core''', which doesn't make much difference for this example, but will as you submit more complex jobs.
* <code>--time=</code> tells how much runtime I need. This can be in the form of "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". This is a very short job, so 1 minute should be plenty. This can't be changed after the job is started please make sure you have requested a sufficient amount of time.
* <code>--cpus-per-task=1</code> tells Slurm that I need only a single core per task. The [[AdvancedSlurm]] page has much more on the "cpus-per-task" switch.
* <code>--ntasks=1</code> tells Slurm that I only need to run 1 task. The [[AdvancedSlurm]] page has much more on the "ntasks" switch.
* <code>--nodes=1</code> tells Slurm that this must be run on one machine. The [[AdvancedSlurm]] page has much more on the "nodes" switch.
* <code>--nodes=4 --ntasks-per-node=16 --constraint=elves</code> requests 4 nodes with 16 cores on each and to only use the Elves.

% '''ls'''
myhost.sh
% '''sbatch --time=1 --mem-per-cpu=512M --cpus-per-task=1 --ntasks=1 --nodes=1 ./myhost.sh'''
salloc: Granted job allocation 1483446

Since this is such a small job, it is likely to be scheduled almost immediately, so a minute or so later, I now see
% '''ls'''
myhost.sh
slurm-1483446.out

% '''cat slurm-1483446.out'''
mage03

== Monitoring Your Job ==

The kstat perl script has been developed at K-State to provide you with all the available information about your jobs on Beocat. kstat --help will give you a full description of how to use it.
The Slurm version of kstat is very similar to the SGE version, with the exception that the actual memory usage of each job is not always available so the
memory requested is reported, and the memory usage on each node is not always accurate since Slurm includes disk cache. We are continuing to look
for better ways to get the memory usage for each job, but at the moment you may need to use [http://ganglia.beocat.ksu.edu/ Ganglia] and look at the
memory graph for the node you are running on to get an accurate idea of the memory being used by your application.

Eos> kstat --help

USAGE: kstat [-q] [-c] [-g] [-l] [-u user] [-p NaMD] [-j 1234567] [--part partition]
kstat alone dumps all info except for the core summaries
choose -q -c for only specific info on queued or core summaries.
then specify any searchables for the user, program name, or job id

kstat info on running and queued jobs
kstat -q info on the queued jobs only
kstat -c core usage for each user
kstat -g gpu nodes only
kstat -l -h long list - prints full node list
kstat -u daveturner job info for one user only
kstat --me job info for my jobs only
kstat -j 1234567 info on a given job id
kstat --nocolor do not use any color

--------------------------------------------------------------------------
Multi-node jobs are highlighted in Magenta
The switch and nodes/switch are on the right
highlighted in Yellow when nodes are spread across multiple switches
Shared jobs are highlighted in Cyan
Memory requested is reported along with the total used when available
Total RSS / Total VMSize / Total requested
Runtime is colorized with yellow then red for jobs nearing their time limit
Time in the queue is colorized yellow then red for jobs waiting long times
--------------------------------------------------------------------------

kstat can be used to give you a summary of your jobs that are running and in the queue: 
Eos> kstat --me 


Hero43      
24 of 24 cores      
Load 23.4 / 24      
495.3 / 512 GB used 
    
daveturner      
unafold        1234567      
1 core               
running               
 4gb req                
 0 d 5 h 35 m 
    
daveturner      
octopus       1234568     
16 core              
running               
 128gb req                
 8 d 15 h 42 m 
 ################################## BeoCat Queue ################################### 
    
daveturner      
NetPIPE       1234569      
2 core    
 PD  
 2h  
 4gb req      
 0 d 1 h 2 m 


kstat produces a separate line for each host. Use kstat -h to see information on all hosts without the jobs.
For the example above we are listing our jobs and the hosts they are on.

Core usage - yellow for empty, red for empty on owned nodes, cyan for partially used, blue for all cores used. 
Load level - yellow or yellow background indicates the node is being inefficiently used. Red just means more threads than cores. 
Memory usage - yellow or red means most memory is used. 
If the node is owned the group name will be in orange on the right. Killable jobs of 24 hours or less can still be run on those nodes. 

Each job line will contain the username, program name, job ID, number of cores, the status which may be colored red for killable jobs,
the maximum memory used or memory requested, and the amount of time the job has run.
Jobs in the queue may contain information on the requested memory and run time, priority access, constraints, and
how long the job has been in the queue.
In this case, I have 2 jobs running on Hero43. unafold is using 1 core while octopus is using 16 cores. Slurm did not provide
any information on the actual memory use so the memory request is reported

Detailed information about a single job

kstat can provide a get a great deal of information on a particular job including a very rough estimate of when it will run. This time is a worst case scenario as this will
be adapted as other jobs finish early. This is a good way to check for job submission problems before contacting us. kstat colorizes the more important
information to make it easier to identify.

Eos> kstat -j 157054

################################## Beocat Queue ###################################
daveturner netpipe 157054 64 cores PD dwarves fabric CS HPC 8gb req 0 d 0 h 0 m

JobId 157054 Job Name netpipe
UserId=daveturner GroupId=daveturner_users(2117) MCS_label=N/A
Priority=11112 Nice=0 Account=ksu-cis-hpc QOS=normal
Status=PENDING Reason=Resources Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=00:40:00 TimeMin=N/A
SubmitTime=2018-02-02T18:18:31 EligibleTime=2018-02-02T18:18:31
Estimated Start Time is 2018-02-03T06:17:49 EndTime=2018-02-03T06:57:49 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partitions killable.q,ksu-cis-hpc.q AllocNode:Sid=eos:1761
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null) SchedNodeList=dwarf[01-02]
NumNodes=2-2 NumCPUs=64 NumTasks=64 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES 2 nodes 64 cores 8192 mem gres/fabric 2
Socks/Node=* NtasksPerN:B:S:C=32:0:*:* CoreSpec=*
MinCPUsNode=32 MinMemoryNode=4G MinTmpDiskNode=0
Constraint=dwarves DelayBoot=00:00:00
Gres=fabric Reservation=(null)
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Slurm script /homes/daveturner/perf/NetPIPE-5.x/sb.np
WorkDir=/homes/daveturner/perf/NetPIPE-5.x
StdErr=/homes/daveturner/perf/NetPIPE-5.x/0.o157054
StdIn=/dev/null
StdOut=/homes/daveturner/perf/NetPIPE-5.x/0.o157054
Switches=1@00:05:00

#!/bin/bash -l
#SBATCH --job-name=netpipe
#SBATCH -o 0.o%j
#SBATCH --time=0:40:00
#SBATCH --mem=4G
#SBATCH --switches=1
#SBATCH --nodes=2
#SBATCH --constraint=dwarves
#SBATCH --ntasks-per-node=32
#SBATCH --gres=fabric:roce:1

host=`echo $SLURM_JOB_NODELIST | sed s/[^a-z0-9]/\ /g | cut -f 1 -d ' '`
nprocs=$SLURM_NTASKS
openmpi_hostfile.pl $SLURM_JOB_NODELIST 1 hf.$host
opts="--printhostnames --quick --pert 3"

echo "*******************************************************************"
echo "Running on $SLURM_NNODES nodes $nprocs cores on nodes $SLURM_JOB_NODELIST"
echo "*******************************************************************"

mpirun -np 2 --hostfile hf.$host NPmpi $opts -o np.${host}.mpi
mpirun -np 2 --hostfile hf.$host NPmpi $opts -o np.${host}.mpi.bi --async --bidir
mpirun -np $nprocs NPmpi $opts -o np.${host}.mpi$nprocs --async --bidir

Completed jobs and memory usage

kstat -d #

This will provide information on the jobs you have currently running and those that have completed
in the last '#' days. This is currently the only reliable way to get the memory used per node for your job.
This also provides information on whether the job completed normally, was canceled with scancel,
timed out, or was killed because it exceeded its memory request.

Eos> kstat -d 10

########################### sacct -u daveturner for 10 days ###########################
max gb used on a node / gb requested per node
193037 ADF dwarf43 1 n 32 c 30.46gb/100gb 05:15:34 COMPLETED
193289 ADF dwarf33 1 n 32 c 26.42gb/100gb 00:50:43 CANCELLED
195171 ADF dwarf44 1 n 32 c 56.81gb/120gb 14:43:35 COMPLETED
209518 matlab dwarf36 1 n 1 c 0.00gb/ 4gb 00:00:02 FAILED

Summary of core usage

kstat can also provide a listing of the core usage and cores requested for each user. 

Eos> kstat -c

############################## Core usage ###############################
antariksh 1512 cores %25.1 used 41528 cores queued
bahadori 432 cores % 7.2 used 80 cores queued
eegoetz 0 cores % 0.0 used 2 cores queued
fahrialkan 24 cores % 0.4 used 32 cores queued
gowri 66 cores % 1.1 used 32 cores queued
jeffcomer 160 cores % 2.7 used 0 cores queued
ldcoates12 80 cores % 1.3 used 112 cores queued
lukesteg 464 cores % 7.7 used 0 cores queued
mike5454 1060 cores %17.6 used 852 cores queued
nilusha 344 cores % 5.7 used 0 cores queued
nnshan2014 136 cores % 2.3 used 0 cores queued
ploetz 264 cores % 4.4 used 60 cores queued
sadish 812 cores %13.5 used 0 cores queued
sandung 72 cores % 1.2 used 56 cores queued
zhiguang 80 cores % 1.3 used 688 cores queued

If you want to read more, continue on to our [[AdvancedSlurm]] page.

AdvancedSlurm

2018-03-13T16:27:19Z

Kylehutson: Clarified how to delete a job

== Resource Requests ==
Aside from the time, RAM, and CPU requirements listed on the [[SlurmBasics]] page, we have a couple other requestable resources:
Valid gres options are:
gpu[[:type]:count]
fabric[[:type]:count]
Generally, if you don't know if you need a particular resource, you should use the default. These can be generated with the command
<tt>srun --gres=help</tt>
=== Fabric ===
We currently offer 3 "fabrics" as request-able resources in Slurm. The "count" specified is the line-rate (in Gigabits-per-second) of the connection on the node.
==== Infiniband ====
First of all, let me state that just because it sounds "cool" doesn't mean you need it or even want it. InfiniBand does absolutely no good if running on a single machine. InfiniBand is a high-speed host-to-host communication fabric. It is (most-often) used in conjunction with MPI jobs (discussed below). Several times we have had jobs which could run just fine, except that the submitter requested InfiniBand, and all the nodes with InfiniBand were currently busy. In fact, some of our fastest nodes do not have InfiniBand, so by requesting it when you don't need it, you are actually slowing down your job. To request Infiniband, add <tt>--gres=fabric:ib:1</tt> to your sbatch command-line.
==== ROCE ====
ROCE, like InfiniBand is a high-speed host-to-host communication layer. Again, used most often with MPI. Most of our nodes are ROCE enabled, but this will let you guarantee the nodes allocated to your job will be able to communicate with ROCE. To request ROCE, add <tt>--gres=fabric:roce:1</tt> to your sbatch command-line.

==== Ethernet ====
Ethernet is another communication fabric. All of our nodes are connected by ethernet, this is simply here to allow you to specify the interconnect speed. Speeds are selected in units of Gbps, with all nodes supporting 1Gbps or above. The currently available speeds for ethernet are: <tt>1, 10, 40, and 100</tt>. To select nodes with 40Gbps and above, you could specify <tt>--gres=fabric:eth:40</tt> on your sbatch command-line. Since ethernet is used to connect to the file server, this can be used to select nodes that have fast access for applications doing heavy IO. The Dwarves and Heroes have 40 Gbps ethernet and we measure single stream performance as high as 20 Gbps, but if your application
requires heavy IO then you'd want to avoid the Moles which are connected to the file server with only 1 Gbps ethernet.

=== CUDA ===
[[CUDA]] is the resource required for GPU computing. We have a very small number of nodes which have GPUs installed. To request one of these gpus on of of these nodes, add <tt>--gres=gpu:1</tt> to your sbatch command-line.
== Parallel Jobs ==
There are two ways jobs can run in parallel, ''intra''node and ''inter''node. '''Note: Beocat will not automatically make a job run in parallel.''' Have I said that enough? It's a common misperception.
=== Intranode jobs ===
Intranode jobs which run on many cores in the same node are easier to code and can take advantage of many common libraries, such as [http://openmp.org/wp/ OpenMP], or Java's threads. Many times, your program will need to know how many cores you want it to use. Many will use all available cores if not told explicitly otherwise. This can be a problem when you are sharing resources, as Beocat does. To request multiple cores, use the sbatch directives '<tt>--cpus-per-task=n</tt>' or '<tt>--nodes=1 --ntasks-per-node=n</tt>', where ''n'' is the number of cores you wish to use. If your command can take an environment variable, you can use $SLURM_CPUS_ON_NODE to tell how many cores you've been allocated.

=== Internode (MPI) jobs ===
Communicating between nodes is trickier than talking between cores on the same node. The specification for doing so is called "[[wikipedia:Message_Passing_Interface|Message Passing Interface]]", or MPI. We have [http://www.open-mpi.org/ OpenMPI] installed on Beocat for this purpose. Most programs written to take advantage of large multi-node systems will use MPI, but MPI also allows an application to run on multiple cores within a node. You can tell if you have an MPI-enabled program because its directions will tell you to run '<tt>mpirun ''program''</tt>'. Requesting MPI resources is only mildly more difficult than requesting single-node jobs. Instead of using '<tt>--cpus-per-task=''n''</tt>', you would use '<tt>--nodes=''n'' --tasks-per-node=''m''</tt>' ''or'' '<tt>--ntasks=''o''</tt>' for your sbatch request, where ''n'' is the number of nodes you want, ''m'' is the number of cores per node you need, and ''o'' is the total number of cores you need.

Some quick examples:

<tt>--nodes=6 --ntasks-per-node=4</tt> will give you 4 cores on each of 6 nodes for a total of 24 cores.

<tt>--ntasks=40</tt> will give you 40 cores spread across any number of nodes.

<tt>--ntasks=100</tt> will give you 100 cores on any number of nodes.

== Requesting memory for multi-core jobs ==
Memory requests are easiest when they are specified '''per core'''. For instance, if you specified the following: '<tt>--tasks=20 --mem-per-core=20G</tt>', your job would have access to 400GB of memory total.
== Other Handy Slurm Features ==
=== Email status changes ===
One of the most commonly used options when submitting jobs not related to resource requests is to have have Slurm email you when a job changes its status. This takes may need two directives to sbatch: <tt>--mail-user</tt> and <tt>--mail-type</tt>.
==== --mail-type ====
<tt>--mail-type</tt> is used to tell Slurm to notify you about certain conditions. Options are comma separated and include the following
{| class="wikitable"
!Option!!Explanation
|-
| NONE || This disables event-based mail
|-
| BEGIN || Sends a notification when the job begins
|-
| END || Sends a notification when the job ends
|-
| FAIL || Sends a notification when the job fails.
|-
| REQUEUE || Sends a notification if the job is put back into the queue from a running state
|-
| STAGE_OUT || Burst buffer stage out and teardown completed
|-
| ALL || Equivalent to BEGIN,END,FAIL,REQUEUE,STAGE_OUT
|-
| TIME_LIMIT || Notifies if the job ran out of time
|-
| TIME_LIMIT_90 || Notifies when the job has used 90% of its allocated time
|-
| TIME_LIMIT_80 || Notifies when the job has used 80% of its allocated time
|-
| TIME_LIMIT_50 || Notifies when the job has used 50% of its allocated time
|-
| ARRAY_TASKS || Modifies the BEGIN, END, and FAIL options to apply to each array task (instead of notifying for the entire job
|}

==== --mail-user ====
<tt>--mail-user</tt> is optional. It is only needed if you intend to send these job status updates to a different e-mail address than what you provided in the [https://acount.beocat.ksu.edu/user Account Request Page]. It is specified with the following arguments to sbatch: <tt>--mail-user=someone@somecompany.com</tt>

=== Job Naming ===
If you have several jobs in the queue, running the same script with different parameters, it's handy to have a different name for each job as it shows up in the queue. This is accomplished with the '<tt>-J ''JobName''</tt>' sbatch directive.

=== Separating Output Streams ===
Normally, Slurm will create one output file, containing both STDERR and STDOUT. If you want both of these to be separated into two files, you can use the sbatch directives '<tt>--output</tt>' and '<tt>--error</tt>'.

{| class="wikitable"
! option !! default !! example
|-
| --output || slurm-%j.out || slurm-206.out
|-
| --error || slurm-%j.out || slurm-206.out
|}
<tt>%j</tt> above indicates that it should be replaced with the job id.

=== Running from the Current Directory ===
By default, jobs run from your home directory. Many programs incorrectly assume that you are running the script from the current directory. You can use the '<tt>-cwd</tt>' directive to change to the "current working directory" you used when submitting the job.
=== Running in a specific class of machine ===
If you want to run on a specific class of machines, e.g., the Dwarves, you can add the flag "--constraint=dwarves" to select any of those machines.

=== Processor Constraints ===
Because Beocat is a heterogenous cluster (we have machines from many years in the cluster), not all of our processors support every new and fancy feature. You might have some applications that require some newer processor features, so we provide a mechanism to request those.

<tt>--contraint</tt> tells the cluster to apply constraints to the types of nodes that the job can run on. For instance, we know of several applications that must be run on chips that have "AVX" processor extensions. To do that, you would specify <tt>--constraint=avx</tt> on you ''<tt>sbatch</tt>'' '''or''' ''<tt>srun</tt>'' command lines.
Using <tt>--constraint=AVX</tt> will prohibit your job from running on the Mages while <tt>--contraint=AVX2</tt> will eliminate the Elves as well as the Mages.

=== Slurm Environment Variables ===
Within an actual job, sometimes you need to know specific things about the running environment to setup your scripts correctly. Here is a listing of environment variables that Slurm makes available to you. Of course the value of these variables will be different based on many different factors.
<syntaxhighlight lang="bash">
CUDA_VISIBLE_DEVICES=NoDevFiles
ENVIRONMENT=BATCH
GPU_DEVICE_ORDINAL=NoDevFiles
HOSTNAME=dwarf37
SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint
SLURM_CLUSTER_NAME=beocat
SLURM_CPUS_ON_NODE=1
SLURM_DISTRIBUTION=cyclic
SLURMD_NODENAME=dwarf37
SLURM_GTIDS=0
SLURM_JOB_CPUS_PER_NODE=1
SLURM_JOB_GID=163587
SLURM_JOB_ID=202
SLURM_JOBID=202
SLURM_JOB_NAME=slurm_simple.sh
SLURM_JOB_NODELIST=dwarf37
SLURM_JOB_NUM_NODES=1
SLURM_JOB_PARTITION=batch.q,killable.q
SLURM_JOB_QOS=normal
SLURM_JOB_UID=163587
SLURM_JOB_USER=mozes
SLURM_LAUNCH_NODE_IPADDR=10.5.16.37
SLURM_LOCALID=0
SLURM_MEM_PER_NODE=1024
SLURM_NNODES=1
SLURM_NODEID=0
SLURM_NODELIST=dwarf37
SLURM_NPROCS=1
SLURM_NTASKS=1
SLURM_PRIO_PROCESS=0
SLURM_PROCID=0
SLURM_SRUN_COMM_HOST=10.5.16.37
SLURM_SRUN_COMM_PORT=37975
SLURM_STEP_ID=0
SLURM_STEPID=0
SLURM_STEP_LAUNCHER_PORT=37975
SLURM_STEP_NODELIST=dwarf37
SLURM_STEP_NUM_NODES=1
SLURM_STEP_NUM_TASKS=1
SLURM_STEP_TASKS_PER_NODE=1
SLURM_SUBMIT_DIR=/homes/mozes
SLURM_SUBMIT_HOST=dwarf37
SLURM_TASK_PID=23408
SLURM_TASKS_PER_NODE=1
SLURM_TOPOLOGY_ADDR=due1121-prod-core-40g-a1,due1121-prod-core-40g-c1.due1121-prod-sw-100g-a9.dwarf37
SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node
SLURM_UMASK=0022
SRUN_DEBUG=3
TERM=screen-256color
TMPDIR=/tmp
USER=mozes
</syntaxhighlight>
Sometimes it is nice to know what hosts you have access to during a job. You would checkout the SLURM_JOB_NODELIST to know that. There are lots of useful Environment Variables there, I will leave it to you to identify the ones you want.

Some of the most commonly-used variables we see used are $SLURM_CPUS_ON_NODE, $HOSTNAME, and $SLURM_JOB_ID.

== Running from a sbatch Submit Script ==
No doubt after you've run a few jobs you get tired of typing something like 'sbatch -l mem=2G,h_rt=10:00 -pe single 8 -n MyJobTitle MyScript.sh'. How are you supposed to remember all of these every time? The answer is to create a 'submit script', which outlines all of these for you. Below is a sample submit script, which you can modify and use for your own purposes.
<syntaxhighlight lang="bash">
#!/bin/bash

## A Sample sbatch script created by Kyle Hutson
##
## Note: Usually a '#" at the beginning of the line is ignored. However, in
## the case of sbatch, lines beginning with #SBATCH are commands for sbatch
## itself, so I have taken the convention here of starting *every* line with a
## '#', just Delete the first one if you want to use that line, and then modify
## it to your own purposes. The only exception here is the first line, which
## *must* be #!/bin/bash (or another valid shell).

## Specify the amount of RAM needed _per_core_. Default is 1G
##SBATCH --mem-per-cpu=1G

## Specify the maximum runtime in DD-HH:MM:SS form. Default is 1 hour (1:00:00)
##SBATCH --time=1:00:00

## Require the use of infiniband. If you don't know what this is, you probably
## don't need it.
##SBATCH --gres=fabric:ib:1

## GPU directive. If You don't know what this is, you probably don't need it
##SBATCH --gres:gpu:1

## number of cores/nodes:
## quick note here. Jobs requesting 16 or fewer cores tend to get scheduled
## fairly quickly. If you need a job that requires more than that, you might
## benefit from emailing us at beocat@cs.ksu.edu to see how we can assist in
## getting your job scheduled in a reasonable amount of time. Default is
##SBATCH --cpus-per-task=1
##SBATCH --cpus-per-task=12
##SBATCH --nodes=2 --tasks-per-node=1
##SBATCH --tasks=20

## Constraints for this job. Maybe you need to run on the elves
##SBATCH --constraint=elves
## or perhaps you just need avx processor extensions
##SBATCH --constraint=avx

## Output file name. Default is slurm-%j.out where %j is the job id.
##SBATCH --output=MyJobTitle.o%j

## Split the errors into a seperate file. Default is the same as output
##SBATCH --error=MyJobTitle.e%j

## Name my job, to make it easier to find in the queue
##SBATCH -J MyJobTitle

## And finally, we run the job we came here to do.
## $HOME/ProgramDir/ProgramName ProgramArguments

## OR, for the case of MPI-capable jobs
## mpirun $HOME/path/MpiJobName

## Send email when certain criteria are met.
## Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to
## BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage
## out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent
## of time limit), TIME_LIMIT_80 (reached 80 percent of time limit),
## TIME_LIMIT_50 (reached 50 percent of time limit) and ARRAY_TASKS (send
## emails for each array task). Multiple type values may be specified in a
## comma separated list. Unless the ARRAY_TASKS option is specified, mail
## notifications on job BEGIN, END and FAIL apply to a job array as a whole
## rather than generating individual email messages for each task in the job
## array.
##SBATCH --mail-type=ALL

## Email address to send the email to based on the above line.
## Default is to send the mail to the e-mail address entered on the account
## request form.
##SBATCH --mail-user myemail@ksu.edu
</syntaxhighlight>

== File Access ==
Beocat has a variety of options for storing and accessing your files.
Every user has a home directory for general use which is limited in size, has decent file access performance,
and will soon be backed up nightly. Larger files should be stored in the /bulk subdirectories which have the same decent performance
but are not backed up. The /scratch file system will soon be implemented on a Lustre file system that will provide very fast
temporary file access. When fast IO is critical to the application performance, access to the local disk on each node or to a
RAM disk are the best options.

===Home directory===

Every user has a <tt>/homes/''username''</tt> directory that they drop into when they log into Beocat.
The home directory is for general use and provides decent performance for most file IO.
Disk space in each home directory is limited to 1 TB, so larger files should be kept in the /bulk
directory, and there is a limit of 100,000 files in each subdirectory in your account.
This file system is fully redundant, so 3 specific hard disks would need to fail before any data was lost.
All files will soon be backed up nightly to a separate file server in Nichols Hall, so if you do accidentally
delete something it can be recovered.

===Bulk directory===

Each user also has a <tt>/bulk/''username''</tt> directory where large files should be stored.
File access is the same speed as for the home directories, and the same limit of 100,000 files
per subdirectory applies. There is no limit to the disk space you can use in your bulk directory,
but the files there will not be backed up. They are still redundantly stored so you don't need to
worry about losing data to hardware failures, just don't delete something by accident. Unused files will be automatically removed after two years.
If you need to back up large files in the bulk directory, talk to Dan Andresen (dan@ksu.edu) about
purchasing some hard disks for archival storage.

===Scratch file system===

The /scratch file system will soon be using the Lustre software which is much faster than the
speed of the file access on /homes or /bulk. In order to use scratch, you first need to make a
directory for yourself. Scratch offers greater speed, no limit to the size of files nor the number
of files in each subdirectory. It is meant as temporary space for prepositioning files and accessing them
during runs. Once runs are completed, any files that need to be kept should be moved to your home
or bulk directories since files on the scratch file system get purged after 30 days. Lustre is faster than
the home and bulk file systems in part because it does not redundantly store files by striping them
across multiple disks, so if a hard disk fails data will be lost. When we get scratch set up to use Lustre
we will post the difference in file access rates.

<syntaxhighlight lang=bash>
mkdir /scratch/$USER
</syntaxhighlight>

===Local disk===

If you are running on a single node, it may also be faster to access your files from the local disk
on that node. Each job creates a subdirectory /tmp/job# where '#' is the job ID number on the
local disk of each node the job uses. This can be accessed simply by writing to /tmp rather than
needing to use /tmp/job#.

You may need to copy files to
local disk at the start of your script, or set the output directory for your application to point
to a file on the local disk, then you'll need to copy any files you want off the local disk before
the job finishes since Slurm will remove all files in your job's directory on /tmp on completion
of the job or when it aborts. When we get the scratch file system working with Lustre, it may
end up being faster than accessing local disk so we will post the access rates for each. Use 'kstat -l -h'
to see how much /tmp space is available on each node.

<syntaxhighlight lang=bash>
# Copy input files to the tmp directory if needed
cp $input_files /tmp

# Make an 'out' directory to pass to the app if needed
mkdir /tmp/out

# Example of running an app and passing the tmp directory in/out
app -input_directory /tmp -output_directory /tmp/out

# Copy the 'out' directory back to the current working directory after the run
cp -rp /tmp/out .
</syntaxhighlight>

===RAM disk===

If you need ultrafast access to files, you can use a RAM disk which is a file system set up in the
memory of the compute node you are running on. The RAM disk is limited to the requested memory on that node, so you should account for this usage when you request
memory for your job. Below is an example of how to use the RAM disk.

<syntaxhighlight lang=bash>
# Copy input files over if necessary
cp $any_input_files /dev/shm/

# Run the application, possibly giving it the path to the RAM disk to use for output files
app -output_directory /dev/shm/

# Copy files from the RAM disk to the current working directory and clean it up
cp /dev/shm/* .
</syntaxhighlight>

===When you leave KSU===

If you are done with your account and leaving KSU, please clean up your directory, move any files
to your supervisor's account that need to be kept after you leave, and notify us so that we can disable your
account. The easiest way to move your files to your supervisor's account is for them to set up
a subdirectory for you with the appropriate write permissions. The example below shows moving
just a user's 'data' subdirectory to their supervisor. The 'nohup' command is used so that the move will
continue even if the window you are doing the move from gets disconnected.

<syntaxhighlight lang=bash>
# Supervisor:
mkdir /bulk/$USER/$STUDENT_USERNAME
chmod ugo+w /bulk/$USER/$STUDENT_USERNAME

# Student:
nohup mv /homes/$USER/data /bulk/$SUPERVISOR_USERNAME/$USER &
</syntaxhighlight>

==File Sharing==

This section will cover methods of sharing files with other users within Beocat and on remote systems.

===Securing your home directory===

By default your home directory is accessible to other users on Beocat for reading but not writing. If you do not want others to have any
access to files in your home directory, you can set the permissions to restrict access to just yourself.

chmod go-rwx /homes/your_user_name

This removes read, write, and execute permission to everyone but yourself. Be aware that it may make it more difficult for us to help you out when
you run into problems.

===Sharing files within your group===

By default all your files and directories have a 'group' that is your user name followed by _users as 'ls -l' shows.
In my case they have the group of daveturner_users.
If your working group owns any nodes on Beocat, then you have a group name that can be used to securely share
files with others within your group. Below is an example of creating a directory called 'share', changing the group
to ksu-cis-hpc (my group is ksu-cis-hpc so I submit jobs to --partition=ksu-cis-hpc.q), then changing the permissions to restrict access to
just that group.

mkdir share
chgrp ksu-cis-hpc share
chmod g+rx share
chmod o-rwx share

This will give people in your group the ability to read files in the 'share' directory. If you also want
them to be able to write or modify files in that directory then use 'chmod g+rwx' instead.

If you want to know what groups you belong to use the line below.

groups

If your group does not own any nodes, you can still request a group name and manage the participants yourself.

===Openly sharing files on the web===

If you create a 'public_html' directory on your home directory, then any files put there will be shared
openly on the web. There is no way to restrict who has access to those files.

cd
mkdir public_html

Then access the data from a web browser using the URL:

http://people.beocat.ksu.edu/~your_user_name

This will show a list of the files you have in your public_html subdirectory.

===Globus===

Kyle will put some Globus stuff here

== Array Jobs ==
One of Slurm's useful options is the ability to run "Array Jobs"

It can be used with the following option to sbatch.

--array=n[-m[:s]]
Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Slurm
almost like a series of jobs. The option argument to --arrat specifies the number of array job tasks and the index number which will be
associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SLURM_ARRAY_TASK_ID. The option
arguments n, and m will be available through the environment variables SLURM_ARRAY_TASK_MIN and SLURM_ARRAY_TASK_MAX.

The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size.
Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SLURM_ARRAY_TASK_ID containing one of the 5 index numbers.

Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The
number of tasks in a array job is unlimited.

STDOUT and STDERR of array job tasks follow a slightly different naming convention (which can be controlled in the same way as mentioned above).

slurm-%A_%a.out

%A is the SLURM_ARRAY_JOB_ID, and %a is the SLURM_ARRAY_TASK_ID

=== Examples ===
==== Change the Size of the Run ====
Array Jobs have a variety of uses, one of the easiest to comprehend is the following:

I have an application, app1 I need to run the exact same way, on the same data set, with only the size of the run changing.

My original script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
RUNSIZE=50
#RUNSIZE=100
#RUNSIZE=150
#RUNSIZE=200
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
For every run of that job I have to change the RUNSIZE variable, and submit each script. This gets tedious.

With Array Jobs the script can be written like so:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=50-200:50
RUNSIZE=$SLURM_ARRAY_TASK_ID
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
I then submit that job, and Slurm understands that it needs to run it 4 times, once for each task. It also knows that it can and should run these tasks in parallel.

==== Choosing a Dataset ====
A slightly more complex use of Array Jobs is the following:

I have an application, app2, that needs to be run against every line of my dataset. Every line changes how app2 runs slightly, but I need to compare the runs against each other.

Originally I had to take each line of my dataset and generate a new submit script and submit the job. This was done with yet another script:

<syntaxhighlight lang="bash">
#!/bin/bash
DATASET=dataset.txt
scriptnum=0
while read LINE
do
echo "app2 $LINE" > ${scriptnum}.sh
sbatch ${scriptnum}.sh
scriptnum=$(( $scriptnum + 1 ))
done < $DATASET
</syntaxhighlight>
Not only is this needlessly complex, it is also slow, as sbatch has to verify each job as it is submitted. This can be done easily with array jobs, as long as you know the number of lines in the dataset. This number can be obtained like so: wc -l dataset.txt in this case lets call it 5000.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=1:5000
app2 `sed -n "${SLURM_ARRAY_TASK_ID}p" dataset.txt`
</syntaxhighlight>
This uses a subshell via `, and has the sed command print out only the line number $SLURM_ARRAY_TASK_ID out of the file dataset.txt.

Not only is this a smaller script, it is also faster to submit because it is one job instead of 5000, so sbatch doesn't have to verify as many.

To give you an idea about time saved: submitting 1 job takes 1-2 seconds. by extension if you are submitting 5000, that is 5,000-10,000 seconds, or 1.5-3 hours.

== Running jobs interactively ==
Some jobs just don't behave like we think they should, or need to be run with somebody sitting at the keyboard and typing in response to the output the computers are generating. Beocat has a facility for this, called 'srun'. srun uses the exact same command-line arguments as sbatch, but you need to add the following arguments at the end: <tt>--pty bash</tt>. If no node is available with your resource requirements, srun will tell you something like the following:
srun --pty bash
srun: Force Terminated job 217
srun: error: CPU count per node can not be satisfied
srun: error: Unable to allocate resources: Requested node configuration is not available
Note that, like sbatch, your interactive job will timeout after your allotted time has passed.

== Connecting to an existing job ==
You can connect to an existing job using srun in the same way that the MonitorNode command
allowed us to in the old cluster. This is essentially like using ssh to get into the node where your job is running which
can be very useful in allowing you to look at files in /tmp/job# or in running htop to view the
activity level for your job.

srun --jobid=# --pty bash where '#' is the job ID number

== Altering Job Requests ==
We generally do not support users to modify job parameters once the job has been submitted. It can be done, but there are numerous catches, and all of the variations can be a bit problematic; it is normally easier to simply delete the job (using '''scancel ''jobid''''') and resubmit it with the right parameters. '''If your job doesn't start after modifying such parameters (after a reasonable amount of time), delete the job and resubmit it.'''

As it is unsupported, this is an excercise left to the reader. A starting point is <tt>man scontrol</tt>
== Killable jobs ==
There are a growing number of machines within Beocat that are owned by a particular person or group. Normally jobs from users that aren't in the group designated by the owner of these machines cannot use them. This is because we have guaranteed that the nodes will be accessible and available to the owner at any given time. We will allow others to use these nodes if they designate their job as "killable." If your job is designated as killable, your job will be able to use these nodes, but can (and will) be killed off at any point in time to make way for the designated owner's jobs. Jobs that are marked killable will be re-queued and may restart on another node.

The way you would designate your job as killable is to add <tt>-p killable.q</tt> to the '''<tt>sbatch</tt> or <tt>srun</tt>''' arguments. This could be either on the command-line or in your script file.

''Note: This is a submit-time only request, it cannot be added by a normal user after the job has been submitted.'' If you would like jobs modified to be '''killable''' after the jobs have been submitted (and it is too much work to <tt>qdel</tt> the jobs and re-submit), send an e-mail to the administrators detailing the job ids and what you would like done.

== Scheduling Priority ==
Some users are members of projects that have contributed to Beocat. When those users have contributed nodes, you will need to include your project's "partition" in your job submission to be able to use those nodes.

To determine the partitions you have access to, run <tt>sinfo -hso '%P'</tt>
That will return a list that looks something like this:
killable.q
batch.q
some-other-partition.q

You can then alter your <tt>#SBATCH</tt> lines to include your new partition:
#SBATCH --partition=some-other-partition.q,batch.q
or
#SBATCH --partition=some-other-partition.q,batch.q,killable.q
You can include 'killable.q' if you would like, reasons for doing so are available [[AdvancedSlurm#Killable_jobs|here]]

== Job Accounting ==
Some people may find it useful to know what their job did during its run. The sacct tool will read Slurm's accounting database and give you summarized or detailed views on jobs that have run within Beocat.
=== sacct ===
This data can usually be used to diagnose two very common job failures.
==== Job debugging ====
It is simplest if you know the job number of the job you are trying to get information on.
<syntaxhighlight lang="bash">
# if you know the jobid, put it here:
sacct -j 1122334455 -l
# if you don't know the job id, you can look at your jobs started since some day:
sacct -S 2017-01-01
</syntaxhighlight>

===== My job didn't do anything when it ran! =====
{{Scrolling table/top}}
{{Scrolling table/mid}}
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|218||218||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||12||00:00:00||FAILED||2:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=12,mem=1G,node=1||cpu=12,mem=1G,node=1
|-
|218.batch||218.batch||batch||||137940K||dwarf37||0||137940K||1576K||dwarf37||0||1576K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||1.36G||0||0||0||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|-
|218.0||218.0||qqqqstat||||204212K||dwarf37||0||204212K||1420K||dwarf37||0||1420K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||196.52M||Unknown||Unknown||Unknown||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
{{Scrolling table/end}}
If you look at the columns showing Elapsed and State, you can see that they show 00:00:00 and FAILED respectively. This means that the job started and then promptly ended. This points to something being wrong with your submission script. Perhaps there is a typo somewhere in it.

===== My job ran but didn't finish! =====
{{Scrolling table/top}}
{{Scrolling table/mid}}
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|220||220||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:01:27||TIMEOUT||0:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=1,mem=1G,node=1||cpu=1,mem=1G,node=1
|-
|220.batch||220.batch||batch||||370716K||dwarf37||0||370716K||7060K||dwarf37||0||7060K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:28||CANCELLED||0:15||1.23G||0||0||0||1Gn||0||0.16M||dwarf37||0||0.16M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|-
|220.0||220.0||sleep||||204212K||dwarf37||0||107916K||1000K||dwarf37||0||620K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:27||CANCELLED||0:15||1.54G||Unknown||Unknown||Unknown||1Gn||0||0.05M||dwarf37||0||0.05M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
{{Scrolling table/end}}
If you look at the column showing State, we can see some pointers to the issue. The job ran out of time (TIMEOUT) and then was killed (CANCELLED).
{{Scrolling table/top}}
{{Scrolling table/mid}}
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|221||221||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:00:00||CANCELLED by 0||0:0||||Unknown||Unknown||Unknown||1Mn||||||||||||||||||||||||cpu=1,mem=1M,node=1||cpu=1,mem=1M,node=1
|-
|221.batch||221.batch||batch||||137940K||dwarf37||0||137940K||1144K||dwarf37||0||1144K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:00:01||CANCELLED||0:15||2.62G||0||0||0||1Mn||0||0||dwarf37||65534||0||0||dwarf37||65534||0||||||||cpu=1,mem=1M,node=1
{{Scrolling table/end}}
If you look at the column showing State, we see it was "CANCELLED by 0", then we look at the AllocTRES column to see our allocated resources, and see that 1MB of memory was granted. Combine that with the column "MaxRSS" and we see that the memory granted was less than the memory we tried to use, thus the job was "CANCELLED".

CUDA

2017-10-16T19:25:21Z

Kylehutson: Changed info to deal with updates from Paladins to Dwarves

== CUDA Overview ==
[[wikipedia:CUDA|CUDA]] is a feature set for programming nVidia [[wikipedia:Graphics_processing_unit|GPUs]]. We have 7 CUDA-enabled nodes. dwarf22, dwarf23, dwarf24, dwarf25, and dwarf35 each have two [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ nVidia 1080 Ti graphics cards]. dwarf38 and dwarf39 each have a single [https://www.geforce.com/hardware/desktop-gpus/geforce-gtx-980-ti/specifications nVidia 980 Ti graphic card]. The former set of nodes is only available to 'killable" jobs for those outside the research group that purchased them. The latter are available for anybody, however, you should send an email to beocat@cs.ksu.edu with a request to be added to the GPU priority group.

Note that both of these graphic cards are consumer-grade rather than the typical GPUs used in most high-performance computing centers. For single-precision computations, these cards are comparable to the high-end cards (at a fraction of the price), however double-precision computations are much slower.

== Training videos ==
CUDA Programming Model Overview: [http://www.youtube.com/watch?v=aveYOlBSe-Y http://www.youtube.com/watch?v=aveYOlBSe-Y]
<HTML5video type="youtube" width="800" height="480" autoplay="false">aveYOlBSe-Y</HTML5video>

CUDA Programming Basics Part I (Host functions): [http://www.youtube.com/watch?v=79VARRFwQgY http://www.youtube.com/watch?v=79VARRFwQgY]
<HTML5video type="youtube" width="800" height="480" autoplay="false">79VARRFwQgY</HTML5video>

CUDA Programming Basics Part II (Device functions): [http://www.youtube.com/watch?v=G5-iI1ogDW4 http://www.youtube.com/watch?v=G5-iI1ogDW4]
<HTML5video type="youtube" width="800" height="480" autoplay="false">G5-iI1ogDW4</HTML5video>
== Compiling CUDA Applications ==
nvcc is the compiler for CUDA applications. When compiling your applications manually you will need to keep 3 things in mind:

* The CUDA development headers are located here: /opt/cuda/sdk/common/inc
* The CUDA architecture is: sm_30
* The CUDA SDK is currently not available on the headnode. (compile on the nodes with CUDA, either in your jobscript or via <tt>qrsh -l cuda=TRUE</tt>)
* '''Do not run your cuda applications on the headnode. I cannot guarantee it will run, and it will give you terrible results if it does run.'''

Putting it all together you can compile CUDA applications as follows:

<syntaxhighlight lang="bash">
nvcc -I /opt/cuda/sdk/common/inc -arch sm_30 <source>.cu -o <output>
</syntaxhighlight>
== Example ==
=== Create your Application ===
Copy the following Application into Beocat as vecadd.cu
<syntaxhighlight lang="c">
// Kernel definition, see also section 4.2.3 of Nvidia Cuda Programming Guide
__global__ void vecAdd(float* A, float* B, float* C)
{
// threadIdx.x is a built-in variable provided by CUDA at runtime
int i = threadIdx.x;
A[i]=0;
B[i]=i;
C[i] = A[i] + B[i];
}

#include <stdio.h>
#define SIZE 10
int main()
{
int N=SIZE;
float A[SIZE], B[SIZE], C[SIZE];
float *devPtrA;
float *devPtrB;
float *devPtrC;
int memsize= SIZE * sizeof(float);

cudaMalloc((void**)&devPtrA, memsize);
cudaMalloc((void**)&devPtrB, memsize);
cudaMalloc((void**)&devPtrC, memsize);
cudaMemcpy(devPtrA, A, memsize, cudaMemcpyHostToDevice);
cudaMemcpy(devPtrB, B, memsize, cudaMemcpyHostToDevice);
// __global__ functions are called: Func<<< Dg, Db, Ns >>>(parameter);
vecAdd<<<1, N>>>(devPtrA, devPtrB, devPtrC);
cudaMemcpy(C, devPtrC, memsize, cudaMemcpyDeviceToHost);

for (int i=0; i<SIZE; i++)
printf("C[%d]=%f\n",i,C[i]);

cudaFree(devPtrA);
cudaFree(devPtrA);
cudaFree(devPtrA);

}
</syntaxhighlight>
=== Gain Access to a CUDA-capable Node ===
<syntaxhighlight lang="bash">
qrsh -l cuda=TRUE
</syntaxhighlight>
=== Compile Your Application ===
<syntaxhighlight lang="bash">
nvcc -I /opt/cuda/sdk/common/inc -arch sm_30 vecadd.cu -o vecadd
</syntaxhighlight>
This will create a program with the name 'vecadd' (specified by the '-o' flag).
=== Run Your Application ===
Run the program as you usually would, namely
<syntaxhighlight lang="bash">
./vecadd
</syntaxhighlight>

Assuming you don't want to run the program interactively because this is a large job, you can submit a job via qsub, just be sure to add the '<tt>-l cuda=true</tt>' directive.

Main Page

2016-12-23T16:31:38Z

Kylehutson: Added "Institute for Computational Research" to the "What is" section

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|HPC]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of Linux servers coordinated by the [https://arc.liv.ac.uk/trac/SGE SGE] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A comparatively small [[Hadoop]] cluster
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''beocat#beocat''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use SGE for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SGEBasics]] page for an introduction on how to submit your first job. If you are already familiar with SGE, we also have an [[AdvancedSGE]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

We are also available on IRC on the [http://freenode.net/using_the_network.shtml freenode chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''m0zes''' and/or '''kylehutson''' in your message, and it should grab our attention. Available from a web browser [[Special:WebChat|here.]]

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat.

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Tips and Tricks

2016-12-01T19:53:42Z

Kylehutson: Update to correct Ganglia link

Beocat has a number of tools to make your work easier, some which you may not know about. This is a simple list of these programs and some basic usage scenarios.

== Submitting your job to run the fastest ==
=== Size your jobs to use the fastest nodes ===
==== Specify the proper number of cores ====
Beocat (nor any other computer or cluster) can make your job run on more than one core at a time if your program isn't designed to take advantage of this. Many people think "I can run this on 40 cores and it will run 40 times faster". This isn't true.

While we have many programs that are designed to take advantage of multiple cores, do not assume this is the case

==== Optimize your jobs for speed, not for number of cores ====
It seems that many people pick an arbitrary large number of cores for their jobs. 20 seems to be a common one. However, some of our fastest nodes have 16 cores. It's quite likely if your job will fit on an Elf (16 cores, 8 GB/RAM/core (64 GB RAM total)), it will run faster with 16 cores than by specifying more cores and having it run on slower nodes.

==== Don't request resources you don't need ====
The most common culprit here is people specifying they need infiniband when the job is run on a single node. This limits the scheduling such that a perfectly good node for your job may be idle while your job is still waiting.

== Programs that make using Beocat easier ==
=== [[wikipedia:nmon|nmon]] ===
The name is short for "Nigel's Monitor", it's a program written by Nigel Griffiths from IBM.
=== [http://www.ibm.com/developerworks/aix/library/au-nmon_analyser/ nmon analyser] ===
A tool for producing graphs and spreadsheets from output generated by nmon.
=== [http://hisham.hm/htop/ htop] ===
A prettier, easier to use top. Shows CPU and memory usage in an easy-to-digest format.
=== [http://www.gnu.org/software/screen/ screen] ===
A sort of terminal multiplexer, allows you to run many terminal programs at once without mixing them up. Also allows you to disconnect and reconnect sessions. There is a good explanation of how to use screen at [http://www.mattcutts.com/blog/a-quick-tutorial-on-screen/ http://www.mattcutts.com/blog/a-quick-tutorial-on-screen/].
=== Ganglia ===
The web-based load monitoring tool for the cluster. [http://ganglia.beocat.cis.ksu.edu http://ganglia.beocat.ksu.edu] . From there, you can see how busy Beocat is.
=== [http://dag.wieers.com/home-made/dstat/ dstat] ===
A very detailed performance analyzer.

== Increasing file write performance ==
Credit for this goes to [http://moo.nac.uci.edu/~hjm/bduc/BDUC_USER_HOWTO.html#writeperfongl http://moo.nac.uci.edu/~hjm/bduc/BDUC_USER_HOWTO.html#writeperfongl]

=== Use gzip ===
If you have written your own code or are using an app that writes zillions of tiny chunks of data to STDOUT, and you are storing the results on Beocat, you should consider passing the output thru gzip to consolidate the writes into a continuous stream. If you don’t do this, each write will be considered a separate IO event and the write performance will suffer.

If, however, the STDOUT is passed thru gzip, the wallclock runtime decreases even below the usual runtime and you end up with an output file that it already compressed to about 1/5 the usual size.

The here’s how to do it:

someapp --opt1 --opt2 --input=/path/to/input_file | gzip > /path/to/output_file
=== Use named pipes ===
Named pipes are special files that don't actually write to the filesystem, and can be used to communicate between processess. Since these pipes are in memory rather than directly to disk, they can be used to buffer writes:

<syntaxhighlight lang="bash">
# Create the named pipe
mkfifo /path/to/MyNamedPipe

# Write some data to it
MyProgram --infile=/path/to/InputData1 --outfile=/path/to/MyNamedPipe &
MyOtherProgram < /path/to/InputData2 > /path/to/MyNamedPipe

# Extract the output
cat < /path/to/MyNamedPipe > $HOME/MyOutput
## OR, we could compress the output
gzip < /path/to/MyNamedPipe > $HOME/MyOutput.gz

# Delete the named pipe like you would a file
rm /path/to/MyNamedPipe
</syntaxhighlight>
One cautionary word. Unlike normal files, named pipes cannot be used between machines, but can be used among processes running on the same machine. So, if you're running an MPI job that will be running completely on one node, you could setup a named pipe and do all your writes to that pipe, and then flush it at the end, but if you're running a multi-node MPI job and your named pipe is on a shared filesystem (like $HOME), each process will need to flush its named pipe to a regular file before the job quits.
=== Use one big file instead of many small ones ===
This may seem to be a non-issue, but it's a performance problem we've seen on Beocat many times. I love the term coined by UCI at the link above. They call making many small files "Zillions Of Tiny files (ZOTfiles)". Using files like this is an inefficient use of our shared resources. A tiny file by itself is no more inefficient than a huge one. If you have only 100bytes to store, store it in single file. However, the problems start compounding when there are many of them. Because of the way data is stored on disk, 10 MB stored in ZOTfiles of 100bytes each can easily take up NOT 10MB, but more than 400MB - 40 times more space. Worse, data stored in this manner makes many operations very slow - instead of looking up 1 directory entry, the OS has to look up 100,000. This means 100,000 times more disk head movement, with a concommittent decrease in performance and disk lifetime. We have had Beocat users with several million files of less than 1kB each. Just creating a directory listing with ls would take nearly 1/2 hour. Not only is that inefficient for you, but it also degrades the performance of everybody using that filesystem and degrades our backups as well.

Please use large files instead of ZOTfiles any chance you can!

As a defense against too much abuse of tiny files, there is a limit of 100,000 entries in any directory in our shared filesystem space.

== Programming for Performance ==
=== BLAS ===
BLAS (Basic Linear Algebra Subroutines) is a standard set of linear algebra subroutines. The standard was set so that software could be written against a standardized library interface, and optimized libraries could be "plug-and-play." There are lots of implementations of the BLAS libraries, with the most common ones being [http://software.intel.com/en-us/intel-mkl/ Intel's MKL] and [http://developer.amd.com/tools/cpu/acml/pages/default.aspx AMD's ACML].

==== Beocat BLAS Libraries ====
Since BLAS is a modular standard, we have installed a few (free) BLAS libraries.

* The BLAS reference library: An unoptimized reference library
* [http://developer.amd.com/tools/cpu/acml/pages/default.aspx AMD's ACML]: Optimized BLAS library for AMD systems
* [http://www.openblas.net OpenBLAS]: Optimized BLAS library for some AMD, and most Intel sytems

The default BLAS library is OpenBLAS.

==== Using a different BLAS library ====
If you want or need to use a different BLAS library, list the available libraries with 'ls -1 /etc/env.d/alternatives/blas' (Ignore _current and _current_list)

$ ls -1 /etc/env.d/alternatives/blas
_current
_current_list
acml-gfortran64
acml-gfortran64-openmp
acml-ifort64
acml-ifort64-openmp
mkl32-dynamic
mkl32-dynamic-openmp
mkl32-gfortran
mkl32-gfortran-openmp
mkl32-intel
mkl32-intel-openmp
mkl64-dynamic
mkl64-dynamic-openmp
mkl64-gfortran
mkl64-gfortran-openmp
mkl64-int64-dynamic
mkl64-int64-dynamic-openmp
mkl64-int64-gfortran
mkl64-int64-gfortran-openmp
mkl64-int64-intel
mkl64-int64-intel-openmp
mkl64-intel
mkl64-intel-openmp
openblas-openmp
reference
To change your default BLAS version you need to determine which shell you are using:

===== CSH or TCSH =====
If your tool simply uses pkg-config to find the right blas, you can just run the following:
<syntaxhighlight lang="bash">
setenv PKG_CONFIG_PATH /etc/env.d/alternatives/blas/openblas-openmp/usr/lib64/pkgconfig
</syntaxhighlight>
Where the openblas-openmp is replaced with name of your preferred BLAS. You can put that line in your job script, or in your ~/.cshrc file.

If it needs actual library names and options for the compiler, after you have run the above you can run these to get the right arguments/library names for your compiler
<syntaxhighlight lang="bash">
pkg-config --cflags blas
pkg-config --libs blas
</syntaxhighlight>
===== SH, BASH, or ZSH =====
If your tool simply uses pkg-config to find the right blas, you can just run the following:
<syntaxhighlight lang="bash">
export PKG_CONFIG_PATH=/etc/env.d/alternatives/blas/openblas-openmp/usr/lib64/pkgconfig
</syntaxhighlight>
Where the openblas-openmp is replaced with name of your preferred BLAS. Put the output of that script in your job script, or in your ~/.bashrc or ~/.zshrc file.

If it needs actual library names and options for the compiler, after you have run the above you can run these to get the right arguments/library names for your compiler
<syntaxhighlight lang="bash">
pkg-config --cflags blas
pkg-config --libs blas
</syntaxhighlight>
=== LAPACK ===
LAPACK (Linear Algebra PACKage) is a standard set of linear algebra subroutines. Like BLAS, these are very optimized, but LAPACK handles a different set of functions. The standard was set so that software could be written against a standardized library interface, and optimized libraries could be "plug-and-play." There are lots of implementations of the LAPACK libraries, with the most common ones being [http://software.intel.com/en-us/intel-mkl/ Intel's MKL] and [http://developer.amd.com/tools/cpu/acml/pages/default.aspx AMD's ACML].

==== Beocat LAPACK Libraries ====
Since LAPACK is a modular standard, we have installed a few (free) LAPACK libraries.
* [http://www.netlib.org/lapack/ The LAPACK reference library]: An unoptimized reference library
* [http://developer.amd.com/tools/cpu/acml/pages/default.aspx AMD's ACML]: Optimized LAPACK library for AMD systems

The default LAPACK library is ACML.

==== Using a different LAPACK library ====
If you want or need to use a different LAPACK library, list the available libraries with 'ls -1 /etc/env.d/alternatives/lapack' (Ignore _current and _current_list)

$ ls -1 /etc/env.d/alternatives/lapack
_current
_current_list
acml-gfortran64
acml-gfortran64-openmp
acml-ifort64
acml-ifort64-openmp
mkl32-dynamic
mkl32-dynamic-openmp
mkl32-gfortran
mkl32-gfortran-openmp
mkl32-intel
mkl32-intel-openmp
mkl64-dynamic
mkl64-dynamic-openmp
mkl64-gfortran
mkl64-gfortran-openmp
mkl64-int64-dynamic
mkl64-int64-dynamic-openmp
mkl64-int64-gfortran
mkl64-int64-gfortran-openmp
mkl64-int64-intel
mkl64-int64-intel-openmp
mkl64-intel
mkl64-intel-openmp
reference
To change your default LAPACK version you need to determine which shell you are using:

===== CSH or TCSH =====
If your tool simply uses pkg-config to find the right lapack, you can just run the following:
<syntaxhighlight lang="bash">
setenv PKG_CONFIG_PATH /etc/env.d/alternatives/lapack/acml-ifort64/usr/lib64/pkgconfig
</syntaxhighlight>
Where the acml-ifort64 is replaced with name of your preferred LAPACK. You can put that line in your job script, or in your ~/.cshrc file.

If it needs actual library names and options for the compiler, after you have run the above you can run these to get the right arguments/library names for your compiler
<syntaxhighlight lang="bash">
pkg-config --cflags lapack
pkg-config --libs lapack
</syntaxhighlight>
===== SH, BASH, or ZSH =====
If your tool simply uses pkg-config to find the right lapack, you can just run the following:
<syntaxhighlight lang="bash">
export PKG_CONFIG_PATH=/etc/env.d/alternatives/lapack/acml-ifort64/usr/lib64/pkgconfig
</syntaxhighlight>
Where the acml-ifort64 is replaced with name of your preferred LAPACK. Put the output of that script in your job script, or in your ~/.bashrc or ~/.zshrc file.

If it needs actual library names and options for the compiler, after you have run the above you can run these to get the right arguments/library names for your compiler
<syntaxhighlight lang="bash">
pkg-config --cflags lapack
pkg-config --libs lapack
</syntaxhighlight>

=== [http://openmp.org/wp/ OpenMP] ===
OpenMP is a set of directives for C, C++, and Fortran which greatly simplifies parallelizing applications on a single node. There is a good tutorial for OpenMP at [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/]
To compile an OpenMP-enabled program, you need to tell GCC that OpenMP is available this is done like:
gcc -fopenmp myOpenMPprogram.c
By default OpenMP will use all available cores for its computation, which is a problem for shared resources like Beocat.

To make use of only the cores assigned to you, you must first make sure you have requested the 'single' parallel environment and in your job script you will need something like the following (before the application you are trying to run):

==== bash, sh, zsh ====
<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=${NSLOTS}
</syntaxhighlight>

==== csh or tcsh ====
<syntaxhighlight lang="bash">
setenv OMP_NUM_THREADS ${NSLOTS}
</syntaxhighlight>

Beocat:About

2016-12-01T19:52:59Z

Kylehutson: Update CIS references to CS

[http://support.beocat.ksu.edu Beocat] is a medium-sized compute cluster in the [http://www.cs.ksu.edu Computer Science Department] of the [http://www.engg.ksu.edu College of Engineering] at [http://www.ksu.edu Kansas State University]. The project is overseen by [[User:dan|Dr. Andresen]] and managed by [[User:mozes|Adam Tygart]] and [[User:kylehutson|Kyle Hutson]].

We support researchers from all over Kansas, and their collaborators (wherever they may be).

OLD DEPRECATED AdvancedSGE

2016-12-01T19:51:25Z

Kylehutson: Updated references

== Resource Requests ==
Aside from the time, RAM, and CPU requirements listed on the [[SGEBasics]] page, we have several other requestable resources. Generally, if you don't know if you need a particular resource, you should use the default. These can be generated with the command
<tt>qconf -sc | awk '{ if ($5 != "NO") { print }}'</tt>
{| class="wikitable sortable"
!name
!shortcut
!type
!relop
!requestable
!consumable
!default
!urgency
|-
|arch
|a
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|avx
|avx
|BOOL
|==
|YES
|NO
|FALSE
|0
|-
|calendar
|c
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|cpu
|cpu
|DOUBLE
|>=
|YES
|NO
|0
|0
|-
|cpu_flags
|c_f
|STRING
|==
|YES
|NO
|NONE
|0
|-
|cuda
|cuda
|INT
|<=
|YES
|JOB
|0
|0
|-
|display_win_gui
|dwg
|BOOL
|==
|YES
|NO
|0
|0
|-
|exclusive
|excl
|BOOL
|EXCL
|YES
|YES
|0
|1000
|-
|h_core
|h_core
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_cpu
|h_cpu
|TIME
|<=
|YES
|NO
|0:0:0
|0
|-
|h_data
|h_data
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_fsize
|h_fsize
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_rss
|h_rss
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_rt
|h_rt
|TIME
|<=
|FORCED
|NO
|0:0:0
|0
|-
|h_stack
|h_stack
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_vmem
|h_vmem
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|hostname
|h
|HOST
|==
|YES
|NO
|NONE
|0
|-
|infiniband
|ib
|BOOL
|==
|YES
|NO
|FALSE
|0
|-
|m_core
|core
|INT
|<=
|YES
|NO
|0
|0
|-
|m_socket
|socket
|INT
|<=
|YES
|NO
|0
|0
|-
|m_thread
|thread
|INT
|<=
|YES
|NO
|0
|0
|-
|m_topology
|topo
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|m_topology_inuse
|utopo
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|mem_free
|mf
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|mem_total
|mt
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|mem_used
|mu
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|memory
|mem
|MEMORY
|<=
|FORCED
|YES
|0
|0
|-
|num_proc
|p
|INT
|==
|YES
|NO
|0
|0
|-
|qname
|q
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|s_core
|s_core
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_cpu
|s_cpu
|TIME
|<=
|YES
|NO
|0:0:0
|0
|-
|s_data
|s_data
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_fsize
|s_fsize
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_rss
|s_rss
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_rt
|s_rt
|TIME
|<=
|YES
|NO
|0:0:0
|0
|-
|s_stack
|s_stack
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_vmem
|s_vmem
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|slots
|s
|INT
|<=
|YES
|YES
|1
|1000
|-
|swap_free
|sf
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|swap_rate
|sr
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|swap_rsvd
|srsv
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|swap_total
|st
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|swap_used
|su
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|virtual_free
|vf
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|virtual_total
|vt
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|virtual_used
|vu
|MEMORY
|>=
|YES
|NO
|0
|0
|}

The good news is that most of these nobody ever uses. There are a couple of exceptions, though:
=== Infiniband ===
First of all, let me state that just because it sounds "cool" doesn't mean you need it or even want it. Infiniband does absolutely no good if running in a 'single' parallel environment. Infiniband is a high-speed host-to-host communication fabric. It is used in conjunction with MPI jobs (discussed below). Several times we have had jobs which could run just fine, except that the submitter requested Infiniband, and all the nodes with Infiniband were currently busy. In fact, some of our fastest nodes do not have Infiniband, so by requesting it when you don't need it, you are actually slowing down your job. To request Infiniband, add <tt>-l ib=true</tt> to your qsub command-line.
=== CUDA ===
[[CUDA]] is the resource required for GPU computing. We have a very small number of nodes which have GPUs installed. To request one of these nodes, add <tt>-l cuda=true</tt> to your qsub command-line.
=== Exclusive ===
Some programs just don't play nicely with others. They will attempt to use all available memory or will try to use all the cores it can use. The way to be a nice neighbor if your program has this problem is to request exclusive use of a node with <tt>-l excl=true</tt>. This can also be useful for benchmarking, where you can be sure that no other jobs are interfering with yours.
== Parallel Jobs ==
There are two ways jobs can run in parallel, ''intra''node and ''inter''node. '''Note: Beocat will not automatically make a job run in parallel.''' Have I said that enough? It's a common misperception.
=== Intranode jobs ===
Intranode jobs are easier to code and can take advantage of many common libraries, such as [http://openmp.org/wp/ OpenMP], or Java's threads. Many times, your program will need to know how many cores you want it to use. Many will use all available cores if not told explicitly otherwise. This can be a problem when you are sharing resources, as Beocat does. To request multiple cores, use the qsub directive '<tt>-pe single ''n''</tt>', where ''n'' is the number of cores you wish to use. If your command can take an environment variable, you can use $nslots to tell how many cores you've been allocated.
=== Internode (MPI) jobs ===
"Talking" between nodes is trickier that talking between cores on the same node. The specification for doing so is called "[[wikipedia:Message_Passing_Interface|Message Passing Interface]]", or MPI. We have [http://www.open-mpi.org/ OpenMPI] installed on Beocat for this purpose. Most programs written to take advantage of large multi-node systems will use MPI. You can tell if you have an MPI-enabled program because its directions will tell you to run '<tt>mpirun ''program''</tt>'. Requesting MPI resources is only mildly more difficult than requesting single-node jobs. Instead of using '<tt>-pe single ''n''</tt>' for your qsub request, you will use one of the following:
{| class="wikitable sortable"
! Parallel Environment !! Description
|-
|mpi-fill
|This environment will use as many slots on each node as it can until it reaches the number of cores you have requested.
|-
|mpi-spread
|This environment will spread itself out over as many nodes as possible until it reaches the number of cores you have requested.
|-
|mpi-1
|This environment will allocate the slots you've requested 1 per node.
|-
|mpi-2
|This environment will allocate the slots you've requested 2 per node. You must request cores as a multiple of 2
|-
|mpi-4
|This environment will allocate the slots you've requested 4 per node. You must request cores as a multiple of 4
|-
|mpi-8
|This environment will allocate the slots you've requested 8 per node. You must request cores as a multiple of 8
|-
|mpi-10
|This environment will allocate the slots you've requested 10 per node. You must request cores as a multiple of 10
|-
|mpi-12
|This environment will allocate the slots you've requested 12 per node. You must request cores as a multiple of 12
|-
|mpi-16
|This environment will allocate the slots you've requested 16 per node. You must request cores as a multiple of 16
|-
|mpi-20
|This environment will allocate the slots you've requested 20 per node. You must request cores as a multiple of 20
|-
|mpi-80
|This environment will allocate the slots you've requested 80 per node. You must request cores as a multiple of 80
|}
Some quick examples:

<tt>-pe mpi-4 16</tt> will give you 4 chunks of 4 cores apiece. They might all happen to be allocated on the same node (16 cores), on 4 different nodes (4 cores each), on 3 nodes (8 cores on one and 4 cores on the other two), or on 2 nodes (8 cores each).

<tt>-pe mpi-fill 40</tt> will give you 40 cores, but will attempt to get them all on the same node.

<tt>-pe mpi-fill 100</tt> will give you 100 cores, and place them on as few nodes as possible. In this case it's likely you would get a full mage (80 cores) and either part of another mage (the remaining 20 cores) or one of the 20-core elves.

<tt>-pe mpi-spread 40</tt> will give you 40 cores, and will attempt to place each on a separate node.
== Requesting memory for multi-core jobs ==
All memory requests are '''per core'''. One of the more common scenarios is where somebody will need, say 20 cores and 400 GB of memory. So they will make a request like '<tt>-pe single 20, -l mem=400G</tt>' This will never run, because what you are really requesting is 20 cores and 8000GB of memory (20 * 400). Since we have no nodes with 8000 terabytes of memory, the job will never run. In this case, you will divide the 400GB total memory request by the number of cores (20), so the correct command would be '<tt>-pe single 20, -l mem=20G</tt>'.
== Other Handy SGE Features ==
=== Email status changes ===
One of the most commonly used options when submitting jobs not related to resource requests is to have have SGE email you when a job changes its status. This takes two directives to qsub: '<tt>-M ''someone@somewhere.com''</tt>' will give the email address to which to send status updates. '<tt>-m abe</tt>' is probably the most common directive given for ''when'' to send updates. This will send email messages when a job (a)borts, (b)egins, or (e)nds. Other possibilities are (s)uspended and (n)ever.
=== Job Naming ===
If you have several jobs in the queue, running the same script with different parameters, it's handy to have a different name for each job as it shows up in the queue. This is accomplished with the '<tt>-N ''JobName''</tt>' qsub directive.
=== Combining Output Streams ===
Normally, SGE will create two files for output. One will be .e''jobnumber'' and the other .o''jobnumber''. If you want both of these to be combined into a single file, you can use the qsub directive '<tt>-j y</tt>'.
=== Running from the Current Directory ===
By default, jobs run from your home directory. Many programs incorrectly assume that you are running the script from the current directory. You can use the '<tt>-cwd</tt>' directive to change to the "current working directory" you used when submitting the job.
=== SGE Environment Variables ===
Within an actual job, sometimes you need to know specific things about the running environment to setup your scripts correctly. Here is a listing of environment variables that SGE makes available to you. Of course the value of these variables will be different based on many different factors.
<syntaxhighlight lang="bash">
HOSTNAME=titan1.beocat
SGE_TASK_STEPSIZE=undefined
SGE_INFOTEXT_MAX_COLUMN=5000
SHELL=/usr/local/bin/sh
NHOSTS=2
SGE_O_WORKDIR=/homes/mozes
TMPDIR=/tmp/105.1.batch.q
SGE_O_HOME=/homes/mozes
SGE_ARCH=lx24-amd64
SGE_CELL=default
RESTARTED=0
ARC=lx24-amd64
USER=mozes
QUEUE=batch.q
PVM_ARCH=LINUX64
SGE_TASK_ID=undefined
SGE_BINARY_PATH=/opt/sge/bin/lx24-amd64
SGE_STDERR_PATH=/homes/mozes/sge_test.sub.e105
SGE_STDOUT_PATH=/homes/mozes/sge_test.sub.o105
SGE_ACCOUNT=sge
SGE_RSH_COMMAND=builtin
JOB_SCRIPT=/opt/sge/default/spool/titan1/job_scripts/105
JOB_NAME=sge_test.sub
SGE_NOMSG=1
SGE_ROOT=/opt/sge
REQNAME=sge_test.sub
SGE_JOB_SPOOL_DIR=/opt/sge/default/spool/titan1/active_jobs/105.1
ENVIRONMENT=BATCH
PE_HOSTFILE=/opt/sge/default/spool/titan1/active_jobs/105.1/pe_hostfile
SGE_CWD_PATH=/homes/mozes
NQUEUES=2
SGE_O_LOGNAME=mozes
SGE_O_MAIL=/var/mail/mozes
TMP=/tmp/105.1.batch.q
JOB_ID=105
LOGNAME=mozes
PE=mpi-fill
SGE_TASK_FIRST=undefined
SGE_O_HOST=loki
SGE_O_SHELL=/bin/bash
SGE_CLUSTER_NAME=beocat
REQUEST=sge_test.sub
NSLOTS=32
SGE_STDIN_PATH=/dev/null
</syntaxhighlight>
Sometimes it is nice to know what hosts you have access to during a PE job. You would checkout the PE_HOSTFILE to know that. If your job has been restarted, it is nice to be able to change what happens rather than redoing all of your work. If this is the case, RESTARTED would equal 1. There are lots of useful Environment Variables there, I will leave it to you to identify the ones you want.

Some of the most commonly-used variables we see used are $NSLOTS, $HOSTNAME, and $SGE_TASK_ID (used for array jobs, discussed below).
== Running from a qsub Submit Script ==
No doubt after you've run a few jobs you get tired of typing something like 'qsub -l mem=2G,h_rt=10:00 -pe single 8 -n MyJobTitle MyScript.sh'. How are you supposed to remember all of these every time? The answer is to create a 'submit script', which outlines all of these for you. Below is a sample submit script, which you can modify and use for your own purposes.
<syntaxhighlight lang="bash">
#!/bin/bash

## A Sample qsub script created by Kyle Hutson
##
## Note: Usually a '#" at the beginning of the line is ignored. However, in
## the case of qsub, lines beginning with #$ are commands for qsub itself, so
## I have taken the convention here of starting *every* line with a '#', just
## Delete the first one if you want to use that line, and then modify it to
## your own purposes. The only exception here is the first line, which *must*
## be #!/bin/bash (or another valid shell).

## Specify the amount of RAM needed _per_core_. Default is 1G
##$ -l mem=1G

## Specify the maximum runtime. Default is 1 hour (1:00:00)
##$ -l h_rt=1:00:00

## Require the use of infiniband. If you don't know what this is, you probably
## don't need it. Default is "FALSE"
##$ ib=TRUE

## CUDA directive. If You don't know what this is, you probably don't need it
## Default is "FALSE"
##$ -l cuda=TRUE

## Parallel environment. Syntax is '-pe Environment NumberOfCores' A list of
## valid environments can be found at
## https://support.beocat.ksu.edu/BeocatDocs/index.php/AdvancedSGE (section 2). One
## quick note here. Jobs requesting 16 or fewer cores tend to get scheduled
## fairly quickly. If you need a job that requires more than that, you might
## benefit from emailing us at beocat@cs.ksu.edu to see how we can assist in
## getting your job scheduled in a reasonable amount of time. Default is
## "single 1"
##$ -pe single 12
##$ -pe mpi-1 2
##$ -pe mpi-fill 20
##$ -pe mpi-spread 16

## Checkpointing. Options are BLCR or dmtcp. Default is no checkpointing.
##$ -ckpt dmtcp

## Use the current working directory instead of your home directory
##$ -cwd

## Merge output and error text streams into a single stream
##$ -j y

## Name my job, to make it easier to find in the queue
##$ -N MyJobTitle

## And finally, we run the job we came here to do.
## $HOME/ProgramDir/ProgramName ProgramArguments

## OR, for the case of MPI-capable jobs
## mpirun $HOME/path/MpiJobName

## Send email when a job is aborted (a), begins (b), and/or ends (e)
##$ -m abe

## Email address to send the email to based on the above line.
##$ -M myemail@ksu.edu
</syntaxhighlight>

== Array Jobs ==
One of SGE's useful options is the ability to run "Array Jobs"

It can be used with the following option to qsub.

-t n[-m[:s]]
Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Grid
Engine almost like a series of jobs. The option argument to -t specifies the number of array job tasks and the index number which will be
associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SGE_TASK_ID. The option arguments
n, m and s will be available through the environment variables SGE_TASK_FIRST, SGE_TASK_LAST and SGE_TASK_STEPSIZE.

Following restrictions apply to the values n and m:

1 <= n <= 1,000,000
1 <= m <= 1,000,000
n <= m

The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size.
Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SGE_TASK_ID containing one of the 5 index numbers.

Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The
number of tasks in a array job is unlimited.

STDOUT and STDERR of array job tasks will be written into different files with the default location

<jobname>.['e'|'o']<job_id>'.'<task_id>

=== Examples ===
==== Change the Size of the Run ====
Array Jobs have a variety of uses, one of the easiest to comprehend is the following:

I have an application, app1 I need to run the exact same way, on the same data set, with only the size of the run changing.

My original script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
RUNSIZE=50
#RUNSIZE=100
#RUNSIZE=150
#RUNSIZE=200
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
For every run of that job I have to change the RUNSIZE variable, and submit each script. This gets tedious.

With Array Jobs the script can be written like so:

<syntaxhighlight lang="bash">
#!/bin/bash
#$ -t 50:200:50
RUNSIZE=$SGE_TASK_ID
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
I then submit that job, and SGE understands that it needs to run it 4 times, once for each task. It also knows that it can and should run these tasks in parallel.

==== Choosing a Dataset ====
A slightly more complex use of Array Jobs is the following:

I have an application, app2, that needs to be run against every line of my dataset. Every line changes how app2 runs slightly, but I need to compare the runs against each other.

Originally I had to take each line of my dataset and generate a new submit script and submit the job. This was done with yet another script:

<syntaxhighlight lang="bash">
#!/bin/bash
DATASET=dataset.txt
scriptnum=0
while read LINE
do
echo "app2 $LINE" > ${scriptnum}.sh
qsub ${scriptnum}.sh
scriptnum=$(( $scriptnum + 1 ))
done < $DATASET
</syntaxhighlight>
Not only is this needlessly complex, it is also slow, as qsub has to verify each job as it is submitted. This can be done easily with array jobs, as long as you know the number of lines in the dataset. This number can be obtained like so: wc -l dataset.txt in this case lets call it 5000.

<syntaxhighlight lang="bash">
#!/bin/bash
#$ -t 1:5000
app2 `sed -n "${SGE_TASK_ID}p" dataset.txt`
</syntaxhighlight>
This uses a subshell via `, and has the sed command print out only the line number $SGE_TASK_ID out of the file dataset.txt.

Not only is this a smaller script, it is also faster to submit because it is one job instead of 5000, so qsub doesn't have to verify as many.

To give you an idea about time saved: submitting 1 job takes 1-2 seconds. by extension if you are submitting 5000, that is 5,000-10,000 seconds, or 1.5-3 hours.
== Running jobs interactively ==
Some jobs just don't behave like we think they should, or need to be run with somebody sitting at the keyboard and typing in response to the output the computers are generating. Beocat has a facility for this, called 'qrsh'. qrsh uses the exact same command-line arguments as qsub. If no node is available with your resource requirements, qrsh will tell you
Your "qrsh" request could not be scheduled, try again later.
Note that, like qsub, your interactive job will timeout after your allotted time has passed.
== Altering Job Requests ==
We generally do not support users to modify job parameters once the job has been submitted. It can be done, but there are numerous catches, and all of the variations can be a bit problematic; it is normally easier to simply delete the job and resubmit it with the right parameters. '''If your job doesn't start after modifying such parameters (after a reasonable amount of time), delete the job and resubmit it.'''
=== qalter ===
<tt>qalter</tt> is the command that can be used to modify parameters of the job after it has been submitted. '''Note: resource requests (memory, runtime, et. al.) can only be modified on jobs that have yet to start running.'''
==== Changing resource requests ====
Syntax:
<syntaxhighlight lang="bash">
qalter -l $all_resources $jobid
</syntaxhighlight>
When modifying resource requests, you '''must''' specify all of the resources your job needs, not just the one you plan to change. If you just specify h_rt, it will drop the memory request. If you just specify memory, it will drop the h_rt. And so on. This leads to jobs failing to start.
==== Changing core requests ====
Syntax:
<syntaxhighlight lang="bash">
qalter -pe $pe_name $number_of_cores $jobid
</syntaxhighlight>
If you request more cores than are available in the parallel environment that you need, the job may fail to start.
: i.e. requesting 400 cores in the single environment will fail due to the fact that we have no machines with 400 cores.
==== Determining why a job is not running ====
Syntax:
<syntaxhighlight lang="bash">
qalter -w v $jobid
</syntaxhighlight>
This will output the scheduler's reasoning as to why the job has not started. Note that lines like:
Job 1122334455 cannot run in PE "single" because it only offers 0 slots
Are usually red herrings. Sometimes they are indicative that the scheduler cannot meet the resources requests for that job at this moment in time.

Sometimes you will see output like this:
Job 1122334455 does not request 'forced' resource "memory" of queue instance batch.q@elf73.beocat
In this case the user performed a qalter and forgot to specify the memory request. The job will never run in this state.

Other times it will have lots of lines like this:
verification: found possible assignment with 1 slots
This indicates that the job should be scheduled shortly.
== Killable jobs ==
There are a growing number of machines within Beocat that are owned by a particular person or group. Normally jobs from users that aren't in the group designated by the owner of these machines cannot use them. This is because we have guaranteed that the nodes will be accessible and available to the owner at any given time. We will allow others to use these nodes if they designate their job as "killable." If your job is designated as killable, your job will be able to use these nodes, but can (and will) be killed off at any point in time to make way for the designated owner's jobs. Jobs that are marked killable will be re-queued and may restart on another node.

The way you would designate your job as killable is to add <tt>-l killable</tt> to the '''<tt>qsub</tt> or <tt>qrsh</tt>''' arguments. This could be either on the command-line or in your script file.

''Note: This is a submit-time only request, it cannot be added by a normal user after the job has been submitted.'' If you would like jobs modified to be '''killable''' after the jobs have been submitted (and it is too much work to <tt>qdel</tt> the jobs and re-submit), send an e-mail to the administrators detailing the job ids and what you would like done.

== Scheduling Priority ==
The scheduler uses a complex formula to determine the order that jobs get scheduled in. Jobs in general get run in the order that they are submitted to the queue with the following exceptions. Jobs for users in a group that owns nodes will immediately get scheduled on those nodes even if that means bumping existing jobs off. Users in groups that have contributed funds to Beocat may have higher scheduling priority. You can check the base scheduling priority of each group using <tt>qconf -sst</tt>. If you do not have a group your jobs are scheduled using BEODEFAULT. The higher the priority, the faster your job will be moved to the front of the queue. A fair scheduling algorithm adjusts this scheduling priority down as users in that group submit more jobs.

Since all users not in a group having higher priority get put into BEODEFAULT, the priority is always very low and each job gets scheduled in the order it was submitted. Groups with a higher priority may jump ahead of the BEODEFAULT jobs, but if these groups are submitting lots of jobs their priority will become low as well. Groups with the highest priority that are submitting the fewest jobs may see those jobs moved to the front of the queue quickly.

When processing cores become available, the scheduler looks at the head of the queue to find jobs that will fit within the resources available. Shorter jobs of 12 hours or less get marked as killable and will be run on nodes owned by other groups. These jobs will jump past longer jobs when resources become available on owned nodes. Many jobs in the queue may require more memory than is available on some nodes, so smaller memory jobs will be scheduled ahead of larger memory jobs on hosts with more limited memory. <tt>kstat -q</tt> will show you the order in the queue and allow you to see jobs marked as "killable" and those that require large memory.

== Job Accounting ==
Some people may find it useful to know what their job did during its run. The qacct tool will read SGE's accounting file and give you summarized or detailed views on jobs that have run within Beocat.
=== qacct ===
This data can usually be used to diagnose two very common job failures.
==== Job debugging ====
It is simplest if you know the job number of the job you are trying to get information on.
<syntaxhighlight lang="bash">
# if you know the jobid, put it here:
qacct -j 1122334455
# if you don't know the job id, you can look at your jobs over some number of days in this case the past 14 days:
qacct -o $USER -d 14 -j
</syntaxhighlight>

===== My job didn't do anything when it ran! =====
<tt>qname batch.q
hostname mage07.beocat
group some_user_users
owner some_user
project BEODEFAULT
department defaultdepartment
jobname my_job_script.sh
jobnumber 1122334455
...
snipped to save space
...
exit_status 1 </tt>
<tt style="color: red">ru_wallclock 1s</tt>
<tt>ru_utime 0.030s
ru_stime 0.030s
...
snipped to save space
...
arid undefined
category -u some_user -q batch.q,long.q -l h_rt=604800,mem_free=1024.0M,memory=2G</tt>
If you look at the line showing ru_wallclock. You can see that it shows 1s. This means that the job started and then promptly ended. This points to something being wrong with your submission script. Perhaps there is a typo somewhere in it.

===== My job ran but didn't finish! =====
<tt>qname batch.q
hostname scout59.beocat
group some_user_users
owner some_user
project BEODEFAULT
department defaultdepartment
jobname my_job_script.sh
jobnumber 1122334455
...
snipped to save space
...
slots 1 </tt>
<tt style="color: red">failed 37 : qmaster enforced h_rt, h_cpu, or h_vmem limit</tt>
<tt>exit_status 0 </tt>
<tt style="color: red">ru_wallclock 21600s</tt>
<tt>ru_utime 0.130s
ru_stime 0.020s
...
snipped to save space
...
arid undefined</tt>
<tt style="color: red">category -u some_user -q batch.q,long.q -l h_rt=21600,mem_free=512.0M,memory=1G</tt>
If you look at the lines showing failed, ru_wallclock and category we can see some pointers to the issue.
It didn't finish because the scheduler (qmaster) enforced some limit. If you look at the category line, the only limit requested was h_rt. So it was a runtime (wallclock) limit.
Comparing ru_wallclock and the h_rt request, we can see that it ran until the h_rt time was hit, and then the scheduler enforce the limit and killed the job. You will need to resubmit the job and ask for more time next time.

FAQ

2016-12-01T19:49:05Z

Kylehutson: Remove section on hostkey failed - no longer relevant

== How do I connect to Beocat ==
{| class="wikitable"
|-
! colspan="2" | Connection Settings
|-
! Hostname
| style="text-align:right" | headnode.beocat.ksu.edu
|-
! Port
| style="text-align:right" | 22
|-
! Username
| style="text-align:right" | <tt>eID</tt>
|-
! Password
| style="text-align:right" | <tt>eID Password</tt>
|}

{| class="wikitable"
!colspan="2" | Supported Connection Software (Latest Versions of Each)
|-
!rowspan="3" | Shell
|-
| [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]
|-
| ssh from openssh
|-
!rowspan="4" | File Transfer Utilities
|-
| [https://filezilla-project.org/ Filezilla]
|-
| [http://winscp.net/ WinSCP]
|-
| scp and sftp from openssh
|-
!rowspan="2" | Combination
|-
| [http://mobaxterm.mobatek.net/ MobaXterm]
|}

== How do I compile my programs? ==
=== Serial programs ===
==== Fortran ====
<tt>ifort</tt> or <tt>gfortran</tt>
==== C/C++ ====
<tt>icc</tt>, <tt>gcc</tt> and <tt>g++</tt>
=== Parallel programs ===
==== Fortran ====
<tt>mpif77</tt> or <tt>mpif90</tt>
==== C/C++ ====
<tt>mpicc</tt> or <tt>mpic++</tt>
== How are the filesystems on Beocat set up? ==

{| class="wikitable"
|-
! Mountpoint !! Local / Shared !! Size !! Filesystem !! Advice
|-
| /bulk || Shared || 2.1PB shared with /bulk and /scratch || cephfs || Slower than /homes; very old files are culled automatically
|-
| /homes || Shared || 2.1PB shared with /bulk and /scratch || cephfs || Good enough for most jobs; limited to 1TB per home directory
|-
| /scratch || Shared || 2.1PB shared with /bulk and /homes || cephfs || Fast shared tmp space; files not used in 30 days are automatically culled
|-
| /tmp || Local || >100GB (varies per node) || ext4 || Good for I/O intensive jobs
|-
|}
=== Usage Advice ===
For most jobs you shouldn't need to worry, your default working directory
is your homedir and it will be fast enough for most tasks.
I/O intensive work should use /tmp, but you will need to remember to copy
your files to and from this partition as part of your job script. This is made
easier through the <tt>$TMPDIR</tt> environment variable in your jobs.

Example usage of <tt>$TMPDIR</tt> in a job script
<syntaxhighlight lang="bash" line>
#!/bin/bash

#copy our input file to $TMPDIR to make processing faster
cp ~/experiments/input.data $TMPDIR

#use the input file we copied over to the local system
#generate the output file in $TMPDIR as well
~/bin/my_program --input-file=$TMPDIR/input.data --output-file=$TMPDIR/output.data

#copy the results back from $TMPDIR
cp $TMPDIR/output.data ~/experiments/results.$SGE_JOBID
</syntaxhighlight>

You need to remember to copy over your data from <tt>$TMPDIR</tt> as part of your job.
That directory and its contents are deleted when the job is complete.

== What is "killable" or "nokillable" ==
On Beocat, some of the machines have been purchased by specific users and/or groups. These users and/or groups get guaranteed access to their machines at any point in time. Often, these machines are sitting idle because the owners have no need for it at the time. This would be a significant waste of computational power if there were no other way to make use of the computing cycles.

=== Enter the "killable" resource ===
Killable (-l killable) jobs are jobs that can be scheduled to these "owned" machines by users outside of the true group of owners. If a "killable" job starts on one of these owned machines and the owner of said machine comes along and submits a job, the "killable" job will be returned to the queue, (killed off as it were), and restarted at some future point in time. The job will still complete at some future point, and if the job makes use of a checkpointing algorithm it may complete even faster. The trade off between marking a job "killable" and not, is that sometimes applications need a significant amount of runtime, and cannot resume running from a partial output, meaning that it may get restarted over and over again, never reaching the finish line. As such, we only auto-enable "killable" for relatively short jobs (<=12:00:00). Some users still feel this is a hinderance, so we created another flag to tell us not to automatically mark short jobs "killable"

=== Enter the "nokillable" resource ===
Nokillable (-l nokillable) simply makes it so that the jsv will not automatically mark the job killable. If you mark both killable and nokillable, killable will win.

=== The trade-off ===
If a job is marked killable, there are a non-trivial amount of additional nodes that the job can run on. If your job checkpoints itself, or is relatively short, there should be no downside to marking the job killable, as the job will probably start sooner. If your job is long-running and doesn't checkpoint (save its state to restart a previous session) itself, it could cause your job to take longer to complete.

== Help! When I submit my jobs I get "Warning To stay compliant with standard unix behavior, there should be a valid #! line in your script i.e. #!/bin/tcsh" ==
Job submission scripts are supposed to have a line similar to '<code>#!/bin/bash</code>' in them to start. We have had problems with people submitting jobs with invalid #! lines, so we enforce that rule. When this happens the job fails and we have to manually clean it up. The warning message is there just to inform you that the job script should have a line in it, in most cases #!/bin/tcsh or #!/bin/bash, to indicate what program should be used to run the script. When the line is missing from a script, by default your default shell is used to execute the script (in your case /usr/local/bin/tcsh). This works in most cases, but may not be what you are wanting.

== Help! When I submit my jobs I get "A #! line exists, but it is not pointing to an executable. Please fix. Job not submitted." ==
Like the above, error says you need a #!/bin/bash or similar line in your job script. This error says that while the line exists, the #! line isn't mentioning an executable file, thus the script will not be able to run. Most likely you wanted #!/bin/bash instead of something else.

== Help! My jobs keep dying after 1 hour and I don't know why ==
Beocat has default runtime limit of 1 hour. If you need more than that, or need more than 1 GB of memory per core, you'll want to look at the documentation [[SGEBasics|here]] to see how to request it.

In short, when you run qsub for your job, you'll want to put something along the lines of '<code>-l h_rt=10:00:00</code>' before the job script if you want your job to run for 10 hours.

== Help my error file has "Warning: no access to tty" ==
The warning message "Warning: no access to tty (Bad file descriptor)" is safe to ignore. It typically happens with the tcsh shell.

== Help! My job isn't going to finish in the time I specified. Can I change the time requirement? ==
Generally speaking, no.

Jobs are scheduled based on execution times (among other things). If it were easy to change your time requirement, one could submit a job with a 15-minute run-time, get it scheduled quickly, and then say "whoops - I meant 15 weeks", effectively gaming the job scheduler. In fact, even the administrators cannot change the run-time requirement of a particular job. In extreme circumstances and depending on the job requirements, we '''may''' be able to manually intervene. This process prevents other users from using the node(s) you are currently using, so are not routinely approved. Contact Beocat support (below) if you feel your circumstances warrant special consideration.

== Help! My perl job runs fine on the head node, but only runs for a few seconds and then quits when submitted to the queue. ==
Perl doesn't like getting called straight from the scheduler. However, there is a fairly easy workaround. Create a shell wrapper script that calls perl and its program.

For instance, I can create a script called runperl.sh that looks like this:

#!/bin/sh
#$ -l h_rt=1:00:00,mem=1G
/usr/bin/perl /path/to/my/perl_program.pl

Make this wrapper program executable:
chmod 755 runperl.sh

Then submit it with
qsub runperl.sh

Of course, the name of this script isn't important, as long as you change the corresponding chmod and qsub commands.

== Help! When using mpi I get 'CMA: no RDMA devices found' or 'A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces' ==
This message simply means that some but not all nodes the job is running on have infiniband cards. The job will still run, but will not use the fastest interconnect we have available. This may or may not be an issue, depending on how message heavy your job is. If you would like to not see this warning, you may request infiniband as a resource when submitting your job. <code>-l infinband=TRUE</code>

== What happens to my data when I leave K-State? ==
First of all, although we use eid credentials, we are not tied in with K-State's central IT policies which apply to employees or students leaving the university. As long as you keep your eid password current, you still have access to Beocat. Once we deem your data to be "stale", we will archive your data and disable your account. We have no written policy on when we do this, because we only do so as necessity dictates, but generally speaking if you have any data which is modified for less than two years will not be marked as stale. If your account is disabled for this reason, you will have to apply for a new account and un-archive your data.

== Common Storage For Projects ==
Sometimes it is useful for groups of people to have a common storage area. If you already have a project you can do the following:

* Create a directory in one of the home directories of someone in your group, ideally the project owner's.
** <tt>mkdir $directory</tt>
* Change the group to the name assigned by the Beocat admins
** <tt>chgrp -R $group_name $directory</tt>
* Set the directory writeable and sticky for the group
** <tt>chmod g+ws $directory</tt>

* Change your umask to 002 (there will probably be a setting for it in your file transfer utilities, also). This step needs to be done by all group members.
** <tt>umask 002</tt> needs to go above <tt><nowiki>if [[ $- != *i* ]] ; then</nowiki></tt> in your <tt>~/.bashrc</tt> file

* Finally logout and log back in

== How do I get more help? ==
There are many sources of help for most Linux systems.

=== Unix man pages ===
Linux provides man pages (short for manual pages). These are simple enough to call, for example: if you need information on submitting jobs to Beocat, you can type '<code>man qsub</code>'. This will bring up the manual for qsub.

=== GNU info system ===
Not all applications have "man pages." Most of the rest have what they call info pages. For example, if you needed information on finding a file you could use '<code>info find</code>'.

=== This documentation ===
This documentation is very thoroughly researched, and has been painstakingly assembled for your benefit. Please use it.

=== Contact support ===
Support can be contacted [mailto:beocat@cis.ksu.edu here]. Please include detailed information about your problem, including the job number, applications you are trying to run, and the current directory that you are in.

Main Page

2016-12-01T19:45:09Z

Kylehutson: Change CIS references to CS

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|HPC]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of Linux servers coordinated by the [https://arc.liv.ac.uk/trac/SGE SGE] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A comparatively small [[Hadoop]] cluster
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''beocat#beocat''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use SGE for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SGEBasics]] page for an introduction on how to submit your first job. If you are already familiar with SGE, we also have an [[AdvancedSGE]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

We are also available on IRC on the [http://freenode.net/using_the_network.shtml freenode chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''m0zes''' and/or '''kylehutson''' in your message, and it should grab our attention. Available from a web browser [[Special:WebChat|here.]]

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat.

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

LinuxBasics

2016-12-01T19:42:09Z

Kylehutson: Change beocat.cis.ksu.edu to headnode.beocat.ksu.edu

'''Disclaimer:''' This is a ''very'' large topic, and much too broad to be covered on a single support page. There are many other sites (yes, entire sites) which cover the topic in more detail. We'll link so some of them below. This page is meant to be just the essentials.

== Logging in for the first time ==
To login to Beocat, you first need an "SSH Client". [[wikipedia:Secure_Shell|SSH]] (short for "secure shell") is a protocol that allows secure communication between two computers. We recommend the following.
* Windows
** [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] is by far the most common SSH client, both for Beocat and in the world.
** [http://mobaxterm.mobatek.net/ MobaXterm] is a fairly new client with some nice features, such as being able to SCP/SFTP (see below), and running X (which isn't terribly useful on Beocat, but might be if you connect to other Linux hosts).
** [http://www.cygwin.com/ Cygwin] is for those that would rather be running Linux but are stuck on Windows. It's purely a text interface.
* Macintosh
** OS-X has SSH a built-in application called "Terminal". It's not great, but it will work for most Beocat users.
** [http://www.iterm2.com/#/section/home iTerm2] is the terminal application we prefer.
* Others
** There are [[wikipedia:Comparison_of_SSH_clients|many SSH clients]] for many different platforms available. While we don't have experience with many of these, any should be sufficient for access to Beocat.

You'll need to connect your client (via the SSH protocol, if your client allows multiple protocols) to headnode.beocat.ksu.edu.

For command-line tools, the command to connect is
ssh ''username''@headnode.beocat.ksu.edu

Your username is your [http://eid.ksu.edu K-State eID] name and the password is your eID password.

'''Note:''' When you type your password, nothing shows up on the screen, not even asterisks.

You'll know you are successfully logged in when you see a prompt that says
(''machinename'':~) ''username''%
where ''machinename'' is the name of the machine you've logged into (currently either 'athena' or 'minerva') and ''username'' is your eID username

== Transferring files (SCP or SFTP) ==
Usually, one of the first things people want to do is to transfer files into or out of Beocat. To do so, you need to use [[wikipedia:Secure_copy|SCP]] (secure copy) or [[wikipedia:SSH_File_Transfer_Protocol|SFTP]] (SSH FTP or Secure FTP). Again, there are multiple programs that do this.
* Windows
** Putty (see above) has PSCP and PSFTP programs (both are included if you run the installer). It is a command-line interface (CLI) rather than a graphical user interface (GUI).
** MobaXterm (see above) has a built-in GUI SFTP client that automatically changes the directories as you change them in your SSH session.
** [https://filezilla-project.org/ FileZilla] (client) has an easy-to-use GUI. Be sure to use 'SFTP' mode rather than 'FTP' mode.
** [http://winscp.net/eng/index.php WinSCP] is another easy-to-use GUI.
** Cygwin (see above) has CLI scp and sftp programs.
* Macintosh
** [https://filezilla-project.org/ FileZilla] is also available for OS-X.
** Within terminal or iTerm, you can use the 'scp' or 'sftp' programs.
* Linux
** FileZilla also has a GUI linux version, in additon to the CLI tools.

=== Using a Command-Line Interface (CLI) ===
You can safely ignore this section if you're using a graphical interface (GUI). We highly recommend using a GUI when first learning how to use Beocat.

First test case: transfer a file called myfile.txt in your current folder to your home directory on Beocat. For these examples, I use bold text to show what you type and plain text to show Beocat's response

Using SCP:
'''scp myfile.txt ''username''@headnode.beocat.ksu.edu:'''
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

Note the colon at the end of the 'scp' line.

Using SFTP
'''sftp ''username''@headnode.beocat.ksu.edu'''
Password: '''(type your password here, it will not show any response on the screen)'''
Connected to headnode.beocat.ksu.edu.
sftp> '''put myfile.txt'''
Uploading myfile.txt to /homes/kylehutson/myfile.txt
myfile.txt 100% 0 0.0KB/s 00:00
sftp> '''exit'''

SFTP is interactive, so this is a two-step process. First, you connect to Beocat, then you transfer the file. As long as the system gives the <code>sftp> </code> prompt, you are in the sftp program, and you will remain there until you type 'exit'.

Second test case: transfer a file called myfile.txt in your current folder to a diretory named 'mydirectory' under your home directory on Beocat.

Here we run into one of the problems with scp - there is no easy way of creating 'mydirectory' if it doesn't already exist. If it does not already exist, you must login via ssh (as seen above) and create the directory using the 'mkdir' command (see Common Linux Commands) below.

'''scp myfile.txt ''username''@headnode.beocat.ksu.edu:mydirectory'''
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

An alternative version. If the colon is immediately followed by a slash, the directory name is taken from the root, rather than your home directory. So, given that your home directory on Beocat is /homes/''username'', we could instead type
'''scp myfile.txt ''username''@headnode.beocat.ksu.edu:/homes/''username''/mydirectory'''
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

Using SFTP:
sftp ''username''@headnode.beocat.ksu.edu
Password: '''(type your password here, it will not show any response on the screen)'''
Connected to headnode.beocat.ksu.edu.
sftp> '''mkdir mydirectory'''
[Note, if this directory already exists, you will get the response "Couldn't create directory: Failure"]
sftp> '''cd mydirectory'''
sftp> '''put myfile.txt'''
Uploading myfile.txt to /homes/''username''/mydirectory/myfile.txt
myfile.txt 100% 0 0.0KB/s 00:00
sftp> '''quit'''

Third test case: copy myfile.txt from your home directory on Beocat to your current folder.

Using scp:
scp ''username''@headnode.beocat.ksu.edu:myfile.txt .
Password: '''(type your password here, it will not show any response on the screen)'''
myfile.txt 100% 0 0.0KB/s 00:00

Using SFTP:
'''sftp ''username''@headnode.beocat.ksu.edu'''
Password: '''(type your password here, it will not show any response on the screen)'''
Connected toheadnode.beocat.ksu.edu.
sftp> '''get myfile.txt'''
Fetching /homes/''username''/myfile.txt to myfile.txt
myfile.txt 100% 0 0.0KB/s 00:00
sftp> '''exit'''

== Basic Linux Commands ==
Again, this guide is very limited, mostly limited to directory navigation and basic file commands. [http://www.ee.surrey.ac.uk/Teaching/Unix/ Here] is a pretty decent tutorial if you want to dig deeper. If you want more, entire books have been written on the subject.

=== The Lingo ===
{| class="wikitable"
!''Term''
!''Definition''
|-
|Directory
|A "Folder" in Windows or OS-X terms. A location where files or other directories are stored. The current directory is sometimes represented as `.` and the parent directory can be referenced as `..`
|-
|Shell
|The interface or environment under which you can run commands. There is a section below on shells
|-
|SSH
|Secure Shell. A protocol that encrypts data and can give access to another system, usually by a username and password
|-
|SCP
|Secure Copy. Copying to or from a remote system using part of SSH
|-
|path
|The list of directories which are searched when you type the name of a program. There is a section below on this
|-
|ownership
|Every file and directory has an user and a group attached to it, called its owners. These affect permissions.
|-
|permissions
|The ability to read, write, and/or execute a file. Permissions are based on ownership
|-
|switches
|Modifiers to a command-line program, usually in the form of -(letter) or --``(word). Several examples are given below, such as the '-a' on the 'ls' command
|-
|pipes and redirects
|Changes the input (often called 'stdin') and/or output (often called stdout) to a program or a file
|}

=== Linux Command Line Cheat Sheet ===
{| class="wikitable"
|+File System Navigation
|-
!''Command''
!''What it does''
!''Example Usage''
!''Example Output''
|-
|pwd
|"Print working directory", Where am I now?
|<code>pwd</code>
|<code>/homes/mozes</code>
|-
|ls
|Lists files and folders
|<code>ls ~/</code>
|<code>NewFile NewFolder</code>
|-
|ls -lh
|Lists files and folders with perms size and ownership
|<code>ls -lh ~/</code>
|<code>-rw-r--r-- 1 mozes mozes_users 1 Jul 13 2011 NewFile
drwxr-xr-x 9 mozes mozes_users 9.0K Apr 12 2010 NewFolder</code>
|-
|ls -a
|Lists all files and folders
|<code>ls -a ~/</code>
|<code>. .. .bashrc .bash_profile .tcshrc NewFile NewFolder</code>
|-
|cd
|Changes directory
|<code>cd NewFolder</code>
|
|-
|cd ..
|Changes to parent directory
|<code>cd ..</code>
|
|-
|cd -
|Changes to the previous directory you were in
|<code>cd -</code>
|
|-
|cd ~
|Changes to your home directory
|<code>cd ~</code>
|
|}

{| class="wikitable"
|+Working with files
|-
!Command'
!What it does
!Example Usage'
!Example Output''
|-
|file
|Identifies the type of object a file is
|<code>file NewFile</code>
|<code>NewFile: a /usr/bin/python script, ASCII text executable</code>
|-
|cat
|Prints the contents of one or more files
|<code>cat NewFile</code>
|<code>This is line one
This is line two</code>
|-
|cp
|copy a file
|<code>cp OldFile NewFile</code>
|
|-
|cp -i
|copy a file, ask to overwrite
|<code>cp -i OldFile NewFile}</code>
|<code>overwrite NewFile? (y/n [n])</code>
|-
|cp -r
|copy a directory, including contents
|<code>cp -r OldFolder NewFolder</code>
|
|-
|mv
|move, or rename, a file
|<code>mv OldFile NewFile</code>
|
|-
|mv -i
|move, or rename, a file, ask to overwrite
|<code>mv -i OldFile NewFile</code>
|<code>overwrite NewFile? (y/n [n])</code>
|-
|rm
|remove a file
|<code>rm NewFile</code>
|
|-
|rm -i
|remove a file, ask to be sure (useful with -r)
|<code>rm -i NewFile</code>
|<code>remove NewFile? (y/n [n])</code>
|-
|rm -r
|remove a direcory and its contents
|<code>rm -r NewFolder</code>
|
|-
|mkdir
|creates a directory
|<code>mkdir TempFolder</code>
|
|-
|rmdir
|removes an empty directory
|<code>rmdir TempFolder</code>
|
|-
|touch
|creates an empty file
|<code>touch TempFile</code>
|
|}

{| class="wikitable"
|+Finding files and directories with [http://linux.die.net/man/1/find find]
|-
!''Command''
!''What it does''
!''Example Usage''
|-
| find <directory>
| finds all files and folders within <directory>
| find ~/
|-
| find <directory> -iname '<filename>'
| finds all files and directories within <directory> that match <filename>
| find ~/ -iname 'hello.qsub'
|-
| find <directory> -iname '*<partialmatch>*'
| finds all files and directories within <directory> that partially match <partialmatch>
| find ~/ -iname '*.qsub*'
|}

Other useful commands include <code>htop</code>, <code>less</code>, and <code>man</code>. <code>man</code> followed by a command name above will give you the manual page for the specified command full of many other useful options for the command. <code>htop</code> will give you an overview of the processes currently being run on the host you are connected to. <code>less</code> allows you to page through files and see their contents using <PgUp> and <PgDn>.

=== Editing Text Files ===
If you're new to Linux, the editor you will probably want to use is 'nano'. It works much the same as 'Notepad' in Windows or 'textedit' on OS-X. Note that you cannot use your mouse to change position within the document as you can with your local computer. You must use the arrow keys, instead.

So, if I wanted to edit my .bashrc (as shown below), and I was already in my home directory (see above), I would type
nano .bashrc

While in nano, there is a list of actions you can take at the bottom of the screen. <Ctrl> is represented by a caret (`^`), so to exit (labeled as `^`X at the bottom of the screen), I would type <ctrl>-x. This action prompts you whether you want to save and exit (Y), lose changes and exit (N), or cancel and go back to editing (<ctrl>-c).

If you do a significant amount of text editing in Linux, you'll probably want to switch to a more powerful editor, such as vim. The usage of vim is beyond the scope of this document. It is not at all intuitive to the beginning user, but with a little practice it becomes a much faster way of editing text files. If you're interested in using vim, [http://www.openvim.com/tutorial.html|there is a nice tutorial here].

=== Shells ===
==== What is a Shell? ====
In this case, I don't believe I can do a better job explaining shells than [[wikipedia:Shell_(computing)|this]].
==== tcsh ====
Elsewhere at Kansas State University, the default Shell is set to tcsh. tcsh stands for "TENEX C SHell." It is considered a replacement for csh and uses many of the same features. If you have experience with either csh or tcsh you'll probably feel right at home. This was the default shell until July of 2013. If you had an account before then, it is probably still tcsh.

But what if you don't want or like tcsh, what can you do? Well, we have other shells available of Beocat as well.
==== bash ====
[http://www.gnu.org/software/bash/ Bash] seems to be the defacto standard shell in most Linux installs today. Bash is common and probably what most of you are used to. As of July 2013, bash is our new default shell. All new users will be set to bash initially.

bash configuration files:

This section gets into some minutiae with the way our job scheduler interacts with bash. If you're trying to solve a problem, read on, otherwise you can probably skip this section.

Bash has 3 user configurable configuration files, <code>~/.bashrc ~/.bash_profile</code> and <code>~/.bash_logout</code>. We'll look at the two more relevant ones <code>~/.bashrc</code> and <code>~/.bash_profile</code>

Bash normally has 3 ways of looking at things, '''login''', '''interactive''', or '''none'''.

Normally what happens is that shells that are '''interactive''' read <code>~/.bash_profile</code>, shells that are '''login''' read <code>~/.bashrc</code>. '''none''' shells read neither.

In Beocat the flow is a little more sane because of our default <code>~/.bash_profile</code>:
We setup a 4th category of shell, '''login+interactive'''. '''login''' and '''login+interactive''' shells read in ''both'' your <code>~/.bash_profile</code> and your <code>~/.bashrc</code>, the difference being that the plain '''login''' shell stops reading <code>~/.bashrc</code> at the following statement:
<syntaxhighlight lang="bash" line>
if [[ $- != *i* ]] ; then
# Shell is non-interactive. Be done now!
return
fi
</syntaxhighlight>

qsub jobs are '''login''', qrsh jobs are '''login+interactive''', logging into Beocat in a way that you can enter commands is '''login+interactive'''. There are very few cases that you will get '''none'''. For any session that isn't '''interactive''', your sourced files cannot output anything to the screen, or else it can break scp or sftp file transfers.

If they are ''quiet'' statements, and you want them in all shells, you can put them ''before'' the aforementioned if statement. If they are not ''quiet'' or they output ''anything'' to the screen, you must put them ''after'' the aforementioned if statement.

==== zsh ====
[http://zsh.sourceforge.net/ zsh] is an alternative to bash and tcsh. It tends to support more complex features than either of the other two while using a syntax remarkably similar to bash. Unless specifically noted, when we specify '''Change your shell to bash''', <tt>zsh</tt> should work as well.

==== Changing Shells ====
Previously, we gave you the option of using a <code>~/.login</code> to modify your shell. This is no longer supported, if you have issues with your shell/paths/environment variables we will ask you to delete your <code>~/.login</code> file and change your shell via the method below.

You can change your shell is via <code>chsh</code> on either of the headnodes (athena/minerva). This does not need to be re-done if you've changed to it to your preferred shell in the past.

Use the appropriate of the following three lines:
<syntaxhighlight lang="bash" line>
/usr/local/bin/chsh -s bash && bash -l
/usr/local/bin/chsh -s tcsh && tcsh -l
/usr/local/bin/chsh -s zsh && zsh -l
</syntaxhighlight>

=== Changing your PATH ===
Typically, you don't have to change your PATH, but it is useful to know what your PATH is and what it does. The PATH is the list of directories which are searched when you type the name of a program. Note that by default the current directory is NOT included in the path, so if you were wanting to run a program called MyProgram in the current directory, you could NOT simply type 'MyProgram', you would instead type <code>'./MyProgram'</code> (where the '.' represents the current directory).

To find your PATH, we need to identify which shell you are using. If you do not know, run the following:
<syntaxhighlight lang="bash" line>
ps | awk '/sh/ {print $4}'
</syntaxhighlight>

==== tcsh ====
You'll need to edit a file in your home directory called .tcshrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.
<syntaxhighlight lang="bash" line>
setenv PATH /usr/local/bin:$PATH
</syntaxhighlight>
==== bash ====
You'll need to edit a file in your home directory called .bashrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.
<syntaxhighlight lang="bash" line>
export PATH=/usr/local/bin:$PATH
</syntaxhighlight>

==== zsh ====
You'll need to edit a file in your home directory called .zshrc, replacing /usr/local/bin with the directory that you want added to your PATH using a text editor as shown above.
<syntaxhighlight lang="bash" line>
export PATH="/usr/local/bin:$PATH"
</syntaxhighlight>

=== Ownership and Permissions ===
Every file and directory has a user and group associated with it. You can view ownership information by using the '-l' switch on ls. By default on Beocat, files you create have a user ownership of your username (i.e., your eID) and a group ownership of your username_users. So, if I were logged in as 'myusername' and I had a single file in my home directory called MyProgram, the result of typing 'ls -l' would be something like this:
total 0
-rwxr-x--- 1 myusername myusername_users 79 May 31 2011 MyProgram
This tells us several things.
* The first column ('-rwxr-x---') is permissions (covered below)
* The second column ('1') is the number of links to this file. You can safely ignore this (unless you're both masochistic and interested in filesystem details)
* The third column ('myusername') shows the user ownership
* The fourth column ('myusername_users') shows the group ownership
* The fifth column ('79') gives the size of the file in bytes
* The next columns ('May 31 2011'), as you have probably guessed, gives the date the file was last changed
* The final column ('MyProgram') is the name of the file

So why is this interesting to us? Because whenever things ''don't'' work, it's usually because of file ownership or permissions. Looking at these often gives us some useful diagnostic information.

The permissions field shows us who has permissions to do what with this file. It is always 10 characters. The first character (-) is usually either a '-' for a regular file or a 'd' for a directory. The next 9 characters are broken into three groups of three, with each group showing read (r), write (w), and execute (x) permissions for the owner, group, and world, in that order.
* The first group (rwx) shows permissions for the owner (myusername). The owner here has read, write, and execute permissions
* The next group (r-x) shows permissions for the group (myusername_users). The group here has read and execute permissions, but cannot write.
* The last group (---) shows permissions for the rest of the world. The world has no permissions to read, write, or execute.

When you create a shell script with a text editor, and sometimes when you copy programs to Beocat via SCP, the execute flag is not set. The permissions string may look more like (-rw-r--r--). To change this, you need to give yourself permission to execute this program. This is done with the 'chmod' (change mode) command. 'chmod' can have a long and confusing syntax, but since by far the most common problem is to give yourself execute permissions, here is the command to change that:
chmod u+x MyProgram
This changes the permissions so that the user ('u', i.e., the owner) adds ('+') execute permission ('x').

For more complex ownership or permissions changes, please feel free to contact the Beocat staff.

=== Manual (man) pages ===
Most commands have a complex set of switches that will modify the amount or type of information they display. To find out what switches are available, or how a program expects data, you can use the manual pages by typing "`man` ''command''". Using one of the most common Linux commands, take a look the output of 'man ls'. It shows that it has over 50 switches available, ranging from which files to include, to how to display file sizes, to sort order and more. (I'm not pasting it here, because it's over 200 lines long!) To navigate a 'manpage', use the up-arrow and down-arrow keys. Press 'q' to quit.

=== Pipes and Redirects ===
Typically a Linux program takes data from the keyboard and outputs data to the screen. In Unix and Linux terminology, the keyboard is the default 'stdin' (pronounced "standard in") and the screen is the default 'stdout' (pronounced "standard out"). Many times, we want to take data from somewhere else (like a file, or the output of another program) and send it to yet another location. These redirectors are:
{|
|cmd > filename
|Redirect output from cmd to filename ||
|-
|cmd >> filename
|Redirect output from cmd and append to filename
|-
|cmd < filename
|Redirect input from cmd to filename
|-
| cmd1 | cmd2
| Use the output from cmd1 as the input to cmd2
|}
Here is a quick example. Let's say I have a thousands of files in a directory, and I want a list of those that end in '.sh'
'ls' by itself scrolls so far I can't see even a fraction of them. So, I redirect the output to a file
ls > ~/filelist.txt
That gives me all the files in the current folder and saves them in my home directory in 'filelist.txt'.
A quick look through the file in my favorite editor tells me this is still going to take too long, so I need another step. The 'grep' program is a commonly-used program to perform pattern matching. The syntax of 'grep' is beyond the scope of this document, but take my word for it that
grep '\.sh$'
will return all lines that end in .sh.

We can now redirect the input from grep to the file we just created:
grep '\.sh$' < ~/filelist.txt
Great! We now have our list. However, we wanted to save this as filelist.txt, and instead we have another list that we have to copy-and-paste. Instead of redirecting to a file, we'll use the vertical bar '|' (which we often term a "pipe") to send the output of one command to another.
ls | grep '\.sh$' > ~/filelist.txt
This time the output of 'ls' is ''not'' redirected to a file, but is redirected to the next command (grep). The output of grep (which is all our .sh files) instead of being sent to the screen is redirected to the file ~/filelist.txt.

This example is a very simple demonstration of how pipes and redirects work. Many more examples with complex structures can be found at http://www.ibm.com/developerworks/linux/library/l-lpic1-v3-103-4/index.html

Main Page

2015-12-18T16:40:05Z

Kylehutson: Added "credits and accolades" section.

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|HPC]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the [http://www.cis.ksu.edu/ Computing and Information Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of Linux servers coordinated by the [https://arc.liv.ac.uk/trac/SGE SGE] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.cis.ksu.edu/ http://ganglia.beocat.cis.ksu.edu/].
* A comparatively small [[Hadoop]] cluster
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.cis.ksu.edu/ https://account.beocat.cis.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''beocat#cis-ksu-edu''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to beocat.cis.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use SGE for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SGEBasics]] page for an introduction on how to submit your first job. If you are already familiar with SGE, we also have an [[AdvancedSGE]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cis.ksu.edu beocat@cis.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

We are also available on IRC on the [http://freenode.net/using_the_network.shtml freenode chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''m0zes''' and/or '''kylehutson''' in your message, and it should grab our attention. Available from a web browser [[Special:WebChat|here.]]

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat.

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Credits

2015-12-18T16:37:45Z

Kylehutson: Initial creation

Beocat has been credited in the following published works:

http://pubs.acs.org/doi/abs/10.1021/acsnano.5b03592 - Predicting Adsorption Affinities of Small Molecules on Carbon Nanotubes Using Molecular Dynamics Simulation

FAQ

2015-12-11T21:46:45Z

Kylehutson: Added FAQ entry for those leaving K-State

== How do I connect to Beocat ==
{| class="wikitable"
|-
! colspan="2" | Connection Settings
|-
! Hostname
| style="text-align:right" | beocat.cis.ksu.edu
|-
! Port
| style="text-align:right" | 22
|-
! Username
| style="text-align:right" | <tt>eID</tt>
|-
! Password
| style="text-align:right" | <tt>eID Password</tt>
|}

{| class="wikitable"
!colspan="2" | Supported Connection Software (Latest Versions of Each)
|-
!rowspan="3" | Shell
|-
| [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]
|-
| ssh from openssh
|-
!rowspan="4" | File Transfer Utilities
|-
| [https://filezilla-project.org/ Filezilla]
|-
| [http://winscp.net/ WinSCP]
|-
| scp and sftp from openssh
|-
!rowspan="2" | Combination
|-
| [http://mobaxterm.mobatek.net/ MobaXterm]
|}

== How do I compile my programs? ==
=== Serial programs ===
==== Fortran ====
<tt>ifort</tt> or <tt>gfortran</tt>
==== C/C++ ====
<tt>icc</tt>, <tt>gcc</tt> and <tt>g++</tt>
=== Parallel programs ===
==== Fortran ====
<tt>mpif77</tt> or <tt>mpif90</tt>
==== C/C++ ====
<tt>mpicc</tt> or <tt>mpic++</tt>
== How are the filesystems on Beocat set up? ==

{| class="wikitable"
|-
! Mountpoint !! Local / Shared !! Size !! Filesystem !! Advice
|-
| /home || Shared || 350TB total || glusterfs on top of xfs || Good enough for most jobs
|-
| /tmp || Local || >100GB (varies per node) || ext4 || Good for I/O intensive jobs
|-
|}
=== Usage Advice ===
For most jobs you shouldn't need to worry, your default working directory
is your homedir and it will be fast enough for most tasks.
I/O intensive work should use /tmp, but you will need to remember to copy
your files to and from this partition as part of your job script. This is made
easier through the <tt>$TMPDIR</tt> environment variable in your jobs.

Example usage of <tt>$TMPDIR</tt> in a job script
<syntaxhighlight lang="bash" line>
#!/bin/bash

#copy our input file to $TMPDIR to make processing faster
cp ~/experiments/input.data $TMPDIR

#use the input file we copied over to the local system
#generate the output file in $TMPDIR as well
~/bin/my_program --input-file=$TMPDIR/input.data --output-file=$TMPDIR/output.data

#copy the results back from $TMPDIR
cp $TMPDIR/output.data ~/experiments/results.$SGE_JOBID
</syntaxhighlight>

You need to remember to copy over your data from <tt>$TMPDIR</tt> as part of your job.
That directory and its contents are deleted when the job is complete.

== What is "killable" or "nokillable" ==
On Beocat, some of the machines have been purchased by specific users and/or groups. These users and/or groups get guaranteed access to their machines at any point in time. Often, these machines are sitting idle because the owners have no need for it at the time. This would be a significant waste of computational power if there were no other way to make use of the computing cycles.

=== Enter the "killable" resource ===
Killable (-l killable) jobs are jobs that can be scheduled to these "owned" machines by users outside of the true group of owners. If a "killable" job starts on one of these owned machines and the owner of said machine comes along and submits a job, the "killable" job will be returned to the queue, (killed off as it were), and restarted at some future point in time. The job will still complete at some future point, and if the job makes use of a checkpointing algorithm it may complete even faster. The trade off between marking a job "killable" and not, is that sometimes applications need a significant amount of runtime, and cannot resume running from a partial output, meaning that it may get restarted over and over again, never reaching the finish line. As such, we only auto-enable "killable" for relatively short jobs (<=12:00:00). Some users still feel this is a hinderance, so we created another flag to tell us not to automatically mark short jobs "killable"

=== Enter the "nokillable" resource ===
Nokillable (-l nokillable) simply makes it so that the jsv will not automatically mark the job killable. If you mark both killable and nokillable, killable will win.

=== The trade-off ===
If a job is marked killable, there are a non-trivial amount of additional nodes that the job can run on. If your job checkpoints itself, or is relatively short, there should be no downside to marking the job killable, as the job will probably start sooner. If your job is long-running and doesn't checkpoint (save its state to restart a previous session) itself, it could cause your job to take longer to complete.

== Help! When I submit my jobs I get "Warning To stay compliant with standard unix behavior, there should be a valid #! line in your script i.e. #!/bin/tcsh" ==
Job submission scripts are supposed to have a line similar to '<code>#!/bin/bash</code>' in them to start. We have had problems with people submitting jobs with invalid #! lines, so we enforce that rule. When this happens the job fails and we have to manually clean it up. The warning message is there just to inform you that the job script should have a line in it, in most cases #!/bin/tcsh or #!/bin/bash, to indicate what program should be used to run the script. When the line is missing from a script, by default your default shell is used to execute the script (in your case /usr/local/bin/tcsh). This works in most cases, but may not be what you are wanting.

== Help! When I submit my jobs I get "A #! line exists, but it is not pointing to an executable. Please fix. Job not submitted." ==
Like the above, error says you need a #!/bin/bash or similar line in your job script. This error says that while the line exists, the #! line isn't mentioning an executable file, thus the script will not be able to run. Most likely you wanted #!/bin/bash instead of something else.

== Help! My jobs keep dying after 1 hour and I don't know why ==
Beocat has default runtime limit of 1 hour. If you need more than that, or need more than 1 GB of memory per core, you'll want to look at the documentation [[SGEBasics|here]] to see how to request it.

In short, when you run qsub for your job, you'll want to put something along the lines of '<code>-l h_rt=10:00:00</code>' before the job script if you want your job to run for 10 hours.

== Help my error file has "Warning: no access to tty" ==
The warning message "Warning: no access to tty (Bad file descriptor)" is safe to ignore. It typically happens with the tcsh shell.

== Help! My job isn't going to finish in the time I specified. Can I change the time requirement? ==
Generally speaking, no.

Jobs are scheduled based on execution times (among other things). If it were easy to change your time requirement, one could submit a job with a 15-minute run-time, get it scheduled quickly, and then say "whoops - I meant 15 weeks", effectively gaming the job scheduler. In fact, even the administrators cannot change the run-time requirement of a particular job. In extreme circumstances and depending on the job requirements, we '''may''' be able to manually intervene. This process prevents other users from using the node(s) you are currently using, so are not routinely approved. Contact Beocat support (below) if you feel your circumstances warrant special consideration.

== Help! My perl job runs fine on the head node, but only runs for a few seconds and then quits when submitted to the queue. ==
Perl doesn't like getting called straight from the scheduler. However, there is a fairly easy workaround. Create a shell wrapper script that calls perl and its program.

For instance, I can create a script called runperl.sh that looks like this:

#!/bin/sh
#$ -l h_rt=1:00:00,mem=1G
/usr/bin/perl /path/to/my/perl_program.pl

Make this wrapper program executable:
chmod 755 runperl.sh

Then submit it with
qsub runperl.sh

Of course, the name of this script isn't important, as long as you change the corresponding chmod and qsub commands.

== Help! When using mpi I get 'CMA: no RDMA devices found' or 'A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces' ==
This message simply means that some but not all nodes the job is running on have infiniband cards. The job will still run, but will not use the fastest interconnect we have available. This may or may not be an issue, depending on how message heavy your job is. If you would like to not see this warning, you may request infiniband as a resource when submitting your job. <code>-l infinband=TRUE</code>

== When I try to connect to Beocat, I get a message that includes "Host key verification failed." ==
There are two ways you might get this.

1) Somebody is trying to intercept your traffic and is impersonating Beocat (this is unlikely, but not impossible)

2) You haven't logged into Beocat since we updated our host keys in May of 2015

The following message was sent to the CIS-BEOCAT listserv in May.
<blockquote>
Hello all,

Another quick note, as you might run into this now, I've created new ssh private/public keys for the headnodes. This means that you will likely see warnings from your ssh/scp/sftp clients that the public key has changed (starting now). This is a good thing, but you will likely have to tell your client in some way that the new key is to be accepted. Here are the public keys, if you would like to verify any of them.

ssh-dss AAAAB3NzaC1kc3MAAACBAOOneUDA3MuU4O9WWGtxCwzjkWn1vx0+b2BZmJyxc99Rpb6mP2Cd3CxvUK5Az+ZP9EoyZM/QnC0dVcckA/qY0RbJaSbLVr0X2SmCDbaBgGRyjVWDzYYPCZYzrtGCAqqijXXGFu/yF1+wZtypI9KUZbimOSQwYBcVJ58sRqLyzqVZAAAAFQCxwPmNASHLDNCU9yftxb/yxOywdwAAAIEA3xJl2vL8UPvy2K55vYnqq0f3ATbPsmRzIggQTOSXaYSB4s25Bbr3TnIrUinf8hWgMx62x3tUaadqEe1ZV0hGWr3sVz1v85OnoyT4mPUNt6znyE/HfgusGBxOXrmxSEUUQoI2N8SoX6suLVE8r4rI+7SvrKLb1nIlE5ZE4EVjjwYAAACALxvBWxnDC49uLE12hC05JFyFl1GMUXsF+SZz5UD/g7h5YDl1VTQnj1dUll8ctbnveQFU8tgoroJerpkK9mlWZ3hWqdHZ39v1lknhUfSsgdX03kZmbU6JPGI3q3kkt5d6EbP/Pty71lHgT7nZuIq2e6Rzy7zA+7nUbYXarGlhRwM=
root@beocat.cis.ksu.edu

ecdsa-sha2-nistp256
AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/sgGK9t6p+qlD2msubeRgdfV27LaH9RBwGjNehWJzKEC/2TG7acjnw844txkpmJp2aVBRXXM4jwI0XMsq3tI4=
root@beocat.cis.ksu.edu

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGQtf9WC7om+SNZe6Zoh5J7qEYY44ci9jOC663P6NUdO
root@beocat.cis.ksu.edu

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnrnLkdAQ+j0YS/cy2DU7magFr5ldB98hs9AdHxTHHI7ynl1Bf9hZrk4J6cDft7OLmNZ215A2pOASFdj40kaOFsalHjKvjlIWUeZ4tqCHTigl0pHJ/Ysr7teF4v8D8xyQqt3SaqxCtYwsWUKjoIo8UAbS3LMzB9RNjsC8c6iVggpl04M+Q9zwqTqS+ApnQGwlA3JhZNiipnVwa/eI49jfouub2JaZNgUBimTW0yaU8f2dshUrDcMTh6BWgxqboaYk7WbyzF0o4fWTTFG8iGQo15B7sqcibBl1IePb4nfQWMPRubxzy2sg9o/+/h9bEubjQoR4luCPW1k85F/pxJevV
root@beocat.cis.ksu.edu
</blockquote>

That was then followed up by this message:
<blockquote>
Sorry for the bother, again, but this has come up a few times. To remove the existing ssh identity, with openssh (Linux and OSX), you should run the following (and then try to login to Beocat again):

<tt>ssh-keygen -R beocat.cis.ksu.edu</tt>

Putty and some of other applications have different methods. Ideally, you will be asked if you want to accept the new public key. If you are, you will probably want to accept.
</blockquote>

== What happens to my data when I leave K-State? ==
First of all, although we use eid credentials, we are not tied in with K-State's central IT policies which apply to employees or students leaving the university. As long as you keep your eid password current, you still have access to Beocat. Once we deem your data to be "stale", we will archive your data and disable your account. We have no written policy on when we do this, because we only do so as necessity dictates, but generally speaking if you have any data which is modified for less than two years will not be marked as stale. If your account is disabled for this reason, you will have to apply for a new account and un-archive your data.

== Common Storage For Projects ==
Sometimes it is useful for groups of people to have a common storage area. If you already have a project you can do the following:

* Create a directory in one of the home directories of someone in your group, ideally the project owner's.
** <tt>mkdir $directory</tt>
* Change the group to the name assigned by the Beocat admins
** <tt>chgrp -R $group_name $directory</tt>
* Set the directory writeable and sticky for the group
** <tt>chmod g+ws $directory</tt>

* Change your umask to 002 (there will probably be a setting for it in your file transfer utilities, also). This step needs to be done by all group members.
** <tt>umask 002</tt> needs to go above <tt><nowiki>if [[ $- != *i* ]] ; then</nowiki></tt> in your <tt>~/.bashrc</tt> file

* Finally logout and log back in

== How do I get more help? ==
There are many sources of help for most Linux systems.

=== Unix man pages ===
Linux provides man pages (short for manual pages). These are simple enough to call, for example: if you need information on submitting jobs to Beocat, you can type '<code>man qsub</code>'. This will bring up the manual for qsub.

=== GNU info system ===
Not all applications have "man pages." Most of the rest have what they call info pages. For example, if you needed information on finding a file you could use '<code>info find</code>'.

=== This documentation ===
This documentation is very thoroughly researched, and has been painstakingly assembled for your benefit. Please use it.

=== Contact support ===
Support can be contacted [mailto:beocat@cis.ksu.edu here]. Please include detailed information about your problem, including the job number, applications you are trying to run, and the current directory that you are in.

FAQ

2015-10-02T20:55:56Z

Kylehutson: Added SSH hostkey FAQ entry since we've been asked several times.

== How do I connect to Beocat ==
{| class="wikitable"
|-
! colspan="2" | Connection Settings
|-
! Hostname
| style="text-align:right" | beocat.cis.ksu.edu
|-
! Port
| style="text-align:right" | 22
|-
! Username
| style="text-align:right" | <tt>eID</tt>
|-
! Password
| style="text-align:right" | <tt>eID Password</tt>
|}

{| class="wikitable"
!colspan="2" | Supported Connection Software (Latest Versions of Each)
|-
!rowspan="3" | Shell
|-
| [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]
|-
| ssh from openssh
|-
!rowspan="4" | File Transfer Utilities
|-
| [https://filezilla-project.org/ Filezilla]
|-
| [http://winscp.net/ WinSCP]
|-
| scp and sftp from openssh
|-
!rowspan="2" | Combination
|-
| [http://mobaxterm.mobatek.net/ MobaXterm]
|}

== How do I compile my programs? ==
=== Serial programs ===
==== Fortran ====
<tt>ifort</tt> or <tt>gfortran</tt>
==== C/C++ ====
<tt>icc</tt>, <tt>gcc</tt> and <tt>g++</tt>
=== Parallel programs ===
==== Fortran ====
<tt>mpif77</tt> or <tt>mpif90</tt>
==== C/C++ ====
<tt>mpicc</tt> or <tt>mpic++</tt>
== How are the filesystems on Beocat set up? ==

{| class="wikitable"
|-
! Mountpoint !! Local / Shared !! Size !! Filesystem !! Advice
|-
| /home || Shared || 350TB total || glusterfs on top of xfs || Good enough for most jobs
|-
| /tmp || Local || >100GB (varies per node) || ext4 || Good for I/O intensive jobs
|-
|}
=== Usage Advice ===
For most jobs you shouldn't need to worry, your default working directory
is your homedir and it will be fast enough for most tasks.
I/O intensive work should use /tmp, but you will need to remember to copy
your files to and from this partition as part of your job script. This is made
easier through the <tt>$TMPDIR</tt> environment variable in your jobs.

Example usage of <tt>$TMPDIR</tt> in a job script
<syntaxhighlight lang="bash" line>
#!/bin/bash

#copy our input file to $TMPDIR to make processing faster
cp ~/experiments/input.data $TMPDIR

#use the input file we copied over to the local system
#generate the output file in $TMPDIR as well
~/bin/my_program --input-file=$TMPDIR/input.data --output-file=$TMPDIR/output.data

#copy the results back from $TMPDIR
cp $TMPDIR/output.data ~/experiments/results.$SGE_JOBID
</syntaxhighlight>

You need to remember to copy over your data from <tt>$TMPDIR</tt> as part of your job.
That directory and its contents are deleted when the job is complete.

== What is "killable" or "nokillable" ==
On Beocat, some of the machines have been purchased by specific users and/or groups. These users and/or groups get guaranteed access to their machines at any point in time. Often, these machines are sitting idle because the owners have no need for it at the time. This would be a significant waste of computational power if there were no other way to make use of the computing cycles.

=== Enter the "killable" resource ===
Killable (-l killable) jobs are jobs that can be scheduled to these "owned" machines by users outside of the true group of owners. If a "killable" job starts on one of these owned machines and the owner of said machine comes along and submits a job, the "killable" job will be returned to the queue, (killed off as it were), and restarted at some future point in time. The job will still complete at some future point, and if the job makes use of a checkpointing algorithm it may complete even faster. The trade off between marking a job "killable" and not, is that sometimes applications need a significant amount of runtime, and cannot resume running from a partial output, meaning that it may get restarted over and over again, never reaching the finish line. As such, we only auto-enable "killable" for relatively short jobs (<=12:00:00). Some users still feel this is a hinderance, so we created another flag to tell us not to automatically mark short jobs "killable"

=== Enter the "nokillable" resource ===
Nokillable (-l nokillable) simply makes it so that the jsv will not automatically mark the job killable. If you mark both killable and nokillable, killable will win.

=== The trade-off ===
If a job is marked killable, there are a non-trivial amount of additional nodes that the job can run on. If your job checkpoints itself, or is relatively short, there should be no downside to marking the job killable, as the job will probably start sooner. If your job is long-running and doesn't checkpoint (save its state to restart a previous session) itself, it could cause your job to take longer to complete.

== Help! When I submit my jobs I get "Warning To stay compliant with standard unix behavior, there should be a valid #! line in your script i.e. #!/bin/tcsh" ==
Job submission scripts are supposed to have a line similar to '<code>#!/bin/bash</code>' in them to start. We have had problems with people submitting jobs with invalid #! lines, so we enforce that rule. When this happens the job fails and we have to manually clean it up. The warning message is there just to inform you that the job script should have a line in it, in most cases #!/bin/tcsh or #!/bin/bash, to indicate what program should be used to run the script. When the line is missing from a script, by default your default shell is used to execute the script (in your case /usr/local/bin/tcsh). This works in most cases, but may not be what you are wanting.

== Help! When I submit my jobs I get "A #! line exists, but it is not pointing to an executable. Please fix. Job not submitted." ==
Like the above, error says you need a #!/bin/bash or similar line in your job script. This error says that while the line exists, the #! line isn't mentioning an executable file, thus the script will not be able to run. Most likely you wanted #!/bin/bash instead of something else.

== Help! My jobs keep dying after 1 hour and I don't know why ==
Beocat has default runtime limit of 1 hour. If you need more than that, or need more than 1 GB of memory per core, you'll want to look at the documentation [[SGEBasics|here]] to see how to request it.

In short, when you run qsub for your job, you'll want to put something along the lines of '<code>-l h_rt=10:00:00</code>' before the job script if you want your job to run for 10 hours.

== Help my error file has "Warning: no access to tty" ==
The warning message "Warning: no access to tty (Bad file descriptor)" is safe to ignore. It typically happens with the tcsh shell.

== Help! My job isn't going to finish in the time I specified. Can I change the time requirement? ==
Generally speaking, no.

Jobs are scheduled based on execution times (among other things). If it were easy to change your time requirement, one could submit a job with a 15-minute run-time, get it scheduled quickly, and then say "whoops - I meant 15 weeks", effectively gaming the job scheduler. In fact, even the administrators cannot change the run-time requirement of a particular job. In extreme circumstances and depending on the job requirements, we '''may''' be able to manually intervene. This process prevents other users from using the node(s) you are currently using, so are not routinely approved. Contact Beocat support (below) if you feel your circumstances warrant special consideration.

== Help! My perl job runs fine on the head node, but only runs for a few seconds and then quits when submitted to the queue. ==
Perl doesn't like getting called straight from the scheduler. However, there is a fairly easy workaround. Create a shell wrapper script that calls perl and its program.

For instance, I can create a script called runperl.sh that looks like this:

#!/bin/sh
#$ -l h_rt=1:00:00,mem=1G
/usr/bin/perl /path/to/my/perl_program.pl

Make this wrapper program executable:
chmod 755 runperl.sh

Then submit it with
qsub runperl.sh

Of course, the name of this script isn't important, as long as you change the corresponding chmod and qsub commands.

== Help! When using mpi I get 'CMA: no RDMA devices found' or 'A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces' ==
This message simply means that some but not all nodes the job is running on have infiniband cards. The job will still run, but will not use the fastest interconnect we have available. This may or may not be an issue, depending on how message heavy your job is. If you would like to not see this warning, you may request infiniband as a resource when submitting your job. <code>-l infinband=TRUE</code>

== When I try to connect to Beocat, I get a message that includes "Host key verification failed." ==
There are two ways you might get this.

1) Somebody is trying to intercept your traffic and is impersonating Beocat (this is unlikely, but not impossible)

2) You haven't logged into Beocat since we updated our host keys in May of 2015

The following message was sent to the CIS-BEOCAT listserv in May.
<blockquote>
Hello all,

Another quick note, as you might run into this now, I've created new ssh private/public keys for the headnodes. This means that you will likely see warnings from your ssh/scp/sftp clients that the public key has changed (starting now). This is a good thing, but you will likely have to tell your client in some way that the new key is to be accepted. Here are the public keys, if you would like to verify any of them.

ssh-dss AAAAB3NzaC1kc3MAAACBAOOneUDA3MuU4O9WWGtxCwzjkWn1vx0+b2BZmJyxc99Rpb6mP2Cd3CxvUK5Az+ZP9EoyZM/QnC0dVcckA/qY0RbJaSbLVr0X2SmCDbaBgGRyjVWDzYYPCZYzrtGCAqqijXXGFu/yF1+wZtypI9KUZbimOSQwYBcVJ58sRqLyzqVZAAAAFQCxwPmNASHLDNCU9yftxb/yxOywdwAAAIEA3xJl2vL8UPvy2K55vYnqq0f3ATbPsmRzIggQTOSXaYSB4s25Bbr3TnIrUinf8hWgMx62x3tUaadqEe1ZV0hGWr3sVz1v85OnoyT4mPUNt6znyE/HfgusGBxOXrmxSEUUQoI2N8SoX6suLVE8r4rI+7SvrKLb1nIlE5ZE4EVjjwYAAACALxvBWxnDC49uLE12hC05JFyFl1GMUXsF+SZz5UD/g7h5YDl1VTQnj1dUll8ctbnveQFU8tgoroJerpkK9mlWZ3hWqdHZ39v1lknhUfSsgdX03kZmbU6JPGI3q3kkt5d6EbP/Pty71lHgT7nZuIq2e6Rzy7zA+7nUbYXarGlhRwM=
root@beocat.cis.ksu.edu

ecdsa-sha2-nistp256
AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/sgGK9t6p+qlD2msubeRgdfV27LaH9RBwGjNehWJzKEC/2TG7acjnw844txkpmJp2aVBRXXM4jwI0XMsq3tI4=
root@beocat.cis.ksu.edu

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGQtf9WC7om+SNZe6Zoh5J7qEYY44ci9jOC663P6NUdO
root@beocat.cis.ksu.edu

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnrnLkdAQ+j0YS/cy2DU7magFr5ldB98hs9AdHxTHHI7ynl1Bf9hZrk4J6cDft7OLmNZ215A2pOASFdj40kaOFsalHjKvjlIWUeZ4tqCHTigl0pHJ/Ysr7teF4v8D8xyQqt3SaqxCtYwsWUKjoIo8UAbS3LMzB9RNjsC8c6iVggpl04M+Q9zwqTqS+ApnQGwlA3JhZNiipnVwa/eI49jfouub2JaZNgUBimTW0yaU8f2dshUrDcMTh6BWgxqboaYk7WbyzF0o4fWTTFG8iGQo15B7sqcibBl1IePb4nfQWMPRubxzy2sg9o/+/h9bEubjQoR4luCPW1k85F/pxJevV
root@beocat.cis.ksu.edu
</blockquote>

That was then followed up by this message:
<blockquote>
Sorry for the bother, again, but this has come up a few times. To remove the existing ssh identity, with openssh (Linux and OSX), you should run the following (and then try to login to Beocat again):

<tt>ssh-keygen -R beocat.cis.ksu.edu</tt>

Putty and some of other applications have different methods. Ideally, you will be asked if you want to accept the new public key. If you are, you will probably want to accept.
</blockquote>

== Common Storage For Projects ==
Sometimes it is useful for groups of people to have a common storage area. If you already have a project you can do the following:

* Create a directory in one of the home directories of someone in your group, ideally the project owner's.
** <tt>mkdir $directory</tt>
* Change the group to the name assigned by the Beocat admins
** <tt>chgrp -R $group_name $directory</tt>
* Set the directory writeable and sticky for the group
** <tt>chmod g+ws $directory</tt>

* Change your umask to 002 (there will probably be a setting for it in your file transfer utilities, also). This step needs to be done by all group members.
** <tt>umask 002</tt> needs to go above <tt><nowiki>if [[ $- != *i* ]] ; then</nowiki></tt> in your <tt>~/.bashrc</tt> file

* Finally logout and log back in

== How do I get more help? ==
There are many sources of help for most Linux systems.

=== Unix man pages ===
Linux provides man pages (short for manual pages). These are simple enough to call, for example: if you need information on submitting jobs to Beocat, you can type '<code>man qsub</code>'. This will bring up the manual for qsub.

=== GNU info system ===
Not all applications have "man pages." Most of the rest have what they call info pages. For example, if you needed information on finding a file you could use '<code>info find</code>'.

=== This documentation ===
This documentation is very thoroughly researched, and has been painstakingly assembled for your benefit. Please use it.

=== Contact support ===
Support can be contacted [mailto:beocat@cis.ksu.edu here]. Please include detailed information about your problem, including the job number, applications you are trying to run, and the current directory that you are in.

Compute Nodes

2015-06-24T16:21:05Z

Kylehutson: Deleted Paladins since they were removed from service.

We currently have three classes of compute nodes. Starting with the oldest first we have

== Mages ==
[1,3,5,7,9,11] - Why are these numbered like this? There are actually 12 physical machines, however each pair (1 and 2, 3 and 4, etc.) is tied together with external [http://en.wikipedia.org/wiki/Intel_QuickPath_Interconnect QPI], making them appear as a single node.

{| class="wikitable"
|Processors
|8x 10-Core Xeon E7-8870
|-
|Ram
|1024GB
|-
|Hard Drive
|2x 300GB Hitachi 10,000rpm SAS
|-
|NIC 0
|Broadcom NetXtreme II BCM5709
|-
|NIC 1
|Broadcom NetXtreme II BCM5709
|-
|NIC 2
|Broadcom NetXtreme II BCM5709
|-
|NIC 3
|Broadcom NetXtreme II BCM5709
|-
| 10GbE and QDR Infiniband
|Mellanox Technologies MT27500 Family [ConnectX-3]
|}

== Elves ==
[1-42]

{| class="wikitable"
|Processors
|2x 8-Core Xeon E5-2690
|-
|Ram
|64GB
|-
|Hard Drive
|1x 250GB 7,200 RPM SATA
|-
|NICs
|4x Intel I350
|-
| 10GbE and QDR Infiniband
|Mellanox Technologies MT27500 Family [ConnectX-3]
|}

[43-56]
{| class="wikitable"
|Processors
|2x 8-Core Xeon E5-2690
|-
|Ram
|64GB
|-
|Hard Drive
|1x 250GB 7,200 RPM SATA
|-
|NICs
|4x Intel I350
|-
|10GbE and QDR Infiniband
|Mellanox Technologies MT27500 Family [ConnectX-3]
|}

[57-72]
{| class="wikitable"
|Processors
|2x 10-Core Xeon E5-2690 v2
|-
|Ram
|96GB
|-
|Hard Drive
|1x 250GB 7,200 RPM SATA
|-
|NICs
|4x Intel I350
|-
|10GbE and QDR Infiniband
|Mellanox Technologies MT27500 Family [ConnectX-3]
|}

[73-76]
{| class="wikitable"
|Processors
|2x 10-Core Xeon E5-2690 v2
|-
|Ram
|384GB
|-
|Hard Drive
|1x 250GB 7,200 RPM SATA
|-
|NICs
|4x Intel I350
|-
|10GbE and QDR Infiniband
|Mellanox Technologies MT27500 Family [ConnectX-3]
|}

== Heroes ==
[1-36]
{| class="wikitable"
| Processors
| 2x 12-Core Xeon E5-2680 v3
|-
| Ram
| 128GB
|-
| Hard Drive
|1x 1TB 7,200 RPM SATA
|-
|NICs
|2x Intel I350
|-
|40GbE
| Mellanox Technologies MT27500 Family [ConnectX-3]
|}

[37-44]
{| class="wikitable"
| Processors
| 2x 12-Core Xeon E5-2680 v3
|-
| Ram
| 512GB
|-
| Hard Drive
|1x 1TB 7,200 RPM SATA
|-
|NICs
|2x Intel I350
|-
|40GbE
| Mellanox Technologies MT27500 Family [ConnectX-3]
|-
| Additional Notes
| 2x Xeon Phi FPU
|}
[[Category:Information]]
[[Category:Hardware]]

Main Page

2014-07-17T20:27:53Z

Kylehutson: Added link to SiPE.

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|HPC]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the [http://www.cis.ksu.edu/ Computing and Information Science] department. Beocat is available to any educational researcher in the state of Kansas without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of Linux servers coordinated by the [https://arc.liv.ac.uk/trac/SGE SGE] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.cis.ksu.edu/ http://ganglia.beocat.cis.ksu.edu/].
* A comparatively small [[Hadoop]] cluster
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.cis.ksu.edu/ https://account.beocat.cis.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''beocat#cis-ksu-edu''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to beocat.cis.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use SGE for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SGEBasics]] page for an introduction on how to submit your first job. If you are already familiar with SGE, we also have an [[AdvancedSGE]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cis.ksu.edu beocat@cis.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc).

We are also available on IRC on the [http://freenode.net/using_the_network.shtml freenode chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''m0zes''' and/or '''kylehutson''' in your message, and it should grab our attention. Available from a web browser [[Special:WebChat|here.]]

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat.

== Policies ==
You can find our policies [[Policy|here]]

Tips and Tricks

2014-07-09T21:44:48Z

Kylehutson: Added "Programming for Performance" section

Beocat has a number of tools to make your work easier, some which you may not know about. This is a simple list of these programs and some basic usage scenarios.

== Submitting your job to run the fastest ==
=== Size your jobs to use the fastest nodes ===
==== Specify the proper number of cores ====
Beocat (nor any other computer or cluster) can make your job run on more than one core at a time if your program isn't designed to take advantage of this. Many people think "I can run this on 40 cores and it will run 40 times faster". This isn't true.

While we have many programs that are designed to take advantage of multiple cores, do not assume this is the case

==== Optimize your jobs for speed, not for number of cores ====
It seems that many people pick an arbitrary large number of cores for their jobs. 20 seems to be a common one. However, some of our fastest nodes have 16 cores. It's quite likely if your job will fit on an Elf (16 cores, 8 GB/RAM/core (64 GB RAM total)), it will run faster with 16 cores than by specifying more cores and having it run on slower nodes.

==== Don't request resources you don't need ====
The most common culprit here is people specifying they need infiniband when the job is run on a single node. This limits the scheduling such that a perfectly good node for your job may be idle while your job is still waiting.

== Programs that make using Beocat easier ==
=== [[wikipedia:nmon|nmon]] ===
The name is short for "Nigel's Monitor", it's a program written by Nigel Griffiths from IBM.
=== [http://www.ibm.com/developerworks/aix/library/au-nmon_analyser/ nmon analyser] ===
A tool for producing graphs and spreadsheets from output generated by nmon.
=== [http://hisham.hm/htop/ htop] ===
A prettier, easier to use top. Shows CPU and memory usage in an easy-to-digest format.
=== [http://www.gnu.org/software/screen/ screen] ===
A sort of terminal multiplexer, allows you to run many terminal programs at once without mixing them up. Also allows you to disconnect and reconnect sessions. There is a good explanation of how to use screen at [http://www.mattcutts.com/blog/a-quick-tutorial-on-screen/ http://www.mattcutts.com/blog/a-quick-tutorial-on-screen/].
=== Ganglia ===
The web-based load monitoring tool for the cluster. [http://ganglia.beocat.cis.ksu.edu http://ganglia.beocat.cis.ksu.edu] . From there, you can see how busy Beocat is.
=== [http://dag.wieers.com/home-made/dstat/ dstat] ===
A very detailed performance analyzer.

== Increasing file write performance ==
Credit for this goes to [http://moo.nac.uci.edu/~hjm/bduc/BDUC_USER_HOWTO.html#writeperfongl http://moo.nac.uci.edu/~hjm/bduc/BDUC_USER_HOWTO.html#writeperfongl]

=== Use gzip ===
If you have written your own code or are using an app that writes zillions of tiny chunks of data to STDOUT, and you are storing the results on Beocat, you should consider passing the output thru gzip to consolidate the writes into a continuous stream. If you don’t do this, each write will be considered a separate IO event and the write performance will suffer.

If, however, the STDOUT is passed thru gzip, the wallclock runtime decreases even below the usual runtime and you end up with an output file that it already compressed to about 1/5 the usual size.

The here’s how to do it:

someapp --opt1 --opt2 --input=/path/to/input_file | gzip > /path/to/output_file
=== Use named pipes ===
Named pipes are special files that don't actually write to the filesystem, and can be used to communicate between processess. Since these pipes are in memory rather than directly to disk, they can be used to buffer writes:

<syntaxhighlight lang="bash">
# Create the named pipe
mkfifo /path/to/MyNamedPipe

# Write some data to it
MyProgram --infile=/path/to/InputData1 --outfile=/path/to/MyNamedPipe &
MyOtherProgram < /path/to/InputData2 > /path/to/MyNamedPipe

# Extract the output
cat < /path/to/MyNamedPipe > $HOME/MyOutput
## OR, we could compress the output
gzip < /path/to/MyNamedPipe > $HOME/MyOutput.gz

# Delete the named pipe like you would a file
rm /path/to/MyNamedPipe
</syntaxhighlight>
One cautionary word. Unlike normal files, named pipes cannot be used between machines, but can be used among processes running on the same machine. So, if you're running an MPI job that will be running completely on one node, you could setup a named pipe and do all your writes to that pipe, and then flush it at the end, but if you're running a multi-node MPI job and your named pipe is on a shared filesystem (like $HOME), each process will need to flush its named pipe to a regular file before the job quits.
=== Use one big file instead of many small ones ===
This may seem to be a non-issue, but it's a performance problem we've seen on Beocat many times. I love the term coined by UCI at the link above. They call making many small files "Zillions Of Tiny files (ZOTfiles)". Using files like this is an inefficient use of our shared resources. A tiny file by itself is no more inefficient than a huge one. If you have only 100bytes to store, store it in single file. However, the problems start compounding when there are many of them. Because of the way data is stored on disk, 10 MB stored in ZOTfiles of 100bytes each can easily take up NOT 10MB, but more than 400MB - 40 times more space. Worse, data stored in this manner makes many operations very slow - instead of looking up 1 directory entry, the OS has to look up 100,000. This means 100,000 times more disk head movement, with a concommittent decrease in performance and disk lifetime. We have had Beocat users with several million files of less than 1kB each. Just creating a directory listing with ls would take nearly 1/2 hour. Not only is that inefficient for you, but it also degrades the performance of everybody using that filesystem and degrades our backups as well.

Please use large files instead of ZOTfiles any chance you can!
== Programming for Performance ==
=== BLAS ===
BLAS (Basic Linear Algebra Subroutines) is a standard set of linear algebra subroutines. The standard was set so that software could be written against a standardized library interface, and optimized libraries could be "plug-and-play." There are lots of implementations of the BLAS libraries, with the most common ones being [http://software.intel.com/en-us/intel-mkl/ Intel's MKL] and [http://developer.amd.com/tools/cpu/acml/pages/default.aspx AMD's ACML]. We have AMD's ACML installed and used by default, as it is free software and doesn't require a paid license. Intel's MKL requires per-user licensing, and thus we haven't paid for it.

==== Beocat BLAS Libraries ====
Since BLAS is a modular standard, we have installed a few (free) BLAS libraries.

* The BLAS reference library: An unoptimized reference library
* AMD's ACML: Optimized BLAS library for AMD systems
* OpenBLAS: Optimized BLAS library for some AMD, and most Intel sytems
The default BLAS library is ACML.

==== Using a different BLAS library ====
If you want or need to use a different BLAS library, list the available libraries with 'eselect blas list'

$ eselect blas list
Available providers for blas:
[1] acml64-ifort *
[2] acml64-ifort-fma4
[3] acml64-ifort-fma4-openmp
[4] acml64-ifort-openmp
[5] openblas-openmp
[6] reference
To change your default BLAS version you need to determine which shell you are using:

===== CSH or TCSH =====
Run the following:
<syntaxhighlight lang="bash">
eselect blas script --csh openblas-openmp
</syntaxhighlight>
Where the openblas-openmp is replaced with name of your preferred BLAS. Put the output of that script in you job script, or in your ~/.cshrc file.

===== SH, BASH, or ZSH =====
Run the following:
<syntaxhighlight lang="bash">
eselect blas script --sh openblas-openmp
</syntaxhighlight>
Where the openblas-openmp is replaced with name of your preferred BLAS. Put the output of that script in your job script, or in your ~/.bashrc or ~/.zshrc file.
=== LAPACK ===
LAPACK (Linear Algebra PACKage) is a standard set of linear algebra subroutines. Like BLAS, these are very optimized, but LAPACK handles a different set of functions. The standard was set so that software could be written against a standardized library interface, and optimized libraries could be "plug-and-play." There are lots of implementations of the LAPACK libraries, with the most common ones being [http://software.intel.com/en-us/intel-mkl/ Intel's MKL] and [http://developer.amd.com/tools/cpu/acml/pages/default.aspx AMD's ACML]. We have AMD's ACML installed and used by default, as it is free software and doesn't require a paid license. Intel's MKL requires per-user licensing, and thus we haven't paid for it.

==== Beocat LAPACK Libraries ====
Since LAPACK is a modular standard, we have installed a few (free) LAPACK libraries.
* [[http://www.netlib.org/lapack/|The LAPACK reference library]]: An unoptimized reference library
* [[http://developer.amd.com/tools/cpu/acml/pages/default.aspx|AMD's ACML]]: Optimized LAPACK library for AMD systems

The default LAPACK library is ACML.

==== Using a different LAPACK library ====
If you want or need to use a different LAPACK library, list the available libraries with 'eselect lapack list'.
<syntaxhighlight lang="bash">
$ eselect lapack list
Available providers for lapack:
[1] acml64-ifort *
[2] acml64-ifort-fma4
[3] acml64-ifort-fma4-openmp
[4] acml64-ifort-openmp
[5] reference
</syntaxhighlight>
To change your default LAPACK version you need to determine which shell you are using:

===== CSH or TCSH =====
Run the following:
<syntaxhighlight lang="bash">
eselect lapack script --csh acml-ifort-openmp
</syntaxhighlight>
Where the acml-ifort-openmp is replaced with name of your preferred LAPACK.
Put the output of that script in you job script, or in your ~/.cshrc file.

===== SH, BASH, or ZSH =====
Run the following:
<syntaxhighlight lang="bash">
eselect lapack script --sh acml-ifort-openmp
</syntaxhighlight>
Where the acml-ifort-openmp is replaced with name of your preferred LAPACK.
Put the output of that script in you job script, or in your ~/.bashrc or ~/.zshrc file.

=== [http://openmp.org/wp/ OpenMP] ===
OpenMP is a set of directives for C, C++, and Fortran which greatly simplifies parallelizing applications on a single node. There is a good tutorial for OpenMP at [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/]
To compile an OpenMP-enabled program, you need to tell GCC that OpenMP is available this is done like:
gcc -fopenmp myOpenMPprogram.c
By default OpenMP will use all available cores for its computation, which is a problem for shared resources like Beocat.

To make use of only the cores assigned to you, you must first make sure you have requested the 'single' parallel environment and in your job script you will need something like the following (before the application you are trying to run):

==== bash, sh, zsh ====
<syntaxhighlight lang="bash">
export OMP_NUM_THREADS=${NSLOTS}
</syntaxhighlight>

==== csh or tcsh ====
<syntaxhighlight lang="bash">
setenv OMP_NUM_THREADS ${NSLOTS}
</syntaxhighlight>

Installed software

2014-07-09T21:36:47Z

Kylehutson: Removed OpenMP to move to Tips and Tricks

== Drinking from the Firehose ==
For a complete list of all installed software, see [[NodePackageList]]

== Most Commonly Used Software ==
=== [http://www.open-mpi.org/ OpenMPI] ===
Version 1.4.3

=== [http://www.scilab.org Scilab] ===
Version 5.4.0

=== [http://www.r-project.org/ R] ===
Version 3.0.3

==== Modules ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own modules ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="rsplus">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="rsplus">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. '<tt>qsub myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSGE#Running_from_a_qsub_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.qsub

<syntaxhighlight lang="bash">
#!/bin/bash
#$ -l mem=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS)
#$ -l h_rt=0:15:00

# Now lets do some actual work. This starts R and loads the file myscript.R
R --no-save -q < myscript.R
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
qsub submit-R.qsub
</syntaxhighlight>
=== [http://www.java.com/ Java] ===
Versions 1.6 and 1.7

We support 4 versions of the Java VM on Beocat. [[wikipedia:IcedTea|IcedTea]] 6 and 7 (based on [[wikipedia:OpenJDK|OpenJDK]]), Sun JDK 1.6 (Java 6), and Oracle JDK 1.7 (Java 7).

We allow each user to select his or her Java version individually. If you do not select one, we default to Sun JDK 1.7.

==== Selecting your Java version ====
First, lets list the available versions. This can be done with the command <code>eselect java-vm list</code>
<pre>
% eselect java-vm list
Available Java Virtual Machines:
[1] icedtea-bin-6
[2] icedtea-bin-7
[3] oracle-jdk-bin-1.7 system-vm
[4] sun-jdk-1.6
</pre>
If you'll note, oracle-jdk-bin-1.7 (marked "system-vm") is the default for all users. If you have a custom version set, it will be marked with "user-vm". Now if you wanted to use icedtea-6, you could run the following:
<syntaxhighlight lang="bash">
eselect java-vm set user 1
</syntaxhighlight>
Now, we see the difference when running the above command
<pre>
% eselect java-vm list
Available Java Virtual Machines:
[1] icedtea-bin-6 user-vm
[2] icedtea-bin-7
[3] oracle-jdk-bin-1.7 system-vm
[4] sun-jdk-1.6
</pre>
To verify you are seeing the correct java, you can run <code>java -version</code>
<pre>
% java -version
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.7) (Gentoo build 1.6.0_27-b27)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
</pre>
=== [http://www.python.org/about/ Python] ===

We have several versions of Python available:
* [http://docs.python.org/2.7/ CPython 2.7]
* [http://docs.python.org/3.2/ CPython 3.2]
* [http://pypy.org/ PyPy] versions 1.9 (Python 2.7.2) and 2.0.2 (Python 2.7.3)

For the uninitiated PyPy provides [[wikipedia:Just-in-time_compilation|just-in-time compilation]] for python code. While it doesn't support all modules, code which does run under PyPy can see a significant performance increase.

If you just need python and its default modules, you can use python2 python3 pypy-c1.9 or pypy-c2.0 as you would any other application.

If, however, you need modules that we do not have installed, you should use [http://www.doughellmann.com/projects/virtualenvwrapper/ virtualenvwrapper] to setup a virtual python environment in your home directory. This will let you install python modules as you please.

==== Setting up your virtual environment ====
* [[LinuxBasics#Shells|Change your shell]] to bash
* Make sure ~/.bash_profile exists
<syntaxhighlight lang="bash">
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
</syntaxhighlight>
* Add a line like <code>source /usr/bin/virtualenvwrapper.sh</code> to your .bash_profile.
<syntaxhighlight lang="bash">
echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.bash_profile
</syntaxhighlight>
* Show your existing environments
<syntaxhighlight lang="bash">
workon
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test', a python2 virtual environment called 'testp2', a python3 virtual environment called 'testp3', and a pypy environment called testpypy. Note that <code>mkvirtualenv --help</code> has many more useful options.
<syntaxhighlight lang="bash">
mkvirtualenv -p $(which python2) testp2
mkvirtualenv -p $(which python3) testp3
mkvirtualenv -p $(which pypy-c2.0) testpypy
</syntaxhighlight>
* Lets look at our virtual environments
<pre>
%workon
testp2
testp3
testpypy
</pre>
* Activate one of these
<pre>
%workon testp2
</pre>
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>
==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment testp2
<syntaxhighlight lang="bash">
#!/bin/bash
source /usr/bin/virtualenvwrapper.sh
workon testp2
~/path/to/your/python/script.py
</syntaxhighlight>
==== A note on [http://www.numpy.org/ NumPy] ====
NumPy is a commonly-used Python package.

Make sure the following is executed before running <code>pip install numpy</code>
<syntaxhighlight lang="bash">
cp /opt/beocat/numpy/.numpy-site.cfg ~/.numpy-site.cfg
</syntaxhighlight>
==== A note on [http://mpi4py.scipy.org/docs/usrman/index.html mpi4py] ====
If you are wanting to use mpi with your python script and are using a virtual environment, you will need to send the correct environment variables to all of the mpi processes to make the virtual environment work.
<syntaxhighlight lang="bash">
#!/bin/bash
# sample mpi4py submit script
source /usr/bin/virtualenvwrapper.sh
workon testp2
# figure out the location of the python interpreter in the virtual environment
PYTHON_BINARY=$(which python)
# mpirun the python interpreter within the virtual environment
# if you don't use the interpreter within the virtual environment, i.e. just using 'python'
# the system python interpreter (without access to your other modules) will be used.
mpirun ${PYTHON_BINARY} ~/path/to/your/mpi-enabled/python/script.py
</syntaxhighlight>

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.
==== Submitting a job with Perl ====
Much like R (above), you cannot simply '<tt>qsub myProgram.pl</tt>', but you must create a [[AdvancedSGE#Running_from_a_qsub_Submit_Script|submit script]] which will call perl. Here is an example:
<syntaxhighlight lang="bash">
#!/bin/bash
#$ -l mem=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS)
#$ -l h_rt=0:15:00
# Now lets do some actual work.
perl /path/to/myProgram.pl
</syntaxhighlight>
==== Getting Perl with threads ====
* Setup perlbrew
** [[LinuxBasics#Shells|Change your shell]] to bash
** Install perlbrew
<syntaxhighlight lang="bash">
curl -L http://install.perlbrew.pl | bash
</syntaxhighlight>
** Make sure that ~/.bash_profile exists
<syntaxhighlight lang="bash">
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
</syntaxhighlight>
** Add <code>source ~/perl5/perlbrew/etc/bashrc</code> to ~/.bash_profile
<syntaxhighlight lang="bash">
echo "source ~/perl5/perlbrew/etc/bashrc" >> ~/.bash_profile
</syntaxhighlight>
** Then source your bash profile
<syntaxhighlight lang="bash">
source ~/.bash_profile
</syntaxhighlight>
* Now, install perl with threads within perlbrew
** Find the current Perl version.
<pre>
% perl -version

This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux
(with 22 registered patches, see perl -V for more detail)
(...several more lines deleted)
</pre>
** In this case the version is 5.16.3, so we run
<syntaxhighlight lang="bash">
perlbrew install -f -n -D usethreads perl-5.16.3
</syntaxhighlight>
** To temporarily use the new version of perl in the current shell, we now run
<syntaxhighlight lang="bash">
perlbrew use perl-5.16.3
</syntaxhighlight>
** To switch versions of perl for every new login or job, run
<syntaxhighlight lang="bash">
perlbrew switch perl-5.16.3
</syntaxhighlight>
** You can reverse this switch with
<syntaxhighlight lang="bash">
perlbrew switch-off
</syntaxhighlight>
== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

Main Page

2014-07-09T20:35:15Z

Kylehutson: Added "writing and installing" section

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|HPC]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the [http://www.cis.ksu.edu/ Computing and Information Science] department. Beocat is available to any educational researcher in the state of Kansas without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of Linux servers coordinated by the [https://arc.liv.ac.uk/trac/SGE SGE] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.cis.ksu.edu/ http://ganglia.beocat.cis.ksu.edu/].
* A comparatively small [[Hadoop]] cluster
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.cis.ksu.edu/ https://account.beocat.cis.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''beocat#cis-ksu-edu''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to beocat.cis.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use SGE for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SGEBasics]] page for an introduction on how to submit your first job. If you are already familiar with SGE, we also have an [[AdvancedSGE]] page where we can adjust the fine-tuning.

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed_software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips_and_Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training_Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training_Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cis.ksu.edu beocat@cis.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc).

We are also available on IRC on the [http://freenode.net/using_the_network.shtml freenode chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''m0zes''' and/or '''kylehutson''' in your message, and it should grab our attention. Available from a web browser [[Special:WebChat|here.]]

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat.

== Policies ==
You can find our policies [[Policy|here]]

Tips and Tricks

2014-07-09T20:14:17Z

Kylehutson: Initial creation / port from old support site.

Installed software

2014-07-09T16:10:16Z

Kylehutson: Updated Perl and R submit scripts

== Drinking from the Firehose ==
For a complete list of all installed software, see [[NodePackageList]]

== Most Commonly Used Software ==
=== [http://www.open-mpi.org/ OpenMPI] ===
Version 1.4.3

=== [http://openmp.org/wp/ OpenMP] ===
OpenMP isn't really a software package itself. It is a set of directives for C, C++, and Fortran which greatly simplifies parallelizing applications on a single node. There is a good tutorial for OpenMP at [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/]

=== [http://www.scilab.org Scilab] ===
Version 5.4.0

=== [http://www.r-project.org/ R] ===
Version 3.0.3

==== Modules ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own modules ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="rsplus">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="rsplus">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. '<tt>qsub myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSGE#Running_from_a_qsub_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.qsub

<syntaxhighlight lang="bash">
#!/bin/bash
#$ -l mem=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS)
#$ -l h_rt=0:15:00

# Now lets do some actual work. This starts R and loads the file myscript.R
R --no-save -q < myscript.R
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
qsub submit-R.qsub
</syntaxhighlight>
=== [http://www.java.com/ Java] ===
Versions 1.6 and 1.7

We support 4 versions of the Java VM on Beocat. [[wikipedia:IcedTea|IcedTea]] 6 and 7 (based on [[wikipedia:OpenJDK|OpenJDK]]), Sun JDK 1.6 (Java 6), and Oracle JDK 1.7 (Java 7).

We allow each user to select his or her Java version individually. If you do not select one, we default to Sun JDK 1.7.

==== Selecting your Java version ====
First, lets list the available versions. This can be done with the command <code>eselect java-vm list</code>
<pre>
% eselect java-vm list
Available Java Virtual Machines:
[1] icedtea-bin-6
[2] icedtea-bin-7
[3] oracle-jdk-bin-1.7 system-vm
[4] sun-jdk-1.6
</pre>
If you'll note, oracle-jdk-bin-1.7 (marked "system-vm") is the default for all users. If you have a custom version set, it will be marked with "user-vm". Now if you wanted to use icedtea-6, you could run the following:
<syntaxhighlight lang="bash">
eselect java-vm set user 1
</syntaxhighlight>
Now, we see the difference when running the above command
<pre>
% eselect java-vm list
Available Java Virtual Machines:
[1] icedtea-bin-6 user-vm
[2] icedtea-bin-7
[3] oracle-jdk-bin-1.7 system-vm
[4] sun-jdk-1.6
</pre>
To verify you are seeing the correct java, you can run <code>java -version</code>
<pre>
% java -version
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.7) (Gentoo build 1.6.0_27-b27)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
</pre>
=== [http://www.python.org/about/ Python] ===

We have several versions of Python available:
* [http://docs.python.org/2.7/ CPython 2.7]
* [http://docs.python.org/3.2/ CPython 3.2]
* [http://pypy.org/ PyPy] versions 1.9 (Python 2.7.2) and 2.0.2 (Python 2.7.3)

For the uninitiated PyPy provides [[wikipedia:Just-in-time_compilation|just-in-time compilation]] for python code. While it doesn't support all modules, code which does run under PyPy can see a significant performance increase.

If you just need python and its default modules, you can use python2 python3 pypy-c1.9 or pypy-c2.0 as you would any other application.

If, however, you need modules that we do not have installed, you should use [http://www.doughellmann.com/projects/virtualenvwrapper/ virtualenvwrapper] to setup a virtual python environment in your home directory. This will let you install python modules as you please.

==== Setting up your virtual environment ====
* [[LinuxBasics#Shells|Change your shell]] to bash
* Make sure ~/.bash_profile exists
<syntaxhighlight lang="bash">
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
</syntaxhighlight>
* Add a line like <code>source /usr/bin/virtualenvwrapper.sh</code> to your .bash_profile.
<syntaxhighlight lang="bash">
echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.bash_profile
</syntaxhighlight>
* Show your existing environments
<syntaxhighlight lang="bash">
workon
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test', a python2 virtual environment called 'testp2', a python3 virtual environment called 'testp3', and a pypy environment called testpypy. Note that <code>mkvirtualenv --help</code> has many more useful options.
<syntaxhighlight lang="bash">
mkvirtualenv -p $(which python2) testp2
mkvirtualenv -p $(which python3) testp3
mkvirtualenv -p $(which pypy-c2.0) testpypy
</syntaxhighlight>
* Lets look at our virtual environments
<pre>
%workon
testp2
testp3
testpypy
</pre>
* Activate one of these
<pre>
%workon testp2
</pre>
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>
==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment testp2
<syntaxhighlight lang="bash">
#!/bin/bash
source /usr/bin/virtualenvwrapper.sh
workon testp2
~/path/to/your/python/script.py
</syntaxhighlight>
==== A note on [http://www.numpy.org/ NumPy] ====
NumPy is a commonly-used Python package.

Make sure the following is executed before running <code>pip install numpy</code>
<syntaxhighlight lang="bash">
cp /opt/beocat/numpy/.numpy-site.cfg ~/.numpy-site.cfg
</syntaxhighlight>
==== A note on [http://mpi4py.scipy.org/docs/usrman/index.html mpi4py] ====
If you are wanting to use mpi with your python script and are using a virtual environment, you will need to send the correct environment variables to all of the mpi processes to make the virtual environment work.
<syntaxhighlight lang="bash">
#!/bin/bash
# sample mpi4py submit script
source /usr/bin/virtualenvwrapper.sh
workon testp2
# figure out the location of the python interpreter in the virtual environment
PYTHON_BINARY=$(which python)
# mpirun the python interpreter within the virtual environment
# if you don't use the interpreter within the virtual environment, i.e. just using 'python'
# the system python interpreter (without access to your other modules) will be used.
mpirun ${PYTHON_BINARY} ~/path/to/your/mpi-enabled/python/script.py
</syntaxhighlight>

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.
==== Submitting a job with Perl ====
Much like R (above), you cannot simply '<tt>qsub myProgram.pl</tt>', but you must create a [[AdvancedSGE#Running_from_a_qsub_Submit_Script|submit script]] which will call perl. Here is an example:
<syntaxhighlight lang="bash">
#!/bin/bash
#$ -l mem=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS)
#$ -l h_rt=0:15:00
# Now lets do some actual work.
perl /path/to/myProgram.pl
</syntaxhighlight>
==== Getting Perl with threads ====
* Setup perlbrew
** [[LinuxBasics#Shells|Change your shell]] to bash
** Install perlbrew
<syntaxhighlight lang="bash">
curl -L http://install.perlbrew.pl | bash
</syntaxhighlight>
** Make sure that ~/.bash_profile exists
<syntaxhighlight lang="bash">
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
</syntaxhighlight>
** Add <code>source ~/perl5/perlbrew/etc/bashrc</code> to ~/.bash_profile
<syntaxhighlight lang="bash">
echo "source ~/perl5/perlbrew/etc/bashrc" >> ~/.bash_profile
</syntaxhighlight>
** Then source your bash profile
<syntaxhighlight lang="bash">
source ~/.bash_profile
</syntaxhighlight>
* Now, install perl with threads within perlbrew
** Find the current Perl version.
<pre>
% perl -version

This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux
(with 22 registered patches, see perl -V for more detail)
(...several more lines deleted)
</pre>
** In this case the version is 5.16.3, so we run
<syntaxhighlight lang="bash">
perlbrew install -f -n -D usethreads perl-5.16.3
</syntaxhighlight>
** To temporarily use the new version of perl in the current shell, we now run
<syntaxhighlight lang="bash">
perlbrew use perl-5.16.3
</syntaxhighlight>
** To switch versions of perl for every new login or job, run
<syntaxhighlight lang="bash">
perlbrew switch perl-5.16.3
</syntaxhighlight>
** You can reverse this switch with
<syntaxhighlight lang="bash">
perlbrew switch-off
</syntaxhighlight>
== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

Installed software

2014-07-09T15:59:57Z

Kylehutson: Add OpenMP and fix some formatting

== Drinking from the Firehose ==
For a complete list of all installed software, see [[NodePackageList]]

== Most Commonly Used Software ==
=== [http://www.open-mpi.org/ OpenMPI] ===
Version 1.4.3

=== [http://openmp.org/wp/ OpenMP] ===
OpenMP isn't really a software package itself. It is a set of directives for C, C++, and Fortran which greatly simplifies parallelizing applications on a single node. There is a good tutorial for OpenMP at [https://computing.llnl.gov/tutorials/openMP/ https://computing.llnl.gov/tutorials/openMP/]

=== [http://www.scilab.org Scilab] ===
Version 5.4.0

=== [http://www.r-project.org/ R] ===
Version 3.0.3

==== Modules ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own modules ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="rsplus">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="rsplus">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. 'qsub myscript.R' will result in an error. Instead, you need to make a bash script that will call R appropriately. Here is a minimal example. We'll save this as submit-R.qsub

<syntaxhighlight lang="bash">
#!/bin/bash
# First, lets tell the qsub command which resources we need
# lets start with memory (in this case I ask for 1 gigabyte).
# For help on these, see [[SGEBasics]]

#$ -l mem=1G
# Now we tell qsub how long we expect our work to take: 15 minutes (H:MM:SS)

#$ -l h_rt=0:15:00

# Lets output a little useful information This will put something like "Starting the job at: Thu Jan 26 10:43:26 CST 2012" in your output file
echo -n "Starting the job at: "
date

# Now lets do some actual work. A lot of our users use R, so we'll go over that
# This starts R and loads the file myscript.R
R --no-save -q < myscript.R

# like before, this is just useful information
echo -n "Ending the job at: "
date
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
qsub submit-R.qsub
</syntaxhighlight>
=== [http://www.java.com/ Java] ===
Versions 1.6 and 1.7

We support 4 versions of the Java VM on Beocat. [[wikipedia:IcedTea|IcedTea]] 6 and 7 (based on [[wikipedia:OpenJDK|OpenJDK]]), Sun JDK 1.6 (Java 6), and Oracle JDK 1.7 (Java 7).

We allow each user to select his or her Java version individually. If you do not select one, we default to Sun JDK 1.7.

==== Selecting your Java version ====
First, lets list the available versions. This can be done with the command <code>eselect java-vm list</code>
<pre>
% eselect java-vm list
Available Java Virtual Machines:
[1] icedtea-bin-6
[2] icedtea-bin-7
[3] oracle-jdk-bin-1.7 system-vm
[4] sun-jdk-1.6
</pre>
If you'll note, oracle-jdk-bin-1.7 (marked "system-vm") is the default for all users. If you have a custom version set, it will be marked with "user-vm". Now if you wanted to use icedtea-6, you could run the following:
<syntaxhighlight lang="bash">
eselect java-vm set user 1
</syntaxhighlight>
Now, we see the difference when running the above command
<pre>
% eselect java-vm list
Available Java Virtual Machines:
[1] icedtea-bin-6 user-vm
[2] icedtea-bin-7
[3] oracle-jdk-bin-1.7 system-vm
[4] sun-jdk-1.6
</pre>
To verify you are seeing the correct java, you can run <code>java -version</code>
<pre>
% java -version
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.7) (Gentoo build 1.6.0_27-b27)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
</pre>
=== [http://www.python.org/about/ Python] ===

We have several versions of Python available:
* [http://docs.python.org/2.7/ CPython 2.7]
* [http://docs.python.org/3.2/ CPython 3.2]
* [http://pypy.org/ PyPy] versions 1.9 (Python 2.7.2) and 2.0.2 (Python 2.7.3)

For the uninitiated PyPy provides [[wikipedia:Just-in-time_compilation|just-in-time compilation]] for python code. While it doesn't support all modules, code which does run under PyPy can see a significant performance increase.

If you just need python and its default modules, you can use python2 python3 pypy-c1.9 or pypy-c2.0 as you would any other application.

If, however, you need modules that we do not have installed, you should use [http://www.doughellmann.com/projects/virtualenvwrapper/ virtualenvwrapper] to setup a virtual python environment in your home directory. This will let you install python modules as you please.

==== Setting up your virtual environment ====
* [[LinuxBasics#Shells|Change your shell]] to bash
* Make sure ~/.bash_profile exists
<syntaxhighlight lang="bash">
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
</syntaxhighlight>
* Add a line like <code>source /usr/bin/virtualenvwrapper.sh</code> to your .bash_profile.
<syntaxhighlight lang="bash">
echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.bash_profile
</syntaxhighlight>
* Show your existing environments
<syntaxhighlight lang="bash">
workon
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test', a python2 virtual environment called 'testp2', a python3 virtual environment called 'testp3', and a pypy environment called testpypy. Note that <code>mkvirtualenv --help</code> has many more useful options.
<syntaxhighlight lang="bash">
mkvirtualenv -p $(which python2) testp2
mkvirtualenv -p $(which python3) testp3
mkvirtualenv -p $(which pypy-c2.0) testpypy
</syntaxhighlight>
* Lets look at our virtual environments
<pre>
%workon
testp2
testp3
testpypy
</pre>
* Activate one of these
<pre>
%workon testp2
</pre>
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>
==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment testp2
<syntaxhighlight lang="bash">
#!/bin/bash
source /usr/bin/virtualenvwrapper.sh
workon testp2
~/path/to/your/python/script.py
</syntaxhighlight>
==== A note on [http://www.numpy.org/ NumPy] ====
NumPy is a commonly-used Python package.

Make sure the following is executed before running <code>pip install numpy</code>
<syntaxhighlight lang="bash">
cp /opt/beocat/numpy/.numpy-site.cfg ~/.numpy-site.cfg
</syntaxhighlight>
==== A note on [http://mpi4py.scipy.org/docs/usrman/index.html mpi4py] ====
If you are wanting to use mpi with your python script and are using a virtual environment, you will need to send the correct environment variables to all of the mpi processes to make the virtual environment work.
<syntaxhighlight lang="bash">
#!/bin/bash
# sample mpi4py submit script
source /usr/bin/virtualenvwrapper.sh
workon testp2
# figure out the location of the python interpreter in the virtual environment
PYTHON_BINARY=$(which python)
# mpirun the python interpreter within the virtual environment
# if you don't use the interpreter within the virtual environment, i.e. just using 'python'
# the system python interpreter (without access to your other modules) will be used.
mpirun ${PYTHON_BINARY} ~/path/to/your/mpi-enabled/python/script.py
</syntaxhighlight>

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.

==== Getting Perl with threads ====
* Setup perlbrew
** [[LinuxBasics#Shells|Change your shell]] to bash
** Install perlbrew
<syntaxhighlight lang="bash">
curl -L http://install.perlbrew.pl | bash
</syntaxhighlight>
** Make sure that ~/.bash_profile exists
<syntaxhighlight lang="bash">
if [ ! -f ~/.bash_profile ]; then cp /etc/skel/.bash_profile ~/.bash_profile; fi
</syntaxhighlight>
** Add <code>source ~/perl5/perlbrew/etc/bashrc</code> to ~/.bash_profile
<syntaxhighlight lang="bash">
echo "source ~/perl5/perlbrew/etc/bashrc" >> ~/.bash_profile
</syntaxhighlight>
** Then source your bash profile
<syntaxhighlight lang="bash">
source ~/.bash_profile
</syntaxhighlight>
* Now, install perl with threads within perlbrew
** Find the current Perl version.
<pre>
% perl -version

This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux
(with 22 registered patches, see perl -V for more detail)
(...several more lines deleted)
</pre>
** In this case the version is 5.16.3, so we run
<syntaxhighlight lang="bash">
perlbrew install -f -n -D usethreads perl-5.16.3
</syntaxhighlight>
** To temporarily use the new version of perl in the current shell, we now run
<syntaxhighlight lang="bash">
perlbrew use perl-5.16.3
</syntaxhighlight>
** To switch versions of perl for every new login or job, run
<syntaxhighlight lang="bash">
perlbrew switch perl-5.16.3
</syntaxhighlight>
** You can reverse this switch with
<syntaxhighlight lang="bash">
perlbrew switch-off
</syntaxhighlight>
== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

Main Page

2014-07-09T15:48:07Z

Kylehutson: Pointed to internal Hadoop documentation

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|HPC]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the [http://www.cis.ksu.edu/ Computing and Information Science] department. Beocat is available to any educational researcher in the state of Kansas without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of Linux servers coordinated by the [https://arc.liv.ac.uk/trac/SGE SGE] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.cis.ksu.edu/ http://ganglia.beocat.cis.ksu.edu/].
* A comparatively small [[Hadoop]] cluster
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.cis.ksu.edu/ https://account.beocat.cis.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''beocat#cis-ksu-edu''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to beocat.cis.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use SGE for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SGEBasics]] page for an introduction on how to submit your first job. If you are already familiar with SGE, we also have an [[AdvancedSGE]] page where we can adjust the fine-tuning.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]].

If your answer isn't there, you can email us at [mailto:beocat@cis.ksu.edu beocat@cis.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc).

We are also available on IRC on the [http://freenode.net/using_the_network.shtml freenode chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''m0zes''' and/or '''kylehutson''' in your message, and it should grab our attention. Available from a web browser [[Special:WebChat|here.]]

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat.

== Policies ==
You can find our policies [[Policy|here]]

Hadoop

2014-07-09T15:47:02Z

Kylehutson:

== Hadoop ==
[[http://hadoop.apache.org/ Hadoop]] is a "Big Data" distributed processing service. It is primarily used for very large data sets (greater than 1 TB).

Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus.

To use Hadoop:
* Login to Beocat
* From there login to the Hadoop headnode, named 'theia'. <tt>ssh theia</tt>
* Copy files into or out of the Hadoop filesystem. Use <tt>hadoop fs put</tt> and <tt>hadoop fs get</tt> to copy files. Note that the Hadoop filesystem is both smaller than the Beocat filesystem and is not backed up. Please copy data back out of Hadoop as soon as you are done using it. '''Data which remains untouched may be deleted with no prior notice.'''
* Run your Hadoop job. <tt>hadoop -jar path/to/file.jar</tt>

Hadoop

2014-07-09T15:44:44Z

Kylehutson: Created page with "== Hadoop == Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to acc..."

== Hadoop ==
Hadoop does not integrate well with SGE (or, for that matter, any other HPC scheduling system). So we have created our own separate Cloudera Hadoop cluster to accommodate the increased usage of Hadoop on campus.

To use Hadoop:
* Login to Beocat
* From there login to the Hadoop headnode, named 'theia'. <tt>ssh theia</tt>
* Copy files into or out of the Hadoop filesystem. Use <tt>hadoop fs put</tt> and <tt>hadoop fs get</tt> to copy files. Note that the Hadoop filesystem is both smaller than the Beocat filesystem and is not backed up. Please copy data back out of Hadoop as soon as you are done using it. '''Data which remains untouched may be deleted with no prior notice.'''
* Run your Hadoop job. <tt>hadoop -jar path/to/file.jar</tt>

OLD DEPRECATED AdvancedSGE

2014-07-09T14:43:06Z

Kylehutson: Added SGE Environment Variables

== Resource Requests ==
Aside from the time, RAM, and CPU requirements listed on the [[SGEBasics]] page, we have several other requestable resources. Generally, if you don't know if you need a particular resource, you should use the default. These can be generated with the command
<tt>qconf -sc | awk '{ if ($5 != "NO") { print }}'</tt>
{| class="wikitable sortable"
!name
!shortcut
!type
!relop
!requestable
!consumable
!default
!urgency
|-
|arch
|a
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|avx
|avx
|BOOL
|==
|YES
|NO
|FALSE
|0
|-
|calendar
|c
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|cpu
|cpu
|DOUBLE
|>=
|YES
|NO
|0
|0
|-
|cpu_flags
|c_f
|STRING
|==
|YES
|NO
|NONE
|0
|-
|cuda
|cuda
|INT
|<=
|YES
|JOB
|0
|0
|-
|display_win_gui
|dwg
|BOOL
|==
|YES
|NO
|0
|0
|-
|exclusive
|excl
|BOOL
|EXCL
|YES
|YES
|0
|1000
|-
|h_core
|h_core
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_cpu
|h_cpu
|TIME
|<=
|YES
|NO
|0:0:0
|0
|-
|h_data
|h_data
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_fsize
|h_fsize
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_rss
|h_rss
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_rt
|h_rt
|TIME
|<=
|FORCED
|NO
|0:0:0
|0
|-
|h_stack
|h_stack
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_vmem
|h_vmem
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|hostname
|h
|HOST
|==
|YES
|NO
|NONE
|0
|-
|infiniband
|ib
|BOOL
|==
|YES
|NO
|FALSE
|0
|-
|m_core
|core
|INT
|<=
|YES
|NO
|0
|0
|-
|m_socket
|socket
|INT
|<=
|YES
|NO
|0
|0
|-
|m_thread
|thread
|INT
|<=
|YES
|NO
|0
|0
|-
|m_topology
|topo
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|m_topology_inuse
|utopo
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|mem_free
|mf
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|mem_total
|mt
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|mem_used
|mu
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|memory
|mem
|MEMORY
|<=
|FORCED
|YES
|0
|0
|-
|num_proc
|p
|INT
|==
|YES
|NO
|0
|0
|-
|qname
|q
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|s_core
|s_core
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_cpu
|s_cpu
|TIME
|<=
|YES
|NO
|0:0:0
|0
|-
|s_data
|s_data
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_fsize
|s_fsize
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_rss
|s_rss
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_rt
|s_rt
|TIME
|<=
|YES
|NO
|0:0:0
|0
|-
|s_stack
|s_stack
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_vmem
|s_vmem
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|slots
|s
|INT
|<=
|YES
|YES
|1
|1000
|-
|swap_free
|sf
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|swap_rate
|sr
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|swap_rsvd
|srsv
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|swap_total
|st
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|swap_used
|su
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|virtual_free
|vf
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|virtual_total
|vt
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|virtual_used
|vu
|MEMORY
|>=
|YES
|NO
|0
|0
|}

The good news is that most of these nobody ever uses. There are a couple of exceptions, though:
=== Infiniband ===
First of all, let me state that just because it sounds "cool" doesn't mean you need it or even want it. Infiniband does absolutely no good if running in a 'single' parallel environment. Infiniband is a high-speed host-to-host communication fabric. It is used in conjunction with MPI jobs (discussed below). Several times we have had jobs which could run just fine, except that the submitter requested Infiniband, and all the nodes with Infiniband were currently busy. In fact, some of our fastest nodes do not have Infiniband, so by requesting it when you don't need it, you are actually slowing down your job. To request Infiniband, add <tt>-l ib=true</tt> to your qsub command-line.
=== CUDA ===
[[CUDA]] is the resource required for GPU computing. We have a very small number of nodes which have GPUs installed. To request one of these nodes, add <tt>-l cuda=true</tt> to your qsub command-line.
=== Exclusive ===
Some programs just don't play nicely with others. They will attempt to use all available memory or will try to use all the cores it can use. The way to be a nice neighbor if your program has this problem is to request exclusive use of a node with <tt>-l excl=true</tt>. This can also be useful for benchmarking, where you can be sure that no other jobs are interfering with yours.
== Parallel Jobs ==
There are two ways jobs can run in parallel, ''intra''node and ''inter''node. '''Note: Beocat will not automatically make a job run in parallel.''' Have I said that enough? It's a common misperception.
=== Intranode jobs ===
Intranode jobs are easier to code and can take advantage of many common libraries, such as [http://openmp.org/wp/ OpenMP], or Java's threads. Many times, your program will need to know how many cores you want it to use. Many will use all available cores if not told explicitly otherwise. This can be a problem when you are sharing resources, as Beocat does. To request multiple cores, use the qsub directive '<tt>-pe single ''n''</tt>', where ''n'' is the number of cores you wish to use. If your command can take an environment variable, you can use $nslots to tell how many cores you've been allocated.
=== Internode (MPI) jobs ===
"Talking" between nodes is trickier that talking between cores on the same node. The specification for doing so is called "[[wikipedia:Message_Passing_Interface|Message Passing Interface]]", or MPI. We have [http://www.open-mpi.org/ OpenMPI] installed on Beocat for this purpose. Most programs written to take advantage of large multi-node systems will use MPI. You can tell if you have an MPI-enabled program because its directions will tell you to run '<tt>mpirun ''program''</tt>'. Requesting MPI resources is only mildly more difficult than requesting single-node jobs. Instead of using '<tt>-pe single ''n''</tt>' for your qsub request, you will use one of the following:
{| class="wikitable sortable"
! Parallel Environment !! Description
|-
|mpi-fill
|This environment will use as many slots on each node as it can until it reaches the number of cores you have requested.
|-
|mpi-spread
|This environment will spread itself out over as many nodes as possible until it reaches the number of cores you have requested.
|-
|mpi-1
|This environment will allocate the slots you've requested 1 per node.
|-
|mpi-2
|This environment will allocate the slots you've requested 2 per node. You must request cores as a multiple of 2
|-
|mpi-4
|This environment will allocate the slots you've requested 4 per node. You must request cores as a multiple of 4
|-
|mpi-8
|This environment will allocate the slots you've requested 8 per node. You must request cores as a multiple of 8
|-
|mpi-10
|This environment will allocate the slots you've requested 10 per node. You must request cores as a multiple of 10
|-
|mpi-12
|This environment will allocate the slots you've requested 12 per node. You must request cores as a multiple of 12
|-
|mpi-16
|This environment will allocate the slots you've requested 16 per node. You must request cores as a multiple of 16
|-
|mpi-80
|This environment will allocate the slots you've requested 80 per node. You must request cores as a multiple of 80
|}
Some quick examples:

<tt>-pe mpi-4 16</tt> will give you 4 chunks of 4 cores apiece. They might all happen to be allocated on the same node (16 cores), on 4 different nodes (4 cores each), on 3 nodes (8 cores on one and 4 cores on the other two), or on 2 nodes (8 cores each).

<tt>-pe mpi-fill 40</tt> will give you 40 cores, but will attempt to get them all on the same node.

<tt>-pe mpi-fill 100</tt> will give you 100 cores, and place them on as few nodes as possible. In this case it's likely you would get a full mage (80 cores) and either part of another mage (the remaining 20 cores) or one of the 20-core elves.

<tt>-pe mpi-spread 40</tt> will give you 40 cores, and will attempt to place each on a separate node.
== Requesting memory for multi-core jobs ==
All memory requests are '''per core'''. One of the more common scenarios is where somebody will need, say 20 cores and 400 GB of memory. So they will make a request like '<tt>-pe single 20, -l mem=400G</tt>' This will never run, because what you are really requesting is 20 cores and 8000GB of memory (20 * 400). Since we have no nodes with 8000 terabytes of memory, the job will never run. In this case, you will divide the 400GB total memory request by the number of cores (20), so the correct command would be '<tt>-pe single 20, -l mem=20G</tt>'.
== Other Handy SGE Features ==
=== Email status changes ===
One of the most commonly used options when submitting jobs not related to resource requests is to have have SGE email you when a job changes its status. This takes two directives to qsub: '<tt>-M ''someone@somewhere.com''</tt>' will give the email address to which to send status updates. '<tt>-m abe</tt>' is probably the most common directive given for ''when'' to send updates. This will send email messages when a job (a)borts, (b)egins, or (e)nds. Other possibilities are (s)uspended and (n)ever.
=== Job Naming ===
If you have several jobs in the queue, running the same script with different parameters, it's handy to have a different name for each job as it shows up in the queue. This is accomplished with the '<tt>-N ''JobName''</tt>' qsub directive.
=== Combining Output Streams ===
Normally, SGE will create two files for output. One will be .e''jobnumber'' and the other .o''jobnumber''. If you want both of these to be combined into a single file, you can use the qsub directive '<tt>-j y</tt>'.
=== Running from the Current Directory ===
By default, jobs run from your home directory. Many programs incorrectly assume that you are running the script from the current directory. You can use the '<tt>-cwd</tt>' directive to change to the "current working directory" you used when submitting the job.
=== SGE Environment Variables ===
Within an actual job, sometimes you need to know specific things about the running environment to setup your scripts correctly. Here is a listing of environment variables that SGE makes available to you. Of course the value of these variables will be different based on many different factors.
<syntaxhighlight lang="bash">
HOSTNAME=titan1.beocat
SGE_TASK_STEPSIZE=undefined
SGE_INFOTEXT_MAX_COLUMN=5000
SHELL=/usr/local/bin/sh
NHOSTS=2
SGE_O_WORKDIR=/homes/mozes
TMPDIR=/tmp/105.1.batch.q
SGE_O_HOME=/homes/mozes
SGE_ARCH=lx24-amd64
SGE_CELL=default
RESTARTED=0
ARC=lx24-amd64
USER=mozes
QUEUE=batch.q
PVM_ARCH=LINUX64
SGE_TASK_ID=undefined
SGE_BINARY_PATH=/opt/sge/bin/lx24-amd64
SGE_STDERR_PATH=/homes/mozes/sge_test.sub.e105
SGE_STDOUT_PATH=/homes/mozes/sge_test.sub.o105
SGE_ACCOUNT=sge
SGE_RSH_COMMAND=builtin
JOB_SCRIPT=/opt/sge/default/spool/titan1/job_scripts/105
JOB_NAME=sge_test.sub
SGE_NOMSG=1
SGE_ROOT=/opt/sge
REQNAME=sge_test.sub
SGE_JOB_SPOOL_DIR=/opt/sge/default/spool/titan1/active_jobs/105.1
ENVIRONMENT=BATCH
PE_HOSTFILE=/opt/sge/default/spool/titan1/active_jobs/105.1/pe_hostfile
SGE_CWD_PATH=/homes/mozes
NQUEUES=2
SGE_O_LOGNAME=mozes
SGE_O_MAIL=/var/mail/mozes
TMP=/tmp/105.1.batch.q
JOB_ID=105
LOGNAME=mozes
PE=mpi-fill
SGE_TASK_FIRST=undefined
SGE_O_HOST=loki
SGE_O_SHELL=/bin/bash
SGE_CLUSTER_NAME=beocat
REQUEST=sge_test.sub
NSLOTS=32
SGE_STDIN_PATH=/dev/null
</syntaxhighlight>
Sometimes it is nice to know what hosts you have access to during a PE job. You would checkout the PE_HOSTFILE to know that. If your job has been restarted, it is nice to be able to change what happens rather than redoing all of your work. If this is the case, RESTARTED would equal 1. There are lots of useful Environment Variables there, I will leave it to you to identify the ones you want.

Some of the most commonly-used variables we see used are $NSLOTS, $HOSTNAME, and $SGE_TASK_ID (used for array jobs, discussed below).
== Running from a qsub Submit Script ==
No doubt after you've run a few jobs you get tired of typing something like 'qsub -l mem=2G,h_rt=10:00 -pe single 8 -n MyJobTitle MyScript.sh'. How are you supposed to remember all of these every time? The answer is to create a 'submit script', which outlines all of these for you. Below is a sample submit script, which you can modify and use for your own purposes.
<syntaxhighlight lang="bash">
#!/bin/bash

## A Sample qsub script created by Kyle Hutson
##
## Note: Usually a '#" at the beginning of the line is ignored. However, in
## the case of qsub, lines beginning with #$ are commands for qsub itself, so
## I have taken the convention here of starting *every* line with a '#', just
## Delete the first one if you want to use that line, and then modify it to
## your own purposes. The only exception here is the first line, which *must*
## be #!/bin/bash (or another valid shell).

## Specify the amount of RAM needed _per_core_. Default is 1G
##$ -l mem=1G

## Specify the maximum runtime. Default is 1 hour (1:00:00)
##$ -l h_rt=1:00:00

## Require the use of infiniband. If you don't know what this is, you probably
## don't need it. Default is "FALSE"
##$ ib=TRUE

## CUDA directive. If You don't know what this is, you probably don't need it
## Default is "FALSE"
##$ -l cuda=TRUE

## Parallel environment. Syntax is '-pe Environment NumberOfCores' A list of
## valid environments can be found at
## http://support.cis.ksu.edu/BeocatDocs/SunGridEngine (section 3.2). One
## quick note here. Jobs requesting 16 or fewer cores tend to get scheduled
## fairly quickly. If you need a job that requires more than that, you might
## benefit from emailing us at beocat@cis.ksu.edu to see how we can assist in
## getting your job scheduled in a reasonable amount of time. Default is
## "single 1"
##$ -pe single 12
##$ -pe mpi-1 2
##$ -pe mpi-fill 20
##$ -pe mpi-spread 16

## Checkpointing. Options are BLCR or dmtcp. Default is no checkpointing.
##$ -ckpt dmtcp

## Use the current working directory instead of your home directory
##$ -cwd

## Merge output and error text streams into a single stream
##$ -j y

## Name my job, to make it easier to find in the queue
##$ -N MyJobTitle

## And finally, we run the job we came here to do.
## $HOME/ProgramDir/ProgramName ProgramArguments

## OR, for the case of MPI-capable jobs
## mpirun $HOME/path/MpiJobName

## Send email when a job is aborted (a), begins (b), and/or ends (e)
##$ -m abe

## Email address to send the email to based on the above line.
##$ -M myemail@ksu.edu
</syntaxhighlight>
== Array Jobs ==
One of SGE's useful options is the ability to run "Array Jobs"

It can be used with the following option to qsub.

-t n[-m[:s]]
Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Grid
Engine almost like a series of jobs. The option argument to -t specifies the number of array job tasks and the index number which will be
associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SGE_TASK_ID. The option arguments
n, m and s will be available through the environment variables SGE_TASK_FIRST, SGE_TASK_LAST and SGE_TASK_STEPSIZE.

Following restrictions apply to the values n and m:

1 <= n <= 1,000,000
1 <= m <= 1,000,000
n <= m

The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size.
Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SGE_TASK_ID containing one of the 5 index numbers.

Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The
number of tasks in a array job is unlimited.

STDOUT and STDERR of array job tasks will be written into different files with the default location

<jobname>.['e'|'o']<job_id>'.'<task_id>

=== Examples ===
==== Change the Size of the Run ====
Array Jobs have a variety of uses, one of the easiest to comprehend is the following:

I have an application, app1 I need to run the exact same way, on the same data set, with only the size of the run changing.

My original script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
RUNSIZE=50
#RUNSIZE=100
#RUNSIZE=150
#RUNSIZE=200
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
For every run of that job I have to change the RUNSIZE variable, and submit each script. This gets tedious.

With Array Jobs the script can be written like so:

<syntaxhighlight lang="bash">
#!/bin/bash
#$ -t 50:200:50
RUNSIZE=$SGE_TASK_ID
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
I then submit that job, and SGE understands that it needs to run it 4 times, once for each task. It also knows that it can and should run these tasks in parallel.

==== Choosing a Dataset ====
A slightly more complex use of Array Jobs is the following:

I have an application, app2, that needs to be run against every line of my dataset. Every line changes how app2 runs slightly, but I need to compare the runs against each other.

Originally I had to take each line of my dataset and generate a new submit script and submit the job. This was done with yet another script:

<syntaxhighlight lang="bash">
#!/bin/bash
DATASET=dataset.txt
scriptnum=0
while read LINE
do
echo "app2 $LINE" > ${scriptnum}.sh
qsub ${scriptnum}.sh
scriptnum=$(( $scriptnum + 1 ))
done < $DATASET
</syntaxhighlight>
Not only is this needlessly complex, it is also slow, as qsub has to verify each job as it is submitted. This can be done easily with array jobs, as long as you know the number of lines in the dataset. This number can be obtained like so: wc -l dataset.txt in this case lets call it 5000.

<syntaxhighlight lang="bash">
#!/bin/bash
#$ -t 1:5000
app2 `sed -n "${SGE_TASK_ID}p" dataset.txt`
</syntaxhighlight>
This uses a subshell via `, and has the sed command print out only the line number $SGE_TASK_ID out of the file dataset.txt.

Not only is this a smaller script, it is also faster to submit because it is one job instead of 5000, so qsub doesn't have to verify as many.

To give you an idea about time saved: submitting 1 job takes 1-2 seconds. by extension if you are submitting 5000, that is 5,000-10,000 seconds, or 1.5-3 hours.
== Running jobs interactively ==
Some jobs just don't behave like we think they should, or need to be run with somebody sitting at the keyboard and typing in response to the output the computers are generating. Beocat has a facility for this, called 'qrsh'. qrsh uses the exact same command-line arguments as qsub. If no node is available with your resource requirements, qrsh will tell you
Your "qrsh" request could not be scheduled, try again later.
Note that, like qsub, your interactive job will timeout after your allotted time has passed.
== Job Accounting ==
Some people may find it useful to know what their job did during its run. The qacct tool will read SGE's accounting file and give you summarized or detailed views on jobs that have run within Beocat.
=== qacct ===
This data can usually be used to diagnose two very common job failures.
==== Job debugging ====
It is simplest if you know the job number of the job you are trying to get information on.
<syntaxhighlight lang="bash">
# if you know the jobid, put it here:
qacct -j 1122334455
# if you don't know the job id, you can look at your jobs over some number of days in this case the past 14 days:
qacct -o $USER -d 14 -j
</syntaxhighlight>

===== My job didn't do anything when it ran! =====
<tt>qname batch.q
hostname mage07.beocat
group some_user_users
owner some_user
project BEODEFAULT
department defaultdepartment
jobname my_job_script.sh
jobnumber 1122334455
...
snipped to save space
...
exit_status 1 </tt>
<tt style="color: red">ru_wallclock 1s</tt>
<tt>ru_utime 0.030s
ru_stime 0.030s
...
snipped to save space
...
arid undefined
category -u some_user -q batch.q,long.q -l h_rt=604800,mem_free=1024.0M,memory=2G</tt>
If you look at the line showing ru_wallclock. You can see that it shows 1s. This means that the job started and then promptly ended. This points to something being wrong with your submission script. Perhaps there is a typo somewhere in it.

===== My job ran but didn't finish! =====
<tt>qname batch.q
hostname scout59.beocat
group some_user_users
owner some_user
project BEODEFAULT
department defaultdepartment
jobname my_job_script.sh
jobnumber 1122334455
...
snipped to save space
...
slots 1 </tt>
<tt style="color: red">failed 37 : qmaster enforced h_rt, h_cpu, or h_vmem limit</tt>
<tt>exit_status 0 </tt>
<tt style="color: red">ru_wallclock 21600s</tt>
<tt>ru_utime 0.130s
ru_stime 0.020s
...
snipped to save space
...
arid undefined</tt>
<tt style="color: red">category -u some_user -q batch.q,long.q -l h_rt=21600,mem_free=512.0M,memory=1G</tt>
If you look at the lines showing failed, ru_wallclock and category we can see some pointers to the issue.
It didn't finish because the scheduler (qmaster) enforced some limit. If you look at the category line, the only limit requested was h_rt. So it was a runtime (wallclock) limit.
Comparing ru_wallclock and the h_rt request, we can see that it ran until the h_rt time was hit, and then the scheduler enforce the limit and killed the job. You will need to resubmit the job and ask for more time next time.

CUDA

2014-07-09T14:22:28Z

Kylehutson: Created page with "== CUDA Overview == CUDA is a feature set for programming nVidia GPUs. We have 16 nodes with nVidia Tesla m2050 GPUs...."

== CUDA Overview ==
[[wikipedia:CUDA|CUDA]] is a feature set for programming nVidia [[wikipedia:Graphics_processing_unit|GPUs]]. We have 16 nodes with nVidia Tesla m2050 GPUs. These GPUs have 448 cores running at 1.15 GHz, and are very fast at floating point math - over a TeraFLOP! However, programming in CUDA is difficult for the uninitiated.

== Training videos ==
CUDA Programming Model Overview: [http://www.youtube.com/watch?v=aveYOlBSe-Y http://www.youtube.com/watch?v=aveYOlBSe-Y]

CUDA Programming Basics Part I (Host functions): [http://www.youtube.com/watch?v=79VARRFwQgY http://www.youtube.com/watch?v=79VARRFwQgY]

CUDA Programming Basics Part II (Device functions): [http://www.youtube.com/watch?v=G5-iI1ogDW4 http://www.youtube.com/watch?v=G5-iI1ogDW4]

== Compiling CUDA Applications ==
nvcc is the compiler for CUDA applications. When compiling your applications manually you will need to keep 3 things in mind:

* The CUDA development headers are located here: /opt/cuda/sdk/C/common/inc
* The CUDA architecture is: sm_20
* The CUDA SDK is currently not available on the headnode. (compile on the nodes with CUDA, either in your jobscript or via <tt>qrsh -l cuda=TRUE</tt>)
* '''Do not run your cuda applications on the headnode. I cannot guarantee it will run, and it will give you terrible results if it does run.'''

Putting it all together you can compile CUDA applications as follows:

<syntaxhighlight lang="bash">
nvcc -I /opt/cuda/sdk/C/common/inc -arch sm_20 <source>.cu -o <output>
</syntaxhighlight>
== Example ==
=== Create your Application ===
Copy the following Application into Beocat as vecadd.cu
<syntaxhighlight lang="c">
// Kernel definition, see also section 4.2.3 of Nvidia Cuda Programming Guide
__global__ void vecAdd(float* A, float* B, float* C)
{
// threadIdx.x is a built-in variable provided by CUDA at runtime
int i = threadIdx.x;
A[i]=0;
B[i]=i;
C[i] = A[i] + B[i];
}

#include <stdio.h>
#define SIZE 10
int main()
{
int N=SIZE;
float A[SIZE], B[SIZE], C[SIZE];
float *devPtrA;
float *devPtrB;
float *devPtrC;
int memsize= SIZE * sizeof(float);

cudaMalloc((void**)&devPtrA, memsize);
cudaMalloc((void**)&devPtrB, memsize);
cudaMalloc((void**)&devPtrC, memsize);
cudaMemcpy(devPtrA, A, memsize, cudaMemcpyHostToDevice);
cudaMemcpy(devPtrB, B, memsize, cudaMemcpyHostToDevice);
// __global__ functions are called: Func<<< Dg, Db, Ns >>>(parameter);
vecAdd<<<1, N>>>(devPtrA, devPtrB, devPtrC);
cudaMemcpy(C, devPtrC, memsize, cudaMemcpyDeviceToHost);

for (int i=0; i<SIZE; i++)
printf("C[%d]=%f\n",i,C[i]);

cudaFree(devPtrA);
cudaFree(devPtrA);
cudaFree(devPtrA);

}
</syntaxhighlight>
=== Gain Access to a CUDA-capable Node ===
<syntaxhighlight lang="bash">
qrsh -l cuda=TRUE
</syntaxhighlight>
=== Compile Your Application ===
<syntaxhighlight lang="bash">
nvcc -I /opt/cuda/sdk/C/common/inc -arch sm_20 vecadd.cu -o vecadd
</syntaxhighlight>
This will create a program with the name 'vecadd' (specified by the '-o' flag).
=== Run Your Application ===
Run the program as you usually would, namely
<syntaxhighlight lang="bash">
./vecadd
</syntaxhighlight>

Assuming you don't want to run the program interactively because this is a large job, you can submit a job via qsub, just be sure to add the '<tt>-l cuda=true</tt>' directive.

OLD DEPRECATED AdvancedSGE

2014-07-08T21:46:48Z

Kylehutson: SGE Features and Array Jobs

== Resource Requests ==
Aside from the time, RAM, and CPU requirements listed on the [[SGEBasics]] page, we have several other requestable resources. Generally, if you don't know if you need a particular resource, you should use the default. These can be generated with the command
<tt>qconf -sc | awk '{ if ($5 != "NO") { print }}'</tt>
{| class="wikitable sortable"
!name
!shortcut
!type
!relop
!requestable
!consumable
!default
!urgency
|-
|arch
|a
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|avx
|avx
|BOOL
|==
|YES
|NO
|FALSE
|0
|-
|calendar
|c
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|cpu
|cpu
|DOUBLE
|>=
|YES
|NO
|0
|0
|-
|cpu_flags
|c_f
|STRING
|==
|YES
|NO
|NONE
|0
|-
|cuda
|cuda
|INT
|<=
|YES
|JOB
|0
|0
|-
|display_win_gui
|dwg
|BOOL
|==
|YES
|NO
|0
|0
|-
|exclusive
|excl
|BOOL
|EXCL
|YES
|YES
|0
|1000
|-
|h_core
|h_core
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_cpu
|h_cpu
|TIME
|<=
|YES
|NO
|0:0:0
|0
|-
|h_data
|h_data
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_fsize
|h_fsize
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_rss
|h_rss
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_rt
|h_rt
|TIME
|<=
|FORCED
|NO
|0:0:0
|0
|-
|h_stack
|h_stack
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|h_vmem
|h_vmem
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|hostname
|h
|HOST
|==
|YES
|NO
|NONE
|0
|-
|infiniband
|ib
|BOOL
|==
|YES
|NO
|FALSE
|0
|-
|m_core
|core
|INT
|<=
|YES
|NO
|0
|0
|-
|m_socket
|socket
|INT
|<=
|YES
|NO
|0
|0
|-
|m_thread
|thread
|INT
|<=
|YES
|NO
|0
|0
|-
|m_topology
|topo
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|m_topology_inuse
|utopo
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|mem_free
|mf
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|mem_total
|mt
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|mem_used
|mu
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|memory
|mem
|MEMORY
|<=
|FORCED
|YES
|0
|0
|-
|num_proc
|p
|INT
|==
|YES
|NO
|0
|0
|-
|qname
|q
|RESTRING
|==
|YES
|NO
|NONE
|0
|-
|s_core
|s_core
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_cpu
|s_cpu
|TIME
|<=
|YES
|NO
|0:0:0
|0
|-
|s_data
|s_data
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_fsize
|s_fsize
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_rss
|s_rss
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_rt
|s_rt
|TIME
|<=
|YES
|NO
|0:0:0
|0
|-
|s_stack
|s_stack
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|s_vmem
|s_vmem
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|slots
|s
|INT
|<=
|YES
|YES
|1
|1000
|-
|swap_free
|sf
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|swap_rate
|sr
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|swap_rsvd
|srsv
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|swap_total
|st
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|swap_used
|su
|MEMORY
|>=
|YES
|NO
|0
|0
|-
|virtual_free
|vf
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|virtual_total
|vt
|MEMORY
|<=
|YES
|NO
|0
|0
|-
|virtual_used
|vu
|MEMORY
|>=
|YES
|NO
|0
|0
|}

The good news is that most of these nobody ever uses. There are a couple of exceptions, though:
=== Infiniband ===
First of all, let me state that just because it sounds "cool" doesn't mean you need it or even want it. Infiniband does absolutely no good if running in a 'single' parallel environment. Infiniband is a high-speed host-to-host communication fabric. It is used in conjunction with MPI jobs (discussed below). Several times we have had jobs which could run just fine, except that the submitter requested Infiniband, and all the nodes with Infiniband were currently busy. In fact, some of our fastest nodes do not have Infiniband, so by requesting it when you don't need it, you are actually slowing down your job. To request Infiniband, add <tt>-l ib=true</tt> to your qsub command-line.
=== CUDA ===
[[CUDA]] is the resource required for GPU computing. We have a very small number of nodes which have GPUs installed. To request one of these nodes, add <tt>-l cuda=true</tt> to your qsub command-line.
=== Exclusive ===
Some programs just don't play nicely with others. They will attempt to use all available memory or will try to use all the cores it can use. The way to be a nice neighbor if your program has this problem is to request exclusive use of a node with <tt>-l excl=true</tt>. This can also be useful for benchmarking, where you can be sure that no other jobs are interfering with yours.
== Parallel Jobs ==
There are two ways jobs can run in parallel, ''intra''node and ''inter''node. '''Note: Beocat will not automatically make a job run in parallel.''' Have I said that enough? It's a common misperception.
=== Intranode jobs ===
Intranode jobs are easier to code and can take advantage of many common libraries, such as [http://openmp.org/wp/ OpenMP], or Java's threads. Many times, your program will need to know how many cores you want it to use. Many will use all available cores if not told explicitly otherwise. This can be a problem when you are sharing resources, as Beocat does. To request multiple cores, use the qsub directive '<tt>-pe single ''n''</tt>', where ''n'' is the number of cores you wish to use. If your command can take an environment variable, you can use $nslots to tell how many cores you've been allocated.
=== Internode (MPI) jobs ===
"Talking" between nodes is trickier that talking between cores on the same node. The specification for doing so is called "[[wikipedia:Message_Passing_Interface|Message Passing Interface]]", or MPI. We have [http://www.open-mpi.org/ OpenMPI] installed on Beocat for this purpose. Most programs written to take advantage of large multi-node systems will use MPI. You can tell if you have an MPI-enabled program because its directions will tell you to run '<tt>mpirun ''program''</tt>'. Requesting MPI resources is only mildly more difficult than requesting single-node jobs. Instead of using '<tt>-pe single ''n''</tt>' for your qsub request, you will use one of the following:
{| class="wikitable sortable"
! Parallel Environment !! Description
|-
|mpi-fill
|This environment will use as many slots on each node as it can until it reaches the number of cores you have requested.
|-
|mpi-spread
|This environment will spread itself out over as many nodes as possible until it reaches the number of cores you have requested.
|-
|mpi-1
|This environment will allocate the slots you've requested 1 per node.
|-
|mpi-2
|This environment will allocate the slots you've requested 2 per node. You must request cores as a multiple of 2
|-
|mpi-4
|This environment will allocate the slots you've requested 4 per node. You must request cores as a multiple of 4
|-
|mpi-8
|This environment will allocate the slots you've requested 8 per node. You must request cores as a multiple of 8
|-
|mpi-10
|This environment will allocate the slots you've requested 10 per node. You must request cores as a multiple of 10
|-
|mpi-12
|This environment will allocate the slots you've requested 12 per node. You must request cores as a multiple of 12
|-
|mpi-16
|This environment will allocate the slots you've requested 16 per node. You must request cores as a multiple of 16
|-
|mpi-80
|This environment will allocate the slots you've requested 80 per node. You must request cores as a multiple of 80
|}
Some quick examples:

<tt>-pe mpi-4 16</tt> will give you 4 chunks of 4 cores apiece. They might all happen to be allocated on the same node (16 cores), on 4 different nodes (4 cores each), on 3 nodes (8 cores on one and 4 cores on the other two), or on 2 nodes (8 cores each).

<tt>-pe mpi-fill 40</tt> will give you 40 cores, but will attempt to get them all on the same node.

<tt>-pe mpi-fill 100</tt> will give you 100 cores, and place them on as few nodes as possible. In this case it's likely you would get a full mage (80 cores) and either part of another mage (the remaining 20 cores) or one of the 20-core elves.

<tt>-pe mpi-spread 40</tt> will give you 40 cores, and will attempt to place each on a separate node.
== Requesting memory for multi-core jobs ==
All memory requests are '''per core'''. One of the more common scenarios is where somebody will need, say 20 cores and 400 GB of memory. So they will make a request like '<tt>-pe single 20, -l mem=400G</tt>' This will never run, because what you are really requesting is 20 cores and 8000GB of memory (20 * 400). Since we have no nodes with 8000 terabytes of memory, the job will never run. In this case, you will divide the 400GB total memory request by the number of cores (20), so the correct command would be '<tt>-pe single 20, -l mem=20G</tt>'.
== Other Handy SGE Features ==
=== Email status changes ===
One of the most commonly used options when submitting jobs not related to resource requests is to have have SGE email you when a job changes its status. This takes two directives to qsub: '<tt>-M ''someone@somewhere.com''</tt>' will give the email address to which to send status updates. '<tt>-m abe</tt>' is probably the most common directive given for ''when'' to send updates. This will send email messages when a job (a)borts, (b)egins, or (e)nds. Other possibilities are (s)uspended and (n)ever.
=== Job Naming ===
If you have several jobs in the queue, running the same script with different parameters, it's handy to have a different name for each job as it shows up in the queue. This is accomplished with the '<tt>-N ''JobName''</tt>' qsub directive.
=== Combining Output Streams ===
Normally, SGE will create two files for output. One will be .e''jobnumber'' and the other .o''jobnumber''. If you want both of these to be combined into a single file, you can use the qsub directive '<tt>-j y</tt>'.
=== Running from the Current Directory ===
By default, jobs run from your home directory. Many programs incorrectly assume that you are running the script from the current directory. You can use the '<tt>-cwd</tt>' directive to change to the "current working directory" you used when submitting the job.
== Running from a qsub Submit Script" ==
No doubt after you've run a few jobs you get tired of typing something like 'qsub -l mem=2G,h_rt=10:00 -pe single 8 -n MyJobTitle MyScript.sh'. How are you supposed to remember all of these every time? The answer is to create a 'submit script', which outlines all of these for you. Below is a sample submit script, which you can modify and use for your own purposes.
<syntaxhighlight lang="bash">
#!/bin/bash

## A Sample qsub script created by Kyle Hutson
##
## Note: Usually a '#" at the beginning of the line is ignored. However, in
## the case of qsub, lines beginning with #$ are commands for qsub itself, so
## I have taken the convention here of starting *every* line with a '#', just
## Delete the first one if you want to use that line, and then modify it to
## your own purposes. The only exception here is the first line, which *must*
## be #!/bin/bash (or another valid shell).

## Specify the amount of RAM needed _per_core_. Default is 1G
##$ -l mem=1G

## Specify the maximum runtime. Default is 1 hour (1:00:00)
##$ -l h_rt=1:00:00

## Require the use of infiniband. If you don't know what this is, you probably
## don't need it. Default is "FALSE"
##$ ib=TRUE

## CUDA directive. If You don't know what this is, you probably don't need it
## Default is "FALSE"
##$ -l cuda=TRUE

## Parallel environment. Syntax is '-pe Environment NumberOfCores' A list of
## valid environments can be found at
## http://support.cis.ksu.edu/BeocatDocs/SunGridEngine (section 3.2). One
## quick note here. Jobs requesting 16 or fewer cores tend to get scheduled
## fairly quickly. If you need a job that requires more than that, you might
## benefit from emailing us at beocat@cis.ksu.edu to see how we can assist in
## getting your job scheduled in a reasonable amount of time. Default is
## "single 1"
##$ -pe single 12
##$ -pe mpi-1 2
##$ -pe mpi-fill 20
##$ -pe mpi-spread 16

## Checkpointing. Options are BLCR or dmtcp. Default is no checkpointing.
##$ -ckpt dmtcp

## Use the current working directory instead of your home directory
##$ -cwd

## Merge output and error text streams into a single stream
##$ -j y

## Name my job, to make it easier to find in the queue
##$ -N MyJobTitle

## And finally, we run the job we came here to do.
## $HOME/ProgramDir/ProgramName ProgramArguments

## OR, for the case of MPI-capable jobs
## mpirun $HOME/path/MpiJobName

## Send email when a job is aborted (a), begins (b), and/or ends (e)
##$ -m abe

## Email address to send the email to based on the above line.
##$ -M myemail@ksu.edu
</syntaxhighlight>
== Array Jobs ==
One of SGE's useful options is the ability to run "Array Jobs"

It can be used with the following option to qsub.

-t n[-m[:s]]
Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Grid
Engine almost like a series of jobs. The option argument to -t specifies the number of array job tasks and the index number which will be
associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SGE_TASK_ID. The option arguments
n, m and s will be available through the environment variables SGE_TASK_FIRST, SGE_TASK_LAST and SGE_TASK_STEPSIZE.

Following restrictions apply to the values n and m:

1 <= n <= 1,000,000
1 <= m <= 1,000,000
n <= m

The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size.
Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SGE_TASK_ID containing one of the 5 index numbers.

Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The
number of tasks in a array job is unlimited.

STDOUT and STDERR of array job tasks will be written into different files with the default location

<jobname>.['e'|'o']<job_id>'.'<task_id>

=== Examples ===
==== Change the Size of the Run ====
Array Jobs have a variety of uses, one of the easiest to comprehend is the following:

I have an application, app1 I need to run the exact same way, on the same data set, with only the size of the run changing.

My original script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
RUNSIZE=50
#RUNSIZE=100
#RUNSIZE=150
#RUNSIZE=200
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
For every run of that job I have to change the RUNSIZE variable, and submit each script. This gets tedious.

With Array Jobs the script can be written like so:

<syntaxhighlight lang="bash">
#!/bin/bash
#$ -t 50:200:50
RUNSIZE=$SGE_TASK_ID
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
I then submit that job, and SGE understands that it needs to run it 4 times, once for each task. It also knows that it can and should run these tasks in parallel.

==== Choosing a Dataset ====
A slightly more complex use of Array Jobs is the following:

I have an application, app2, that needs to be run against every line of my dataset. Every line changes how app2 runs slightly, but I need to compare the runs against each other.

Originally I had to take each line of my dataset and generate a new submit script and submit the job. This was done with yet another script:

<syntaxhighlight lang="bash">
#!/bin/bash
DATASET=dataset.txt
scriptnum=0
while read LINE
do
echo "app2 $LINE" > ${scriptnum}.sh
qsub ${scriptnum}.sh
scriptnum=$(( $scriptnum + 1 ))
done < $DATASET
</syntaxhighlight>
Not only is this needlessly complex, it is also slow, as qsub has to verify each job as it is submitted. This can be done easily with array jobs, as long as you know the number of lines in the dataset. This number can be obtained like so: wc -l dataset.txt in this case lets call it 5000.

<syntaxhighlight lang="bash">
#!/bin/bash
#$ -t 1:5000
app2 `sed -n "${SGE_TASK_ID}p" dataset.txt`
</syntaxhighlight>
This uses a subshell via `, and has the sed command print out only the line number $SGE_TASK_ID out of the file dataset.txt.

Not only is this a smaller script, it is also faster to submit because it is one job instead of 5000, so qsub doesn't have to verify as many.

To give you an idea about time saved: submitting 1 job takes 1-2 seconds. by extension if you are submitting 5000, that is 5,000-10,000 seconds, or 1.5-3 hours.
== Running jobs interactively ==
Some jobs just don't behave like we think they should, or need to be run with somebody sitting at the keyboard and typing in response to the output the computers are generating. Beocat has a facility for this, called 'qrsh'. qrsh uses the exact same command-line arguments as qsub. If no node is available with your resource requirements, qrsh will tell you
Your "qrsh" request could not be scheduled, try again later.
Note that, like qsub, your interactive job will timeout after your allotted time has passed.
== Job Accounting ==
Some people may find it useful to know what their job did during its run. The qacct tool will read SGE's accounting file and give you summarized or detailed views on jobs that have run within Beocat.
=== qacct ===
This data can usually be used to diagnose two very common job failures.
==== Job debugging ====
It is simplest if you know the job number of the job you are trying to get information on.
<syntaxhighlight lang="bash">
# if you know the jobid, put it here:
qacct -j 1122334455
# if you don't know the job id, you can look at your jobs over some number of days in this case the past 14 days:
qacct -o $USER -d 14 -j
</syntaxhighlight>

===== My job didn't do anything when it ran! =====
<tt>qname batch.q
hostname mage07.beocat
group some_user_users
owner some_user
project BEODEFAULT
department defaultdepartment
jobname my_job_script.sh
jobnumber 1122334455
...
snipped to save space
...
exit_status 1 </tt>
<tt style="color: red">ru_wallclock 1s</tt>
<tt>ru_utime 0.030s
ru_stime 0.030s
...
snipped to save space
...
arid undefined
category -u some_user -q batch.q,long.q -l h_rt=604800,mem_free=1024.0M,memory=2G</tt>
If you look at the line showing ru_wallclock. You can see that it shows 1s. This means that the job started and then promptly ended. This points to something being wrong with your submission script. Perhaps there is a typo somewhere in it.

===== My job ran but didn't finish! =====
<tt>qname batch.q
hostname scout59.beocat
group some_user_users
owner some_user
project BEODEFAULT
department defaultdepartment
jobname my_job_script.sh
jobnumber 1122334455
...
snipped to save space
...
slots 1 </tt>
<tt style="color: red">failed 37 : qmaster enforced h_rt, h_cpu, or h_vmem limit</tt>
<tt>exit_status 0 </tt>
<tt style="color: red">ru_wallclock 21600s</tt>
<tt>ru_utime 0.130s
ru_stime 0.020s
...
snipped to save space
...
arid undefined</tt>
<tt style="color: red">category -u some_user -q batch.q,long.q -l h_rt=21600,mem_free=512.0M,memory=1G</tt>
If you look at the lines showing failed, ru_wallclock and category we can see some pointers to the issue.
It didn't finish because the scheduler (qmaster) enforced some limit. If you look at the category line, the only limit requested was h_rt. So it was a runtime (wallclock) limit.
Comparing ru_wallclock and the h_rt request, we can see that it ran until the h_rt time was hit, and then the scheduler enforce the limit and killed the job. You will need to resubmit the job and ask for more time next time.

OLD DEPRECATED AdvancedSGE

2014-07-07T21:42:00Z

Kylehutson: Added MPI instructions.

OLD DEPRECATED AdvancedSGE

2014-07-03T21:58:41Z

Kylehutson: Added Resource Requests section. Still needs MPI documented.