Beocat - User contributions [en]

Nautilus

2025-09-23T16:32:28Z

Nathanrwells:

= NOTICE =

The Nautilus folks have made changes to how we access the nautilus portal. While we work to make the system compatible, access to Nautilus through Fiona will be offline.

== Nautilus ==
To access the Nautilus namespace, login using K-State SSO at https://portal.nrp-nautilus.io/ . Once you have done so, contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] and request to be added to the Beocat Nautilus namespace (ksu-nrp-cluster). Once you have received notification that you have been added to the namespace, you can continue with the following steps to get set up to use the cluster resources.
<ol>
<li>SSH into headnode.beocat.ksu.edu</li>
<li>SSH into fiona (fiona hosts the kubectl tool we will use for this)</li>
<li>Once on fiona, use the command ‘cd ~’ to ensure you are in your home directory. If you
are not, this will return you to the top level of your home directory.<li>
<li>From there you will need to create a .kube directory inside of your home directory. Use
the command ‘mkdir ~/.kube’</li>
<li>Login to https://portal.nrp-nautilus.io/ using the same login previously used to create your
account (this will be your K-State EID login)</li>
<li>From here it is MANDATORY to read the cluster policy documentation provided by the
National Research Platform for the Nautilus program. You can find this here.
https://docs.nationalresearchplatform.org/userdocs/start/policies/ </li>
a. This is to ensure we do not break any of the rules put in place by the NRP.
 b. Return to https://portal.nrp-nautilus.io/ and accept the Acceptable Use Policy (AUP)
<li>Next, return to the website specified in step 5, in the top right corner of the page press
the “Get Config” option. </li>
a. This will download a file called ‘config’
<li>You will need to move the file to your ~/.kube directory created in step 4.</li>
a. To do this you can copy and paste the contents through the command line
 b. You can also utilize the OpenOnDemand tool to upload the file through the web
interface. Information for this tool can be found here:
https://support.beocat.ksu.edu/Docs/OpenOnDemand
 c. You can also use other means of moving the contents to the Beocat
headnodes/your home directory, but these are just a few examples.
 d. NOTE: Because we added a period before the directory name it is now a hidden directory,
and the directory will not appear when running a normal ‘ls’, to see the directory you will
need to run “ls -a” or “ls -la”.
<li>Once you have read the required documentation, created the .kube directory in your
home directory, and placed the config file in the '~/.kube' directory, you are now ready to continue!</li>
<li>Below is an example pod that can be used. It does not request much in the way of resources so you will likely need to change some things. Be sure to change the “name:” field
underneath “metadata:”. Change the text “test-pod” to “{eid}-pod” where ‘{eid}’ is your
K-State ID. It will look something like this “dan-pod”.</li>

<syntaxhighlight lang=yaml>
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: mypod
image: ubuntu
resources:
limits:
memory: 400Mi
cpu: 100m
requests:
memory: 100Mi
cpu: 100m
command: ["sh", "-c", "echo 'Im a new pod'"]
</syntaxhighlight>
<li>Place your .yaml file in the same directory created earlier (~/.kube).</li>
<li>If you are not already in the .kube directory enter the command “cd ~/.kube” to change
your current directory.</li>
<li>Now we are going to create our ‘pod’. This will request a ubuntu pc using the
specifications from above.</li>
a. To do this enter the command “kubectl create -f pod1.yaml” NOTE: You must be
in the same directory that you placed the pod1.yaml file in (in this situation, the above pod config was put into a file named pod1.yaml).
 b. If the command is successful you will see an output of “pod/{eid}-pod created”.
<li>You will need to wait until the container for the pod is finished creating. You can check
this by running “kubectl get pods”</li>
a. Once you run this command, it will output all the pods currently running or being
created in the namespace. Look for yours among the list of pods, the name will
be the same name specified in step 10.
 b. Once you locate your pod, check its STATUS. If the pod says Running, then you
are good to proceed. If it says Container Creating, then you will need to wait just a
bit. It should not take long.
<li>You can now execute and enter the pod by running “kubectl exec -it {eid}-pod --
/bin/bash”. Where ‘{eid}-pod’ is the pod created in step 13/the name specified in step 10.</li>
a. Executing this command will open the pod you created and run a bash console
on the pod.
 b. NOTE: If you have trouble logging into the pod, and are met with a “You must be
logged in to the server, you can run “kubectl proxy”, and after a moment, you can
cancel the command with a “crtl+c”. This should remedy the error.
</ol>

Additional documentation for Kubernetes can be found on the Kubernetes website https://kubernetes.io/docs/home

Main Page

2025-08-20T15:46:03Z

Nathanrwells: /* Online Documentations */

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

=== Online Documentations ===

* Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/]
* Read about [[Installed software]] and languages
* Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] and download the [[Media:Slurm-quick-reference.pdf|Slurm Quick Reference PDF]]
* Run interactive jobs with [[OpenOnDemand]]
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]]
* Big Data course on Beocat! [[BigDataOnBeocat]]
* Interested in web-based computational biology research? Check out [[GalaxyDocs|Galaxy!]]
* Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]]
* Need Bulk data storage on Beocat? We provide a no-backup large data storage solution, more information can be found here: [https://support.beocat.ksu.edu/Docs/AdvancedSlurm#Bulk_directory Beocat Bulk Directories]

=== Training Videos and Slides ===

* [https://www.youtube.com/watch?v=7NOB_HGQE0U Beocat Introduction] and [[Media:Beocat-Beoshock-Intro.pdf|slides]]
* [https://www.youtube.com/watch?v=b_yawpwFRdk Linux and Bash Introduction] and [[Media:Linux-Intro-cheatsheet.pdf|Linux Quick Reference PDF]]
* [https://www.youtube.com/watch?v=vcC-DURbH6c Advanced HPC Usage] and [[Media:HPC-Advanced-Usage.pdf|slides]]
* [https://www.youtube.com/watch?v=inJbYdZacjs HPC Parallel Computing] and [[Media:HPC-Parallel-Computing.pdf|slides]]

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
 With the recent changes to how KState handles DUO authentication, we recommend you use Globus to transfer files in and out of Beocat
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address or through TDX and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or email us at beocat@cs.ksu.edu, please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

AdvancedSlurm

2025-08-20T15:43:29Z

Nathanrwells: /* Bulk directory */

== Resource Requests ==
Aside from the time, RAM, and CPU requirements listed on the [[SlurmBasics]] page, we have a couple other requestable resources:
Valid gres options are:
gpu[[:type]:count]
fabric[[:type]:count]
Generally, if you don't know if you need a particular resource, you should use the default. These can be generated with the command
<tt>srun --gres=help</tt>
=== Fabric ===
We currently offer 3 "fabrics" as request-able resources in Slurm. The "count" specified is the line-rate (in Gigabits-per-second) of the connection on the node.
==== Infiniband ====
First of all, let me state that just because it sounds "cool" doesn't mean you need it or even want it. InfiniBand does absolutely no good if running on a single machine. InfiniBand is a high-speed host-to-host communication fabric. It is (most-often) used in conjunction with MPI jobs (discussed below). Several times we have had jobs which could run just fine, except that the submitter requested InfiniBand, and all the nodes with InfiniBand were currently busy. In fact, some of our fastest nodes do not have InfiniBand, so by requesting it when you don't need it, you are actually slowing down your job. To request Infiniband, add <tt>--gres=fabric:ib:1</tt> to your sbatch command-line.
==== ROCE ====
ROCE, like InfiniBand is a high-speed host-to-host communication layer. Again, used most often with MPI. Most of our nodes are ROCE enabled, but this will let you guarantee the nodes allocated to your job will be able to communicate with ROCE. To request ROCE, add <tt>--gres=fabric:roce:1</tt> to your sbatch command-line.

==== Ethernet ====
Ethernet is another communication fabric. All of our nodes are connected by ethernet, this is simply here to allow you to specify the interconnect speed. Speeds are selected in units of Gbps, with all nodes supporting 1Gbps or above. The currently available speeds for ethernet are: <tt>1, 10, 40, and 100</tt>. To select nodes with 40Gbps and above, you could specify <tt>--gres=fabric:eth:40</tt> on your sbatch command-line. Since ethernet is used to connect to the file server, this can be used to select nodes that have fast access for applications doing heavy IO. The Dwarves and Heroes have 40 Gbps ethernet and we measure single stream performance as high as 20 Gbps, but if your application
requires heavy IO then you'd want to avoid the Moles which are connected to the file server with only 1 Gbps ethernet.

=== CUDA ===
[[CUDA]] is the resource required for GPU computing. 'kstat -g' will show you the GPU nodes and the jobs running on them. To request a GPU node, add <tt>--gres=gpu:1</tt> for example to request 1 GPU for your job; if your job uses multiple nodes, the number of GPUs requested is per-node. You can also request a given type of GPU (kstat -g -l to show types) by using <tt>--gres=gpu:geforce_gtx_1080_ti:1</tt> for a 1080Ti GPU on the Wizards or Dwarves, <tt>--gres=gpu:quadro_gp100:1</tt> for the P100 GPUs on Wizard20-21 that are best for 64-bit codes like Vasp. Most of these GPU nodes are owned by various groups. If you want access to GPU nodes and your group does not own any, we can add you to the <tt>--partition=ksu-gen-gpu.q</tt> group that has priority on Dwarf36-39. For more information on compiling CUDA code click on this [[CUDA]] link.

A listing of the current types of gpus can be gathered with this command:
<syntaxhighlight lang=bash>
scontrol show nodes | grep CfgTRES | tr ',' '\n' | awk -F '[:=]' '/gres\/gpu:/ { print $2 }' | sort -u
</syntaxhighlight>
At the time of this writing, that command produces this list:
* geforce_gtx_1080_ti
* geforce_rtx_2080_ti
* geforce_rtx_3090
* l40s
* quadro_gp100
* rtx_a4000
* rtx_a6000

== Parallel Jobs ==
There are two ways jobs can run in parallel, ''intra''node and ''inter''node. '''Note: Beocat will not automatically make a job run in parallel.''' Have I said that enough? It's a common misperception.
=== Intranode jobs ===
''Intra''node jobs run on many cores in the same node. These jobs can take advantage of many common libraries, such as [http://openmp.org/wp/ OpenMP], or any programming language that has the concept of ''threads''. Often, your program will need to know how many cores you want it to use, and many will use all available cores if not told explicitly otherwise. This can be a problem when you are sharing resources, as Beocat does. To request multiple cores, use the sbatch directives '<tt>--nodes=1 --cpus-per-task=n</tt>' or '<tt>--nodes=1 --ntasks-per-node=n</tt>', where ''n'' is the number of cores you wish to use. If your command can take an environment variable, you can use $SLURM_CPUS_ON_NODE to tell how many cores you've been allocated.

=== Internode (MPI) jobs ===
''Inter''node jobs can utilize many cores on one or more nodes. Communicating between nodes is trickier than talking between cores on the same node. The specification for doing so is called "[[wikipedia:Message_Passing_Interface|Message Passing Interface]]", or MPI. We have [http://www.open-mpi.org/ OpenMPI] installed on Beocat for this purpose. Most programs written to take advantage of large multi-node systems will use MPI, but MPI also allows an application to run on multiple cores within a node. You can tell if you have an MPI-enabled program because its directions will tell you to run '<tt>mpirun ''program''</tt>'. Requesting MPI resources is only mildly more difficult than requesting single-node jobs. Instead of using '<tt>--cpus-per-task=''n''</tt>', you would use '<tt>--nodes=''n'' --tasks-per-node=''m''</tt>' ''or'' '<tt>--nodes=''n'' --ntasks=''o''</tt>' for your sbatch request, where ''n'' is the number of nodes you want, ''m'' is the number of cores per node you need, and ''o'' is the total number of cores you need.

Some quick examples:

<tt>--nodes=6 --ntasks-per-node=4</tt> will give you 4 cores on each of 6 nodes for a total of 24 cores.

<tt>--ntasks=40</tt> will give you 40 cores spread across any number of nodes.

<tt>--nodes=10 --ntasks=100</tt> will give you a total of 100 cores across 10 nodes.

== Requesting memory for multi-core jobs ==
Memory requests are easiest when they are specified '''per core'''. For instance, if you specified the following: '<tt>--tasks=20 --mem-per-core=20G</tt>', your job would have access to 400GB of memory total.
== Other Handy Slurm Features ==
=== Email status changes ===
One of the most commonly used options when submitting jobs not related to resource requests is to have have Slurm email you when a job changes its status. This takes may need two directives to sbatch: <tt>--mail-user</tt> and <tt>--mail-type</tt>.
==== --mail-type ====
<tt>--mail-type</tt> is used to tell Slurm to notify you about certain conditions. Options are comma separated and include the following
{| class="wikitable"
!Option!!Explanation
|-
| NONE || This disables event-based mail
|-
| BEGIN || Sends a notification when the job begins
|-
| END || Sends a notification when the job ends
|-
| FAIL || Sends a notification when the job fails.
|-
| REQUEUE || Sends a notification if the job is put back into the queue from a running state
|-
| STAGE_OUT || Burst buffer stage out and teardown completed
|-
| ALL || Equivalent to BEGIN,END,FAIL,REQUEUE,STAGE_OUT
|-
| TIME_LIMIT || Notifies if the job ran out of time
|-
| TIME_LIMIT_90 || Notifies when the job has used 90% of its allocated time
|-
| TIME_LIMIT_80 || Notifies when the job has used 80% of its allocated time
|-
| TIME_LIMIT_50 || Notifies when the job has used 50% of its allocated time
|-
| ARRAY_TASKS || Modifies the BEGIN, END, and FAIL options to apply to each array task (instead of notifying for the entire job
|}

==== --mail-user ====
<tt>--mail-user</tt> is optional. It is only needed if you intend to send these job status updates to a different e-mail address than what you provided in the [https://acount.beocat.ksu.edu/user Account Request Page]. It is specified with the following arguments to sbatch: <tt>--mail-user=someone@somecompany.com</tt>

=== Job Naming ===
If you have several jobs in the queue, running the same script with different parameters, it's handy to have a different name for each job as it shows up in the queue. This is accomplished with the '<tt>-J ''JobName''</tt>' sbatch directive.

=== Separating Output Streams ===
Normally, Slurm will create one output file, containing both STDERR and STDOUT. If you want both of these to be separated into two files, you can use the sbatch directives '<tt>--output</tt>' and '<tt>--error</tt>'.

{| class="wikitable"
! option !! default !! example
|-
| --output || slurm-%j.out || slurm-206.out
|-
| --error || slurm-%j.out || slurm-206.out
|}
<tt>%j</tt> above indicates that it should be replaced with the job id.

=== Running from the Current Directory ===
By default, jobs run from your home directory. Many programs incorrectly assume that you are running the script from the current directory. You can use the '<tt>-cwd</tt>' directive to change to the "current working directory" you used when submitting the job.
=== Running in a specific class of machine ===
If you want to run on a specific class of machines, e.g., the Dwarves, you can add the flag "--constraint=dwarves" to select any of those machines.

=== Processor Constraints ===
Because Beocat is a heterogenous cluster (we have machines from many years in the cluster), not all of our processors support every new and fancy feature. You might have some applications that require some newer processor features, so we provide a mechanism to request those.

<tt>--contraint</tt> tells the cluster to apply constraints to the types of nodes that the job can run on. For instance, we know of several applications that must be run on chips that have "AVX" processor extensions. To do that, you would specify <tt>--constraint=avx</tt> on you ''<tt>sbatch</tt>'' '''or''' ''<tt>srun</tt>'' command lines.
Using <tt>--constraint=avx</tt> will prohibit your job from running on the Mages while <tt>--contraint=avx2</tt> will eliminate the Elves as well as the Mages.

=== Slurm Environment Variables ===
Within an actual job, sometimes you need to know specific things about the running environment to setup your scripts correctly. Here is a listing of environment variables that Slurm makes available to you. Of course the value of these variables will be different based on many different factors.
<syntaxhighlight lang="bash">
CUDA_VISIBLE_DEVICES=NoDevFiles
ENVIRONMENT=BATCH
GPU_DEVICE_ORDINAL=NoDevFiles
HOSTNAME=dwarf37
SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint
SLURM_CLUSTER_NAME=beocat
SLURM_CPUS_ON_NODE=1
SLURM_DISTRIBUTION=cyclic
SLURMD_NODENAME=dwarf37
SLURM_GTIDS=0
SLURM_JOB_CPUS_PER_NODE=1
SLURM_JOB_GID=163587
SLURM_JOB_ID=202
SLURM_JOBID=202
SLURM_JOB_NAME=slurm_simple.sh
SLURM_JOB_NODELIST=dwarf37
SLURM_JOB_NUM_NODES=1
SLURM_JOB_PARTITION=batch.q,killable.q
SLURM_JOB_QOS=normal
SLURM_JOB_UID=163587
SLURM_JOB_USER=mozes
SLURM_LAUNCH_NODE_IPADDR=10.5.16.37
SLURM_LOCALID=0
SLURM_MEM_PER_NODE=1024
SLURM_NNODES=1
SLURM_NODEID=0
SLURM_NODELIST=dwarf37
SLURM_NPROCS=1
SLURM_NTASKS=1
SLURM_PRIO_PROCESS=0
SLURM_PROCID=0
SLURM_SRUN_COMM_HOST=10.5.16.37
SLURM_SRUN_COMM_PORT=37975
SLURM_STEP_ID=0
SLURM_STEPID=0
SLURM_STEP_LAUNCHER_PORT=37975
SLURM_STEP_NODELIST=dwarf37
SLURM_STEP_NUM_NODES=1
SLURM_STEP_NUM_TASKS=1
SLURM_STEP_TASKS_PER_NODE=1
SLURM_SUBMIT_DIR=/homes/mozes
SLURM_SUBMIT_HOST=dwarf37
SLURM_TASK_PID=23408
SLURM_TASKS_PER_NODE=1
SLURM_TOPOLOGY_ADDR=due1121-prod-core-40g-a1,due1121-prod-core-40g-c1.due1121-prod-sw-100g-a9.dwarf37
SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node
SLURM_UMASK=0022
SRUN_DEBUG=3
TERM=screen-256color
TMPDIR=/tmp
USER=mozes
</syntaxhighlight>
Sometimes it is nice to know what hosts you have access to during a job. You would checkout the SLURM_JOB_NODELIST to know that. There are lots of useful Environment Variables there, I will leave it to you to identify the ones you want.

Some of the most commonly-used variables we see used are $SLURM_CPUS_ON_NODE, $HOSTNAME, and $SLURM_JOB_ID.

== Running from a sbatch Submit Script ==
No doubt after you've run a few jobs you get tired of typing something like 'sbatch -l mem=2G,h_rt=10:00 -pe single 8 -n MyJobTitle MyScript.sh'. How are you supposed to remember all of these every time? The answer is to create a 'submit script', which outlines all of these for you. Below is a sample submit script, which you can modify and use for your own purposes.
<syntaxhighlight lang="bash">
#!/bin/bash

## A Sample sbatch script created by Kyle Hutson
##
## Note: Usually a '#" at the beginning of the line is ignored. However, in
## the case of sbatch, lines beginning with #SBATCH are commands for sbatch
## itself, so I have taken the convention here of starting *every* line with a
## '#', just Delete the first one if you want to use that line, and then modify
## it to your own purposes. The only exception here is the first line, which
## *must* be #!/bin/bash (or another valid shell).

## There is one strict rule for guaranteeing Slurm reads all of your options:
## Do not put *any* lines above your resource requests that aren't either:
## 1) blank. (no other characters)
## 2) comments (lines must begin with '#')

## Specify the amount of RAM needed _per_core_. Default is 1G
##SBATCH --mem-per-cpu=1G

## Specify the maximum runtime in DD-HH:MM:SS form. Default is 1 hour (1:00:00)
##SBATCH --time=1:00:00

## Require the use of infiniband. If you don't know what this is, you probably
## don't need it.
##SBATCH --gres=fabric:ib:1

## GPU directive. If You don't know what this is, you probably don't need it
##SBATCH --gres=gpu:1

## number of cores/nodes:
## quick note here. Jobs requesting 16 or fewer cores tend to get scheduled
## fairly quickly. If you need a job that requires more than that, you might
## benefit from contacting Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]
## to see how we can assist in getting your
## job scheduled in a reasonable amount of time. Default is
##SBATCH --cpus-per-task=1
##SBATCH --cpus-per-task=12
##SBATCH --nodes=2 --tasks-per-node=1
##SBATCH --tasks=20

## Constraints for this job. Maybe you need to run on the elves
##SBATCH --constraint=elves
## or perhaps you just need avx processor extensions
##SBATCH --constraint=avx

## Output file name. Default is slurm-%j.out where %j is the job id.
##SBATCH --output=MyJobTitle.o%j

## Split the errors into a seperate file. Default is the same as output
##SBATCH --error=MyJobTitle.e%j

## Name my job, to make it easier to find in the queue
##SBATCH -J MyJobTitle

## Send email when certain criteria are met.
## Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to
## BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage
## out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent
## of time limit), TIME_LIMIT_80 (reached 80 percent of time limit),
## TIME_LIMIT_50 (reached 50 percent of time limit) and ARRAY_TASKS (send
## emails for each array task). Multiple type values may be specified in a
## comma separated list. Unless the ARRAY_TASKS option is specified, mail
## notifications on job BEGIN, END and FAIL apply to a job array as a whole
## rather than generating individual email messages for each task in the job
## array.
##SBATCH --mail-type=ALL

## Email address to send the email to based on the above line.
## Default is to send the mail to the e-mail address entered on the account
## request form.
##SBATCH --mail-user myemail@ksu.edu

## And finally, we run the job we came here to do.
## $HOME/ProgramDir/ProgramName ProgramArguments

## OR, for the case of MPI-capable jobs
## mpirun $HOME/path/MpiJobName
</syntaxhighlight>

== File Access ==
Beocat has a variety of options for storing and accessing your files.
Every user has a home directory for general use which is limited in size, has decent file access performance. Those needing more storage may purchase /bulk subdirectories which have the same decent performance
but are not backed up. The /fastscratch file system is a zfs host with lots of NVME drives provide much faster
temporary file access. When fast IO is critical to the application performance, access to /fastscratch, the local disk on each node, or to a
RAM disk are the best options.

===Home directory===

Every user has a <tt>/homes/''username''</tt> directory that they drop into when they log into Beocat.
The home directory is for general use and provides decent performance for most file IO.
Disk space in each home directory is limited to 1 TB, so larger files should be kept in a purchased /bulk
directory, and there is a limit of 100,000 files in each subdirectory in your account.
This file system is fully redundant, so 3 specific hard disks would need to fail before any data was lost.
All files will soon be backed up nightly to a separate file server in Nichols Hall, so if you do accidentally
delete something it can be recovered.

===Bulk directory===

Bulk data storage may be provided at a cost of $45/TB/year billed monthly. A bulk directory is provided once you have contacted Beocat systems administrators in a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] and provided us with the appropriate billing information through your accounting department. We do not allow payments to be made through personal bank accounts or checks, bulk directories require a monthly billable account.

===Fast Scratch file system===

The /fastscratch file system is faster than /bulk or /homes.
In order to use fastscratch, you first need to make a directory for yourself.
Fast Scratch is meant as temporary space for prepositioning files and accessing them
during runs. Once runs are completed, any files that need to be kept should be moved to your home
or bulk directories since files on the fastscratch file system may get purged after 30 days.

<syntaxhighlight lang=bash>
mkdir /fastscratch/$USER
</syntaxhighlight>

===Local disk===

If you are running on a single node, it may also be faster to access your files from the local disk
on that node. Each job creates a subdirectory /tmp/job# where '#' is the job ID number on the
local disk of each node the job uses. This can be accessed simply by writing to /tmp rather than
needing to use /tmp/job#.

You may need to copy files to
local disk at the start of your script, or set the output directory for your application to point
to a file on the local disk, then you'll need to copy any files you want off the local disk before
the job finishes since Slurm will remove all files in your job's directory on /tmp on completion
of the job or when it aborts. Use 'kstat -l -h' to see how much /tmp space is available on each node.

<syntaxhighlight lang=bash>
# Copy input files to the tmp directory if needed
cp $input_files /tmp

# Make an 'out' directory to pass to the app if needed
mkdir /tmp/out

# Example of running an app and passing the tmp directory in/out
app -input_directory /tmp -output_directory /tmp/out

# Copy the 'out' directory back to the current working directory after the run
cp -rp /tmp/out .
</syntaxhighlight>

===RAM disk===

If you need ultrafast access to files, you can use a RAM disk which is a file system set up in the
memory of the compute node you are running on. The RAM disk is limited to the requested memory on that node, so you should account for this usage when you request
memory for your job. Below is an example of how to use the RAM disk.

<syntaxhighlight lang=bash>
# Copy input files over if necessary
cp $any_input_files /dev/shm/

# Run the application, possibly giving it the path to the RAM disk to use for output files
app -output_directory /dev/shm/

# Copy files from the RAM disk to the current working directory and clean it up
cp /dev/shm/* .
</syntaxhighlight>

===When you leave KSU===

If you are done with your account and leaving KSU, please clean up your directory, move any files
to your supervisor's account that need to be kept after you leave, and notify us so that we can disable your
account. The easiest way to move your files to your supervisor's account is for them to set up
a subdirectory for you with the appropriate write permissions. The example below shows moving
just a user's 'data' subdirectory to their supervisor. The 'nohup' command is used so that the move will
continue even if the window you are doing the move from gets disconnected.

<syntaxhighlight lang=bash>
# Supervisor:
mkdir /bulk/$USER/$STUDENT_USERNAME
setfacl -d -m u:$USER:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -m u:$USER:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -d -m u:$STUDENT_USERNAME:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -m u:$STUDENT_USERNAME:rwX -R /bulk/$USER/$STUDENT_USERNAME

# Student:
nohup mv /homes/$USER/data /bulk/$SUPERVISOR_USERNAME/$USER &

# Once the move is complete, the Supervisor should limit the permissions for the directory again by removing the student's access:
chown $USER: -R /bulk/$USER/$STUDENT_USERNAME
setfacl -d -x u:$STUDENT_USERNAME -R /bulk/$USER/$STUDENT_USERNAME
setfacl -x u:$STUDENT_USERNAME -R /bulk/$USER/$STUDENT_USERNAME
</syntaxhighlight>

==File Sharing==

This section will cover methods of sharing files with other users within Beocat and on remote systems.
In the past, Beocat users have been allowed to keep their
/homes and /bulk directories open so that any other user could
access files. In order to bring Beocat into alignment with
State of Kansas regulations and industry norms, all users must now have their /homes /bulk /scratch and /fastscratch directories
locked down from other users, but can still share files and directories within their group or with individual users
using group and individual ACLs (Access Control Lists) which will be explained below.
Beocat staff will be exempted from this
policy as we need to work freely with all users and will manage our
subdirectories to minimize access.

===Securing your home directory with the setacls script===

If you do not wish to share files or directories with other users, you do not need to do anything
as rwx access to others has already been removed.
If you want to share files or directories you can either use the '''setacls''' script or configure
the ACLs (Access Control Lists) manually.

The '''setacls -h''' will show how to use the script.

Eos: setacls -h
setacls [-r] [-w] [-g group] [-u user] -d /full/path/to/directory
Execute pemission will always be applied, you may also choose r or w
Must specify at least one group or user
Must specify at least one directory, and it must be the full path
Example: setacls -r -g ksu-cis-hpc -u mozes -d /homes/daveturner/shared_dir

You can specify the permissions to be either -r for read or -w for write or you can specify both.
You can provide a priority group to share with, which is the same as the group used in a --partition=
statement in a job submission script. You can also specify users.
You can specify a file or a directory to share. If the directory is specified then all files in that
directory will also be shared, and all files created in the directory laster will also be shared.

The script will set everything up for you, telling you the commands it is executing along the way,
then show the resulting ACLs at the end with the '''getfacl''' command. Below is an example of
sharing the directory '''test_directory''' in my /bulk/daveturner directory with Nathan.

<syntaxhighlight>
Beocat> cd /bulk/daveturner
Beocat> mkdir test_directory
Beocat> setacls -r -w -u nathanrwells -d /bulk/daveturner/test_directory

Opening up base directory /bulk/daveturner with X execute permission only
setfacl -m u:nathanrwells:X /bulk/daveturner

Setting Xrw for directory/file /bulk/daveturner/test_directory
setfacl -m u:nathanrwells:Xrw -R /bulk/daveturner/test_directory
setfacl -d -m u:nathanrwells:Xrw -R /bulk/daveturner/test_directory

The ACLs on directory /bulk/daveturner/test_directory are set to:

getfacl: Removing leading '/' from absolute path names
# file: bulk/daveturner/test_directory
USER daveturner rwx rwx
user nathanrwells rwx rwx
GROUP daveturner_users r-x r-x
group beocat_support r-x r-x
mask rwx rwx
other --- ---
</syntaxhighlight>

The '''getfacl''' run by the script now shows that user '''nathanrwells''' has
read and write permissions to that directory and execute access to all directories
leading up to it.

====Manually configuring your ACLs====

If you want to manually configure the ACLs you can use the directions below to do what the '''setacls'''
script would do for you.
You first need to provide the minimum execute access to your /homes
or /bulk directory before sharing individual subdirectories. Setting the ACL to execute only will allow those
in your group to get access to subdirectories while not including read access will mean they will not
be able to see other files or subdirectories on your main directory, but do keep in mind that they can still access them
so you may want to still lock them down manually. Below is an example of how I would change my
/homes/daveturner directory to allow ksu-cis-hpc group execute access.

<syntaxhighlight lang=bash>
setfacl -m g:ksu-cis-hpc:X /homes/daveturner
</syntaxhighlight>

If your research group owns any nodes on Beocat, then you have a group name that can be used to securely share
files with others within your group. Below is an example of creating a directory called 'share_hpc',
then providing access to my ksu-cis.hpc group
(my group is ksu-cis-hpc so I submit jobs to --partition=ksu-cis-hpc.q).
Using -R will make these changes recursively to all files and directories in that subdirectory while changing the defaults with the setfacl -d command will ensure that files and directories created
later will be done so with these same ACLs.

<syntaxhighlight lang=bash>
mkdir share_hpc
# ACLs are used here for setting default permissions
setfacl -d -m g:ksu-cis-hpc:rX -R share_hpc
# ACLs are used here for setting actual permissions
setfacl -m g:ksu-cis-hpc:rX -R share_hpc
</syntaxhighlight>

This will give people in your group the ability to read files in the 'share_hpc' directory. If you also want
them to be able to write or modify files in that directory then change the ':rX' to ':rwX' instead. e.g. 'setfacl -d -m g:ksu-cis-hpc:rwX -R share_hpc'

If you want to know what groups you belong to use the line below.
<syntaxhighlight lang=bash>
groups
</syntaxhighlight>
If your group does not own any nodes, you can still request a group name and manage the participants yourself
by emailing us at
or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or emailing us at beocat@cs.ksu.edu
.
If you want to share a directory with only a few people you can manage your ACLs using individual usernames
instead of with a group.

You can use the '''getfacl''' command to see groups have access to a given directory.

<syntaxhighlight lang=bash>
getfacl share_hpc

# file: share_hpc
# owner: daveturner
# group: daveturner_users
user::rwx
group::r-x
group:ksu-cis-hpc:r-x
mask::r-x
other::---
default:user::rwx
default:group::r-x
default:group:ksu-cis-hpc:r-x
default:mask::r-x
default:other::---
</syntaxhighlight>

ACLs give you great flexibility in controlling file access at the
group level. Below is a more advanced example where I set up a directory to be shared with
my ksu-cis-hpc group, Dan's ksu-cis-dan group, and an individual user 'mozes' who I also want
to have write access.

<syntaxhighlight lang=bash>
mkdir share_hpc_dan_mozes
# acls are used here for setting default permissions
setfacl -d -m g:ksu-cis-hpc:rX -R share_hpc_dan_mozes
setfacl -d -m g:ksu-cis-dan:rX -R share_hpc_dan_mozes
setfacl -d -m u:mozes:rwX -R share_hpc_dan_mozes
# ACLs are used here for setting actual permissions
setfacl -m g:ksu-cis-hpc:rX -R share_hpc_dan_mozes
setfacl -m g:ksu-cis-dan:rX -R share_hpc_dan_mozes
setfacl -m u:mozes:rwX -R share_hpc_dan_mozes

getfacl share_hpc_dan_mozes

# file: share_hpc_dan_mozes
# owner: daveturner
# group: daveturner_users
user::rwx
user:mozes:rwx
group::r-x
group:ksu-cis-hpc:r-x
group:ksu-cis-dan:r-x
mask::r-x
other::---
default:user::rwx
default:user:mozes:rwx
default:group::r-x
default:group:ksu-cis-hpc:r-x
default:group:ksu-cis-dan:r-x
default:mask::r-x
default:other::--x
</syntaxhighlight>

===Openly sharing files on the web===

If you create a 'public_html' directory on your home directory, then any files put there will be shared
openly on the web. There is no way to restrict who has access to those files.

<syntaxhighlight lang=bash>
cd
mkdir public_html
# Opt-in to letting the webserver access your home directory:
setfacl -m g:public_html:x ~/
</syntaxhighlight>

Then access the data from a web browser using the URL:

http://people.beocat.ksu.edu/~your_user_name

This will show a list of the files you have in your public_html subdirectory.

===Globus===

We have a page here dedicated to [[Globus]]

== Array Jobs ==
One of Slurm's useful options is the ability to run "Array Jobs"

It can be used with the following option to sbatch.

--array=n[-m[:s]]
Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Slurm
almost like a series of jobs. The option argument to --array specifies the number of array job tasks and the index number which will be
associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SLURM_ARRAY_TASK_ID. The option
arguments n, and m will be available through the environment variables SLURM_ARRAY_TASK_MIN and SLURM_ARRAY_TASK_MAX.

The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size.
Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SLURM_ARRAY_TASK_ID containing one of the 5 index numbers.

Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The
number of tasks in a array job is unlimited.

STDOUT and STDERR of array job tasks follow a slightly different naming convention (which can be controlled in the same way as mentioned above).

slurm-%A_%a.out

%A is the SLURM_ARRAY_JOB_ID, and %a is the SLURM_ARRAY_TASK_ID

=== Examples ===
==== Change the Size of the Run ====
Array Jobs have a variety of uses, one of the easiest to comprehend is the following:

I have an application, app1 I need to run the exact same way, on the same data set, with only the size of the run changing.

My original script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
RUNSIZE=50
#RUNSIZE=100
#RUNSIZE=150
#RUNSIZE=200
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
For every run of that job I have to change the RUNSIZE variable, and submit each script. This gets tedious.

With Array Jobs the script can be written like so:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=50-200:50
RUNSIZE=$SLURM_ARRAY_TASK_ID
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
I then submit that job, and Slurm understands that it needs to run it 4 times, once for each task. It also knows that it can and should run these tasks in parallel.

==== Choosing a Dataset ====
A slightly more complex use of Array Jobs is the following:

I have an application, app2, that needs to be run against every line of my dataset. Every line changes how app2 runs slightly, but I need to compare the runs against each other.

Originally I had to take each line of my dataset and generate a new submit script and submit the job. This was done with yet another script:

<syntaxhighlight lang="bash">
#!/bin/bash
DATASET=dataset.txt
scriptnum=0
while read LINE
do
echo "app2 $LINE" > ${scriptnum}.sh
sbatch ${scriptnum}.sh
scriptnum=$(( $scriptnum + 1 ))
done < $DATASET
</syntaxhighlight>
Not only is this needlessly complex, it is also slow, as sbatch has to verify each job as it is submitted. This can be done easily with array jobs, as long as you know the number of lines in the dataset. This number can be obtained like so: wc -l dataset.txt in this case lets call it 5000.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=1-5000
app2 `sed -n "${SLURM_ARRAY_TASK_ID}p" dataset.txt`
</syntaxhighlight>
This uses a subshell via `, and has the sed command print out only the line number $SLURM_ARRAY_TASK_ID out of the file dataset.txt.

Not only is this a smaller script, it is also faster to submit because it is one job instead of 5000, so sbatch doesn't have to verify as many.

To give you an idea about time saved: submitting 1 job takes 1-2 seconds. by extension if you are submitting 5000, that is 5,000-10,000 seconds, or 1.5-3 hours.

== Checkpoint/Restart using DMTCP ==

DMTCP is Distributed Multi-Threaded CheckPoint software that will checkpoint your application without modification, and
can be set up to automatically restart your job from the last checkpoint if for example the node you are running on fails.
This has been tested successfully
on Beocat for some scalar and OpenMP codes, but has failed on all MPI tests so far. We would like to encourage users to
try DMTCP out if their non-MPI jobs run longer than 24 hours. If you want to try this, please contact us first since we are still
experimenting with DMTCP.

The sample job submission script below shows how dmtcp_launch is used to start the application, then dmtcp_restart is used to start from a checkpoint if the job has failed and been rescheduled.
If you are putting this in an array script, then add the Slurm array task ID to the end of the ckeckpoint directory name
like ckptdir=ckpt-$SLURM_ARRAY_TASK_ID.

#!/bin/bash -l
#SBATCH --job-name=gromacs
#SBATCH --mem=50G
#SBATCH --time=24:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module reset
module load GROMACS/2016.4-foss-2017beocatb-hybrid
module load DMTCP
module list

ckptdir=ckpt
mkdir -p $ckptdir
export DMTCP_CHECKPOINT_DIR=$ckptdir

if ! ls -1 $ckptdir | grep -c dmtcp_restart_script > /dev/null
then
echo "Using dmtcp_launch to start the app the first time"
dmtcp_launch --no-coordinator mpirun -np 1 -x OMP_NUM_THREADS=4 gmx_mpi mdrun -nsteps 50000 -ntomp 4 -v -deffnm 1ns -c 1ns.pdb -nice 0
else
echo "Using dmtcp_restart from $ckptdir to continue from a checkpoint"
dmtcp_restart $ckptdir/*.dmtcp
fi

You will need to run several tests to verify that DMTCP is working properly with your application.
First, run a short test without DMTCP and another with DMTCP with the checkpoint interval set to 5 minutes
by adding the line export DMTCP_CHECKPOINT_INTERVAL=300 to your script. Then use kstat -d 1 to
check that the memory in both runs is close to the same. Also use this information to calculate the time
that each checkpoint takes. In most cases I've seen times less than a minute for checkpointing that will normally
be done once each hour. If your application is taking more time, let us know. Sometimes this can be sped up
by simply turning off compression by adding the line export DMTCP_GZIP=0. Make sure to remove the
line where you set the checkpoint interval to 300 seconds so that the default time of once per hour will be used.

After verifying that your code completes using DMTCP and does not take significantly more time or memory, you
will need to start a run then scancel it after the first checkpoint, then resubmit the same script to make
sure that it restarts and runs to completion. If you are working with an array job script, the last is to try a few
array tasks at once to make sure there is no conflict between the jobs.

== Running jobs interactively ==
Some jobs just don't behave like we think they should, or need to be run with somebody sitting at the keyboard and typing in response to the output the computers are generating. Beocat has a facility for this, called 'srun'. srun uses the exact same command-line arguments as sbatch, but you need to add the following arguments at the end: <tt>--pty bash</tt>. If no node is available with your resource requirements, srun will tell you something like the following:
srun --pty bash
srun: Force Terminated job 217
srun: error: CPU count per node can not be satisfied
srun: error: Unable to allocate resources: Requested node configuration is not available
Note that, like sbatch, your interactive job will timeout after your allotted time has passed.

== Connecting to an existing job ==
You can connect to an existing job using srun in the same way that the MonitorNode command
allowed us to in the old cluster. This is essentially like using ssh to get into the node where your job is running which
can be very useful in allowing you to look at files in /tmp/job# or in running htop to view the
activity level for your job.

srun --jobid=# --pty bash where '#' is the job ID number

== Altering Job Requests ==
We generally do not support users to modify job parameters once the job has been submitted. It can be done, but there are numerous catches, and all of the variations can be a bit problematic; it is normally easier to simply delete the job (using '''scancel ''jobid''''') and resubmit it with the right parameters. '''If your job doesn't start after modifying such parameters (after a reasonable amount of time), delete the job and resubmit it.'''

As it is unsupported, this is an excercise left to the reader. A starting point is <tt>man scontrol</tt>
== Killable jobs ==
There are a growing number of machines within Beocat that are owned by a particular person or group. Normally jobs from users that aren't in the group designated by the owner of these machines cannot use them. This is because we have guaranteed that the nodes will be accessible and available to the owner at any given time. We will allow others to use these nodes if they designate their job as "killable." If your job is designated as killable, your job will be able to use these nodes, but can (and will) be killed off at any point in time to make way for the designated owner's jobs. Jobs that are marked killable will be re-queued and may restart on another node.

The way you would designate your job as killable is to add <tt>--gres=killable:1</tt> to the '''<tt>sbatch</tt> or <tt>srun</tt>''' arguments. This could be either on the command-line or in your script file.

''Note: This is a submit-time only request, it cannot be added by a normal user after the job has been submitted.'' If you would like jobs modified to be '''killable''' after the jobs have been submitted (and it is too much work to <tt>scancel</tt> the jobs and re-submit), send an e-mail to the administrators detailing the job ids and what you would like done.

== Scheduling Priority ==
Some users are members of projects that have contributed to Beocat. When those users have contributed nodes, the group gets access to a "partition" giving you priority on those nodes.

In most situations, the scheduler will automatically add those priority partitions to the jobs as submitted. You should not need to include a partition list in your job submission.

There are currently just a few exceptions that we will not automatically add:
* ksu-chem-mri.q
* ksu-gen-gpu.q
* ksu-gen-highmem.q

If you have access to those any of the non-automatic partitions, and have need of the resources in that partition, you can then alter your <tt>#SBATCH</tt> lines to include your new partition:
#SBATCH --partition=ksu-gen-highmem.q

Otherwise, you shouldn't modify the partition line at all unless you really know what you're doing.

== Graphical Applications ==
Some applications are graphical and need to have some graphical input/output. We currently accomplish this with X11 forwarding or [[OpenOnDemand]]
=== OpenOnDemand ===
[[OpenOnDemand]] is likely the easier and more performant way to run a graphical application on the cluster.
# visit [https://ondemand.beocat.ksu.edu/ ondemand] and login with your cluster credentials.
# Check the "Interactive Apps" dropdown. We may have a workflow ready for you. If not choose the desktop.
# Select the resources you need
# Select launch
# A job is now submitted to the cluster and once the job is started you'll see a Connect button
# use the app as needed. If using the desktop, start your graphical application.

=== X11 Forwarding ===
==== Connecting with an X11 client ====
===== Windows =====
If you are running Windows, we recommend MobaXTerm as your file/ssh manager, this is because it is one relatively simple tool to do everything. MobaXTerm also automatically connects with X11 forwarding enabled.
===== Linux/OSX =====
Both Linux and OSX can connect in an X11 forwarding mode. Linux will have all of the tools you need installed already, OSX will need [https://www.xquartz.org/ XQuartz] installed.

Then you will need to change your 'ssh' command slightly:

ssh -Y eid@headnode.beocat.ksu.edu

The '''-Y''' argument tells ssh to setup X11 forwarding.
==== Starting an Graphical job ====
All graphical jobs, by design, must be interactive, so we'll use the srun command. On a headnode, we run the following:
# load an X11 enabled application
module load Octave
# start an X11 job, sbatch arguments are accepted for srun as well, 1 node, 1 hour, 1 gb of memory
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 octave --gui

Because these jobs are interactive, they may not be able to run at all times, depending on how busy the scheduler is at any point in time. '''--pty --x11''' are required arguments setting up the job, and '''octave --gui''' is the command to run inside the job.

== Job Accounting ==
Some people may find it useful to know what their job did during its run. The sacct tool will read Slurm's accounting database and give you summarized or detailed views on jobs that have run within Beocat.
=== sacct ===
This data can usually be used to diagnose two very common job failures.
==== Job debugging ====
It is simplest if you know the job number of the job you are trying to get information on.
<syntaxhighlight lang="bash">
# if you know the jobid, put it here:
sacct -j 1122334455 -l
# if you don't know the job id, you can look at your jobs started since some day:
sacct -S 2017-01-01
</syntaxhighlight>

===== My job didn't do anything when it ran! =====
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|-
|3
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|218||218||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||12||00:00:00||FAILED||2:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=12,mem=1G,node=1||cpu=12,mem=1G,node=1
|-
|218.batch||218.batch||batch||||137940K||dwarf37||0||137940K||1576K||dwarf37||0||1576K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||1.36G||0||0||0||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|-
|218.0||218.0||qqqqstat||||204212K||dwarf37||0||204212K||1420K||dwarf37||0||1420K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||196.52M||Unknown||Unknown||Unknown||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|}</div> 
If you look at the columns showing Elapsed and State, you can see that they show 00:00:00 and FAILED respectively. This means that the job started and then promptly ended. This points to something being wrong with your submission script. Perhaps there is a typo somewhere in it.

===== My job ran but didn't finish! =====
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|-
|3
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|220||220||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:01:27||TIMEOUT||0:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=1,mem=1G,node=1||cpu=1,mem=1G,node=1
|-
|220.batch||220.batch||batch||||370716K||dwarf37||0||370716K||7060K||dwarf37||0||7060K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:28||CANCELLED||0:15||1.23G||0||0||0||1Gn||0||0.16M||dwarf37||0||0.16M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|-
|220.0||220.0||sleep||||204212K||dwarf37||0||107916K||1000K||dwarf37||0||620K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:27||CANCELLED||0:15||1.54G||Unknown||Unknown||Unknown||1Gn||0||0.05M||dwarf37||0||0.05M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|}</div> 
If you look at the column showing State, we can see some pointers to the issue. The job ran out of time (TIMEOUT) and then was killed (CANCELLED).
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|221||221||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:00:00||CANCELLED by 0||0:0||||Unknown||Unknown||Unknown||1Mn||||||||||||||||||||||||cpu=1,mem=1M,node=1||cpu=1,mem=1M,node=1
|-
|221.batch||221.batch||batch||||137940K||dwarf37||0||137940K||1144K||dwarf37||0||1144K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:00:01||CANCELLED||0:15||2.62G||0||0||0||1Mn||0||0||dwarf37||65534||0||0||dwarf37||65534||0||||||||cpu=1,mem=1M,node=1
|}</div> 
If you look at the column showing State, we see it was "CANCELLED by 0", then we look at the AllocTRES column to see our allocated resources, and see that 1MB of memory was granted. Combine that with the column "MaxRSS" and we see that the memory granted was less than the memory we tried to use, thus the job was "CANCELLED".

FAQ

2025-06-26T13:48:52Z

Nathanrwells: /* Common Storage For Projects */

== How do I connect to Beocat ==
{| class="wikitable"
|-
! colspan="2" | Connection Settings
|-
! Hostname
| style="text-align:right" | headnode.beocat.ksu.edu
|-
! Port
| style="text-align:right" | 22
|-
! Username
| style="text-align:right" | <tt>eID</tt>
|-
! Password
| style="text-align:right" | <tt>eID Password</tt>
|}

{| class="wikitable"
!colspan="2" | Supported Connection Software (Latest Versions of Each)
|-
!rowspan="3" | Shell
|-
| [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]
|-
| ssh from openssh
|-
!rowspan="4" | File Transfer Utilities
|-
| [https://filezilla-project.org/ Filezilla]
|-
| [http://winscp.net/ WinSCP]
|-
| scp and sftp from openssh
|-
!rowspan="2" | Combination
|-
| [http://mobaxterm.mobatek.net/ MobaXterm]
|}

===Duo===
If your account is Duo Enabled, you will be asked to approve ''each'' connection through Duo's push system to your smart device by default for any non-interactive protocols. If you don't have a smart device, or your smart device is not currently able to be contacted by Duo, there are options.

====Automating Duo Method====
You would need to configure your connection client to send an ''Environment'' variable called <tt>DUO_PASSCODE</tt>. Its value could be the currently valid passcode from Duo, <tt>push</tt> or it could be set to <tt>phone</tt>. <tt>push</tt> will push the prompt to your smart device. <tt>phone</tt> will have duo call your phone number to approve.

===== OpenSSH =====
With OpenSSH (Linux or Mac command-line), to automatically set the Duo method to "push", use the command
<syntaxhighlight lang="bash">DUO_PASSCODE=push ssh -o SendEnv=DUO_PASSCODE headnode.beocat.ksu.edu</syntaxhighlight>

If you would like to put this in your ~/.ssh/config file, it will send the environment variable whenever it is set to Beocat upon connection:
Host headnode.beocat.ksu.edu
HostName headnode.beocat.ksu.edu
User YOUR_EID_GOES_HERE
SendEnv DUO_PASSCODE

From there you would simply do the following:
<syntaxhighlight lang=bash>
export DUO_PASSCODE=push
ssh headnode.beocat.ksu.edu
</syntaxhighlight>

===== PuTTY =====
In PuTTY to automatically set the Duo method to "push", expand "Connection" (if it isn't already), then click "Data". Under Environment variables, enter '''<tt>DUO_PASSCODE</tt>''' beside ''Variable'' and '''<tt>push</tt>''' beside ''Value''. Click the "Add" button and it will show up underneath. Be sure to go back to "Session" to save this change for PuTTY to remember this change.

===== MobaXTerm =====
There doesn't seem to be a way to send an environment variable in MobaXTerm, so you won't be able to set DUO_PASSCODE to an actual valid temporary key. To get MobaXterm to push automatically, you can edit your SSH session and on the "Advanced SSH Settings" tab, change the "Execute command" to <syntaxhighlight lang="bash">DUO_PASSCODE=push bash</syntaxhighlight>

==== Common issues ====
; Duo Pushes sometimes don't show up in a timely manner.
: If you open the Duo MFA application on your smart device when you're expecting an authentication challenge, the prompts seem to show up faster.
; MobaXTerm has excessive prompts for managing files.
: MobaXTerm has a sidebar browser for managing your files. Unfortunately, that sidebar browser initiates another SSH connection for every file transfer, which triggers a Duo push that you need to approve. MobaXTerm's dedicated SFTP Session doesn't have this same issue, it initiates a connection, keeps it open and re-uses it as needed, so you will have much fewer Duo approvals to respond to. If you choose to use the dedicated SFTP Session, you might consider disabling the sidebar file browser. "Advanced SSH settings" -> "SSH-browser type" -> "None"
; WinSCP has auto-reconnect enabled by default.
: Auto-reconnect is a useful function when actively transferring files, but if you have an idle session and the connection drops it will reconnect, sending you a Duo MFA prompt. If you don't approve it soon enough, WinSCP will attempt it again. Miss enough prompts and Duo will lock your account. It may be best to disable [https://winscp.net/eng/docs/ui_pref_resume reconnections during idle periods] if you do not wish be locked out of all services at K-State using Duo.
; FileZilla has auto-reconnect enabled by default.
: Auto-reconnect is a useful function when actively transferring files, but if you have an idle session and the connection drops it will reconnect, sending you a Duo MFA prompt. If you don't approve it soon enough, FileZilla will attempt it again. Miss enough prompts and Duo will lock your account. It may be best to disable timeouts and/or connection retries under the <tt>Edit -> Settings -> Connection</tt> menu if you do not wish to be locked out of all services at K-State using Duo.
; FileZilla has excessive prompts for managing files.
: Filezilla opens one connection for browsing the system. Transferring files opens 1-4 additional connections when the transfers start. Once they finish, those connections disconnect. If you start additional transfers, new connections will be opened. Every one of those connections must be approved through Duo MFA on your smart device. You can adjust the number of connections that FileZilla opens for transfers if you like. <tt>File -> Site Manager -> (choose the site you're changing) -> Transfer Settings -> Limit number of simultaneous connections</tt>.
: Another option is to disable processing the transfer queue, add the things to it you want to transfer and then re-enable the transfer queue. Then at least it will re-use the connections until the queue is empty.

== How do I compile my programs? ==
=== Serial programs ===
; Fortran
: <tt>ifort</tt> or <tt>gfortran</tt>
; C/C++
: <tt>icc</tt>, <tt>gcc</tt> and <tt>g++</tt>

=== Parallel programs ===
; Fortran
: <tt>mpif77</tt> or <tt>mpif90</tt>
; C/C++
: <tt>mpicc</tt> or <tt>mpic++</tt>

== Do Beocat jobs have a maximum Time Limit ==
Yes, there is a time limit, the scheduler will reject jobs longer than 28 days. The other side of that is that we reserve the right to a maintenance period every 14 days. Unless it is an emergency, we will give at least 2 weeks notice before these maintenance periods actually occur. Jobs 14 days or less that have started when we announce a maintenance period should be able to complete before it begins.

With that being said, there is no guarantee that any physical piece of hardware and the software that runs on it will behave for any significant length of time. Memory, processors, disk drives can all fail with little to no warning. Software may have bugs. We have had issues with the shared filesystem that resulted in several nodes losing connectivity and forced reboots. If you can, we always recommend that you write your jobs so that they can be resumed if they get interrupted.

{{Note|The 28 day limit can be overridden on a temporary and per-user basis provided there is enough justification|reminder|inline=1}}

== How are the filesystems on Beocat set up? ==

{| class="wikitable"
|-
! Mountpoint !! Local / Shared !! Size !! Filesystem !! Advice
|-
| /bulk || Shared || 3.1PB shared with /homes || cephfs || Slower than /homes; costs $45/TB/year
|-
| /homes || Shared || 3.1PB shared with /bulk || cephfs || Good enough for most jobs; limited to 1TB per home directory
|-
| /fastscratch || Shared || 280TB || nfs on top of ZFS || Faster than /homes or /bulk, built with all NVME disks; files not used in 30 days are automatically culled.
|-
| /tmp || Local || >100GB (varies per node) || XFS || Good for I/O intensive jobs. Unique per job, culled with the job finishes.
|-
|}
=== Usage Advice ===
For most jobs you shouldn't need to worry, your default working directory
is your homedir and it will be fast enough for most tasks.
I/O intensive work should use /tmp, but you will need to remember to copy
your files to and from this partition as part of your job script. This is made
easier through the <tt>$TMPDIR</tt> environment variable in your jobs.

Example usage of <tt>$TMPDIR</tt> in a job script
<syntaxhighlight lang="bash" line>
#!/bin/bash

#copy our input file to $TMPDIR to make processing faster
cp ~/experiments/input.data $TMPDIR

#use the input file we copied over to the local system
#generate the output file in $TMPDIR as well
~/bin/my_program --input-file=$TMPDIR/input.data --output-file=$TMPDIR/output.data

#copy the results back from $TMPDIR
cp $TMPDIR/output.data ~/experiments/results.$SLURM_JOB_ID
</syntaxhighlight>

You need to remember to copy over your data from <tt>$TMPDIR</tt> as part of your job.
That directory and its contents are deleted when the job is complete.

== What is "killable:1" or "killable:0" ==
On Beocat, some of the machines have been purchased by specific users and/or groups. These users and/or groups get guaranteed access to their machines at any point in time. Often, these machines are sitting idle because the owners have no need for it at the time. This would be a significant waste of computational power if there were no other way to make use of the computing cycles.

If you're wondering why a job may have the exit status of <tt>PREEMPTED</tt> from kstat or sacct, this is the reason.

=== Enter the "killable" resource ===
Killable (--gres=killable:1) jobs are jobs that can be scheduled to these "owned" machines by users outside of the true group of owners. If a "killable" job starts on one of these owned machines and the owner of said machine comes along and submits a job, the "killable" job will be returned to the queue, (killed off as it were), and restarted at some future point in time. The job will still complete at some future point, and if the job makes use of a checkpointing algorithm it may complete even faster. The trade off between marking a job "killable" and not, is that sometimes applications need a significant amount of runtime, and cannot resume running from a partial output, meaning that it may get restarted over and over again, never reaching the finish line. As such, we only auto-enable "killable" for relatively short jobs (<=168:00:00). Some users still feel this is a hindrance, so we created a way to tell us not to automatically mark short jobs "killable"

=== Disabling killable ===
Specifying --gres=killable:0 will tell us to not mark your job as killable.

=== The trade-off ===
If a job is marked killable, there are a non-trivial amount of additional nodes that the job can run on. If your job checkpoints itself, or is relatively short, there should be no downside to marking the job killable, as the job will probably start sooner. If your job is long-running and doesn't checkpoint (save its state to restart a previous session) itself, it could cause your job to take longer to complete.

== Help! When I submit my jobs I get "Warning To stay compliant with standard unix behavior, there should be a valid #! line in your script i.e. #!/bin/tcsh" ==
Job submission scripts are supposed to have a line similar to '<code>#!/bin/bash</code>' in them to start. We have had problems with people submitting jobs with invalid #! lines, so we enforce that rule. When this happens the job fails and we have to manually clean it up. The warning message is there just to inform you that the job script should have a line in it, in most cases #!/bin/tcsh or #!/bin/bash, to indicate what program should be used to run the script. When the line is missing from a script, by default your default shell is used to execute the script (in your case /usr/local/bin/tcsh). This works in most cases, but may not be what you are wanting.

== Help! When I submit my jobs I get "A #! line exists, but it is not pointing to an executable. Please fix. Job not submitted." ==
Like the above, error says you need a #!/bin/bash or similar line in your job script. This error says that while the line exists, the #! line isn't mentioning an executable file, thus the script will not be able to run. Most likely you wanted #!/bin/bash instead of something else.

== Help! My jobs keep dying after 1 hour and I don't know why ==
Beocat has default runtime limit of 1 hour. If you need more than that, or need more than 1 GB of memory per core, you'll want to look at the documentation [[SlurmBasics|here]] to see how to request it.

In short, when you run sbatch for your job, you'll want to put something along the lines of '<code>--time=0-10:00:00</code>' before the job script if you want your job to run for 10 hours.

== Help my error file has "Warning: no access to tty" ==
The warning message "Warning: no access to tty (Bad file descriptor)" is safe to ignore. It typically happens with the tcsh shell.

== Help! My job isn't going to finish in the time I specified. Can I change the time requirement? ==
Generally speaking, no.

Jobs are scheduled based on execution times (among other things). If it were easy to change your time requirement, one could submit a job with a 15-minute run-time, get it scheduled quickly, and then say "whoops - I meant 15 weeks", effectively gaming the job scheduler. In extreme circumstances and depending on the job requirements, we '''may''' be able to manually intervene. This process prevents other users from using the node(s) you are currently using, so are not routinely approved. Contact Beocat support (below) if you feel your circumstances warrant special consideration.

== Help! My perl job runs fine on the head node, but only runs for a few seconds and then quits when submitted to the queue. ==
Take a look at our documentation on [[Installed_software#Perl|Perl]]

== Help! When using mpi I get 'CMA: no RDMA devices found' or 'A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces' ==
This message simply means that some but not all nodes the job is running on have infiniband cards. The job will still run, but will not use the fastest interconnect we have available. This may or may not be an issue, depending on how message heavy your job is. If you would like to not see this warning, you may request infiniband as a resource when submitting your job. <code>--gres=fabric:ib:1</code>

== Help! when I use sbatch I get an error about line breaks ==
Beocat is a Linux system. Operating Systems use certain patterns of characters to indicate line breaks in their files. Linux and operating systems like it use '\n' as their line break character. Windows uses '\r\n' for their line breaks.

If you're getting an error that looks like this:
sbatch: error: Batch script contains DOS line breaks (\r\n)
sbatch: error: instead of expected UNIX line breaks (\n).

It means that your script is using the windows line endings. You can convert it with the <tt>dos2unix</tt> command
dos2unix myscript.sh

It would probably be beneficial for your editor to save the files with UNIX line breaks in the future.
* Visual Studio Code -- “Text Editor” > “Files” > “Eol”
* Notepad++ -- "Edit" > "EOL Conversion"

== Help! when logging into OnDemand I get a '400 Bad request' message ==
Unfortunately, there are some known issues with OnDemand and how it handles some of the complexities behind the scenes. This involves browser cookies that (occasionally) get too large and make it so you get these messages upon login.

The only work around is to clear your browser cookies (although you can limit it to simply clearing the ksu.edu ones).

Details for specific browsers are below

* [https://support.mozilla.org/en-US/kb/clear-cookies-and-site-data-firefox Firefox]
* [https://support.microsoft.com/en-us/microsoft-edge/delete-cookies-in-microsoft-edge-63947406-40ac-c3b8-57b9-2a946a29ae09 Edge]
* [https://support.google.com/chrome/answer/95647?sjid=1537101898131489753-NA#zippy=%2Cdelete-cookies-from-a-site Chrome]
* [https://support.apple.com/guide/safari/manage-cookies-sfri11471/mac Safari]
* If you are using some other browser, I would recommend searching google for <tt>$browsername clear site cookies</tt>

== Common Storage For Projects ==
Sometimes it is useful for groups of people to have a common storage area.

If you do not have a project, send a request to Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]. Note that these projects are generally reserved for tenure-track faculty and with a single project per eID.

If you already have a project you can do the following:

'''Note:''' The <tt>$group_name</tt> variable in the commands below needs to be replaced with the lower case name of your project. Membership of the projects can be done using our [[Group Management]] application.
* Create a directory in one of the home directories of someone in your group, ideally the project owner's.
** <tt>mkdir $directory</tt>
* Set the default permissions for new files and directories created in the directory:
** <tt>setfacl -d -m g:$group_name:rX -R $directory</tt>
* Set the permissions for the existing files and directories:
** <tt>setfacl -m g:$group_name:rX -R $directory</tt>

This will give people in your group the ability to read files in the 'share' directory. If you also want them to be able to write or modify files in that directory then use change the ':rX' to ':rwX' instead. e.g. 'setfacl -d -m g:$group_name:rwX -R $directory' for both setfacl commands. As with other permissions, the individuals will need access through every level of the directory hierarchy. [[LinuxBasics#Access_Control_Lists|It may be best to review our more in-depth topic on Access Control Lists.]]

== How do I get more help? ==
There are many sources of help for most Linux systems.

=== Unix man pages ===
Linux provides man pages (short for manual pages). These are simple enough to call, for example: if you need information on submitting jobs to Beocat, you can type '<code>man sbatch</code>'. This will bring up the manual for sbatch.

=== GNU info system ===
Not all applications have "man pages." Most of the rest have what they call info pages. For example, if you needed information on finding a file you could use '<code>info find</code>'.

=== This documentation ===
This documentation is very thoroughly researched, and has been painstakingly assembled for your benefit. Please use it.

=== Contact support ===
Support can be contacted [mailto:beocat@cis.ksu.edu here]. Please include detailed information about your problem, including the job number, applications you are trying to run, and the current directory that you are in.

SlurmBasics

2025-06-26T13:48:17Z

Nathanrwells: /* The Rocky/Slurm nodes */

== The Rocky/Slurm nodes ==

We have converted Beocat from CentOS Linux to Rocky Linux on April 1st of 2024. Any applications or libraries from the old system must be recompiled.

=== Using Modules ===

If you're using a common code that others may also be using, we may already have it compiled in a module. You can list the modules available and load an application as in the example below for Vasp.

eos> module avail
eos> module load GROMACS
eos> module list

When a module gets loaded, all the necessary libraries are also loaded and the paths to the libraries and executables are automatically set up. Loading GROMACS for example also loads the OpenMPI library needed to run it and adds the path to the MPI commands and Grimaces executables. To see how the path is set up, try executing which gmx. The module system allows you to easily switch between different version of applications, libraries, or languages as well.

If you are using a custom code or one that is not installed in a module, you'll need to recompile it yourself. This process is easier under CentOS as some of the work just involves loading the necessary set of modules. The first step is to decide whether to use the Intel compiler toolchain or the GNU toolchain, each of which includes the compilers and other math libraries. The module commands for each are below, and you can load these automatically when you log in by adding one of these module load statements to your .bashrc file. See /homes/daveturner/.bashrc as an example, where I put the module load statements .

To load the Intel compiler tool chain including the Intel Math Kernel Library (and OpenMPI):
icr-helios> module load iomkl

To load the GNU compiler tool chain including OpenMPI, OpenBLAS, FFTW, and ScalaPack load foss (free open source software):
icr-helios> module load foss

Modules provide an easy way to set up the compilers and libraries you may need to compile your code. Beyond that there are many different ways to compile codes so you'll just need to follow the directions. If you need help you can always contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket].

=== Submitting jobs to Slurm ===

You can submit your job script using the sbatch command.

icr-helios> sbatch sbatch_script.sh
icr-helios> kstat --me

This will submit the script and show you a list of your jobs that are running and the jobs you have in the queue. By default the output for each job will go into a slurm-###.out file where ### is the job ID number. If you need to kill a job, you can use the scancel command with the job ID number.

== Submitting your first job ==
To submit a job to run under Slurm, we use the sbatch (submit batch) command. The scheduler finds the optimum place for your job to run. With over 300 nodes and 7500 cores to schedule, as well as differing priorities, hardware, and individual resources, the scheduler's job is not trivial and it can take some time for a job to start even when there are empty nodes available.

There are a few things you'll need to know before running sbatch.
* How many cores you need. Note that unless your program is created to use multiple cores (called "threading"), asking for more cores will not speed up your job. This is a common misperception. '''Beocat will not magically make your program use multiple cores!''' For this reason the default is 1 core.
* How much time you need. Many users when beginning to use Beocat neglect to specify a time requirement. The default is one hour, and we get asked why their job died after one hour. We usually point them to the [[FAQ]].
* How much memory you need. The default is 1 GB. If your job uses significantly more than you ask, your job will be killed off.
* Any advanced options. See the [[AdvancedSlurm]] page for these requests. For our basic examples here, we will ignore these.

So let's now create a small script to test our ability to submit jobs. Create the following file (either by copying it to Beocat or by editing a text file and we'll name it <code>myhost.sh</code>. Both of these methods are documented on our [[LinuxBasics]] page.
<syntaxhighlight lang="bash" line>
#!/bin/sh
hostname
</syntaxhighlight>

Be sure to make it executable
chmod u+x myhost.sh

So, now lets submit it as a job and see what happens. Here I'm going to use five options
* <code>--mem-per-cpu=</code> tells how much memory I need. In my example, I'm using our system minimum of 512 MB, which is more than enough. Note that your memory request is '''per core''', which doesn't make much difference for this example, but will as you submit more complex jobs.
* <code>--time=</code> tells how much runtime I need. This can be in the form of "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". This is a very short job, so 1 minute should be plenty. This can't be changed after the job is started please make sure you have requested a sufficient amount of time.
* <code>--nodes=1</code> tells Slurm that this must be run on one machine. The [[AdvancedSlurm]] page has much more on the "nodes" switch.
* <code>--ntasks-per-node=16 </code> Request 16 cores on each node.
* <code>--constraint=moles</code> Request to only run on the Mole class of compute nodes.

% '''ls'''
myhost.sh
% '''sbatch --time=1 --mem-per-cpu=512M --cpus-per-task=1 --ntasks=1 --nodes=1 ./myhost.sh'''
salloc: Granted job allocation 1483446

Since this is such a small job, it is likely to be scheduled almost immediately, so a minute or so later, I now see
% '''ls'''
myhost.sh
slurm-1483446.out

% '''cat slurm-1483446.out'''
mage03

== Monitoring Your Job ==

The kstat perl script has been developed at K-State to provide you with all the available information about your jobs on Beocat. kstat --help will give you a full description of how to use it.

Eos> kstat --help

USAGE: kstat [-q] [-c] [-g] [-l] [-u user] [-p NaMD] [-j 1234567] [--part partition]
kstat alone dumps all info except for the core summaries
choose -q -c for only specific info on queued or core summaries.
then specify any searchables for the user, program name, or job id

kstat info on running and queued jobs
kstat -h list host info only, no jobs
kstat -q info on the queued jobs only
kstat -c core usage for each user
kstat -d # show jobs run in the last # days
Memory per node - used/allocated/requested
Red is close to or over requested amount
Yellow is under utilized for large jobs
kstat -g Only show GPU nodes
kstat -o Turner Only show info for a given owner
kstat -o CS_HPC Same but sub _ for spaces
kstat -l long list - node features and performance
Node hardware and node CPU usage
job nodelist and switchlist
job current and max memory
job CPU utilizations
kstat -u daveturner job info for one user only
kstat --me job info for my jobs only
kstat -j 1234567 info on a given job id
kstat --osg show OSG background jobs also
kstat --nocolor do not use any color
kstat --name display full names instead of eIDs

---------------- Graphs and Tables ---------------------------------------
Specify graph/table, CPU or GPU or host, usage or memory, and optional time
kstat --graph-cpu-memory # gnuplot CPU memory for job #
kstat --table-gpu-usage-5min # GPU usage table every 5 min for job #
kstat --table-cpu-60min # CPU usage, memory, swap table every 60 min for job #
kstat --table-node [nodename] cores, load, CPU usage, memory table for a node

--------------------------------------------------------------------------
Multi-node jobs are highlighted in Magenta
kstat -l also provides a node list and switch list
highlighted in Yellow when nodes are spread across multiple switches
Run time is colorized yellow then red for jobs nearing their time limit
Queue time is colorized yellow then red for jobs waiting longer times
--------------------------------------------------------------------------

kstat can be used to give you a summary of your jobs that are running and in the queue:
Eos> kstat --me


Hero43      
24 of 24 cores      
Load 23.4 / 24      
495.3 / 512 GB used 
    
daveturner      
unafold        1234567      
1 core               
running               
 4gb req                
 0 d 5 h 35 m 
    
daveturner      
octopus       1234568     
16 core              
running               
 128gb req                
 8 d 15 h 42 m 
 ################################## BeoCat Queue ################################### 
    
daveturner      
NetPIPE       1234569      
2 core    
 PD  
 2h  
 4gb req      
 0 d 1 h 2 m 


kstat produces a separate line for each host. Use kstat -h to see information on all hosts without the jobs.
For the example above we are listing our jobs and the hosts they are on.

Core usage - yellow for empty, red for empty on owned nodes, cyan for partially used, blue for all cores used. 
Load level - yellow or yellow background indicates the node is being inefficiently used. Red just means more threads than cores. 
Memory usage - yellow or red means most memory is used. 
If the node is owned the group name will be in orange on the right. Killable jobs can still be run on those nodes. 

Each job line will contain the username, program name, job ID, number of cores, the status which may be colored red for killable jobs,
the maximum memory used or memory requested, and the amount of time the job has run.
Jobs in the queue may contain information on the requested memory and run time, priority access, constraints, and
how long the job has been in the queue.
In this case, I have 2 jobs running on Hero43. unafold is using 1 core while octopus is using 16 cores. Slurm did not provide
any information on the actual memory use so the memory request is reported

=== Detailed information about a single job ===

kstat can provide a great deal of information on a particular job including a very rough estimate of when it will run. This time is a worst case scenario as this will
be adapted as other jobs finish early. This is a good way to check for job submission problems before contacting us. kstat colorizes the more important
information to make it easier to identify.

Eos> kstat -j 157054

################################## Beocat Queue ###################################
daveturner netpipe 157054 64 cores PD dwarves fabric CS HPC 8gb req 0 d 0 h 0 m

JobId 157054 Job Name netpipe
UserId=daveturner GroupId=daveturner_users(2117) MCS_label=N/A
Priority=11112 Nice=0 Account=ksu-cis-hpc QOS=normal
Status=PENDING Reason=Resources Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=00:40:00 TimeMin=N/A
SubmitTime=2018-02-02T18:18:31 EligibleTime=2018-02-02T18:18:31
Estimated Start Time is 2018-02-03T06:17:49 EndTime=2018-02-03T06:57:49 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partitions killable.q,ksu-cis-hpc.q AllocNode:Sid=eos:1761
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null) SchedNodeList=dwarf[01-02]
NumNodes=2-2 NumCPUs=64 NumTasks=64 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES 2 nodes 64 cores 8192 mem gres/fabric 2
Socks/Node=* NtasksPerN:B:S:C=32:0:*:* CoreSpec=*
MinCPUsNode=32 MinMemoryNode=4G MinTmpDiskNode=0
Constraint=dwarves DelayBoot=00:00:00
Gres=fabric Reservation=(null)
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Slurm script /homes/daveturner/perf/NetPIPE-5.x/sb.np
WorkDir=/homes/daveturner/perf/NetPIPE-5.x
StdErr=/homes/daveturner/perf/NetPIPE-5.x/0.o157054
StdIn=/dev/null
StdOut=/homes/daveturner/perf/NetPIPE-5.x/0.o157054
Switches=1@00:05:00
<syntaxhighlight lang=bash>
#!/bin/bash -l
#SBATCH --job-name=netpipe
#SBATCH -o 0.o%j
#SBATCH --time=0:40:00
#SBATCH --mem=4G
#SBATCH --switches=1
#SBATCH --nodes=2
#SBATCH --constraint=dwarves
#SBATCH --ntasks-per-node=32
#SBATCH --gres=fabric:roce:1

host=`echo $SLURM_JOB_NODELIST | sed s/[^a-z0-9]/\ /g | cut -f 1 -d ' '`
nprocs=$SLURM_NTASKS
openmpi_hostfile.pl $SLURM_JOB_NODELIST 1 hf.$host
opts="--printhostnames --quick --pert 3"

echo "*******************************************************************"
echo "Running on $SLURM_NNODES nodes $nprocs cores on nodes $SLURM_JOB_NODELIST"
echo "*******************************************************************"

mpirun -np 2 --hostfile hf.$host NPmpi $opts -o np.${host}.mpi
mpirun -np 2 --hostfile hf.$host NPmpi $opts -o np.${host}.mpi.bi --async --bidir
mpirun -np $nprocs NPmpi $opts -o np.${host}.mpi$nprocs --async --bidir
</syntaxhighlight>

=== Completed jobs and memory usage ===

kstat -d #

This will provide information on the jobs you have currently running and those that have completed
in the last '#' days. This is currently the only reliable way to get the memory used per node for your job.
This also provides information on whether the job completed normally, was canceled with scancel,
timed out, or was killed because it exceeded its memory request.

Eos> kstat -d 10

########################### sacct -u daveturner for 10 days ###########################
max gb used on a node / gb requested per node
193037 ADF dwarf43 1 n 32 c 30.46gb/100gb 05:15:34 COMPLETED
193289 ADF dwarf33 1 n 32 c 26.42gb/100gb 00:50:43 CANCELLED
195171 ADF dwarf44 1 n 32 c 56.81gb/120gb 14:43:35 COMPLETED
209518 matlab dwarf36 1 n 1 c 0.00gb/ 4gb 00:00:02 FAILED

=== Summary of core usage ===

kstat can also provide a listing of the core usage and cores requested for each user.
Eos> kstat -c

############################## Core usage ###############################
antariksh 1512 cores %25.1 used 41528 cores queued
bahadori 432 cores % 7.2 used 80 cores queued
eegoetz 0 cores % 0.0 used 2 cores queued
fahrialkan 24 cores % 0.4 used 32 cores queued
gowri 66 cores % 1.1 used 32 cores queued
jeffcomer 160 cores % 2.7 used 0 cores queued
ldcoates12 80 cores % 1.3 used 112 cores queued
lukesteg 464 cores % 7.7 used 0 cores queued
mike5454 1060 cores %17.6 used 852 cores queued
nilusha 344 cores % 5.7 used 0 cores queued
nnshan2014 136 cores % 2.3 used 0 cores queued
ploetz 264 cores % 4.4 used 60 cores queued
sadish 812 cores %13.5 used 0 cores queued
sandung 72 cores % 1.2 used 56 cores queued
zhiguang 80 cores % 1.3 used 688 cores queued

=== Producing memory and CPU utilization tables and graphs ===

kstat can now produce tables or graphs for the memory or CPU utilization
for a job. In order to view graphs you must set up X11 forwarding on your
ssh connection by using the -X parameter.

If you want to read more, continue on to our [[AdvancedSlurm]] page.

=== kstat is now available to download and install on other clusters ===

https://gitlab.beocat.ksu.edu/Admin-Public/kstat

This software has been installed and used on several clusters for many years.
It should be considered Beta software and may take some slight modifications
to install on some clusters. Please contact the author if you want to give
it a try (daveturner@ksu.edu).

AdvancedSlurm

2025-06-26T13:47:39Z

Nathanrwells: /* Manually configuring your ACLs */

== Resource Requests ==
Aside from the time, RAM, and CPU requirements listed on the [[SlurmBasics]] page, we have a couple other requestable resources:
Valid gres options are:
gpu[[:type]:count]
fabric[[:type]:count]
Generally, if you don't know if you need a particular resource, you should use the default. These can be generated with the command
<tt>srun --gres=help</tt>
=== Fabric ===
We currently offer 3 "fabrics" as request-able resources in Slurm. The "count" specified is the line-rate (in Gigabits-per-second) of the connection on the node.
==== Infiniband ====
First of all, let me state that just because it sounds "cool" doesn't mean you need it or even want it. InfiniBand does absolutely no good if running on a single machine. InfiniBand is a high-speed host-to-host communication fabric. It is (most-often) used in conjunction with MPI jobs (discussed below). Several times we have had jobs which could run just fine, except that the submitter requested InfiniBand, and all the nodes with InfiniBand were currently busy. In fact, some of our fastest nodes do not have InfiniBand, so by requesting it when you don't need it, you are actually slowing down your job. To request Infiniband, add <tt>--gres=fabric:ib:1</tt> to your sbatch command-line.
==== ROCE ====
ROCE, like InfiniBand is a high-speed host-to-host communication layer. Again, used most often with MPI. Most of our nodes are ROCE enabled, but this will let you guarantee the nodes allocated to your job will be able to communicate with ROCE. To request ROCE, add <tt>--gres=fabric:roce:1</tt> to your sbatch command-line.

==== Ethernet ====
Ethernet is another communication fabric. All of our nodes are connected by ethernet, this is simply here to allow you to specify the interconnect speed. Speeds are selected in units of Gbps, with all nodes supporting 1Gbps or above. The currently available speeds for ethernet are: <tt>1, 10, 40, and 100</tt>. To select nodes with 40Gbps and above, you could specify <tt>--gres=fabric:eth:40</tt> on your sbatch command-line. Since ethernet is used to connect to the file server, this can be used to select nodes that have fast access for applications doing heavy IO. The Dwarves and Heroes have 40 Gbps ethernet and we measure single stream performance as high as 20 Gbps, but if your application
requires heavy IO then you'd want to avoid the Moles which are connected to the file server with only 1 Gbps ethernet.

=== CUDA ===
[[CUDA]] is the resource required for GPU computing. 'kstat -g' will show you the GPU nodes and the jobs running on them. To request a GPU node, add <tt>--gres=gpu:1</tt> for example to request 1 GPU for your job; if your job uses multiple nodes, the number of GPUs requested is per-node. You can also request a given type of GPU (kstat -g -l to show types) by using <tt>--gres=gpu:geforce_gtx_1080_ti:1</tt> for a 1080Ti GPU on the Wizards or Dwarves, <tt>--gres=gpu:quadro_gp100:1</tt> for the P100 GPUs on Wizard20-21 that are best for 64-bit codes like Vasp. Most of these GPU nodes are owned by various groups. If you want access to GPU nodes and your group does not own any, we can add you to the <tt>--partition=ksu-gen-gpu.q</tt> group that has priority on Dwarf36-39. For more information on compiling CUDA code click on this [[CUDA]] link.

A listing of the current types of gpus can be gathered with this command:
<syntaxhighlight lang=bash>
scontrol show nodes | grep CfgTRES | tr ',' '\n' | awk -F '[:=]' '/gres\/gpu:/ { print $2 }' | sort -u
</syntaxhighlight>
At the time of this writing, that command produces this list:
* geforce_gtx_1080_ti
* geforce_rtx_2080_ti
* geforce_rtx_3090
* l40s
* quadro_gp100
* rtx_a4000
* rtx_a6000

== Parallel Jobs ==
There are two ways jobs can run in parallel, ''intra''node and ''inter''node. '''Note: Beocat will not automatically make a job run in parallel.''' Have I said that enough? It's a common misperception.
=== Intranode jobs ===
''Intra''node jobs run on many cores in the same node. These jobs can take advantage of many common libraries, such as [http://openmp.org/wp/ OpenMP], or any programming language that has the concept of ''threads''. Often, your program will need to know how many cores you want it to use, and many will use all available cores if not told explicitly otherwise. This can be a problem when you are sharing resources, as Beocat does. To request multiple cores, use the sbatch directives '<tt>--nodes=1 --cpus-per-task=n</tt>' or '<tt>--nodes=1 --ntasks-per-node=n</tt>', where ''n'' is the number of cores you wish to use. If your command can take an environment variable, you can use $SLURM_CPUS_ON_NODE to tell how many cores you've been allocated.

=== Internode (MPI) jobs ===
''Inter''node jobs can utilize many cores on one or more nodes. Communicating between nodes is trickier than talking between cores on the same node. The specification for doing so is called "[[wikipedia:Message_Passing_Interface|Message Passing Interface]]", or MPI. We have [http://www.open-mpi.org/ OpenMPI] installed on Beocat for this purpose. Most programs written to take advantage of large multi-node systems will use MPI, but MPI also allows an application to run on multiple cores within a node. You can tell if you have an MPI-enabled program because its directions will tell you to run '<tt>mpirun ''program''</tt>'. Requesting MPI resources is only mildly more difficult than requesting single-node jobs. Instead of using '<tt>--cpus-per-task=''n''</tt>', you would use '<tt>--nodes=''n'' --tasks-per-node=''m''</tt>' ''or'' '<tt>--nodes=''n'' --ntasks=''o''</tt>' for your sbatch request, where ''n'' is the number of nodes you want, ''m'' is the number of cores per node you need, and ''o'' is the total number of cores you need.

Some quick examples:

<tt>--nodes=6 --ntasks-per-node=4</tt> will give you 4 cores on each of 6 nodes for a total of 24 cores.

<tt>--ntasks=40</tt> will give you 40 cores spread across any number of nodes.

<tt>--nodes=10 --ntasks=100</tt> will give you a total of 100 cores across 10 nodes.

== Requesting memory for multi-core jobs ==
Memory requests are easiest when they are specified '''per core'''. For instance, if you specified the following: '<tt>--tasks=20 --mem-per-core=20G</tt>', your job would have access to 400GB of memory total.
== Other Handy Slurm Features ==
=== Email status changes ===
One of the most commonly used options when submitting jobs not related to resource requests is to have have Slurm email you when a job changes its status. This takes may need two directives to sbatch: <tt>--mail-user</tt> and <tt>--mail-type</tt>.
==== --mail-type ====
<tt>--mail-type</tt> is used to tell Slurm to notify you about certain conditions. Options are comma separated and include the following
{| class="wikitable"
!Option!!Explanation
|-
| NONE || This disables event-based mail
|-
| BEGIN || Sends a notification when the job begins
|-
| END || Sends a notification when the job ends
|-
| FAIL || Sends a notification when the job fails.
|-
| REQUEUE || Sends a notification if the job is put back into the queue from a running state
|-
| STAGE_OUT || Burst buffer stage out and teardown completed
|-
| ALL || Equivalent to BEGIN,END,FAIL,REQUEUE,STAGE_OUT
|-
| TIME_LIMIT || Notifies if the job ran out of time
|-
| TIME_LIMIT_90 || Notifies when the job has used 90% of its allocated time
|-
| TIME_LIMIT_80 || Notifies when the job has used 80% of its allocated time
|-
| TIME_LIMIT_50 || Notifies when the job has used 50% of its allocated time
|-
| ARRAY_TASKS || Modifies the BEGIN, END, and FAIL options to apply to each array task (instead of notifying for the entire job
|}

==== --mail-user ====
<tt>--mail-user</tt> is optional. It is only needed if you intend to send these job status updates to a different e-mail address than what you provided in the [https://acount.beocat.ksu.edu/user Account Request Page]. It is specified with the following arguments to sbatch: <tt>--mail-user=someone@somecompany.com</tt>

=== Job Naming ===
If you have several jobs in the queue, running the same script with different parameters, it's handy to have a different name for each job as it shows up in the queue. This is accomplished with the '<tt>-J ''JobName''</tt>' sbatch directive.

=== Separating Output Streams ===
Normally, Slurm will create one output file, containing both STDERR and STDOUT. If you want both of these to be separated into two files, you can use the sbatch directives '<tt>--output</tt>' and '<tt>--error</tt>'.

{| class="wikitable"
! option !! default !! example
|-
| --output || slurm-%j.out || slurm-206.out
|-
| --error || slurm-%j.out || slurm-206.out
|}
<tt>%j</tt> above indicates that it should be replaced with the job id.

=== Running from the Current Directory ===
By default, jobs run from your home directory. Many programs incorrectly assume that you are running the script from the current directory. You can use the '<tt>-cwd</tt>' directive to change to the "current working directory" you used when submitting the job.
=== Running in a specific class of machine ===
If you want to run on a specific class of machines, e.g., the Dwarves, you can add the flag "--constraint=dwarves" to select any of those machines.

=== Processor Constraints ===
Because Beocat is a heterogenous cluster (we have machines from many years in the cluster), not all of our processors support every new and fancy feature. You might have some applications that require some newer processor features, so we provide a mechanism to request those.

<tt>--contraint</tt> tells the cluster to apply constraints to the types of nodes that the job can run on. For instance, we know of several applications that must be run on chips that have "AVX" processor extensions. To do that, you would specify <tt>--constraint=avx</tt> on you ''<tt>sbatch</tt>'' '''or''' ''<tt>srun</tt>'' command lines.
Using <tt>--constraint=avx</tt> will prohibit your job from running on the Mages while <tt>--contraint=avx2</tt> will eliminate the Elves as well as the Mages.

=== Slurm Environment Variables ===
Within an actual job, sometimes you need to know specific things about the running environment to setup your scripts correctly. Here is a listing of environment variables that Slurm makes available to you. Of course the value of these variables will be different based on many different factors.
<syntaxhighlight lang="bash">
CUDA_VISIBLE_DEVICES=NoDevFiles
ENVIRONMENT=BATCH
GPU_DEVICE_ORDINAL=NoDevFiles
HOSTNAME=dwarf37
SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint
SLURM_CLUSTER_NAME=beocat
SLURM_CPUS_ON_NODE=1
SLURM_DISTRIBUTION=cyclic
SLURMD_NODENAME=dwarf37
SLURM_GTIDS=0
SLURM_JOB_CPUS_PER_NODE=1
SLURM_JOB_GID=163587
SLURM_JOB_ID=202
SLURM_JOBID=202
SLURM_JOB_NAME=slurm_simple.sh
SLURM_JOB_NODELIST=dwarf37
SLURM_JOB_NUM_NODES=1
SLURM_JOB_PARTITION=batch.q,killable.q
SLURM_JOB_QOS=normal
SLURM_JOB_UID=163587
SLURM_JOB_USER=mozes
SLURM_LAUNCH_NODE_IPADDR=10.5.16.37
SLURM_LOCALID=0
SLURM_MEM_PER_NODE=1024
SLURM_NNODES=1
SLURM_NODEID=0
SLURM_NODELIST=dwarf37
SLURM_NPROCS=1
SLURM_NTASKS=1
SLURM_PRIO_PROCESS=0
SLURM_PROCID=0
SLURM_SRUN_COMM_HOST=10.5.16.37
SLURM_SRUN_COMM_PORT=37975
SLURM_STEP_ID=0
SLURM_STEPID=0
SLURM_STEP_LAUNCHER_PORT=37975
SLURM_STEP_NODELIST=dwarf37
SLURM_STEP_NUM_NODES=1
SLURM_STEP_NUM_TASKS=1
SLURM_STEP_TASKS_PER_NODE=1
SLURM_SUBMIT_DIR=/homes/mozes
SLURM_SUBMIT_HOST=dwarf37
SLURM_TASK_PID=23408
SLURM_TASKS_PER_NODE=1
SLURM_TOPOLOGY_ADDR=due1121-prod-core-40g-a1,due1121-prod-core-40g-c1.due1121-prod-sw-100g-a9.dwarf37
SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node
SLURM_UMASK=0022
SRUN_DEBUG=3
TERM=screen-256color
TMPDIR=/tmp
USER=mozes
</syntaxhighlight>
Sometimes it is nice to know what hosts you have access to during a job. You would checkout the SLURM_JOB_NODELIST to know that. There are lots of useful Environment Variables there, I will leave it to you to identify the ones you want.

Some of the most commonly-used variables we see used are $SLURM_CPUS_ON_NODE, $HOSTNAME, and $SLURM_JOB_ID.

== Running from a sbatch Submit Script ==
No doubt after you've run a few jobs you get tired of typing something like 'sbatch -l mem=2G,h_rt=10:00 -pe single 8 -n MyJobTitle MyScript.sh'. How are you supposed to remember all of these every time? The answer is to create a 'submit script', which outlines all of these for you. Below is a sample submit script, which you can modify and use for your own purposes.
<syntaxhighlight lang="bash">
#!/bin/bash

## A Sample sbatch script created by Kyle Hutson
##
## Note: Usually a '#" at the beginning of the line is ignored. However, in
## the case of sbatch, lines beginning with #SBATCH are commands for sbatch
## itself, so I have taken the convention here of starting *every* line with a
## '#', just Delete the first one if you want to use that line, and then modify
## it to your own purposes. The only exception here is the first line, which
## *must* be #!/bin/bash (or another valid shell).

## There is one strict rule for guaranteeing Slurm reads all of your options:
## Do not put *any* lines above your resource requests that aren't either:
## 1) blank. (no other characters)
## 2) comments (lines must begin with '#')

## Specify the amount of RAM needed _per_core_. Default is 1G
##SBATCH --mem-per-cpu=1G

## Specify the maximum runtime in DD-HH:MM:SS form. Default is 1 hour (1:00:00)
##SBATCH --time=1:00:00

## Require the use of infiniband. If you don't know what this is, you probably
## don't need it.
##SBATCH --gres=fabric:ib:1

## GPU directive. If You don't know what this is, you probably don't need it
##SBATCH --gres=gpu:1

## number of cores/nodes:
## quick note here. Jobs requesting 16 or fewer cores tend to get scheduled
## fairly quickly. If you need a job that requires more than that, you might
## benefit from contacting Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]
## to see how we can assist in getting your
## job scheduled in a reasonable amount of time. Default is
##SBATCH --cpus-per-task=1
##SBATCH --cpus-per-task=12
##SBATCH --nodes=2 --tasks-per-node=1
##SBATCH --tasks=20

## Constraints for this job. Maybe you need to run on the elves
##SBATCH --constraint=elves
## or perhaps you just need avx processor extensions
##SBATCH --constraint=avx

## Output file name. Default is slurm-%j.out where %j is the job id.
##SBATCH --output=MyJobTitle.o%j

## Split the errors into a seperate file. Default is the same as output
##SBATCH --error=MyJobTitle.e%j

## Name my job, to make it easier to find in the queue
##SBATCH -J MyJobTitle

## Send email when certain criteria are met.
## Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to
## BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage
## out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent
## of time limit), TIME_LIMIT_80 (reached 80 percent of time limit),
## TIME_LIMIT_50 (reached 50 percent of time limit) and ARRAY_TASKS (send
## emails for each array task). Multiple type values may be specified in a
## comma separated list. Unless the ARRAY_TASKS option is specified, mail
## notifications on job BEGIN, END and FAIL apply to a job array as a whole
## rather than generating individual email messages for each task in the job
## array.
##SBATCH --mail-type=ALL

## Email address to send the email to based on the above line.
## Default is to send the mail to the e-mail address entered on the account
## request form.
##SBATCH --mail-user myemail@ksu.edu

## And finally, we run the job we came here to do.
## $HOME/ProgramDir/ProgramName ProgramArguments

## OR, for the case of MPI-capable jobs
## mpirun $HOME/path/MpiJobName
</syntaxhighlight>

== File Access ==
Beocat has a variety of options for storing and accessing your files.
Every user has a home directory for general use which is limited in size, has decent file access performance. Those needing more storage may purchase /bulk subdirectories which have the same decent performance
but are not backed up. The /fastscratch file system is a zfs host with lots of NVME drives provide much faster
temporary file access. When fast IO is critical to the application performance, access to /fastscratch, the local disk on each node, or to a
RAM disk are the best options.

===Home directory===

Every user has a <tt>/homes/''username''</tt> directory that they drop into when they log into Beocat.
The home directory is for general use and provides decent performance for most file IO.
Disk space in each home directory is limited to 1 TB, so larger files should be kept in a purchased /bulk
directory, and there is a limit of 100,000 files in each subdirectory in your account.
This file system is fully redundant, so 3 specific hard disks would need to fail before any data was lost.
All files will soon be backed up nightly to a separate file server in Nichols Hall, so if you do accidentally
delete something it can be recovered.

===Bulk directory===

Bulk data storage may be provided at a cost of $45/TB/year billed monthly. Due to the cost, directories will be provided when we are contacted and provided with payment information.

===Fast Scratch file system===

The /fastscratch file system is faster than /bulk or /homes.
In order to use fastscratch, you first need to make a directory for yourself.
Fast Scratch is meant as temporary space for prepositioning files and accessing them
during runs. Once runs are completed, any files that need to be kept should be moved to your home
or bulk directories since files on the fastscratch file system may get purged after 30 days.

<syntaxhighlight lang=bash>
mkdir /fastscratch/$USER
</syntaxhighlight>

===Local disk===

If you are running on a single node, it may also be faster to access your files from the local disk
on that node. Each job creates a subdirectory /tmp/job# where '#' is the job ID number on the
local disk of each node the job uses. This can be accessed simply by writing to /tmp rather than
needing to use /tmp/job#.

You may need to copy files to
local disk at the start of your script, or set the output directory for your application to point
to a file on the local disk, then you'll need to copy any files you want off the local disk before
the job finishes since Slurm will remove all files in your job's directory on /tmp on completion
of the job or when it aborts. Use 'kstat -l -h' to see how much /tmp space is available on each node.

<syntaxhighlight lang=bash>
# Copy input files to the tmp directory if needed
cp $input_files /tmp

# Make an 'out' directory to pass to the app if needed
mkdir /tmp/out

# Example of running an app and passing the tmp directory in/out
app -input_directory /tmp -output_directory /tmp/out

# Copy the 'out' directory back to the current working directory after the run
cp -rp /tmp/out .
</syntaxhighlight>

===RAM disk===

If you need ultrafast access to files, you can use a RAM disk which is a file system set up in the
memory of the compute node you are running on. The RAM disk is limited to the requested memory on that node, so you should account for this usage when you request
memory for your job. Below is an example of how to use the RAM disk.

<syntaxhighlight lang=bash>
# Copy input files over if necessary
cp $any_input_files /dev/shm/

# Run the application, possibly giving it the path to the RAM disk to use for output files
app -output_directory /dev/shm/

# Copy files from the RAM disk to the current working directory and clean it up
cp /dev/shm/* .
</syntaxhighlight>

===When you leave KSU===

If you are done with your account and leaving KSU, please clean up your directory, move any files
to your supervisor's account that need to be kept after you leave, and notify us so that we can disable your
account. The easiest way to move your files to your supervisor's account is for them to set up
a subdirectory for you with the appropriate write permissions. The example below shows moving
just a user's 'data' subdirectory to their supervisor. The 'nohup' command is used so that the move will
continue even if the window you are doing the move from gets disconnected.

<syntaxhighlight lang=bash>
# Supervisor:
mkdir /bulk/$USER/$STUDENT_USERNAME
setfacl -d -m u:$USER:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -m u:$USER:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -d -m u:$STUDENT_USERNAME:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -m u:$STUDENT_USERNAME:rwX -R /bulk/$USER/$STUDENT_USERNAME

# Student:
nohup mv /homes/$USER/data /bulk/$SUPERVISOR_USERNAME/$USER &

# Once the move is complete, the Supervisor should limit the permissions for the directory again by removing the student's access:
chown $USER: -R /bulk/$USER/$STUDENT_USERNAME
setfacl -d -x u:$STUDENT_USERNAME -R /bulk/$USER/$STUDENT_USERNAME
setfacl -x u:$STUDENT_USERNAME -R /bulk/$USER/$STUDENT_USERNAME
</syntaxhighlight>

==File Sharing==

This section will cover methods of sharing files with other users within Beocat and on remote systems.
In the past, Beocat users have been allowed to keep their
/homes and /bulk directories open so that any other user could
access files. In order to bring Beocat into alignment with
State of Kansas regulations and industry norms, all users must now have their /homes /bulk /scratch and /fastscratch directories
locked down from other users, but can still share files and directories within their group or with individual users
using group and individual ACLs (Access Control Lists) which will be explained below.
Beocat staff will be exempted from this
policy as we need to work freely with all users and will manage our
subdirectories to minimize access.

===Securing your home directory with the setacls script===

If you do not wish to share files or directories with other users, you do not need to do anything
as rwx access to others has already been removed.
If you want to share files or directories you can either use the '''setacls''' script or configure
the ACLs (Access Control Lists) manually.

The '''setacls -h''' will show how to use the script.

Eos: setacls -h
setacls [-r] [-w] [-g group] [-u user] -d /full/path/to/directory
Execute pemission will always be applied, you may also choose r or w
Must specify at least one group or user
Must specify at least one directory, and it must be the full path
Example: setacls -r -g ksu-cis-hpc -u mozes -d /homes/daveturner/shared_dir

You can specify the permissions to be either -r for read or -w for write or you can specify both.
You can provide a priority group to share with, which is the same as the group used in a --partition=
statement in a job submission script. You can also specify users.
You can specify a file or a directory to share. If the directory is specified then all files in that
directory will also be shared, and all files created in the directory laster will also be shared.

The script will set everything up for you, telling you the commands it is executing along the way,
then show the resulting ACLs at the end with the '''getfacl''' command. Below is an example of
sharing the directory '''test_directory''' in my /bulk/daveturner directory with Nathan.

<syntaxhighlight>
Beocat> cd /bulk/daveturner
Beocat> mkdir test_directory
Beocat> setacls -r -w -u nathanrwells -d /bulk/daveturner/test_directory

Opening up base directory /bulk/daveturner with X execute permission only
setfacl -m u:nathanrwells:X /bulk/daveturner

Setting Xrw for directory/file /bulk/daveturner/test_directory
setfacl -m u:nathanrwells:Xrw -R /bulk/daveturner/test_directory
setfacl -d -m u:nathanrwells:Xrw -R /bulk/daveturner/test_directory

The ACLs on directory /bulk/daveturner/test_directory are set to:

getfacl: Removing leading '/' from absolute path names
# file: bulk/daveturner/test_directory
USER daveturner rwx rwx
user nathanrwells rwx rwx
GROUP daveturner_users r-x r-x
group beocat_support r-x r-x
mask rwx rwx
other --- ---
</syntaxhighlight>

The '''getfacl''' run by the script now shows that user '''nathanrwells''' has
read and write permissions to that directory and execute access to all directories
leading up to it.

====Manually configuring your ACLs====

If you want to manually configure the ACLs you can use the directions below to do what the '''setacls'''
script would do for you.
You first need to provide the minimum execute access to your /homes
or /bulk directory before sharing individual subdirectories. Setting the ACL to execute only will allow those
in your group to get access to subdirectories while not including read access will mean they will not
be able to see other files or subdirectories on your main directory, but do keep in mind that they can still access them
so you may want to still lock them down manually. Below is an example of how I would change my
/homes/daveturner directory to allow ksu-cis-hpc group execute access.

<syntaxhighlight lang=bash>
setfacl -m g:ksu-cis-hpc:X /homes/daveturner
</syntaxhighlight>

If your research group owns any nodes on Beocat, then you have a group name that can be used to securely share
files with others within your group. Below is an example of creating a directory called 'share_hpc',
then providing access to my ksu-cis.hpc group
(my group is ksu-cis-hpc so I submit jobs to --partition=ksu-cis-hpc.q).
Using -R will make these changes recursively to all files and directories in that subdirectory while changing the defaults with the setfacl -d command will ensure that files and directories created
later will be done so with these same ACLs.

<syntaxhighlight lang=bash>
mkdir share_hpc
# ACLs are used here for setting default permissions
setfacl -d -m g:ksu-cis-hpc:rX -R share_hpc
# ACLs are used here for setting actual permissions
setfacl -m g:ksu-cis-hpc:rX -R share_hpc
</syntaxhighlight>

This will give people in your group the ability to read files in the 'share_hpc' directory. If you also want
them to be able to write or modify files in that directory then change the ':rX' to ':rwX' instead. e.g. 'setfacl -d -m g:ksu-cis-hpc:rwX -R share_hpc'

If you want to know what groups you belong to use the line below.
<syntaxhighlight lang=bash>
groups
</syntaxhighlight>
If your group does not own any nodes, you can still request a group name and manage the participants yourself
by emailing us at
or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or emailing us at beocat@cs.ksu.edu
.
If you want to share a directory with only a few people you can manage your ACLs using individual usernames
instead of with a group.

You can use the '''getfacl''' command to see groups have access to a given directory.

<syntaxhighlight lang=bash>
getfacl share_hpc

# file: share_hpc
# owner: daveturner
# group: daveturner_users
user::rwx
group::r-x
group:ksu-cis-hpc:r-x
mask::r-x
other::---
default:user::rwx
default:group::r-x
default:group:ksu-cis-hpc:r-x
default:mask::r-x
default:other::---
</syntaxhighlight>

ACLs give you great flexibility in controlling file access at the
group level. Below is a more advanced example where I set up a directory to be shared with
my ksu-cis-hpc group, Dan's ksu-cis-dan group, and an individual user 'mozes' who I also want
to have write access.

<syntaxhighlight lang=bash>
mkdir share_hpc_dan_mozes
# acls are used here for setting default permissions
setfacl -d -m g:ksu-cis-hpc:rX -R share_hpc_dan_mozes
setfacl -d -m g:ksu-cis-dan:rX -R share_hpc_dan_mozes
setfacl -d -m u:mozes:rwX -R share_hpc_dan_mozes
# ACLs are used here for setting actual permissions
setfacl -m g:ksu-cis-hpc:rX -R share_hpc_dan_mozes
setfacl -m g:ksu-cis-dan:rX -R share_hpc_dan_mozes
setfacl -m u:mozes:rwX -R share_hpc_dan_mozes

getfacl share_hpc_dan_mozes

# file: share_hpc_dan_mozes
# owner: daveturner
# group: daveturner_users
user::rwx
user:mozes:rwx
group::r-x
group:ksu-cis-hpc:r-x
group:ksu-cis-dan:r-x
mask::r-x
other::---
default:user::rwx
default:user:mozes:rwx
default:group::r-x
default:group:ksu-cis-hpc:r-x
default:group:ksu-cis-dan:r-x
default:mask::r-x
default:other::--x
</syntaxhighlight>

===Openly sharing files on the web===

If you create a 'public_html' directory on your home directory, then any files put there will be shared
openly on the web. There is no way to restrict who has access to those files.

<syntaxhighlight lang=bash>
cd
mkdir public_html
# Opt-in to letting the webserver access your home directory:
setfacl -m g:public_html:x ~/
</syntaxhighlight>

Then access the data from a web browser using the URL:

http://people.beocat.ksu.edu/~your_user_name

This will show a list of the files you have in your public_html subdirectory.

===Globus===

We have a page here dedicated to [[Globus]]

== Array Jobs ==
One of Slurm's useful options is the ability to run "Array Jobs"

It can be used with the following option to sbatch.

--array=n[-m[:s]]
Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Slurm
almost like a series of jobs. The option argument to --array specifies the number of array job tasks and the index number which will be
associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SLURM_ARRAY_TASK_ID. The option
arguments n, and m will be available through the environment variables SLURM_ARRAY_TASK_MIN and SLURM_ARRAY_TASK_MAX.

The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size.
Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SLURM_ARRAY_TASK_ID containing one of the 5 index numbers.

Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The
number of tasks in a array job is unlimited.

STDOUT and STDERR of array job tasks follow a slightly different naming convention (which can be controlled in the same way as mentioned above).

slurm-%A_%a.out

%A is the SLURM_ARRAY_JOB_ID, and %a is the SLURM_ARRAY_TASK_ID

=== Examples ===
==== Change the Size of the Run ====
Array Jobs have a variety of uses, one of the easiest to comprehend is the following:

I have an application, app1 I need to run the exact same way, on the same data set, with only the size of the run changing.

My original script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
RUNSIZE=50
#RUNSIZE=100
#RUNSIZE=150
#RUNSIZE=200
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
For every run of that job I have to change the RUNSIZE variable, and submit each script. This gets tedious.

With Array Jobs the script can be written like so:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=50-200:50
RUNSIZE=$SLURM_ARRAY_TASK_ID
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
I then submit that job, and Slurm understands that it needs to run it 4 times, once for each task. It also knows that it can and should run these tasks in parallel.

==== Choosing a Dataset ====
A slightly more complex use of Array Jobs is the following:

I have an application, app2, that needs to be run against every line of my dataset. Every line changes how app2 runs slightly, but I need to compare the runs against each other.

Originally I had to take each line of my dataset and generate a new submit script and submit the job. This was done with yet another script:

<syntaxhighlight lang="bash">
#!/bin/bash
DATASET=dataset.txt
scriptnum=0
while read LINE
do
echo "app2 $LINE" > ${scriptnum}.sh
sbatch ${scriptnum}.sh
scriptnum=$(( $scriptnum + 1 ))
done < $DATASET
</syntaxhighlight>
Not only is this needlessly complex, it is also slow, as sbatch has to verify each job as it is submitted. This can be done easily with array jobs, as long as you know the number of lines in the dataset. This number can be obtained like so: wc -l dataset.txt in this case lets call it 5000.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=1-5000
app2 `sed -n "${SLURM_ARRAY_TASK_ID}p" dataset.txt`
</syntaxhighlight>
This uses a subshell via `, and has the sed command print out only the line number $SLURM_ARRAY_TASK_ID out of the file dataset.txt.

Not only is this a smaller script, it is also faster to submit because it is one job instead of 5000, so sbatch doesn't have to verify as many.

To give you an idea about time saved: submitting 1 job takes 1-2 seconds. by extension if you are submitting 5000, that is 5,000-10,000 seconds, or 1.5-3 hours.

== Checkpoint/Restart using DMTCP ==

DMTCP is Distributed Multi-Threaded CheckPoint software that will checkpoint your application without modification, and
can be set up to automatically restart your job from the last checkpoint if for example the node you are running on fails.
This has been tested successfully
on Beocat for some scalar and OpenMP codes, but has failed on all MPI tests so far. We would like to encourage users to
try DMTCP out if their non-MPI jobs run longer than 24 hours. If you want to try this, please contact us first since we are still
experimenting with DMTCP.

The sample job submission script below shows how dmtcp_launch is used to start the application, then dmtcp_restart is used to start from a checkpoint if the job has failed and been rescheduled.
If you are putting this in an array script, then add the Slurm array task ID to the end of the ckeckpoint directory name
like ckptdir=ckpt-$SLURM_ARRAY_TASK_ID.

#!/bin/bash -l
#SBATCH --job-name=gromacs
#SBATCH --mem=50G
#SBATCH --time=24:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module reset
module load GROMACS/2016.4-foss-2017beocatb-hybrid
module load DMTCP
module list

ckptdir=ckpt
mkdir -p $ckptdir
export DMTCP_CHECKPOINT_DIR=$ckptdir

if ! ls -1 $ckptdir | grep -c dmtcp_restart_script > /dev/null
then
echo "Using dmtcp_launch to start the app the first time"
dmtcp_launch --no-coordinator mpirun -np 1 -x OMP_NUM_THREADS=4 gmx_mpi mdrun -nsteps 50000 -ntomp 4 -v -deffnm 1ns -c 1ns.pdb -nice 0
else
echo "Using dmtcp_restart from $ckptdir to continue from a checkpoint"
dmtcp_restart $ckptdir/*.dmtcp
fi

You will need to run several tests to verify that DMTCP is working properly with your application.
First, run a short test without DMTCP and another with DMTCP with the checkpoint interval set to 5 minutes
by adding the line export DMTCP_CHECKPOINT_INTERVAL=300 to your script. Then use kstat -d 1 to
check that the memory in both runs is close to the same. Also use this information to calculate the time
that each checkpoint takes. In most cases I've seen times less than a minute for checkpointing that will normally
be done once each hour. If your application is taking more time, let us know. Sometimes this can be sped up
by simply turning off compression by adding the line export DMTCP_GZIP=0. Make sure to remove the
line where you set the checkpoint interval to 300 seconds so that the default time of once per hour will be used.

After verifying that your code completes using DMTCP and does not take significantly more time or memory, you
will need to start a run then scancel it after the first checkpoint, then resubmit the same script to make
sure that it restarts and runs to completion. If you are working with an array job script, the last is to try a few
array tasks at once to make sure there is no conflict between the jobs.

== Running jobs interactively ==
Some jobs just don't behave like we think they should, or need to be run with somebody sitting at the keyboard and typing in response to the output the computers are generating. Beocat has a facility for this, called 'srun'. srun uses the exact same command-line arguments as sbatch, but you need to add the following arguments at the end: <tt>--pty bash</tt>. If no node is available with your resource requirements, srun will tell you something like the following:
srun --pty bash
srun: Force Terminated job 217
srun: error: CPU count per node can not be satisfied
srun: error: Unable to allocate resources: Requested node configuration is not available
Note that, like sbatch, your interactive job will timeout after your allotted time has passed.

== Connecting to an existing job ==
You can connect to an existing job using srun in the same way that the MonitorNode command
allowed us to in the old cluster. This is essentially like using ssh to get into the node where your job is running which
can be very useful in allowing you to look at files in /tmp/job# or in running htop to view the
activity level for your job.

srun --jobid=# --pty bash where '#' is the job ID number

== Altering Job Requests ==
We generally do not support users to modify job parameters once the job has been submitted. It can be done, but there are numerous catches, and all of the variations can be a bit problematic; it is normally easier to simply delete the job (using '''scancel ''jobid''''') and resubmit it with the right parameters. '''If your job doesn't start after modifying such parameters (after a reasonable amount of time), delete the job and resubmit it.'''

As it is unsupported, this is an excercise left to the reader. A starting point is <tt>man scontrol</tt>
== Killable jobs ==
There are a growing number of machines within Beocat that are owned by a particular person or group. Normally jobs from users that aren't in the group designated by the owner of these machines cannot use them. This is because we have guaranteed that the nodes will be accessible and available to the owner at any given time. We will allow others to use these nodes if they designate their job as "killable." If your job is designated as killable, your job will be able to use these nodes, but can (and will) be killed off at any point in time to make way for the designated owner's jobs. Jobs that are marked killable will be re-queued and may restart on another node.

The way you would designate your job as killable is to add <tt>--gres=killable:1</tt> to the '''<tt>sbatch</tt> or <tt>srun</tt>''' arguments. This could be either on the command-line or in your script file.

''Note: This is a submit-time only request, it cannot be added by a normal user after the job has been submitted.'' If you would like jobs modified to be '''killable''' after the jobs have been submitted (and it is too much work to <tt>scancel</tt> the jobs and re-submit), send an e-mail to the administrators detailing the job ids and what you would like done.

== Scheduling Priority ==
Some users are members of projects that have contributed to Beocat. When those users have contributed nodes, the group gets access to a "partition" giving you priority on those nodes.

In most situations, the scheduler will automatically add those priority partitions to the jobs as submitted. You should not need to include a partition list in your job submission.

There are currently just a few exceptions that we will not automatically add:
* ksu-chem-mri.q
* ksu-gen-gpu.q
* ksu-gen-highmem.q

If you have access to those any of the non-automatic partitions, and have need of the resources in that partition, you can then alter your <tt>#SBATCH</tt> lines to include your new partition:
#SBATCH --partition=ksu-gen-highmem.q

Otherwise, you shouldn't modify the partition line at all unless you really know what you're doing.

== Graphical Applications ==
Some applications are graphical and need to have some graphical input/output. We currently accomplish this with X11 forwarding or [[OpenOnDemand]]
=== OpenOnDemand ===
[[OpenOnDemand]] is likely the easier and more performant way to run a graphical application on the cluster.
# visit [https://ondemand.beocat.ksu.edu/ ondemand] and login with your cluster credentials.
# Check the "Interactive Apps" dropdown. We may have a workflow ready for you. If not choose the desktop.
# Select the resources you need
# Select launch
# A job is now submitted to the cluster and once the job is started you'll see a Connect button
# use the app as needed. If using the desktop, start your graphical application.

=== X11 Forwarding ===
==== Connecting with an X11 client ====
===== Windows =====
If you are running Windows, we recommend MobaXTerm as your file/ssh manager, this is because it is one relatively simple tool to do everything. MobaXTerm also automatically connects with X11 forwarding enabled.
===== Linux/OSX =====
Both Linux and OSX can connect in an X11 forwarding mode. Linux will have all of the tools you need installed already, OSX will need [https://www.xquartz.org/ XQuartz] installed.

Then you will need to change your 'ssh' command slightly:

ssh -Y eid@headnode.beocat.ksu.edu

The '''-Y''' argument tells ssh to setup X11 forwarding.
==== Starting an Graphical job ====
All graphical jobs, by design, must be interactive, so we'll use the srun command. On a headnode, we run the following:
# load an X11 enabled application
module load Octave
# start an X11 job, sbatch arguments are accepted for srun as well, 1 node, 1 hour, 1 gb of memory
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 octave --gui

Because these jobs are interactive, they may not be able to run at all times, depending on how busy the scheduler is at any point in time. '''--pty --x11''' are required arguments setting up the job, and '''octave --gui''' is the command to run inside the job.

== Job Accounting ==
Some people may find it useful to know what their job did during its run. The sacct tool will read Slurm's accounting database and give you summarized or detailed views on jobs that have run within Beocat.
=== sacct ===
This data can usually be used to diagnose two very common job failures.
==== Job debugging ====
It is simplest if you know the job number of the job you are trying to get information on.
<syntaxhighlight lang="bash">
# if you know the jobid, put it here:
sacct -j 1122334455 -l
# if you don't know the job id, you can look at your jobs started since some day:
sacct -S 2017-01-01
</syntaxhighlight>

===== My job didn't do anything when it ran! =====
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|-
|3
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|218||218||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||12||00:00:00||FAILED||2:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=12,mem=1G,node=1||cpu=12,mem=1G,node=1
|-
|218.batch||218.batch||batch||||137940K||dwarf37||0||137940K||1576K||dwarf37||0||1576K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||1.36G||0||0||0||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|-
|218.0||218.0||qqqqstat||||204212K||dwarf37||0||204212K||1420K||dwarf37||0||1420K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||196.52M||Unknown||Unknown||Unknown||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|}</div> 
If you look at the columns showing Elapsed and State, you can see that they show 00:00:00 and FAILED respectively. This means that the job started and then promptly ended. This points to something being wrong with your submission script. Perhaps there is a typo somewhere in it.

===== My job ran but didn't finish! =====
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|-
|3
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|220||220||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:01:27||TIMEOUT||0:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=1,mem=1G,node=1||cpu=1,mem=1G,node=1
|-
|220.batch||220.batch||batch||||370716K||dwarf37||0||370716K||7060K||dwarf37||0||7060K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:28||CANCELLED||0:15||1.23G||0||0||0||1Gn||0||0.16M||dwarf37||0||0.16M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|-
|220.0||220.0||sleep||||204212K||dwarf37||0||107916K||1000K||dwarf37||0||620K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:27||CANCELLED||0:15||1.54G||Unknown||Unknown||Unknown||1Gn||0||0.05M||dwarf37||0||0.05M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|}</div> 
If you look at the column showing State, we can see some pointers to the issue. The job ran out of time (TIMEOUT) and then was killed (CANCELLED).
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|221||221||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:00:00||CANCELLED by 0||0:0||||Unknown||Unknown||Unknown||1Mn||||||||||||||||||||||||cpu=1,mem=1M,node=1||cpu=1,mem=1M,node=1
|-
|221.batch||221.batch||batch||||137940K||dwarf37||0||137940K||1144K||dwarf37||0||1144K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:00:01||CANCELLED||0:15||2.62G||0||0||0||1Mn||0||0||dwarf37||65534||0||0||dwarf37||65534||0||||||||cpu=1,mem=1M,node=1
|}</div> 
If you look at the column showing State, we see it was "CANCELLED by 0", then we look at the AllocTRES column to see our allocated resources, and see that 1MB of memory was granted. Combine that with the column "MaxRSS" and we see that the memory granted was less than the memory we tried to use, thus the job was "CANCELLED".

AdvancedSlurm

2025-06-26T13:46:48Z

Nathanrwells: /* Running from a sbatch Submit Script */

== Resource Requests ==
Aside from the time, RAM, and CPU requirements listed on the [[SlurmBasics]] page, we have a couple other requestable resources:
Valid gres options are:
gpu[[:type]:count]
fabric[[:type]:count]
Generally, if you don't know if you need a particular resource, you should use the default. These can be generated with the command
<tt>srun --gres=help</tt>
=== Fabric ===
We currently offer 3 "fabrics" as request-able resources in Slurm. The "count" specified is the line-rate (in Gigabits-per-second) of the connection on the node.
==== Infiniband ====
First of all, let me state that just because it sounds "cool" doesn't mean you need it or even want it. InfiniBand does absolutely no good if running on a single machine. InfiniBand is a high-speed host-to-host communication fabric. It is (most-often) used in conjunction with MPI jobs (discussed below). Several times we have had jobs which could run just fine, except that the submitter requested InfiniBand, and all the nodes with InfiniBand were currently busy. In fact, some of our fastest nodes do not have InfiniBand, so by requesting it when you don't need it, you are actually slowing down your job. To request Infiniband, add <tt>--gres=fabric:ib:1</tt> to your sbatch command-line.
==== ROCE ====
ROCE, like InfiniBand is a high-speed host-to-host communication layer. Again, used most often with MPI. Most of our nodes are ROCE enabled, but this will let you guarantee the nodes allocated to your job will be able to communicate with ROCE. To request ROCE, add <tt>--gres=fabric:roce:1</tt> to your sbatch command-line.

==== Ethernet ====
Ethernet is another communication fabric. All of our nodes are connected by ethernet, this is simply here to allow you to specify the interconnect speed. Speeds are selected in units of Gbps, with all nodes supporting 1Gbps or above. The currently available speeds for ethernet are: <tt>1, 10, 40, and 100</tt>. To select nodes with 40Gbps and above, you could specify <tt>--gres=fabric:eth:40</tt> on your sbatch command-line. Since ethernet is used to connect to the file server, this can be used to select nodes that have fast access for applications doing heavy IO. The Dwarves and Heroes have 40 Gbps ethernet and we measure single stream performance as high as 20 Gbps, but if your application
requires heavy IO then you'd want to avoid the Moles which are connected to the file server with only 1 Gbps ethernet.

=== CUDA ===
[[CUDA]] is the resource required for GPU computing. 'kstat -g' will show you the GPU nodes and the jobs running on them. To request a GPU node, add <tt>--gres=gpu:1</tt> for example to request 1 GPU for your job; if your job uses multiple nodes, the number of GPUs requested is per-node. You can also request a given type of GPU (kstat -g -l to show types) by using <tt>--gres=gpu:geforce_gtx_1080_ti:1</tt> for a 1080Ti GPU on the Wizards or Dwarves, <tt>--gres=gpu:quadro_gp100:1</tt> for the P100 GPUs on Wizard20-21 that are best for 64-bit codes like Vasp. Most of these GPU nodes are owned by various groups. If you want access to GPU nodes and your group does not own any, we can add you to the <tt>--partition=ksu-gen-gpu.q</tt> group that has priority on Dwarf36-39. For more information on compiling CUDA code click on this [[CUDA]] link.

A listing of the current types of gpus can be gathered with this command:
<syntaxhighlight lang=bash>
scontrol show nodes | grep CfgTRES | tr ',' '\n' | awk -F '[:=]' '/gres\/gpu:/ { print $2 }' | sort -u
</syntaxhighlight>
At the time of this writing, that command produces this list:
* geforce_gtx_1080_ti
* geforce_rtx_2080_ti
* geforce_rtx_3090
* l40s
* quadro_gp100
* rtx_a4000
* rtx_a6000

== Parallel Jobs ==
There are two ways jobs can run in parallel, ''intra''node and ''inter''node. '''Note: Beocat will not automatically make a job run in parallel.''' Have I said that enough? It's a common misperception.
=== Intranode jobs ===
''Intra''node jobs run on many cores in the same node. These jobs can take advantage of many common libraries, such as [http://openmp.org/wp/ OpenMP], or any programming language that has the concept of ''threads''. Often, your program will need to know how many cores you want it to use, and many will use all available cores if not told explicitly otherwise. This can be a problem when you are sharing resources, as Beocat does. To request multiple cores, use the sbatch directives '<tt>--nodes=1 --cpus-per-task=n</tt>' or '<tt>--nodes=1 --ntasks-per-node=n</tt>', where ''n'' is the number of cores you wish to use. If your command can take an environment variable, you can use $SLURM_CPUS_ON_NODE to tell how many cores you've been allocated.

=== Internode (MPI) jobs ===
''Inter''node jobs can utilize many cores on one or more nodes. Communicating between nodes is trickier than talking between cores on the same node. The specification for doing so is called "[[wikipedia:Message_Passing_Interface|Message Passing Interface]]", or MPI. We have [http://www.open-mpi.org/ OpenMPI] installed on Beocat for this purpose. Most programs written to take advantage of large multi-node systems will use MPI, but MPI also allows an application to run on multiple cores within a node. You can tell if you have an MPI-enabled program because its directions will tell you to run '<tt>mpirun ''program''</tt>'. Requesting MPI resources is only mildly more difficult than requesting single-node jobs. Instead of using '<tt>--cpus-per-task=''n''</tt>', you would use '<tt>--nodes=''n'' --tasks-per-node=''m''</tt>' ''or'' '<tt>--nodes=''n'' --ntasks=''o''</tt>' for your sbatch request, where ''n'' is the number of nodes you want, ''m'' is the number of cores per node you need, and ''o'' is the total number of cores you need.

Some quick examples:

<tt>--nodes=6 --ntasks-per-node=4</tt> will give you 4 cores on each of 6 nodes for a total of 24 cores.

<tt>--ntasks=40</tt> will give you 40 cores spread across any number of nodes.

<tt>--nodes=10 --ntasks=100</tt> will give you a total of 100 cores across 10 nodes.

== Requesting memory for multi-core jobs ==
Memory requests are easiest when they are specified '''per core'''. For instance, if you specified the following: '<tt>--tasks=20 --mem-per-core=20G</tt>', your job would have access to 400GB of memory total.
== Other Handy Slurm Features ==
=== Email status changes ===
One of the most commonly used options when submitting jobs not related to resource requests is to have have Slurm email you when a job changes its status. This takes may need two directives to sbatch: <tt>--mail-user</tt> and <tt>--mail-type</tt>.
==== --mail-type ====
<tt>--mail-type</tt> is used to tell Slurm to notify you about certain conditions. Options are comma separated and include the following
{| class="wikitable"
!Option!!Explanation
|-
| NONE || This disables event-based mail
|-
| BEGIN || Sends a notification when the job begins
|-
| END || Sends a notification when the job ends
|-
| FAIL || Sends a notification when the job fails.
|-
| REQUEUE || Sends a notification if the job is put back into the queue from a running state
|-
| STAGE_OUT || Burst buffer stage out and teardown completed
|-
| ALL || Equivalent to BEGIN,END,FAIL,REQUEUE,STAGE_OUT
|-
| TIME_LIMIT || Notifies if the job ran out of time
|-
| TIME_LIMIT_90 || Notifies when the job has used 90% of its allocated time
|-
| TIME_LIMIT_80 || Notifies when the job has used 80% of its allocated time
|-
| TIME_LIMIT_50 || Notifies when the job has used 50% of its allocated time
|-
| ARRAY_TASKS || Modifies the BEGIN, END, and FAIL options to apply to each array task (instead of notifying for the entire job
|}

==== --mail-user ====
<tt>--mail-user</tt> is optional. It is only needed if you intend to send these job status updates to a different e-mail address than what you provided in the [https://acount.beocat.ksu.edu/user Account Request Page]. It is specified with the following arguments to sbatch: <tt>--mail-user=someone@somecompany.com</tt>

=== Job Naming ===
If you have several jobs in the queue, running the same script with different parameters, it's handy to have a different name for each job as it shows up in the queue. This is accomplished with the '<tt>-J ''JobName''</tt>' sbatch directive.

=== Separating Output Streams ===
Normally, Slurm will create one output file, containing both STDERR and STDOUT. If you want both of these to be separated into two files, you can use the sbatch directives '<tt>--output</tt>' and '<tt>--error</tt>'.

{| class="wikitable"
! option !! default !! example
|-
| --output || slurm-%j.out || slurm-206.out
|-
| --error || slurm-%j.out || slurm-206.out
|}
<tt>%j</tt> above indicates that it should be replaced with the job id.

=== Running from the Current Directory ===
By default, jobs run from your home directory. Many programs incorrectly assume that you are running the script from the current directory. You can use the '<tt>-cwd</tt>' directive to change to the "current working directory" you used when submitting the job.
=== Running in a specific class of machine ===
If you want to run on a specific class of machines, e.g., the Dwarves, you can add the flag "--constraint=dwarves" to select any of those machines.

=== Processor Constraints ===
Because Beocat is a heterogenous cluster (we have machines from many years in the cluster), not all of our processors support every new and fancy feature. You might have some applications that require some newer processor features, so we provide a mechanism to request those.

<tt>--contraint</tt> tells the cluster to apply constraints to the types of nodes that the job can run on. For instance, we know of several applications that must be run on chips that have "AVX" processor extensions. To do that, you would specify <tt>--constraint=avx</tt> on you ''<tt>sbatch</tt>'' '''or''' ''<tt>srun</tt>'' command lines.
Using <tt>--constraint=avx</tt> will prohibit your job from running on the Mages while <tt>--contraint=avx2</tt> will eliminate the Elves as well as the Mages.

=== Slurm Environment Variables ===
Within an actual job, sometimes you need to know specific things about the running environment to setup your scripts correctly. Here is a listing of environment variables that Slurm makes available to you. Of course the value of these variables will be different based on many different factors.
<syntaxhighlight lang="bash">
CUDA_VISIBLE_DEVICES=NoDevFiles
ENVIRONMENT=BATCH
GPU_DEVICE_ORDINAL=NoDevFiles
HOSTNAME=dwarf37
SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint
SLURM_CLUSTER_NAME=beocat
SLURM_CPUS_ON_NODE=1
SLURM_DISTRIBUTION=cyclic
SLURMD_NODENAME=dwarf37
SLURM_GTIDS=0
SLURM_JOB_CPUS_PER_NODE=1
SLURM_JOB_GID=163587
SLURM_JOB_ID=202
SLURM_JOBID=202
SLURM_JOB_NAME=slurm_simple.sh
SLURM_JOB_NODELIST=dwarf37
SLURM_JOB_NUM_NODES=1
SLURM_JOB_PARTITION=batch.q,killable.q
SLURM_JOB_QOS=normal
SLURM_JOB_UID=163587
SLURM_JOB_USER=mozes
SLURM_LAUNCH_NODE_IPADDR=10.5.16.37
SLURM_LOCALID=0
SLURM_MEM_PER_NODE=1024
SLURM_NNODES=1
SLURM_NODEID=0
SLURM_NODELIST=dwarf37
SLURM_NPROCS=1
SLURM_NTASKS=1
SLURM_PRIO_PROCESS=0
SLURM_PROCID=0
SLURM_SRUN_COMM_HOST=10.5.16.37
SLURM_SRUN_COMM_PORT=37975
SLURM_STEP_ID=0
SLURM_STEPID=0
SLURM_STEP_LAUNCHER_PORT=37975
SLURM_STEP_NODELIST=dwarf37
SLURM_STEP_NUM_NODES=1
SLURM_STEP_NUM_TASKS=1
SLURM_STEP_TASKS_PER_NODE=1
SLURM_SUBMIT_DIR=/homes/mozes
SLURM_SUBMIT_HOST=dwarf37
SLURM_TASK_PID=23408
SLURM_TASKS_PER_NODE=1
SLURM_TOPOLOGY_ADDR=due1121-prod-core-40g-a1,due1121-prod-core-40g-c1.due1121-prod-sw-100g-a9.dwarf37
SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node
SLURM_UMASK=0022
SRUN_DEBUG=3
TERM=screen-256color
TMPDIR=/tmp
USER=mozes
</syntaxhighlight>
Sometimes it is nice to know what hosts you have access to during a job. You would checkout the SLURM_JOB_NODELIST to know that. There are lots of useful Environment Variables there, I will leave it to you to identify the ones you want.

Some of the most commonly-used variables we see used are $SLURM_CPUS_ON_NODE, $HOSTNAME, and $SLURM_JOB_ID.

== Running from a sbatch Submit Script ==
No doubt after you've run a few jobs you get tired of typing something like 'sbatch -l mem=2G,h_rt=10:00 -pe single 8 -n MyJobTitle MyScript.sh'. How are you supposed to remember all of these every time? The answer is to create a 'submit script', which outlines all of these for you. Below is a sample submit script, which you can modify and use for your own purposes.
<syntaxhighlight lang="bash">
#!/bin/bash

## A Sample sbatch script created by Kyle Hutson
##
## Note: Usually a '#" at the beginning of the line is ignored. However, in
## the case of sbatch, lines beginning with #SBATCH are commands for sbatch
## itself, so I have taken the convention here of starting *every* line with a
## '#', just Delete the first one if you want to use that line, and then modify
## it to your own purposes. The only exception here is the first line, which
## *must* be #!/bin/bash (or another valid shell).

## There is one strict rule for guaranteeing Slurm reads all of your options:
## Do not put *any* lines above your resource requests that aren't either:
## 1) blank. (no other characters)
## 2) comments (lines must begin with '#')

## Specify the amount of RAM needed _per_core_. Default is 1G
##SBATCH --mem-per-cpu=1G

## Specify the maximum runtime in DD-HH:MM:SS form. Default is 1 hour (1:00:00)
##SBATCH --time=1:00:00

## Require the use of infiniband. If you don't know what this is, you probably
## don't need it.
##SBATCH --gres=fabric:ib:1

## GPU directive. If You don't know what this is, you probably don't need it
##SBATCH --gres=gpu:1

## number of cores/nodes:
## quick note here. Jobs requesting 16 or fewer cores tend to get scheduled
## fairly quickly. If you need a job that requires more than that, you might
## benefit from contacting Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]
## to see how we can assist in getting your
## job scheduled in a reasonable amount of time. Default is
##SBATCH --cpus-per-task=1
##SBATCH --cpus-per-task=12
##SBATCH --nodes=2 --tasks-per-node=1
##SBATCH --tasks=20

## Constraints for this job. Maybe you need to run on the elves
##SBATCH --constraint=elves
## or perhaps you just need avx processor extensions
##SBATCH --constraint=avx

## Output file name. Default is slurm-%j.out where %j is the job id.
##SBATCH --output=MyJobTitle.o%j

## Split the errors into a seperate file. Default is the same as output
##SBATCH --error=MyJobTitle.e%j

## Name my job, to make it easier to find in the queue
##SBATCH -J MyJobTitle

## Send email when certain criteria are met.
## Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to
## BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage
## out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent
## of time limit), TIME_LIMIT_80 (reached 80 percent of time limit),
## TIME_LIMIT_50 (reached 50 percent of time limit) and ARRAY_TASKS (send
## emails for each array task). Multiple type values may be specified in a
## comma separated list. Unless the ARRAY_TASKS option is specified, mail
## notifications on job BEGIN, END and FAIL apply to a job array as a whole
## rather than generating individual email messages for each task in the job
## array.
##SBATCH --mail-type=ALL

## Email address to send the email to based on the above line.
## Default is to send the mail to the e-mail address entered on the account
## request form.
##SBATCH --mail-user myemail@ksu.edu

## And finally, we run the job we came here to do.
## $HOME/ProgramDir/ProgramName ProgramArguments

## OR, for the case of MPI-capable jobs
## mpirun $HOME/path/MpiJobName
</syntaxhighlight>

== File Access ==
Beocat has a variety of options for storing and accessing your files.
Every user has a home directory for general use which is limited in size, has decent file access performance. Those needing more storage may purchase /bulk subdirectories which have the same decent performance
but are not backed up. The /fastscratch file system is a zfs host with lots of NVME drives provide much faster
temporary file access. When fast IO is critical to the application performance, access to /fastscratch, the local disk on each node, or to a
RAM disk are the best options.

===Home directory===

Every user has a <tt>/homes/''username''</tt> directory that they drop into when they log into Beocat.
The home directory is for general use and provides decent performance for most file IO.
Disk space in each home directory is limited to 1 TB, so larger files should be kept in a purchased /bulk
directory, and there is a limit of 100,000 files in each subdirectory in your account.
This file system is fully redundant, so 3 specific hard disks would need to fail before any data was lost.
All files will soon be backed up nightly to a separate file server in Nichols Hall, so if you do accidentally
delete something it can be recovered.

===Bulk directory===

Bulk data storage may be provided at a cost of $45/TB/year billed monthly. Due to the cost, directories will be provided when we are contacted and provided with payment information.

===Fast Scratch file system===

The /fastscratch file system is faster than /bulk or /homes.
In order to use fastscratch, you first need to make a directory for yourself.
Fast Scratch is meant as temporary space for prepositioning files and accessing them
during runs. Once runs are completed, any files that need to be kept should be moved to your home
or bulk directories since files on the fastscratch file system may get purged after 30 days.

<syntaxhighlight lang=bash>
mkdir /fastscratch/$USER
</syntaxhighlight>

===Local disk===

If you are running on a single node, it may also be faster to access your files from the local disk
on that node. Each job creates a subdirectory /tmp/job# where '#' is the job ID number on the
local disk of each node the job uses. This can be accessed simply by writing to /tmp rather than
needing to use /tmp/job#.

You may need to copy files to
local disk at the start of your script, or set the output directory for your application to point
to a file on the local disk, then you'll need to copy any files you want off the local disk before
the job finishes since Slurm will remove all files in your job's directory on /tmp on completion
of the job or when it aborts. Use 'kstat -l -h' to see how much /tmp space is available on each node.

<syntaxhighlight lang=bash>
# Copy input files to the tmp directory if needed
cp $input_files /tmp

# Make an 'out' directory to pass to the app if needed
mkdir /tmp/out

# Example of running an app and passing the tmp directory in/out
app -input_directory /tmp -output_directory /tmp/out

# Copy the 'out' directory back to the current working directory after the run
cp -rp /tmp/out .
</syntaxhighlight>

===RAM disk===

If you need ultrafast access to files, you can use a RAM disk which is a file system set up in the
memory of the compute node you are running on. The RAM disk is limited to the requested memory on that node, so you should account for this usage when you request
memory for your job. Below is an example of how to use the RAM disk.

<syntaxhighlight lang=bash>
# Copy input files over if necessary
cp $any_input_files /dev/shm/

# Run the application, possibly giving it the path to the RAM disk to use for output files
app -output_directory /dev/shm/

# Copy files from the RAM disk to the current working directory and clean it up
cp /dev/shm/* .
</syntaxhighlight>

===When you leave KSU===

If you are done with your account and leaving KSU, please clean up your directory, move any files
to your supervisor's account that need to be kept after you leave, and notify us so that we can disable your
account. The easiest way to move your files to your supervisor's account is for them to set up
a subdirectory for you with the appropriate write permissions. The example below shows moving
just a user's 'data' subdirectory to their supervisor. The 'nohup' command is used so that the move will
continue even if the window you are doing the move from gets disconnected.

<syntaxhighlight lang=bash>
# Supervisor:
mkdir /bulk/$USER/$STUDENT_USERNAME
setfacl -d -m u:$USER:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -m u:$USER:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -d -m u:$STUDENT_USERNAME:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -m u:$STUDENT_USERNAME:rwX -R /bulk/$USER/$STUDENT_USERNAME

# Student:
nohup mv /homes/$USER/data /bulk/$SUPERVISOR_USERNAME/$USER &

# Once the move is complete, the Supervisor should limit the permissions for the directory again by removing the student's access:
chown $USER: -R /bulk/$USER/$STUDENT_USERNAME
setfacl -d -x u:$STUDENT_USERNAME -R /bulk/$USER/$STUDENT_USERNAME
setfacl -x u:$STUDENT_USERNAME -R /bulk/$USER/$STUDENT_USERNAME
</syntaxhighlight>

==File Sharing==

This section will cover methods of sharing files with other users within Beocat and on remote systems.
In the past, Beocat users have been allowed to keep their
/homes and /bulk directories open so that any other user could
access files. In order to bring Beocat into alignment with
State of Kansas regulations and industry norms, all users must now have their /homes /bulk /scratch and /fastscratch directories
locked down from other users, but can still share files and directories within their group or with individual users
using group and individual ACLs (Access Control Lists) which will be explained below.
Beocat staff will be exempted from this
policy as we need to work freely with all users and will manage our
subdirectories to minimize access.

===Securing your home directory with the setacls script===

If you do not wish to share files or directories with other users, you do not need to do anything
as rwx access to others has already been removed.
If you want to share files or directories you can either use the '''setacls''' script or configure
the ACLs (Access Control Lists) manually.

The '''setacls -h''' will show how to use the script.

Eos: setacls -h
setacls [-r] [-w] [-g group] [-u user] -d /full/path/to/directory
Execute pemission will always be applied, you may also choose r or w
Must specify at least one group or user
Must specify at least one directory, and it must be the full path
Example: setacls -r -g ksu-cis-hpc -u mozes -d /homes/daveturner/shared_dir

You can specify the permissions to be either -r for read or -w for write or you can specify both.
You can provide a priority group to share with, which is the same as the group used in a --partition=
statement in a job submission script. You can also specify users.
You can specify a file or a directory to share. If the directory is specified then all files in that
directory will also be shared, and all files created in the directory laster will also be shared.

The script will set everything up for you, telling you the commands it is executing along the way,
then show the resulting ACLs at the end with the '''getfacl''' command. Below is an example of
sharing the directory '''test_directory''' in my /bulk/daveturner directory with Nathan.

<syntaxhighlight>
Beocat> cd /bulk/daveturner
Beocat> mkdir test_directory
Beocat> setacls -r -w -u nathanrwells -d /bulk/daveturner/test_directory

Opening up base directory /bulk/daveturner with X execute permission only
setfacl -m u:nathanrwells:X /bulk/daveturner

Setting Xrw for directory/file /bulk/daveturner/test_directory
setfacl -m u:nathanrwells:Xrw -R /bulk/daveturner/test_directory
setfacl -d -m u:nathanrwells:Xrw -R /bulk/daveturner/test_directory

The ACLs on directory /bulk/daveturner/test_directory are set to:

getfacl: Removing leading '/' from absolute path names
# file: bulk/daveturner/test_directory
USER daveturner rwx rwx
user nathanrwells rwx rwx
GROUP daveturner_users r-x r-x
group beocat_support r-x r-x
mask rwx rwx
other --- ---
</syntaxhighlight>

The '''getfacl''' run by the script now shows that user '''nathanrwells''' has
read and write permissions to that directory and execute access to all directories
leading up to it.

====Manually configuring your ACLs====

If you want to manually configure the ACLs you can use the directions below to do what the '''setacls'''
script would do for you.
You first need to provide the minimum execute access to your /homes
or /bulk directory before sharing individual subdirectories. Setting the ACL to execute only will allow those
in your group to get access to subdirectories while not including read access will mean they will not
be able to see other files or subdirectories on your main directory, but do keep in mind that they can still access them
so you may want to still lock them down manually. Below is an example of how I would change my
/homes/daveturner directory to allow ksu-cis-hpc group execute access.

<syntaxhighlight lang=bash>
setfacl -m g:ksu-cis-hpc:X /homes/daveturner
</syntaxhighlight>

If your research group owns any nodes on Beocat, then you have a group name that can be used to securely share
files with others within your group. Below is an example of creating a directory called 'share_hpc',
then providing access to my ksu-cis.hpc group
(my group is ksu-cis-hpc so I submit jobs to --partition=ksu-cis-hpc.q).
Using -R will make these changes recursively to all files and directories in that subdirectory while changing the defaults with the setfacl -d command will ensure that files and directories created
later will be done so with these same ACLs.

<syntaxhighlight lang=bash>
mkdir share_hpc
# ACLs are used here for setting default permissions
setfacl -d -m g:ksu-cis-hpc:rX -R share_hpc
# ACLs are used here for setting actual permissions
setfacl -m g:ksu-cis-hpc:rX -R share_hpc
</syntaxhighlight>

This will give people in your group the ability to read files in the 'share_hpc' directory. If you also want
them to be able to write or modify files in that directory then change the ':rX' to ':rwX' instead. e.g. 'setfacl -d -m g:ksu-cis-hpc:rwX -R share_hpc'

If you want to know what groups you belong to use the line below.
<syntaxhighlight lang=bash>
groups
</syntaxhighlight>
If your group does not own any nodes, you can still request a group name and manage the participants yourself
by emailing us at
beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]
.
If you want to share a directory with only a few people you can manage your ACLs using individual usernames
instead of with a group.

You can use the '''getfacl''' command to see groups have access to a given directory.

<syntaxhighlight lang=bash>
getfacl share_hpc

# file: share_hpc
# owner: daveturner
# group: daveturner_users
user::rwx
group::r-x
group:ksu-cis-hpc:r-x
mask::r-x
other::---
default:user::rwx
default:group::r-x
default:group:ksu-cis-hpc:r-x
default:mask::r-x
default:other::---
</syntaxhighlight>

ACLs give you great flexibility in controlling file access at the
group level. Below is a more advanced example where I set up a directory to be shared with
my ksu-cis-hpc group, Dan's ksu-cis-dan group, and an individual user 'mozes' who I also want
to have write access.

<syntaxhighlight lang=bash>
mkdir share_hpc_dan_mozes
# acls are used here for setting default permissions
setfacl -d -m g:ksu-cis-hpc:rX -R share_hpc_dan_mozes
setfacl -d -m g:ksu-cis-dan:rX -R share_hpc_dan_mozes
setfacl -d -m u:mozes:rwX -R share_hpc_dan_mozes
# ACLs are used here for setting actual permissions
setfacl -m g:ksu-cis-hpc:rX -R share_hpc_dan_mozes
setfacl -m g:ksu-cis-dan:rX -R share_hpc_dan_mozes
setfacl -m u:mozes:rwX -R share_hpc_dan_mozes

getfacl share_hpc_dan_mozes

# file: share_hpc_dan_mozes
# owner: daveturner
# group: daveturner_users
user::rwx
user:mozes:rwx
group::r-x
group:ksu-cis-hpc:r-x
group:ksu-cis-dan:r-x
mask::r-x
other::---
default:user::rwx
default:user:mozes:rwx
default:group::r-x
default:group:ksu-cis-hpc:r-x
default:group:ksu-cis-dan:r-x
default:mask::r-x
default:other::--x
</syntaxhighlight>

===Openly sharing files on the web===

If you create a 'public_html' directory on your home directory, then any files put there will be shared
openly on the web. There is no way to restrict who has access to those files.

<syntaxhighlight lang=bash>
cd
mkdir public_html
# Opt-in to letting the webserver access your home directory:
setfacl -m g:public_html:x ~/
</syntaxhighlight>

Then access the data from a web browser using the URL:

http://people.beocat.ksu.edu/~your_user_name

This will show a list of the files you have in your public_html subdirectory.

===Globus===

We have a page here dedicated to [[Globus]]

== Array Jobs ==
One of Slurm's useful options is the ability to run "Array Jobs"

It can be used with the following option to sbatch.

--array=n[-m[:s]]
Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Slurm
almost like a series of jobs. The option argument to --array specifies the number of array job tasks and the index number which will be
associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SLURM_ARRAY_TASK_ID. The option
arguments n, and m will be available through the environment variables SLURM_ARRAY_TASK_MIN and SLURM_ARRAY_TASK_MAX.

The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size.
Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SLURM_ARRAY_TASK_ID containing one of the 5 index numbers.

Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The
number of tasks in a array job is unlimited.

STDOUT and STDERR of array job tasks follow a slightly different naming convention (which can be controlled in the same way as mentioned above).

slurm-%A_%a.out

%A is the SLURM_ARRAY_JOB_ID, and %a is the SLURM_ARRAY_TASK_ID

=== Examples ===
==== Change the Size of the Run ====
Array Jobs have a variety of uses, one of the easiest to comprehend is the following:

I have an application, app1 I need to run the exact same way, on the same data set, with only the size of the run changing.

My original script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
RUNSIZE=50
#RUNSIZE=100
#RUNSIZE=150
#RUNSIZE=200
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
For every run of that job I have to change the RUNSIZE variable, and submit each script. This gets tedious.

With Array Jobs the script can be written like so:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=50-200:50
RUNSIZE=$SLURM_ARRAY_TASK_ID
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
I then submit that job, and Slurm understands that it needs to run it 4 times, once for each task. It also knows that it can and should run these tasks in parallel.

==== Choosing a Dataset ====
A slightly more complex use of Array Jobs is the following:

I have an application, app2, that needs to be run against every line of my dataset. Every line changes how app2 runs slightly, but I need to compare the runs against each other.

Originally I had to take each line of my dataset and generate a new submit script and submit the job. This was done with yet another script:

<syntaxhighlight lang="bash">
#!/bin/bash
DATASET=dataset.txt
scriptnum=0
while read LINE
do
echo "app2 $LINE" > ${scriptnum}.sh
sbatch ${scriptnum}.sh
scriptnum=$(( $scriptnum + 1 ))
done < $DATASET
</syntaxhighlight>
Not only is this needlessly complex, it is also slow, as sbatch has to verify each job as it is submitted. This can be done easily with array jobs, as long as you know the number of lines in the dataset. This number can be obtained like so: wc -l dataset.txt in this case lets call it 5000.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=1-5000
app2 `sed -n "${SLURM_ARRAY_TASK_ID}p" dataset.txt`
</syntaxhighlight>
This uses a subshell via `, and has the sed command print out only the line number $SLURM_ARRAY_TASK_ID out of the file dataset.txt.

Not only is this a smaller script, it is also faster to submit because it is one job instead of 5000, so sbatch doesn't have to verify as many.

To give you an idea about time saved: submitting 1 job takes 1-2 seconds. by extension if you are submitting 5000, that is 5,000-10,000 seconds, or 1.5-3 hours.

== Checkpoint/Restart using DMTCP ==

DMTCP is Distributed Multi-Threaded CheckPoint software that will checkpoint your application without modification, and
can be set up to automatically restart your job from the last checkpoint if for example the node you are running on fails.
This has been tested successfully
on Beocat for some scalar and OpenMP codes, but has failed on all MPI tests so far. We would like to encourage users to
try DMTCP out if their non-MPI jobs run longer than 24 hours. If you want to try this, please contact us first since we are still
experimenting with DMTCP.

The sample job submission script below shows how dmtcp_launch is used to start the application, then dmtcp_restart is used to start from a checkpoint if the job has failed and been rescheduled.
If you are putting this in an array script, then add the Slurm array task ID to the end of the ckeckpoint directory name
like ckptdir=ckpt-$SLURM_ARRAY_TASK_ID.

#!/bin/bash -l
#SBATCH --job-name=gromacs
#SBATCH --mem=50G
#SBATCH --time=24:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module reset
module load GROMACS/2016.4-foss-2017beocatb-hybrid
module load DMTCP
module list

ckptdir=ckpt
mkdir -p $ckptdir
export DMTCP_CHECKPOINT_DIR=$ckptdir

if ! ls -1 $ckptdir | grep -c dmtcp_restart_script > /dev/null
then
echo "Using dmtcp_launch to start the app the first time"
dmtcp_launch --no-coordinator mpirun -np 1 -x OMP_NUM_THREADS=4 gmx_mpi mdrun -nsteps 50000 -ntomp 4 -v -deffnm 1ns -c 1ns.pdb -nice 0
else
echo "Using dmtcp_restart from $ckptdir to continue from a checkpoint"
dmtcp_restart $ckptdir/*.dmtcp
fi

You will need to run several tests to verify that DMTCP is working properly with your application.
First, run a short test without DMTCP and another with DMTCP with the checkpoint interval set to 5 minutes
by adding the line export DMTCP_CHECKPOINT_INTERVAL=300 to your script. Then use kstat -d 1 to
check that the memory in both runs is close to the same. Also use this information to calculate the time
that each checkpoint takes. In most cases I've seen times less than a minute for checkpointing that will normally
be done once each hour. If your application is taking more time, let us know. Sometimes this can be sped up
by simply turning off compression by adding the line export DMTCP_GZIP=0. Make sure to remove the
line where you set the checkpoint interval to 300 seconds so that the default time of once per hour will be used.

After verifying that your code completes using DMTCP and does not take significantly more time or memory, you
will need to start a run then scancel it after the first checkpoint, then resubmit the same script to make
sure that it restarts and runs to completion. If you are working with an array job script, the last is to try a few
array tasks at once to make sure there is no conflict between the jobs.

== Running jobs interactively ==
Some jobs just don't behave like we think they should, or need to be run with somebody sitting at the keyboard and typing in response to the output the computers are generating. Beocat has a facility for this, called 'srun'. srun uses the exact same command-line arguments as sbatch, but you need to add the following arguments at the end: <tt>--pty bash</tt>. If no node is available with your resource requirements, srun will tell you something like the following:
srun --pty bash
srun: Force Terminated job 217
srun: error: CPU count per node can not be satisfied
srun: error: Unable to allocate resources: Requested node configuration is not available
Note that, like sbatch, your interactive job will timeout after your allotted time has passed.

== Connecting to an existing job ==
You can connect to an existing job using srun in the same way that the MonitorNode command
allowed us to in the old cluster. This is essentially like using ssh to get into the node where your job is running which
can be very useful in allowing you to look at files in /tmp/job# or in running htop to view the
activity level for your job.

srun --jobid=# --pty bash where '#' is the job ID number

== Altering Job Requests ==
We generally do not support users to modify job parameters once the job has been submitted. It can be done, but there are numerous catches, and all of the variations can be a bit problematic; it is normally easier to simply delete the job (using '''scancel ''jobid''''') and resubmit it with the right parameters. '''If your job doesn't start after modifying such parameters (after a reasonable amount of time), delete the job and resubmit it.'''

As it is unsupported, this is an excercise left to the reader. A starting point is <tt>man scontrol</tt>
== Killable jobs ==
There are a growing number of machines within Beocat that are owned by a particular person or group. Normally jobs from users that aren't in the group designated by the owner of these machines cannot use them. This is because we have guaranteed that the nodes will be accessible and available to the owner at any given time. We will allow others to use these nodes if they designate their job as "killable." If your job is designated as killable, your job will be able to use these nodes, but can (and will) be killed off at any point in time to make way for the designated owner's jobs. Jobs that are marked killable will be re-queued and may restart on another node.

The way you would designate your job as killable is to add <tt>--gres=killable:1</tt> to the '''<tt>sbatch</tt> or <tt>srun</tt>''' arguments. This could be either on the command-line or in your script file.

''Note: This is a submit-time only request, it cannot be added by a normal user after the job has been submitted.'' If you would like jobs modified to be '''killable''' after the jobs have been submitted (and it is too much work to <tt>scancel</tt> the jobs and re-submit), send an e-mail to the administrators detailing the job ids and what you would like done.

== Scheduling Priority ==
Some users are members of projects that have contributed to Beocat. When those users have contributed nodes, the group gets access to a "partition" giving you priority on those nodes.

In most situations, the scheduler will automatically add those priority partitions to the jobs as submitted. You should not need to include a partition list in your job submission.

There are currently just a few exceptions that we will not automatically add:
* ksu-chem-mri.q
* ksu-gen-gpu.q
* ksu-gen-highmem.q

If you have access to those any of the non-automatic partitions, and have need of the resources in that partition, you can then alter your <tt>#SBATCH</tt> lines to include your new partition:
#SBATCH --partition=ksu-gen-highmem.q

Otherwise, you shouldn't modify the partition line at all unless you really know what you're doing.

== Graphical Applications ==
Some applications are graphical and need to have some graphical input/output. We currently accomplish this with X11 forwarding or [[OpenOnDemand]]
=== OpenOnDemand ===
[[OpenOnDemand]] is likely the easier and more performant way to run a graphical application on the cluster.
# visit [https://ondemand.beocat.ksu.edu/ ondemand] and login with your cluster credentials.
# Check the "Interactive Apps" dropdown. We may have a workflow ready for you. If not choose the desktop.
# Select the resources you need
# Select launch
# A job is now submitted to the cluster and once the job is started you'll see a Connect button
# use the app as needed. If using the desktop, start your graphical application.

=== X11 Forwarding ===
==== Connecting with an X11 client ====
===== Windows =====
If you are running Windows, we recommend MobaXTerm as your file/ssh manager, this is because it is one relatively simple tool to do everything. MobaXTerm also automatically connects with X11 forwarding enabled.
===== Linux/OSX =====
Both Linux and OSX can connect in an X11 forwarding mode. Linux will have all of the tools you need installed already, OSX will need [https://www.xquartz.org/ XQuartz] installed.

Then you will need to change your 'ssh' command slightly:

ssh -Y eid@headnode.beocat.ksu.edu

The '''-Y''' argument tells ssh to setup X11 forwarding.
==== Starting an Graphical job ====
All graphical jobs, by design, must be interactive, so we'll use the srun command. On a headnode, we run the following:
# load an X11 enabled application
module load Octave
# start an X11 job, sbatch arguments are accepted for srun as well, 1 node, 1 hour, 1 gb of memory
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 octave --gui

Because these jobs are interactive, they may not be able to run at all times, depending on how busy the scheduler is at any point in time. '''--pty --x11''' are required arguments setting up the job, and '''octave --gui''' is the command to run inside the job.

== Job Accounting ==
Some people may find it useful to know what their job did during its run. The sacct tool will read Slurm's accounting database and give you summarized or detailed views on jobs that have run within Beocat.
=== sacct ===
This data can usually be used to diagnose two very common job failures.
==== Job debugging ====
It is simplest if you know the job number of the job you are trying to get information on.
<syntaxhighlight lang="bash">
# if you know the jobid, put it here:
sacct -j 1122334455 -l
# if you don't know the job id, you can look at your jobs started since some day:
sacct -S 2017-01-01
</syntaxhighlight>

===== My job didn't do anything when it ran! =====
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|-
|3
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|218||218||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||12||00:00:00||FAILED||2:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=12,mem=1G,node=1||cpu=12,mem=1G,node=1
|-
|218.batch||218.batch||batch||||137940K||dwarf37||0||137940K||1576K||dwarf37||0||1576K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||1.36G||0||0||0||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|-
|218.0||218.0||qqqqstat||||204212K||dwarf37||0||204212K||1420K||dwarf37||0||1420K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||196.52M||Unknown||Unknown||Unknown||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|}</div> 
If you look at the columns showing Elapsed and State, you can see that they show 00:00:00 and FAILED respectively. This means that the job started and then promptly ended. This points to something being wrong with your submission script. Perhaps there is a typo somewhere in it.

===== My job ran but didn't finish! =====
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|-
|3
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|220||220||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:01:27||TIMEOUT||0:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=1,mem=1G,node=1||cpu=1,mem=1G,node=1
|-
|220.batch||220.batch||batch||||370716K||dwarf37||0||370716K||7060K||dwarf37||0||7060K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:28||CANCELLED||0:15||1.23G||0||0||0||1Gn||0||0.16M||dwarf37||0||0.16M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|-
|220.0||220.0||sleep||||204212K||dwarf37||0||107916K||1000K||dwarf37||0||620K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:27||CANCELLED||0:15||1.54G||Unknown||Unknown||Unknown||1Gn||0||0.05M||dwarf37||0||0.05M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|}</div> 
If you look at the column showing State, we can see some pointers to the issue. The job ran out of time (TIMEOUT) and then was killed (CANCELLED).
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|221||221||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:00:00||CANCELLED by 0||0:0||||Unknown||Unknown||Unknown||1Mn||||||||||||||||||||||||cpu=1,mem=1M,node=1||cpu=1,mem=1M,node=1
|-
|221.batch||221.batch||batch||||137940K||dwarf37||0||137940K||1144K||dwarf37||0||1144K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:00:01||CANCELLED||0:15||2.62G||0||0||0||1Mn||0||0||dwarf37||65534||0||0||dwarf37||65534||0||||||||cpu=1,mem=1M,node=1
|}</div> 
If you look at the column showing State, we see it was "CANCELLED by 0", then we look at the AllocTRES column to see our allocated resources, and see that 1MB of memory was granted. Combine that with the column "MaxRSS" and we see that the memory granted was less than the memory we tried to use, thus the job was "CANCELLED".

GalaxyDocs

2025-06-26T13:46:01Z

Nathanrwells: /* Requesting Help */

== What is Galaxy? ==
[https://galaxyproject.org/ Galaxy] is a scientific workflow, data integration, and data and analysis persistence and publishing platform that aims to make computational biology accessible to research scientists that do not have computer programming or systems administration experience.

== How do I access Galaxy? ==
Access to Beocat's local instance of Galaxy is easy. Simply navigate to [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/] and sign in if prompted using the Keycloak login. This will use your Beocat EID and Password to login, you will also be prompted to authenticate against DUO.

Please note that this utilizes Beocat's /bulk directory. This is a billed directory, and as such, if usage in your respective upload directory for Galaxy exceeds the billing threshold (1T), we will contact you regarding remediation.

== Upload Larger Files to Galaxy ==
Have some larger files that need to be uploaded to Galaxy? We provide some documentation on how to upload files directly to Beocat and then import them to Galaxy for use. It can be found here: [[Galaxy_File_Upload| Galaxy File Upload]]

== How do I use Galaxy? ==

After accessing our local instance of Galaxy you should have access to all installed tools which will be in the tool panel on the left-hand side of your home screen on Galaxy. It should look like this:

[[File:Galaxy Toolbox.png|Tool Panel]]

Each category (Get Data, Send Data, MetaGenomics Tools, etc) is expandable reveal subcategories that help sort out many installed tools. Each tool will be placed under its respective category once it is installed.

From here you can select a tool to use and submit to the Slurm cluster. After opening a tool you will be given options (if available) to configure the tool, including its inputs, what the tool will do (within its constraints) outputs, and what compute resources the tool will submit with. This means that you can specify the resources given to slurm jobs by expanding the drop-down menu "Job Resource Paramters" and selecting "Specify Job resource parameters" which will show the following options to modify.

Processors: Number of processing cores, 'ppn' value (1-128) this is equivalent to slurms "--cpus-per-task". 
Memory: Memory size in gigabytes, 'pmem' value (1-1500). Note that the job scheduler uses --mem-per-cpu to allocate memory for your slurm job. This mean the number given for memory will be multiplied by the Processors count from above. I.e 2 Processors with 5 Memory will be 10GB of memory. 
Priority Queue: If you have access to a priority queue, and would like to use it, enter the partition name here. 
Runtime: How long you want your job to run for in hours.

If you fail to switch your job resource parameters, your job will submit with the default resources. This means it will submit with 5Gb of Memory and 1 CPU with 1 hour of Runtime and no Priority queue.

=== Canceling a running job ===
While Galaxy does submit to slurm, you will not be able to cancel the job in the same way you typically do. With Galaxy, to cancel your upcoming/currently running job simply press the trash can icon next the name of your job.

== Tool Requests ==
If you are missing a specific tool and would like to have it added to Galaxy, please contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] with a link to the tool. Additionally, you can browse through Galaxy's own [https://toolshed.g2.bx.psu.edu/ toolshed] to make a recommendation.

== Data Management ==

Galaxy follows the typical costs for bulk data storage, as Galaxy utilizes /bulk/galaxy for storage. Bulk data storage may be provided at a cost of $45/TB/year billed monthly. Billing starting at 1TB of usage. Users can easily see how much data they are utilizing in galaxy by checking the top right corner of the 'home' page of galaxy. This will say "Using ####MB/GB/TB", above your histories.

[[File:Galaxy_Data_usage_example.png|Usage Data]]

Clicking on this usage will bring you to a storage dashboard where you can easily manage your files and derelict dataset histories.

[[File:Storage_dashboard.png|Storage Dashboard]]

== Requesting Help ==
To request help with [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/], please contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]. 
When requesting help, it is best to give as much information as possible so that we may solve your issue in a timely manner.

== Acknowledgements ==
Beocat's installation of UseGalaxy is funded through K-INBRE with an Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under grant number P20GM103418.

This initiative was started through the Data Science Core group to bring easy to use GUI-based computational biology research to students and researchers at Kansas State University through Beocat.

Additional information on K-INBRE can be found [https://www.k-inbre.org/pages/k-inbre_about_bio-core.html here]

Nautilus

2025-06-26T13:45:03Z

Nathanrwells: /* Nautilus */

== Nautilus ==
To access the Nautilus namespace, login using K-State SSO at https://portal.nrp-nautilus.io/ . Once you have done so, contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] and request to be added to the Beocat Nautilus namespace (ksu-nrp-cluster). Once you have received notification that you have been added to the namespace, you can continue with the following steps to get set up to use the cluster resources.
<ol>
<li>SSH into headnode.beocat.ksu.edu</li>
<li>SSH into fiona (fiona hosts the kubectl tool we will use for this)</li>
<li>Once on fiona, use the command ‘cd ~’ to ensure you are in your home directory. If you
are not, this will return you to the top level of your home directory.<li>
<li>From there you will need to create a .kube directory inside of your home directory. Use
the command ‘mkdir ~/.kube’</li>
<li>Login to https://portal.nrp-nautilus.io/ using the same login previously used to create your
account (this will be your K-State EID login)</li>
<li>From here it is MANDATORY to read the cluster policy documentation provided by the
National Research Platform for the Nautilus program. You can find this here.
https://docs.nationalresearchplatform.org/userdocs/start/policies/ </li>
a. This is to ensure we do not break any of the rules put in place by the NRP.
 b. Return to https://portal.nrp-nautilus.io/ and accept the Acceptable Use Policy (AUP)
<li>Next, return to the website specified in step 5, in the top right corner of the page press
the “Get Config” option. </li>
a. This will download a file called ‘config’
<li>You will need to move the file to your ~/.kube directory created in step 4.</li>
a. To do this you can copy and paste the contents through the command line
 b. You can also utilize the OpenOnDemand tool to upload the file through the web
interface. Information for this tool can be found here:
https://support.beocat.ksu.edu/Docs/OpenOnDemand
 c. You can also use other means of moving the contents to the Beocat
headnodes/your home directory, but these are just a few examples.
 d. NOTE: Because we added a period before the directory name it is now a hidden directory,
and the directory will not appear when running a normal ‘ls’, to see the directory you will
need to run “ls -a” or “ls -la”.
<li>Once you have read the required documentation, created the .kube directory in your
home directory, and placed the config file in the '~/.kube' directory, you are now ready to continue!</li>
<li>Below is an example pod that can be used. It does not request much in the way of resources so you will likely need to change some things. Be sure to change the “name:” field
underneath “metadata:”. Change the text “test-pod” to “{eid}-pod” where ‘{eid}’ is your
K-State ID. It will look something like this “dan-pod”.</li>

<syntaxhighlight lang=yaml>
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: mypod
image: ubuntu
resources:
limits:
memory: 400Mi
cpu: 100m
requests:
memory: 100Mi
cpu: 100m
command: ["sh", "-c", "echo 'Im a new pod'"]
</syntaxhighlight>
<li>Place your .yaml file in the same directory created earlier (~/.kube).</li>
<li>If you are not already in the .kube directory enter the command “cd ~/.kube” to change
your current directory.</li>
<li>Now we are going to create our ‘pod’. This will request a ubuntu pc using the
specifications from above.</li>
a. To do this enter the command “kubectl create -f pod1.yaml” NOTE: You must be
in the same directory that you placed the pod1.yaml file in (in this situation, the above pod config was put into a file named pod1.yaml).
 b. If the command is successful you will see an output of “pod/{eid}-pod created”.
<li>You will need to wait until the container for the pod is finished creating. You can check
this by running “kubectl get pods”</li>
a. Once you run this command, it will output all the pods currently running or being
created in the namespace. Look for yours among the list of pods, the name will
be the same name specified in step 10.
 b. Once you locate your pod, check its STATUS. If the pod says Running, then you
are good to proceed. If it says Container Creating, then you will need to wait just a
bit. It should not take long.
<li>You can now execute and enter the pod by running “kubectl exec -it {eid}-pod --
/bin/bash”. Where ‘{eid}-pod’ is the pod created in step 13/the name specified in step 10.</li>
a. Executing this command will open the pod you created and run a bash console
on the pod.
 b. NOTE: If you have trouble logging into the pod, and are met with a “You must be
logged in to the server, you can run “kubectl proxy”, and after a moment, you can
cancel the command with a “crtl+c”. This should remedy the error.
</ol>

Additional documentation for Kubernetes can be found on the Kubernetes website https://kubernetes.io/docs/home

CUDA

2025-06-26T13:44:37Z

Nathanrwells: /* CUDA Overview */

== CUDA Overview ==
[[wikipedia:CUDA|CUDA]] is a feature set for programming nVidia [[wikipedia:Graphics_processing_unit|GPUs]]. We have many dwarf nodes that are CUDA-enabled with 1-2 GPUs and most of the Wizard nodes have 4 GPUs each. Most of these are consumer grade [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ nVidia 1080 Ti graphics cards] that are good for accelerating 32-bit calculations. Dwarf36-38 have two [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ nVidia RTX A4000 graphic cards] and dwarf39 has two [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ nVidia 1080 Ti graphics cards] that are available for anybody to use but you'll need to contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or email beocat@cs.ksu.edu to request being added to the GPU priority group then you'll need to submit jobs with --partition=ksu-gen-gpu.q. Wizard20 and wizard21 each have two [https://www.nvidia.com/object/quadro-graphics-with-pascal.html nVidia P100 cards] that are much more costly than the consumer grade 1080Ti cards but can accelerate 64-bit calculations much better.

== Training videos ==
CUDA Programming Model Overview: [http://www.youtube.com/watch?v=aveYOlBSe-Y http://www.youtube.com/watch?v=aveYOlBSe-Y]

{{#widget:YouTube|id=aveYOlBSe-Y|width=800|height=600}}

CUDA Programming Basics Part I (Host functions): [http://www.youtube.com/watch?v=79VARRFwQgY http://www.youtube.com/watch?v=79VARRFwQgY]

{{#widget:YouTube|id=79VARRFwQgY|width=800|height=600}}

CUDA Programming Basics Part II (Device functions): [http://www.youtube.com/watch?v=G5-iI1ogDW4 http://www.youtube.com/watch?v=G5-iI1ogDW4]

{{#widget:YouTube|id=G5-iI1ogDW4|width=800|height=600}}
== Compiling CUDA Applications ==
nvcc is the compiler for CUDA applications. When compiling your applications manually you will need to load a CUDA enabled compiler toolchain (e.g. fosscuda):

* module load fosscuda
* '''Do not run your cuda applications on the headnode. I cannot guarantee it will run, and it will give you terrible results if it does run.'''

With those two things in mind, you can compile CUDA applications as follows:

<syntaxhighlight lang="bash">
module load fosscuda
nvcc <source>.cu -o <output>
</syntaxhighlight>

== Example ==
=== Create your Application ===
Copy the following Application into Beocat as vecadd.cu
<syntaxhighlight lang="c">
// Kernel definition, see also section 4.2.3 of Nvidia Cuda Programming Guide
__global__ void vecAdd(float* A, float* B, float* C)
{
// threadIdx.x is a built-in variable provided by CUDA at runtime
int i = threadIdx.x;
A[i]=0;
B[i]=i;
C[i] = A[i] + B[i];
}

#include <stdio.h>
#define SIZE 10
int main()
{
int N=SIZE;
float A[SIZE], B[SIZE], C[SIZE];
float *devPtrA;
float *devPtrB;
float *devPtrC;
int memsize= SIZE * sizeof(float);

cudaMalloc((void**)&devPtrA, memsize);
cudaMalloc((void**)&devPtrB, memsize);
cudaMalloc((void**)&devPtrC, memsize);
cudaMemcpy(devPtrA, A, memsize, cudaMemcpyHostToDevice);
cudaMemcpy(devPtrB, B, memsize, cudaMemcpyHostToDevice);
// __global__ functions are called: Func<<< Dg, Db, Ns >>>(parameter);
vecAdd<<<1, N>>>(devPtrA, devPtrB, devPtrC);
cudaMemcpy(C, devPtrC, memsize, cudaMemcpyDeviceToHost);

for (int i=0; i<SIZE; i++)
printf("C[%d]=%f\n",i,C[i]);

cudaFree(devPtrA);
cudaFree(devPtrA);
cudaFree(devPtrA);

}
</syntaxhighlight>
=== Gain Access to a CUDA-capable Node ===
See our [[AdvancedSlurm|advanced scheduler documentation]]
=== Compile Your Application ===
<syntaxhighlight lang="bash">
module load fosscuda
nvcc vecadd.cu -o vecadd
</syntaxhighlight>
This will create a program with the name 'vecadd' (specified by the '-o' flag).

=== Run Your Application ===
Run the program as you usually would, namely
<syntaxhighlight lang="bash">
./vecadd
</syntaxhighlight>

Assuming you don't want to run the program interactively because this is a large job, you can submit a job via sbatch, just be sure to add '<tt>--gres=gpu:1</tt>' to the '''sbatch''' directive.

Galaxy File Upload

2025-06-26T13:43:37Z

Nathanrwells: /* Upload Files */

= Large File Uploads to Galaxy =
The Galaxy web UI is sometimes inconsistent with large downloads. As such, we have made user directory imports available on Galaxy. You can upload files to /bulk/galaxy/user-space/, and locating the user space that was created for you on first login. The folder is usually named "eid@ksu.edu" or if you are from another college (VetMed for instance) "eid@vet.k-state.edu". You can use the command "ls /bulk/galaxy/user-space" to find the name of your directory.

From here, we have written some instructions on how to utilize this upload method.
== Upload Files ==
First, you need to transfer the data onto Beocat, this can be done through any number of ways. We have some documentation on uploading data into Beocat that can be found here (for large files, we suggest using Globus, scp or an FTP program): https://support.beocat.ksu.edu/Docs/Main_Page#Transferring_data_to_Beocat

Next, due to how Galaxy handles data exposure to the web UI, we need to move this data to a userspace that was created for you in Galaxy on your first login:

Move the data to the following directory (You should be able to move data here, if you are unable to, please contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]) or email us at beocat@cs.ksu.edu. Note that the backwards slash is a necessary character to escape the @ symbol from your email address:
/bulk/galaxy/user-space/eid\@vet.k-state.edu/

Next, login to galaxy.beocat.ksu.edu with your eID through KeyCloak.

Then, navigate to the "Shared Data" tab at the top of the page, in the drop-down menu, navigate to "Data Libraries". This should take you to a relatively empty page that allows you to create a library with a "+ Library" in the top left of the web page. For reference, I have included a screenshot of my Data Libraries:

[[File:Data library creation.png|CreatingDataLibrary]]

Create a new Library with any name, and then open it by clicking on its name. In this case, from the photo above, you could click on "Test2" or "Upload".

This takes you to a page that will let you manage the data inside of that library. It should look like this:

[[File:Library explanation.png|ExplainingDataLibrary]]

From here you can either add a folder to help manage your data, upload data to the library, or add the current data in the library to your History so that it can be accessed. We are going to upload data to the library. Clicking "+ Datasets" will expand a drop-down menu, from here, select "from User Directory". This will allow you to upload any data to galaxy from that /bulk directory we moved your data to earlier on Beocat. That process looks something like this:

[[File:Import files.png|ImportToDataLibrary]]

In this, I just uploaded two empty text files. From here, it has to do some auto-magic to get the data to work inside galaxy, with the "state" of the data. I am not exactly sure what this, or how long it might take. I would imagine not long.

From here we finally need to publish the data to our history as a so that we can utilize it. I am going to import this data as a Dataset, though if a collection suits better, use that. In the photo below, I am uploading "iamanothertextfile.txt" to my new history called "IAmANewHistory".

Once that is added a notification should pop up in the bottom right hand corner of the browser. If not, navigate to the homepage and manually open the History your data was added to. From here you should be able to process your data like normal.

ProposalDescription

2025-06-26T13:42:46Z

Nathanrwells: /* Description of Beocat for Proposals or Information */

== Description of Beocat for Proposals or Information ==

Below is a current description of Beocat Compute resources and availability that can be used within proposals or to provide information. If you have any questions regarding this, please contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or reach out to us at beocat@cs.ksu.edu.

=== Compute Resources Description (Updated Oct. 1 2024) ===

Beocat, the K-State research computing cluster, is currently the largest academic supercomputer in Kansas. Its hardware includes nearly 400 researcher-funded computers, approximately 3.3PB of storage, ~10,000 processor cores on machines ranging from dual-processor Xeon e5 nodes with 128GB RAM and 100GbE to 128 core AMD nodes with 2TB RAM connected by 40-100 Gbps networks, and a total of 170 different GPUs ranging from GTX 1080ti's to NVIDIA L40S's. Beocat and its staff have provided tours demonstrating the value of K-State research and a high-tech look at our research facilities for over 3,000 participants, including USD383 StarBase, current and prospective students, funding agencies, faculty recruitment, and outreach activities. Classes supported include topics such as bioinformatics, business analytics, cybersecurity, data science, deep learning, economics, chemistry, and genetics. Beocat is supported by many NSF and university grants, and it acts as the central computing resource for multiple departments across campus. Beocat staff includes one full-time system administrators, a full-time applications scientist with a PhD in Physics and 35 years’ experience optimizing parallel programs and assisting researchers, and a part-time director.

Beocat is available to any academic researcher in Kansas and their partners under the statewide KanShare MOU. Under current policy, heavy users are expected to buy in through adding computational or personnel resources for the cluster (condo computing). Their jobs, then, are given guaranteed priority on any contributed machines, and they have access to other resources in the cluster on an as-available basis. Thus, projects can preserve a guaranteed base level of computation while utilizing the larger cluster for major computations. Users can also purchase archival data storage as needed. Dr. Daniel Andresen is the K-State XSEDE Campus Champion in the event national-class computational resources are required.

Main Page

2025-06-26T13:42:06Z

Nathanrwells: /* How do I get help? */

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

=== Online Documentations ===

* Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/]
* Read about [[Installed software]] and languages
* Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] and download the [[Media:Slurm-quick-reference.pdf|Slurm Quick Reference PDF]]
* Run interactive jobs with [[OpenOnDemand]]
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]]
* Big Data course on Beocat! [[BigDataOnBeocat]]
* Interested in web-based computational biology research? Check out [[GalaxyDocs|Galaxy!]]
* Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]]

=== Training Videos and Slides ===

* [https://www.youtube.com/watch?v=7NOB_HGQE0U Beocat Introduction] and [[Media:Beocat-Beoshock-Intro.pdf|slides]]
* [https://www.youtube.com/watch?v=b_yawpwFRdk Linux and Bash Introduction] and [[Media:Linux-Intro-cheatsheet.pdf|Linux Quick Reference PDF]]
* [https://www.youtube.com/watch?v=vcC-DURbH6c Advanced HPC Usage] and [[Media:HPC-Advanced-Usage.pdf|slides]]
* [https://www.youtube.com/watch?v=inJbYdZacjs HPC Parallel Computing] and [[Media:HPC-Parallel-Computing.pdf|slides]]

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
 With the recent changes to how KState handles DUO authentication, we recommend you use Globus to transfer files in and out of Beocat
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address or through TDX and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or email us at beocat@cs.ksu.edu, please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Main Page

2025-06-26T13:41:23Z

Nathanrwells:

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

=== Online Documentations ===

* Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/]
* Read about [[Installed software]] and languages
* Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] and download the [[Media:Slurm-quick-reference.pdf|Slurm Quick Reference PDF]]
* Run interactive jobs with [[OpenOnDemand]]
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]]
* Big Data course on Beocat! [[BigDataOnBeocat]]
* Interested in web-based computational biology research? Check out [[GalaxyDocs|Galaxy!]]
* Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]]

=== Training Videos and Slides ===

* [https://www.youtube.com/watch?v=7NOB_HGQE0U Beocat Introduction] and [[Media:Beocat-Beoshock-Intro.pdf|slides]]
* [https://www.youtube.com/watch?v=b_yawpwFRdk Linux and Bash Introduction] and [[Media:Linux-Intro-cheatsheet.pdf|Linux Quick Reference PDF]]
* [https://www.youtube.com/watch?v=vcC-DURbH6c Advanced HPC Usage] and [[Media:HPC-Advanced-Usage.pdf|slides]]
* [https://www.youtube.com/watch?v=inJbYdZacjs HPC Parallel Computing] and [[Media:HPC-Parallel-Computing.pdf|slides]]

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
 With the recent changes to how KState handles DUO authentication, we recommend you use Globus to transfer files in and out of Beocat
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address or through TDX and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you email us at beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket], please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

GalaxyDocs

2025-06-26T13:40:27Z

Nathanrwells: /* Tool Requests */

== What is Galaxy? ==
[https://galaxyproject.org/ Galaxy] is a scientific workflow, data integration, and data and analysis persistence and publishing platform that aims to make computational biology accessible to research scientists that do not have computer programming or systems administration experience.

== How do I access Galaxy? ==
Access to Beocat's local instance of Galaxy is easy. Simply navigate to [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/] and sign in if prompted using the Keycloak login. This will use your Beocat EID and Password to login, you will also be prompted to authenticate against DUO.

Please note that this utilizes Beocat's /bulk directory. This is a billed directory, and as such, if usage in your respective upload directory for Galaxy exceeds the billing threshold (1T), we will contact you regarding remediation.

== Upload Larger Files to Galaxy ==
Have some larger files that need to be uploaded to Galaxy? We provide some documentation on how to upload files directly to Beocat and then import them to Galaxy for use. It can be found here: [[Galaxy_File_Upload| Galaxy File Upload]]

== How do I use Galaxy? ==

After accessing our local instance of Galaxy you should have access to all installed tools which will be in the tool panel on the left-hand side of your home screen on Galaxy. It should look like this:

[[File:Galaxy Toolbox.png|Tool Panel]]

Each category (Get Data, Send Data, MetaGenomics Tools, etc) is expandable reveal subcategories that help sort out many installed tools. Each tool will be placed under its respective category once it is installed.

From here you can select a tool to use and submit to the Slurm cluster. After opening a tool you will be given options (if available) to configure the tool, including its inputs, what the tool will do (within its constraints) outputs, and what compute resources the tool will submit with. This means that you can specify the resources given to slurm jobs by expanding the drop-down menu "Job Resource Paramters" and selecting "Specify Job resource parameters" which will show the following options to modify.

Processors: Number of processing cores, 'ppn' value (1-128) this is equivalent to slurms "--cpus-per-task". 
Memory: Memory size in gigabytes, 'pmem' value (1-1500). Note that the job scheduler uses --mem-per-cpu to allocate memory for your slurm job. This mean the number given for memory will be multiplied by the Processors count from above. I.e 2 Processors with 5 Memory will be 10GB of memory. 
Priority Queue: If you have access to a priority queue, and would like to use it, enter the partition name here. 
Runtime: How long you want your job to run for in hours.

If you fail to switch your job resource parameters, your job will submit with the default resources. This means it will submit with 5Gb of Memory and 1 CPU with 1 hour of Runtime and no Priority queue.

=== Canceling a running job ===
While Galaxy does submit to slurm, you will not be able to cancel the job in the same way you typically do. With Galaxy, to cancel your upcoming/currently running job simply press the trash can icon next the name of your job.

== Tool Requests ==
If you are missing a specific tool and would like to have it added to Galaxy, please contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] with a link to the tool. Additionally, you can browse through Galaxy's own [https://toolshed.g2.bx.psu.edu/ toolshed] to make a recommendation.

== Data Management ==

Galaxy follows the typical costs for bulk data storage, as Galaxy utilizes /bulk/galaxy for storage. Bulk data storage may be provided at a cost of $45/TB/year billed monthly. Billing starting at 1TB of usage. Users can easily see how much data they are utilizing in galaxy by checking the top right corner of the 'home' page of galaxy. This will say "Using ####MB/GB/TB", above your histories.

[[File:Galaxy_Data_usage_example.png|Usage Data]]

Clicking on this usage will bring you to a storage dashboard where you can easily manage your files and derelict dataset histories.

[[File:Storage_dashboard.png|Storage Dashboard]]

== Requesting Help ==
To request help with [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/], please contact beocathelp@ksu.edu,or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]. 
When requesting help, it is best to give as much information as possible so that we may solve your issue in a timely manner.

== Acknowledgements ==
Beocat's installation of UseGalaxy is funded through K-INBRE with an Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under grant number P20GM103418.

This initiative was started through the Data Science Core group to bring easy to use GUI-based computational biology research to students and researchers at Kansas State University through Beocat.

Additional information on K-INBRE can be found [https://www.k-inbre.org/pages/k-inbre_about_bio-core.html here]

RSICC

2025-06-26T13:40:02Z

Nathanrwells:

== RSICC Codes ==
RSICC requires administrators of the system to be licensed for every code (and version) a user wishes to be on the system.

Let us know which RSICC software you'd like to run on Beocat. We will have to become licensed for it before it can be put on the system.

If it is an RSICC software that we've already obtained, you *must* provide proof of a license by providing us with an electronic copy of the email message from RSICC’s Request History tool or a copy of the user’s License Agreement. The email generated by RSICC’s Request History tool lists all of the software the individual has a license to use. The Request History link is available on RSICC’s Customer Service webpage, https://rsicc.ornl.gov/CustomerService.aspx Contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] or send those e-mails to beocat@cs.ksu.edu

FAQ

2025-06-23T20:19:58Z

Nathanrwells: /* Common Storage For Projects */

== How do I connect to Beocat ==
{| class="wikitable"
|-
! colspan="2" | Connection Settings
|-
! Hostname
| style="text-align:right" | headnode.beocat.ksu.edu
|-
! Port
| style="text-align:right" | 22
|-
! Username
| style="text-align:right" | <tt>eID</tt>
|-
! Password
| style="text-align:right" | <tt>eID Password</tt>
|}

{| class="wikitable"
!colspan="2" | Supported Connection Software (Latest Versions of Each)
|-
!rowspan="3" | Shell
|-
| [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]
|-
| ssh from openssh
|-
!rowspan="4" | File Transfer Utilities
|-
| [https://filezilla-project.org/ Filezilla]
|-
| [http://winscp.net/ WinSCP]
|-
| scp and sftp from openssh
|-
!rowspan="2" | Combination
|-
| [http://mobaxterm.mobatek.net/ MobaXterm]
|}

===Duo===
If your account is Duo Enabled, you will be asked to approve ''each'' connection through Duo's push system to your smart device by default for any non-interactive protocols. If you don't have a smart device, or your smart device is not currently able to be contacted by Duo, there are options.

====Automating Duo Method====
You would need to configure your connection client to send an ''Environment'' variable called <tt>DUO_PASSCODE</tt>. Its value could be the currently valid passcode from Duo, <tt>push</tt> or it could be set to <tt>phone</tt>. <tt>push</tt> will push the prompt to your smart device. <tt>phone</tt> will have duo call your phone number to approve.

===== OpenSSH =====
With OpenSSH (Linux or Mac command-line), to automatically set the Duo method to "push", use the command
<syntaxhighlight lang="bash">DUO_PASSCODE=push ssh -o SendEnv=DUO_PASSCODE headnode.beocat.ksu.edu</syntaxhighlight>

If you would like to put this in your ~/.ssh/config file, it will send the environment variable whenever it is set to Beocat upon connection:
Host headnode.beocat.ksu.edu
HostName headnode.beocat.ksu.edu
User YOUR_EID_GOES_HERE
SendEnv DUO_PASSCODE

From there you would simply do the following:
<syntaxhighlight lang=bash>
export DUO_PASSCODE=push
ssh headnode.beocat.ksu.edu
</syntaxhighlight>

===== PuTTY =====
In PuTTY to automatically set the Duo method to "push", expand "Connection" (if it isn't already), then click "Data". Under Environment variables, enter '''<tt>DUO_PASSCODE</tt>''' beside ''Variable'' and '''<tt>push</tt>''' beside ''Value''. Click the "Add" button and it will show up underneath. Be sure to go back to "Session" to save this change for PuTTY to remember this change.

===== MobaXTerm =====
There doesn't seem to be a way to send an environment variable in MobaXTerm, so you won't be able to set DUO_PASSCODE to an actual valid temporary key. To get MobaXterm to push automatically, you can edit your SSH session and on the "Advanced SSH Settings" tab, change the "Execute command" to <syntaxhighlight lang="bash">DUO_PASSCODE=push bash</syntaxhighlight>

==== Common issues ====
; Duo Pushes sometimes don't show up in a timely manner.
: If you open the Duo MFA application on your smart device when you're expecting an authentication challenge, the prompts seem to show up faster.
; MobaXTerm has excessive prompts for managing files.
: MobaXTerm has a sidebar browser for managing your files. Unfortunately, that sidebar browser initiates another SSH connection for every file transfer, which triggers a Duo push that you need to approve. MobaXTerm's dedicated SFTP Session doesn't have this same issue, it initiates a connection, keeps it open and re-uses it as needed, so you will have much fewer Duo approvals to respond to. If you choose to use the dedicated SFTP Session, you might consider disabling the sidebar file browser. "Advanced SSH settings" -> "SSH-browser type" -> "None"
; WinSCP has auto-reconnect enabled by default.
: Auto-reconnect is a useful function when actively transferring files, but if you have an idle session and the connection drops it will reconnect, sending you a Duo MFA prompt. If you don't approve it soon enough, WinSCP will attempt it again. Miss enough prompts and Duo will lock your account. It may be best to disable [https://winscp.net/eng/docs/ui_pref_resume reconnections during idle periods] if you do not wish be locked out of all services at K-State using Duo.
; FileZilla has auto-reconnect enabled by default.
: Auto-reconnect is a useful function when actively transferring files, but if you have an idle session and the connection drops it will reconnect, sending you a Duo MFA prompt. If you don't approve it soon enough, FileZilla will attempt it again. Miss enough prompts and Duo will lock your account. It may be best to disable timeouts and/or connection retries under the <tt>Edit -> Settings -> Connection</tt> menu if you do not wish to be locked out of all services at K-State using Duo.
; FileZilla has excessive prompts for managing files.
: Filezilla opens one connection for browsing the system. Transferring files opens 1-4 additional connections when the transfers start. Once they finish, those connections disconnect. If you start additional transfers, new connections will be opened. Every one of those connections must be approved through Duo MFA on your smart device. You can adjust the number of connections that FileZilla opens for transfers if you like. <tt>File -> Site Manager -> (choose the site you're changing) -> Transfer Settings -> Limit number of simultaneous connections</tt>.
: Another option is to disable processing the transfer queue, add the things to it you want to transfer and then re-enable the transfer queue. Then at least it will re-use the connections until the queue is empty.

== How do I compile my programs? ==
=== Serial programs ===
; Fortran
: <tt>ifort</tt> or <tt>gfortran</tt>
; C/C++
: <tt>icc</tt>, <tt>gcc</tt> and <tt>g++</tt>

=== Parallel programs ===
; Fortran
: <tt>mpif77</tt> or <tt>mpif90</tt>
; C/C++
: <tt>mpicc</tt> or <tt>mpic++</tt>

== Do Beocat jobs have a maximum Time Limit ==
Yes, there is a time limit, the scheduler will reject jobs longer than 28 days. The other side of that is that we reserve the right to a maintenance period every 14 days. Unless it is an emergency, we will give at least 2 weeks notice before these maintenance periods actually occur. Jobs 14 days or less that have started when we announce a maintenance period should be able to complete before it begins.

With that being said, there is no guarantee that any physical piece of hardware and the software that runs on it will behave for any significant length of time. Memory, processors, disk drives can all fail with little to no warning. Software may have bugs. We have had issues with the shared filesystem that resulted in several nodes losing connectivity and forced reboots. If you can, we always recommend that you write your jobs so that they can be resumed if they get interrupted.

{{Note|The 28 day limit can be overridden on a temporary and per-user basis provided there is enough justification|reminder|inline=1}}

== How are the filesystems on Beocat set up? ==

{| class="wikitable"
|-
! Mountpoint !! Local / Shared !! Size !! Filesystem !! Advice
|-
| /bulk || Shared || 3.1PB shared with /homes || cephfs || Slower than /homes; costs $45/TB/year
|-
| /homes || Shared || 3.1PB shared with /bulk || cephfs || Good enough for most jobs; limited to 1TB per home directory
|-
| /fastscratch || Shared || 280TB || nfs on top of ZFS || Faster than /homes or /bulk, built with all NVME disks; files not used in 30 days are automatically culled.
|-
| /tmp || Local || >100GB (varies per node) || XFS || Good for I/O intensive jobs. Unique per job, culled with the job finishes.
|-
|}
=== Usage Advice ===
For most jobs you shouldn't need to worry, your default working directory
is your homedir and it will be fast enough for most tasks.
I/O intensive work should use /tmp, but you will need to remember to copy
your files to and from this partition as part of your job script. This is made
easier through the <tt>$TMPDIR</tt> environment variable in your jobs.

Example usage of <tt>$TMPDIR</tt> in a job script
<syntaxhighlight lang="bash" line>
#!/bin/bash

#copy our input file to $TMPDIR to make processing faster
cp ~/experiments/input.data $TMPDIR

#use the input file we copied over to the local system
#generate the output file in $TMPDIR as well
~/bin/my_program --input-file=$TMPDIR/input.data --output-file=$TMPDIR/output.data

#copy the results back from $TMPDIR
cp $TMPDIR/output.data ~/experiments/results.$SLURM_JOB_ID
</syntaxhighlight>

You need to remember to copy over your data from <tt>$TMPDIR</tt> as part of your job.
That directory and its contents are deleted when the job is complete.

== What is "killable:1" or "killable:0" ==
On Beocat, some of the machines have been purchased by specific users and/or groups. These users and/or groups get guaranteed access to their machines at any point in time. Often, these machines are sitting idle because the owners have no need for it at the time. This would be a significant waste of computational power if there were no other way to make use of the computing cycles.

If you're wondering why a job may have the exit status of <tt>PREEMPTED</tt> from kstat or sacct, this is the reason.

=== Enter the "killable" resource ===
Killable (--gres=killable:1) jobs are jobs that can be scheduled to these "owned" machines by users outside of the true group of owners. If a "killable" job starts on one of these owned machines and the owner of said machine comes along and submits a job, the "killable" job will be returned to the queue, (killed off as it were), and restarted at some future point in time. The job will still complete at some future point, and if the job makes use of a checkpointing algorithm it may complete even faster. The trade off between marking a job "killable" and not, is that sometimes applications need a significant amount of runtime, and cannot resume running from a partial output, meaning that it may get restarted over and over again, never reaching the finish line. As such, we only auto-enable "killable" for relatively short jobs (<=168:00:00). Some users still feel this is a hindrance, so we created a way to tell us not to automatically mark short jobs "killable"

=== Disabling killable ===
Specifying --gres=killable:0 will tell us to not mark your job as killable.

=== The trade-off ===
If a job is marked killable, there are a non-trivial amount of additional nodes that the job can run on. If your job checkpoints itself, or is relatively short, there should be no downside to marking the job killable, as the job will probably start sooner. If your job is long-running and doesn't checkpoint (save its state to restart a previous session) itself, it could cause your job to take longer to complete.

== Help! When I submit my jobs I get "Warning To stay compliant with standard unix behavior, there should be a valid #! line in your script i.e. #!/bin/tcsh" ==
Job submission scripts are supposed to have a line similar to '<code>#!/bin/bash</code>' in them to start. We have had problems with people submitting jobs with invalid #! lines, so we enforce that rule. When this happens the job fails and we have to manually clean it up. The warning message is there just to inform you that the job script should have a line in it, in most cases #!/bin/tcsh or #!/bin/bash, to indicate what program should be used to run the script. When the line is missing from a script, by default your default shell is used to execute the script (in your case /usr/local/bin/tcsh). This works in most cases, but may not be what you are wanting.

== Help! When I submit my jobs I get "A #! line exists, but it is not pointing to an executable. Please fix. Job not submitted." ==
Like the above, error says you need a #!/bin/bash or similar line in your job script. This error says that while the line exists, the #! line isn't mentioning an executable file, thus the script will not be able to run. Most likely you wanted #!/bin/bash instead of something else.

== Help! My jobs keep dying after 1 hour and I don't know why ==
Beocat has default runtime limit of 1 hour. If you need more than that, or need more than 1 GB of memory per core, you'll want to look at the documentation [[SlurmBasics|here]] to see how to request it.

In short, when you run sbatch for your job, you'll want to put something along the lines of '<code>--time=0-10:00:00</code>' before the job script if you want your job to run for 10 hours.

== Help my error file has "Warning: no access to tty" ==
The warning message "Warning: no access to tty (Bad file descriptor)" is safe to ignore. It typically happens with the tcsh shell.

== Help! My job isn't going to finish in the time I specified. Can I change the time requirement? ==
Generally speaking, no.

Jobs are scheduled based on execution times (among other things). If it were easy to change your time requirement, one could submit a job with a 15-minute run-time, get it scheduled quickly, and then say "whoops - I meant 15 weeks", effectively gaming the job scheduler. In extreme circumstances and depending on the job requirements, we '''may''' be able to manually intervene. This process prevents other users from using the node(s) you are currently using, so are not routinely approved. Contact Beocat support (below) if you feel your circumstances warrant special consideration.

== Help! My perl job runs fine on the head node, but only runs for a few seconds and then quits when submitted to the queue. ==
Take a look at our documentation on [[Installed_software#Perl|Perl]]

== Help! When using mpi I get 'CMA: no RDMA devices found' or 'A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces' ==
This message simply means that some but not all nodes the job is running on have infiniband cards. The job will still run, but will not use the fastest interconnect we have available. This may or may not be an issue, depending on how message heavy your job is. If you would like to not see this warning, you may request infiniband as a resource when submitting your job. <code>--gres=fabric:ib:1</code>

== Help! when I use sbatch I get an error about line breaks ==
Beocat is a Linux system. Operating Systems use certain patterns of characters to indicate line breaks in their files. Linux and operating systems like it use '\n' as their line break character. Windows uses '\r\n' for their line breaks.

If you're getting an error that looks like this:
sbatch: error: Batch script contains DOS line breaks (\r\n)
sbatch: error: instead of expected UNIX line breaks (\n).

It means that your script is using the windows line endings. You can convert it with the <tt>dos2unix</tt> command
dos2unix myscript.sh

It would probably be beneficial for your editor to save the files with UNIX line breaks in the future.
* Visual Studio Code -- “Text Editor” > “Files” > “Eol”
* Notepad++ -- "Edit" > "EOL Conversion"

== Help! when logging into OnDemand I get a '400 Bad request' message ==
Unfortunately, there are some known issues with OnDemand and how it handles some of the complexities behind the scenes. This involves browser cookies that (occasionally) get too large and make it so you get these messages upon login.

The only work around is to clear your browser cookies (although you can limit it to simply clearing the ksu.edu ones).

Details for specific browsers are below

* [https://support.mozilla.org/en-US/kb/clear-cookies-and-site-data-firefox Firefox]
* [https://support.microsoft.com/en-us/microsoft-edge/delete-cookies-in-microsoft-edge-63947406-40ac-c3b8-57b9-2a946a29ae09 Edge]
* [https://support.google.com/chrome/answer/95647?sjid=1537101898131489753-NA#zippy=%2Cdelete-cookies-from-a-site Chrome]
* [https://support.apple.com/guide/safari/manage-cookies-sfri11471/mac Safari]
* If you are using some other browser, I would recommend searching google for <tt>$browsername clear site cookies</tt>

== Common Storage For Projects ==
Sometimes it is useful for groups of people to have a common storage area.

If you do not have a project, send a request via email to beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]. Note that these projects are generally reserved for tenure-track faculty and with a single project per eID.

If you already have a project you can do the following:

'''Note:''' The <tt>$group_name</tt> variable in the commands below needs to be replaced with the lower case name of your project. Membership of the projects can be done using our [[Group Management]] application.
* Create a directory in one of the home directories of someone in your group, ideally the project owner's.
** <tt>mkdir $directory</tt>
* Set the default permissions for new files and directories created in the directory:
** <tt>setfacl -d -m g:$group_name:rX -R $directory</tt>
* Set the permissions for the existing files and directories:
** <tt>setfacl -m g:$group_name:rX -R $directory</tt>

This will give people in your group the ability to read files in the 'share' directory. If you also want them to be able to write or modify files in that directory then use change the ':rX' to ':rwX' instead. e.g. 'setfacl -d -m g:$group_name:rwX -R $directory' for both setfacl commands. As with other permissions, the individuals will need access through every level of the directory hierarchy. [[LinuxBasics#Access_Control_Lists|It may be best to review our more in-depth topic on Access Control Lists.]]

== How do I get more help? ==
There are many sources of help for most Linux systems.

=== Unix man pages ===
Linux provides man pages (short for manual pages). These are simple enough to call, for example: if you need information on submitting jobs to Beocat, you can type '<code>man sbatch</code>'. This will bring up the manual for sbatch.

=== GNU info system ===
Not all applications have "man pages." Most of the rest have what they call info pages. For example, if you needed information on finding a file you could use '<code>info find</code>'.

=== This documentation ===
This documentation is very thoroughly researched, and has been painstakingly assembled for your benefit. Please use it.

=== Contact support ===
Support can be contacted [mailto:beocat@cis.ksu.edu here]. Please include detailed information about your problem, including the job number, applications you are trying to run, and the current directory that you are in.

SlurmBasics

2025-06-23T20:19:13Z

Nathanrwells:

== The Rocky/Slurm nodes ==

We have converted Beocat from CentOS Linux to Rocky Linux on April 1st of 2024. Any applications or libraries from the old system must be recompiled.

=== Using Modules ===

If you're using a common code that others may also be using, we may already have it compiled in a module. You can list the modules available and load an application as in the example below for Vasp.

eos> module avail
eos> module load GROMACS
eos> module list

When a module gets loaded, all the necessary libraries are also loaded and the paths to the libraries and executables are automatically set up. Loading GROMACS for example also loads the OpenMPI library needed to run it and adds the path to the MPI commands and Grimaces executables. To see how the path is set up, try executing which gmx. The module system allows you to easily switch between different version of applications, libraries, or languages as well.

If you are using a custom code or one that is not installed in a module, you'll need to recompile it yourself. This process is easier under CentOS as some of the work just involves loading the necessary set of modules. The first step is to decide whether to use the Intel compiler toolchain or the GNU toolchain, each of which includes the compilers and other math libraries. The module commands for each are below, and you can load these automatically when you log in by adding one of these module load statements to your .bashrc file. See /homes/daveturner/.bashrc as an example, where I put the module load statements .

To load the Intel compiler tool chain including the Intel Math Kernel Library (and OpenMPI):
icr-helios> module load iomkl

To load the GNU compiler tool chain including OpenMPI, OpenBLAS, FFTW, and ScalaPack load foss (free open source software):
icr-helios> module load foss

Modules provide an easy way to set up the compilers and libraries you may need to compile your code. Beyond that there are many different ways to compile codes so you'll just need to follow the directions. If you need help you can always email us at beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket].

=== Submitting jobs to Slurm ===

You can submit your job script using the sbatch command.

icr-helios> sbatch sbatch_script.sh
icr-helios> kstat --me

This will submit the script and show you a list of your jobs that are running and the jobs you have in the queue. By default the output for each job will go into a slurm-###.out file where ### is the job ID number. If you need to kill a job, you can use the scancel command with the job ID number.

== Submitting your first job ==
To submit a job to run under Slurm, we use the sbatch (submit batch) command. The scheduler finds the optimum place for your job to run. With over 300 nodes and 7500 cores to schedule, as well as differing priorities, hardware, and individual resources, the scheduler's job is not trivial and it can take some time for a job to start even when there are empty nodes available.

There are a few things you'll need to know before running sbatch.
* How many cores you need. Note that unless your program is created to use multiple cores (called "threading"), asking for more cores will not speed up your job. This is a common misperception. '''Beocat will not magically make your program use multiple cores!''' For this reason the default is 1 core.
* How much time you need. Many users when beginning to use Beocat neglect to specify a time requirement. The default is one hour, and we get asked why their job died after one hour. We usually point them to the [[FAQ]].
* How much memory you need. The default is 1 GB. If your job uses significantly more than you ask, your job will be killed off.
* Any advanced options. See the [[AdvancedSlurm]] page for these requests. For our basic examples here, we will ignore these.

So let's now create a small script to test our ability to submit jobs. Create the following file (either by copying it to Beocat or by editing a text file and we'll name it <code>myhost.sh</code>. Both of these methods are documented on our [[LinuxBasics]] page.
<syntaxhighlight lang="bash" line>
#!/bin/sh
hostname
</syntaxhighlight>

Be sure to make it executable
chmod u+x myhost.sh

So, now lets submit it as a job and see what happens. Here I'm going to use five options
* <code>--mem-per-cpu=</code> tells how much memory I need. In my example, I'm using our system minimum of 512 MB, which is more than enough. Note that your memory request is '''per core''', which doesn't make much difference for this example, but will as you submit more complex jobs.
* <code>--time=</code> tells how much runtime I need. This can be in the form of "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". This is a very short job, so 1 minute should be plenty. This can't be changed after the job is started please make sure you have requested a sufficient amount of time.
* <code>--nodes=1</code> tells Slurm that this must be run on one machine. The [[AdvancedSlurm]] page has much more on the "nodes" switch.
* <code>--ntasks-per-node=16 </code> Request 16 cores on each node.
* <code>--constraint=moles</code> Request to only run on the Mole class of compute nodes.

% '''ls'''
myhost.sh
% '''sbatch --time=1 --mem-per-cpu=512M --cpus-per-task=1 --ntasks=1 --nodes=1 ./myhost.sh'''
salloc: Granted job allocation 1483446

Since this is such a small job, it is likely to be scheduled almost immediately, so a minute or so later, I now see
% '''ls'''
myhost.sh
slurm-1483446.out

% '''cat slurm-1483446.out'''
mage03

== Monitoring Your Job ==

The kstat perl script has been developed at K-State to provide you with all the available information about your jobs on Beocat. kstat --help will give you a full description of how to use it.

Eos> kstat --help

USAGE: kstat [-q] [-c] [-g] [-l] [-u user] [-p NaMD] [-j 1234567] [--part partition]
kstat alone dumps all info except for the core summaries
choose -q -c for only specific info on queued or core summaries.
then specify any searchables for the user, program name, or job id

kstat info on running and queued jobs
kstat -h list host info only, no jobs
kstat -q info on the queued jobs only
kstat -c core usage for each user
kstat -d # show jobs run in the last # days
Memory per node - used/allocated/requested
Red is close to or over requested amount
Yellow is under utilized for large jobs
kstat -g Only show GPU nodes
kstat -o Turner Only show info for a given owner
kstat -o CS_HPC Same but sub _ for spaces
kstat -l long list - node features and performance
Node hardware and node CPU usage
job nodelist and switchlist
job current and max memory
job CPU utilizations
kstat -u daveturner job info for one user only
kstat --me job info for my jobs only
kstat -j 1234567 info on a given job id
kstat --osg show OSG background jobs also
kstat --nocolor do not use any color
kstat --name display full names instead of eIDs

---------------- Graphs and Tables ---------------------------------------
Specify graph/table, CPU or GPU or host, usage or memory, and optional time
kstat --graph-cpu-memory # gnuplot CPU memory for job #
kstat --table-gpu-usage-5min # GPU usage table every 5 min for job #
kstat --table-cpu-60min # CPU usage, memory, swap table every 60 min for job #
kstat --table-node [nodename] cores, load, CPU usage, memory table for a node

--------------------------------------------------------------------------
Multi-node jobs are highlighted in Magenta
kstat -l also provides a node list and switch list
highlighted in Yellow when nodes are spread across multiple switches
Run time is colorized yellow then red for jobs nearing their time limit
Queue time is colorized yellow then red for jobs waiting longer times
--------------------------------------------------------------------------

kstat can be used to give you a summary of your jobs that are running and in the queue:
Eos> kstat --me


Hero43      
24 of 24 cores      
Load 23.4 / 24      
495.3 / 512 GB used 
    
daveturner      
unafold        1234567      
1 core               
running               
 4gb req                
 0 d 5 h 35 m 
    
daveturner      
octopus       1234568     
16 core              
running               
 128gb req                
 8 d 15 h 42 m 
 ################################## BeoCat Queue ################################### 
    
daveturner      
NetPIPE       1234569      
2 core    
 PD  
 2h  
 4gb req      
 0 d 1 h 2 m 


kstat produces a separate line for each host. Use kstat -h to see information on all hosts without the jobs.
For the example above we are listing our jobs and the hosts they are on.

Core usage - yellow for empty, red for empty on owned nodes, cyan for partially used, blue for all cores used. 
Load level - yellow or yellow background indicates the node is being inefficiently used. Red just means more threads than cores. 
Memory usage - yellow or red means most memory is used. 
If the node is owned the group name will be in orange on the right. Killable jobs can still be run on those nodes. 

Each job line will contain the username, program name, job ID, number of cores, the status which may be colored red for killable jobs,
the maximum memory used or memory requested, and the amount of time the job has run.
Jobs in the queue may contain information on the requested memory and run time, priority access, constraints, and
how long the job has been in the queue.
In this case, I have 2 jobs running on Hero43. unafold is using 1 core while octopus is using 16 cores. Slurm did not provide
any information on the actual memory use so the memory request is reported

=== Detailed information about a single job ===

kstat can provide a great deal of information on a particular job including a very rough estimate of when it will run. This time is a worst case scenario as this will
be adapted as other jobs finish early. This is a good way to check for job submission problems before contacting us. kstat colorizes the more important
information to make it easier to identify.

Eos> kstat -j 157054

################################## Beocat Queue ###################################
daveturner netpipe 157054 64 cores PD dwarves fabric CS HPC 8gb req 0 d 0 h 0 m

JobId 157054 Job Name netpipe
UserId=daveturner GroupId=daveturner_users(2117) MCS_label=N/A
Priority=11112 Nice=0 Account=ksu-cis-hpc QOS=normal
Status=PENDING Reason=Resources Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=00:40:00 TimeMin=N/A
SubmitTime=2018-02-02T18:18:31 EligibleTime=2018-02-02T18:18:31
Estimated Start Time is 2018-02-03T06:17:49 EndTime=2018-02-03T06:57:49 Deadline=N/A
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partitions killable.q,ksu-cis-hpc.q AllocNode:Sid=eos:1761
ReqNodeList=(null) ExcNodeList=(null)
NodeList=(null) SchedNodeList=dwarf[01-02]
NumNodes=2-2 NumCPUs=64 NumTasks=64 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES 2 nodes 64 cores 8192 mem gres/fabric 2
Socks/Node=* NtasksPerN:B:S:C=32:0:*:* CoreSpec=*
MinCPUsNode=32 MinMemoryNode=4G MinTmpDiskNode=0
Constraint=dwarves DelayBoot=00:00:00
Gres=fabric Reservation=(null)
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Slurm script /homes/daveturner/perf/NetPIPE-5.x/sb.np
WorkDir=/homes/daveturner/perf/NetPIPE-5.x
StdErr=/homes/daveturner/perf/NetPIPE-5.x/0.o157054
StdIn=/dev/null
StdOut=/homes/daveturner/perf/NetPIPE-5.x/0.o157054
Switches=1@00:05:00
<syntaxhighlight lang=bash>
#!/bin/bash -l
#SBATCH --job-name=netpipe
#SBATCH -o 0.o%j
#SBATCH --time=0:40:00
#SBATCH --mem=4G
#SBATCH --switches=1
#SBATCH --nodes=2
#SBATCH --constraint=dwarves
#SBATCH --ntasks-per-node=32
#SBATCH --gres=fabric:roce:1

host=`echo $SLURM_JOB_NODELIST | sed s/[^a-z0-9]/\ /g | cut -f 1 -d ' '`
nprocs=$SLURM_NTASKS
openmpi_hostfile.pl $SLURM_JOB_NODELIST 1 hf.$host
opts="--printhostnames --quick --pert 3"

echo "*******************************************************************"
echo "Running on $SLURM_NNODES nodes $nprocs cores on nodes $SLURM_JOB_NODELIST"
echo "*******************************************************************"

mpirun -np 2 --hostfile hf.$host NPmpi $opts -o np.${host}.mpi
mpirun -np 2 --hostfile hf.$host NPmpi $opts -o np.${host}.mpi.bi --async --bidir
mpirun -np $nprocs NPmpi $opts -o np.${host}.mpi$nprocs --async --bidir
</syntaxhighlight>

=== Completed jobs and memory usage ===

kstat -d #

This will provide information on the jobs you have currently running and those that have completed
in the last '#' days. This is currently the only reliable way to get the memory used per node for your job.
This also provides information on whether the job completed normally, was canceled with scancel,
timed out, or was killed because it exceeded its memory request.

Eos> kstat -d 10

########################### sacct -u daveturner for 10 days ###########################
max gb used on a node / gb requested per node
193037 ADF dwarf43 1 n 32 c 30.46gb/100gb 05:15:34 COMPLETED
193289 ADF dwarf33 1 n 32 c 26.42gb/100gb 00:50:43 CANCELLED
195171 ADF dwarf44 1 n 32 c 56.81gb/120gb 14:43:35 COMPLETED
209518 matlab dwarf36 1 n 1 c 0.00gb/ 4gb 00:00:02 FAILED

=== Summary of core usage ===

kstat can also provide a listing of the core usage and cores requested for each user.
Eos> kstat -c

############################## Core usage ###############################
antariksh 1512 cores %25.1 used 41528 cores queued
bahadori 432 cores % 7.2 used 80 cores queued
eegoetz 0 cores % 0.0 used 2 cores queued
fahrialkan 24 cores % 0.4 used 32 cores queued
gowri 66 cores % 1.1 used 32 cores queued
jeffcomer 160 cores % 2.7 used 0 cores queued
ldcoates12 80 cores % 1.3 used 112 cores queued
lukesteg 464 cores % 7.7 used 0 cores queued
mike5454 1060 cores %17.6 used 852 cores queued
nilusha 344 cores % 5.7 used 0 cores queued
nnshan2014 136 cores % 2.3 used 0 cores queued
ploetz 264 cores % 4.4 used 60 cores queued
sadish 812 cores %13.5 used 0 cores queued
sandung 72 cores % 1.2 used 56 cores queued
zhiguang 80 cores % 1.3 used 688 cores queued

=== Producing memory and CPU utilization tables and graphs ===

kstat can now produce tables or graphs for the memory or CPU utilization
for a job. In order to view graphs you must set up X11 forwarding on your
ssh connection by using the -X parameter.

If you want to read more, continue on to our [[AdvancedSlurm]] page.

=== kstat is now available to download and install on other clusters ===

https://gitlab.beocat.ksu.edu/Admin-Public/kstat

This software has been installed and used on several clusters for many years.
It should be considered Beta software and may take some slight modifications
to install on some clusters. Please contact the author if you want to give
it a try (daveturner@ksu.edu).

AdvancedSlurm

2025-06-23T20:18:45Z

Nathanrwells:

== Resource Requests ==
Aside from the time, RAM, and CPU requirements listed on the [[SlurmBasics]] page, we have a couple other requestable resources:
Valid gres options are:
gpu[[:type]:count]
fabric[[:type]:count]
Generally, if you don't know if you need a particular resource, you should use the default. These can be generated with the command
<tt>srun --gres=help</tt>
=== Fabric ===
We currently offer 3 "fabrics" as request-able resources in Slurm. The "count" specified is the line-rate (in Gigabits-per-second) of the connection on the node.
==== Infiniband ====
First of all, let me state that just because it sounds "cool" doesn't mean you need it or even want it. InfiniBand does absolutely no good if running on a single machine. InfiniBand is a high-speed host-to-host communication fabric. It is (most-often) used in conjunction with MPI jobs (discussed below). Several times we have had jobs which could run just fine, except that the submitter requested InfiniBand, and all the nodes with InfiniBand were currently busy. In fact, some of our fastest nodes do not have InfiniBand, so by requesting it when you don't need it, you are actually slowing down your job. To request Infiniband, add <tt>--gres=fabric:ib:1</tt> to your sbatch command-line.
==== ROCE ====
ROCE, like InfiniBand is a high-speed host-to-host communication layer. Again, used most often with MPI. Most of our nodes are ROCE enabled, but this will let you guarantee the nodes allocated to your job will be able to communicate with ROCE. To request ROCE, add <tt>--gres=fabric:roce:1</tt> to your sbatch command-line.

==== Ethernet ====
Ethernet is another communication fabric. All of our nodes are connected by ethernet, this is simply here to allow you to specify the interconnect speed. Speeds are selected in units of Gbps, with all nodes supporting 1Gbps or above. The currently available speeds for ethernet are: <tt>1, 10, 40, and 100</tt>. To select nodes with 40Gbps and above, you could specify <tt>--gres=fabric:eth:40</tt> on your sbatch command-line. Since ethernet is used to connect to the file server, this can be used to select nodes that have fast access for applications doing heavy IO. The Dwarves and Heroes have 40 Gbps ethernet and we measure single stream performance as high as 20 Gbps, but if your application
requires heavy IO then you'd want to avoid the Moles which are connected to the file server with only 1 Gbps ethernet.

=== CUDA ===
[[CUDA]] is the resource required for GPU computing. 'kstat -g' will show you the GPU nodes and the jobs running on them. To request a GPU node, add <tt>--gres=gpu:1</tt> for example to request 1 GPU for your job; if your job uses multiple nodes, the number of GPUs requested is per-node. You can also request a given type of GPU (kstat -g -l to show types) by using <tt>--gres=gpu:geforce_gtx_1080_ti:1</tt> for a 1080Ti GPU on the Wizards or Dwarves, <tt>--gres=gpu:quadro_gp100:1</tt> for the P100 GPUs on Wizard20-21 that are best for 64-bit codes like Vasp. Most of these GPU nodes are owned by various groups. If you want access to GPU nodes and your group does not own any, we can add you to the <tt>--partition=ksu-gen-gpu.q</tt> group that has priority on Dwarf36-39. For more information on compiling CUDA code click on this [[CUDA]] link.

A listing of the current types of gpus can be gathered with this command:
<syntaxhighlight lang=bash>
scontrol show nodes | grep CfgTRES | tr ',' '\n' | awk -F '[:=]' '/gres\/gpu:/ { print $2 }' | sort -u
</syntaxhighlight>
At the time of this writing, that command produces this list:
* geforce_gtx_1080_ti
* geforce_rtx_2080_ti
* geforce_rtx_3090
* l40s
* quadro_gp100
* rtx_a4000
* rtx_a6000

== Parallel Jobs ==
There are two ways jobs can run in parallel, ''intra''node and ''inter''node. '''Note: Beocat will not automatically make a job run in parallel.''' Have I said that enough? It's a common misperception.
=== Intranode jobs ===
''Intra''node jobs run on many cores in the same node. These jobs can take advantage of many common libraries, such as [http://openmp.org/wp/ OpenMP], or any programming language that has the concept of ''threads''. Often, your program will need to know how many cores you want it to use, and many will use all available cores if not told explicitly otherwise. This can be a problem when you are sharing resources, as Beocat does. To request multiple cores, use the sbatch directives '<tt>--nodes=1 --cpus-per-task=n</tt>' or '<tt>--nodes=1 --ntasks-per-node=n</tt>', where ''n'' is the number of cores you wish to use. If your command can take an environment variable, you can use $SLURM_CPUS_ON_NODE to tell how many cores you've been allocated.

=== Internode (MPI) jobs ===
''Inter''node jobs can utilize many cores on one or more nodes. Communicating between nodes is trickier than talking between cores on the same node. The specification for doing so is called "[[wikipedia:Message_Passing_Interface|Message Passing Interface]]", or MPI. We have [http://www.open-mpi.org/ OpenMPI] installed on Beocat for this purpose. Most programs written to take advantage of large multi-node systems will use MPI, but MPI also allows an application to run on multiple cores within a node. You can tell if you have an MPI-enabled program because its directions will tell you to run '<tt>mpirun ''program''</tt>'. Requesting MPI resources is only mildly more difficult than requesting single-node jobs. Instead of using '<tt>--cpus-per-task=''n''</tt>', you would use '<tt>--nodes=''n'' --tasks-per-node=''m''</tt>' ''or'' '<tt>--nodes=''n'' --ntasks=''o''</tt>' for your sbatch request, where ''n'' is the number of nodes you want, ''m'' is the number of cores per node you need, and ''o'' is the total number of cores you need.

Some quick examples:

<tt>--nodes=6 --ntasks-per-node=4</tt> will give you 4 cores on each of 6 nodes for a total of 24 cores.

<tt>--ntasks=40</tt> will give you 40 cores spread across any number of nodes.

<tt>--nodes=10 --ntasks=100</tt> will give you a total of 100 cores across 10 nodes.

== Requesting memory for multi-core jobs ==
Memory requests are easiest when they are specified '''per core'''. For instance, if you specified the following: '<tt>--tasks=20 --mem-per-core=20G</tt>', your job would have access to 400GB of memory total.
== Other Handy Slurm Features ==
=== Email status changes ===
One of the most commonly used options when submitting jobs not related to resource requests is to have have Slurm email you when a job changes its status. This takes may need two directives to sbatch: <tt>--mail-user</tt> and <tt>--mail-type</tt>.
==== --mail-type ====
<tt>--mail-type</tt> is used to tell Slurm to notify you about certain conditions. Options are comma separated and include the following
{| class="wikitable"
!Option!!Explanation
|-
| NONE || This disables event-based mail
|-
| BEGIN || Sends a notification when the job begins
|-
| END || Sends a notification when the job ends
|-
| FAIL || Sends a notification when the job fails.
|-
| REQUEUE || Sends a notification if the job is put back into the queue from a running state
|-
| STAGE_OUT || Burst buffer stage out and teardown completed
|-
| ALL || Equivalent to BEGIN,END,FAIL,REQUEUE,STAGE_OUT
|-
| TIME_LIMIT || Notifies if the job ran out of time
|-
| TIME_LIMIT_90 || Notifies when the job has used 90% of its allocated time
|-
| TIME_LIMIT_80 || Notifies when the job has used 80% of its allocated time
|-
| TIME_LIMIT_50 || Notifies when the job has used 50% of its allocated time
|-
| ARRAY_TASKS || Modifies the BEGIN, END, and FAIL options to apply to each array task (instead of notifying for the entire job
|}

==== --mail-user ====
<tt>--mail-user</tt> is optional. It is only needed if you intend to send these job status updates to a different e-mail address than what you provided in the [https://acount.beocat.ksu.edu/user Account Request Page]. It is specified with the following arguments to sbatch: <tt>--mail-user=someone@somecompany.com</tt>

=== Job Naming ===
If you have several jobs in the queue, running the same script with different parameters, it's handy to have a different name for each job as it shows up in the queue. This is accomplished with the '<tt>-J ''JobName''</tt>' sbatch directive.

=== Separating Output Streams ===
Normally, Slurm will create one output file, containing both STDERR and STDOUT. If you want both of these to be separated into two files, you can use the sbatch directives '<tt>--output</tt>' and '<tt>--error</tt>'.

{| class="wikitable"
! option !! default !! example
|-
| --output || slurm-%j.out || slurm-206.out
|-
| --error || slurm-%j.out || slurm-206.out
|}
<tt>%j</tt> above indicates that it should be replaced with the job id.

=== Running from the Current Directory ===
By default, jobs run from your home directory. Many programs incorrectly assume that you are running the script from the current directory. You can use the '<tt>-cwd</tt>' directive to change to the "current working directory" you used when submitting the job.
=== Running in a specific class of machine ===
If you want to run on a specific class of machines, e.g., the Dwarves, you can add the flag "--constraint=dwarves" to select any of those machines.

=== Processor Constraints ===
Because Beocat is a heterogenous cluster (we have machines from many years in the cluster), not all of our processors support every new and fancy feature. You might have some applications that require some newer processor features, so we provide a mechanism to request those.

<tt>--contraint</tt> tells the cluster to apply constraints to the types of nodes that the job can run on. For instance, we know of several applications that must be run on chips that have "AVX" processor extensions. To do that, you would specify <tt>--constraint=avx</tt> on you ''<tt>sbatch</tt>'' '''or''' ''<tt>srun</tt>'' command lines.
Using <tt>--constraint=avx</tt> will prohibit your job from running on the Mages while <tt>--contraint=avx2</tt> will eliminate the Elves as well as the Mages.

=== Slurm Environment Variables ===
Within an actual job, sometimes you need to know specific things about the running environment to setup your scripts correctly. Here is a listing of environment variables that Slurm makes available to you. Of course the value of these variables will be different based on many different factors.
<syntaxhighlight lang="bash">
CUDA_VISIBLE_DEVICES=NoDevFiles
ENVIRONMENT=BATCH
GPU_DEVICE_ORDINAL=NoDevFiles
HOSTNAME=dwarf37
SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint
SLURM_CLUSTER_NAME=beocat
SLURM_CPUS_ON_NODE=1
SLURM_DISTRIBUTION=cyclic
SLURMD_NODENAME=dwarf37
SLURM_GTIDS=0
SLURM_JOB_CPUS_PER_NODE=1
SLURM_JOB_GID=163587
SLURM_JOB_ID=202
SLURM_JOBID=202
SLURM_JOB_NAME=slurm_simple.sh
SLURM_JOB_NODELIST=dwarf37
SLURM_JOB_NUM_NODES=1
SLURM_JOB_PARTITION=batch.q,killable.q
SLURM_JOB_QOS=normal
SLURM_JOB_UID=163587
SLURM_JOB_USER=mozes
SLURM_LAUNCH_NODE_IPADDR=10.5.16.37
SLURM_LOCALID=0
SLURM_MEM_PER_NODE=1024
SLURM_NNODES=1
SLURM_NODEID=0
SLURM_NODELIST=dwarf37
SLURM_NPROCS=1
SLURM_NTASKS=1
SLURM_PRIO_PROCESS=0
SLURM_PROCID=0
SLURM_SRUN_COMM_HOST=10.5.16.37
SLURM_SRUN_COMM_PORT=37975
SLURM_STEP_ID=0
SLURM_STEPID=0
SLURM_STEP_LAUNCHER_PORT=37975
SLURM_STEP_NODELIST=dwarf37
SLURM_STEP_NUM_NODES=1
SLURM_STEP_NUM_TASKS=1
SLURM_STEP_TASKS_PER_NODE=1
SLURM_SUBMIT_DIR=/homes/mozes
SLURM_SUBMIT_HOST=dwarf37
SLURM_TASK_PID=23408
SLURM_TASKS_PER_NODE=1
SLURM_TOPOLOGY_ADDR=due1121-prod-core-40g-a1,due1121-prod-core-40g-c1.due1121-prod-sw-100g-a9.dwarf37
SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.node
SLURM_UMASK=0022
SRUN_DEBUG=3
TERM=screen-256color
TMPDIR=/tmp
USER=mozes
</syntaxhighlight>
Sometimes it is nice to know what hosts you have access to during a job. You would checkout the SLURM_JOB_NODELIST to know that. There are lots of useful Environment Variables there, I will leave it to you to identify the ones you want.

Some of the most commonly-used variables we see used are $SLURM_CPUS_ON_NODE, $HOSTNAME, and $SLURM_JOB_ID.

== Running from a sbatch Submit Script ==
No doubt after you've run a few jobs you get tired of typing something like 'sbatch -l mem=2G,h_rt=10:00 -pe single 8 -n MyJobTitle MyScript.sh'. How are you supposed to remember all of these every time? The answer is to create a 'submit script', which outlines all of these for you. Below is a sample submit script, which you can modify and use for your own purposes.
<syntaxhighlight lang="bash">
#!/bin/bash

## A Sample sbatch script created by Kyle Hutson
##
## Note: Usually a '#" at the beginning of the line is ignored. However, in
## the case of sbatch, lines beginning with #SBATCH are commands for sbatch
## itself, so I have taken the convention here of starting *every* line with a
## '#', just Delete the first one if you want to use that line, and then modify
## it to your own purposes. The only exception here is the first line, which
## *must* be #!/bin/bash (or another valid shell).

## There is one strict rule for guaranteeing Slurm reads all of your options:
## Do not put *any* lines above your resource requests that aren't either:
## 1) blank. (no other characters)
## 2) comments (lines must begin with '#')

## Specify the amount of RAM needed _per_core_. Default is 1G
##SBATCH --mem-per-cpu=1G

## Specify the maximum runtime in DD-HH:MM:SS form. Default is 1 hour (1:00:00)
##SBATCH --time=1:00:00

## Require the use of infiniband. If you don't know what this is, you probably
## don't need it.
##SBATCH --gres=fabric:ib:1

## GPU directive. If You don't know what this is, you probably don't need it
##SBATCH --gres=gpu:1

## number of cores/nodes:
## quick note here. Jobs requesting 16 or fewer cores tend to get scheduled
## fairly quickly. If you need a job that requires more than that, you might
## benefit from emailing us at beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]
## to see how we can assist in getting your
## job scheduled in a reasonable amount of time. Default is
##SBATCH --cpus-per-task=1
##SBATCH --cpus-per-task=12
##SBATCH --nodes=2 --tasks-per-node=1
##SBATCH --tasks=20

## Constraints for this job. Maybe you need to run on the elves
##SBATCH --constraint=elves
## or perhaps you just need avx processor extensions
##SBATCH --constraint=avx

## Output file name. Default is slurm-%j.out where %j is the job id.
##SBATCH --output=MyJobTitle.o%j

## Split the errors into a seperate file. Default is the same as output
##SBATCH --error=MyJobTitle.e%j

## Name my job, to make it easier to find in the queue
##SBATCH -J MyJobTitle

## Send email when certain criteria are met.
## Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to
## BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage
## out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent
## of time limit), TIME_LIMIT_80 (reached 80 percent of time limit),
## TIME_LIMIT_50 (reached 50 percent of time limit) and ARRAY_TASKS (send
## emails for each array task). Multiple type values may be specified in a
## comma separated list. Unless the ARRAY_TASKS option is specified, mail
## notifications on job BEGIN, END and FAIL apply to a job array as a whole
## rather than generating individual email messages for each task in the job
## array.
##SBATCH --mail-type=ALL

## Email address to send the email to based on the above line.
## Default is to send the mail to the e-mail address entered on the account
## request form.
##SBATCH --mail-user myemail@ksu.edu

## And finally, we run the job we came here to do.
## $HOME/ProgramDir/ProgramName ProgramArguments

## OR, for the case of MPI-capable jobs
## mpirun $HOME/path/MpiJobName
</syntaxhighlight>

== File Access ==
Beocat has a variety of options for storing and accessing your files.
Every user has a home directory for general use which is limited in size, has decent file access performance. Those needing more storage may purchase /bulk subdirectories which have the same decent performance
but are not backed up. The /fastscratch file system is a zfs host with lots of NVME drives provide much faster
temporary file access. When fast IO is critical to the application performance, access to /fastscratch, the local disk on each node, or to a
RAM disk are the best options.

===Home directory===

Every user has a <tt>/homes/''username''</tt> directory that they drop into when they log into Beocat.
The home directory is for general use and provides decent performance for most file IO.
Disk space in each home directory is limited to 1 TB, so larger files should be kept in a purchased /bulk
directory, and there is a limit of 100,000 files in each subdirectory in your account.
This file system is fully redundant, so 3 specific hard disks would need to fail before any data was lost.
All files will soon be backed up nightly to a separate file server in Nichols Hall, so if you do accidentally
delete something it can be recovered.

===Bulk directory===

Bulk data storage may be provided at a cost of $45/TB/year billed monthly. Due to the cost, directories will be provided when we are contacted and provided with payment information.

===Fast Scratch file system===

The /fastscratch file system is faster than /bulk or /homes.
In order to use fastscratch, you first need to make a directory for yourself.
Fast Scratch is meant as temporary space for prepositioning files and accessing them
during runs. Once runs are completed, any files that need to be kept should be moved to your home
or bulk directories since files on the fastscratch file system may get purged after 30 days.

<syntaxhighlight lang=bash>
mkdir /fastscratch/$USER
</syntaxhighlight>

===Local disk===

If you are running on a single node, it may also be faster to access your files from the local disk
on that node. Each job creates a subdirectory /tmp/job# where '#' is the job ID number on the
local disk of each node the job uses. This can be accessed simply by writing to /tmp rather than
needing to use /tmp/job#.

You may need to copy files to
local disk at the start of your script, or set the output directory for your application to point
to a file on the local disk, then you'll need to copy any files you want off the local disk before
the job finishes since Slurm will remove all files in your job's directory on /tmp on completion
of the job or when it aborts. Use 'kstat -l -h' to see how much /tmp space is available on each node.

<syntaxhighlight lang=bash>
# Copy input files to the tmp directory if needed
cp $input_files /tmp

# Make an 'out' directory to pass to the app if needed
mkdir /tmp/out

# Example of running an app and passing the tmp directory in/out
app -input_directory /tmp -output_directory /tmp/out

# Copy the 'out' directory back to the current working directory after the run
cp -rp /tmp/out .
</syntaxhighlight>

===RAM disk===

If you need ultrafast access to files, you can use a RAM disk which is a file system set up in the
memory of the compute node you are running on. The RAM disk is limited to the requested memory on that node, so you should account for this usage when you request
memory for your job. Below is an example of how to use the RAM disk.

<syntaxhighlight lang=bash>
# Copy input files over if necessary
cp $any_input_files /dev/shm/

# Run the application, possibly giving it the path to the RAM disk to use for output files
app -output_directory /dev/shm/

# Copy files from the RAM disk to the current working directory and clean it up
cp /dev/shm/* .
</syntaxhighlight>

===When you leave KSU===

If you are done with your account and leaving KSU, please clean up your directory, move any files
to your supervisor's account that need to be kept after you leave, and notify us so that we can disable your
account. The easiest way to move your files to your supervisor's account is for them to set up
a subdirectory for you with the appropriate write permissions. The example below shows moving
just a user's 'data' subdirectory to their supervisor. The 'nohup' command is used so that the move will
continue even if the window you are doing the move from gets disconnected.

<syntaxhighlight lang=bash>
# Supervisor:
mkdir /bulk/$USER/$STUDENT_USERNAME
setfacl -d -m u:$USER:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -m u:$USER:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -d -m u:$STUDENT_USERNAME:rwX -R /bulk/$USER/$STUDENT_USERNAME
setfacl -m u:$STUDENT_USERNAME:rwX -R /bulk/$USER/$STUDENT_USERNAME

# Student:
nohup mv /homes/$USER/data /bulk/$SUPERVISOR_USERNAME/$USER &

# Once the move is complete, the Supervisor should limit the permissions for the directory again by removing the student's access:
chown $USER: -R /bulk/$USER/$STUDENT_USERNAME
setfacl -d -x u:$STUDENT_USERNAME -R /bulk/$USER/$STUDENT_USERNAME
setfacl -x u:$STUDENT_USERNAME -R /bulk/$USER/$STUDENT_USERNAME
</syntaxhighlight>

==File Sharing==

This section will cover methods of sharing files with other users within Beocat and on remote systems.
In the past, Beocat users have been allowed to keep their
/homes and /bulk directories open so that any other user could
access files. In order to bring Beocat into alignment with
State of Kansas regulations and industry norms, all users must now have their /homes /bulk /scratch and /fastscratch directories
locked down from other users, but can still share files and directories within their group or with individual users
using group and individual ACLs (Access Control Lists) which will be explained below.
Beocat staff will be exempted from this
policy as we need to work freely with all users and will manage our
subdirectories to minimize access.

===Securing your home directory with the setacls script===

If you do not wish to share files or directories with other users, you do not need to do anything
as rwx access to others has already been removed.
If you want to share files or directories you can either use the '''setacls''' script or configure
the ACLs (Access Control Lists) manually.

The '''setacls -h''' will show how to use the script.

Eos: setacls -h
setacls [-r] [-w] [-g group] [-u user] -d /full/path/to/directory
Execute pemission will always be applied, you may also choose r or w
Must specify at least one group or user
Must specify at least one directory, and it must be the full path
Example: setacls -r -g ksu-cis-hpc -u mozes -d /homes/daveturner/shared_dir

You can specify the permissions to be either -r for read or -w for write or you can specify both.
You can provide a priority group to share with, which is the same as the group used in a --partition=
statement in a job submission script. You can also specify users.
You can specify a file or a directory to share. If the directory is specified then all files in that
directory will also be shared, and all files created in the directory laster will also be shared.

The script will set everything up for you, telling you the commands it is executing along the way,
then show the resulting ACLs at the end with the '''getfacl''' command. Below is an example of
sharing the directory '''test_directory''' in my /bulk/daveturner directory with Nathan.

<syntaxhighlight>
Beocat> cd /bulk/daveturner
Beocat> mkdir test_directory
Beocat> setacls -r -w -u nathanrwells -d /bulk/daveturner/test_directory

Opening up base directory /bulk/daveturner with X execute permission only
setfacl -m u:nathanrwells:X /bulk/daveturner

Setting Xrw for directory/file /bulk/daveturner/test_directory
setfacl -m u:nathanrwells:Xrw -R /bulk/daveturner/test_directory
setfacl -d -m u:nathanrwells:Xrw -R /bulk/daveturner/test_directory

The ACLs on directory /bulk/daveturner/test_directory are set to:

getfacl: Removing leading '/' from absolute path names
# file: bulk/daveturner/test_directory
USER daveturner rwx rwx
user nathanrwells rwx rwx
GROUP daveturner_users r-x r-x
group beocat_support r-x r-x
mask rwx rwx
other --- ---
</syntaxhighlight>

The '''getfacl''' run by the script now shows that user '''nathanrwells''' has
read and write permissions to that directory and execute access to all directories
leading up to it.

====Manually configuring your ACLs====

If you want to manually configure the ACLs you can use the directions below to do what the '''setacls'''
script would do for you.
You first need to provide the minimum execute access to your /homes
or /bulk directory before sharing individual subdirectories. Setting the ACL to execute only will allow those
in your group to get access to subdirectories while not including read access will mean they will not
be able to see other files or subdirectories on your main directory, but do keep in mind that they can still access them
so you may want to still lock them down manually. Below is an example of how I would change my
/homes/daveturner directory to allow ksu-cis-hpc group execute access.

<syntaxhighlight lang=bash>
setfacl -m g:ksu-cis-hpc:X /homes/daveturner
</syntaxhighlight>

If your research group owns any nodes on Beocat, then you have a group name that can be used to securely share
files with others within your group. Below is an example of creating a directory called 'share_hpc',
then providing access to my ksu-cis.hpc group
(my group is ksu-cis-hpc so I submit jobs to --partition=ksu-cis-hpc.q).
Using -R will make these changes recursively to all files and directories in that subdirectory while changing the defaults with the setfacl -d command will ensure that files and directories created
later will be done so with these same ACLs.

<syntaxhighlight lang=bash>
mkdir share_hpc
# ACLs are used here for setting default permissions
setfacl -d -m g:ksu-cis-hpc:rX -R share_hpc
# ACLs are used here for setting actual permissions
setfacl -m g:ksu-cis-hpc:rX -R share_hpc
</syntaxhighlight>

This will give people in your group the ability to read files in the 'share_hpc' directory. If you also want
them to be able to write or modify files in that directory then change the ':rX' to ':rwX' instead. e.g. 'setfacl -d -m g:ksu-cis-hpc:rwX -R share_hpc'

If you want to know what groups you belong to use the line below.
<syntaxhighlight lang=bash>
groups
</syntaxhighlight>
If your group does not own any nodes, you can still request a group name and manage the participants yourself
by emailing us at
beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]
.
If you want to share a directory with only a few people you can manage your ACLs using individual usernames
instead of with a group.

You can use the '''getfacl''' command to see groups have access to a given directory.

<syntaxhighlight lang=bash>
getfacl share_hpc

# file: share_hpc
# owner: daveturner
# group: daveturner_users
user::rwx
group::r-x
group:ksu-cis-hpc:r-x
mask::r-x
other::---
default:user::rwx
default:group::r-x
default:group:ksu-cis-hpc:r-x
default:mask::r-x
default:other::---
</syntaxhighlight>

ACLs give you great flexibility in controlling file access at the
group level. Below is a more advanced example where I set up a directory to be shared with
my ksu-cis-hpc group, Dan's ksu-cis-dan group, and an individual user 'mozes' who I also want
to have write access.

<syntaxhighlight lang=bash>
mkdir share_hpc_dan_mozes
# acls are used here for setting default permissions
setfacl -d -m g:ksu-cis-hpc:rX -R share_hpc_dan_mozes
setfacl -d -m g:ksu-cis-dan:rX -R share_hpc_dan_mozes
setfacl -d -m u:mozes:rwX -R share_hpc_dan_mozes
# ACLs are used here for setting actual permissions
setfacl -m g:ksu-cis-hpc:rX -R share_hpc_dan_mozes
setfacl -m g:ksu-cis-dan:rX -R share_hpc_dan_mozes
setfacl -m u:mozes:rwX -R share_hpc_dan_mozes

getfacl share_hpc_dan_mozes

# file: share_hpc_dan_mozes
# owner: daveturner
# group: daveturner_users
user::rwx
user:mozes:rwx
group::r-x
group:ksu-cis-hpc:r-x
group:ksu-cis-dan:r-x
mask::r-x
other::---
default:user::rwx
default:user:mozes:rwx
default:group::r-x
default:group:ksu-cis-hpc:r-x
default:group:ksu-cis-dan:r-x
default:mask::r-x
default:other::--x
</syntaxhighlight>

===Openly sharing files on the web===

If you create a 'public_html' directory on your home directory, then any files put there will be shared
openly on the web. There is no way to restrict who has access to those files.

<syntaxhighlight lang=bash>
cd
mkdir public_html
# Opt-in to letting the webserver access your home directory:
setfacl -m g:public_html:x ~/
</syntaxhighlight>

Then access the data from a web browser using the URL:

http://people.beocat.ksu.edu/~your_user_name

This will show a list of the files you have in your public_html subdirectory.

===Globus===

We have a page here dedicated to [[Globus]]

== Array Jobs ==
One of Slurm's useful options is the ability to run "Array Jobs"

It can be used with the following option to sbatch.

--array=n[-m[:s]]
Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Slurm
almost like a series of jobs. The option argument to --array specifies the number of array job tasks and the index number which will be
associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SLURM_ARRAY_TASK_ID. The option
arguments n, and m will be available through the environment variables SLURM_ARRAY_TASK_MIN and SLURM_ARRAY_TASK_MAX.

The task id range specified in the option argument may be a single number, a simple range of the form n-m or a range with a step size.
Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each
with the environment variable SLURM_ARRAY_TASK_ID containing one of the 5 index numbers.

Array jobs are commonly used to execute the same type of operation on varying input data sets correlated with the task index number. The
number of tasks in a array job is unlimited.

STDOUT and STDERR of array job tasks follow a slightly different naming convention (which can be controlled in the same way as mentioned above).

slurm-%A_%a.out

%A is the SLURM_ARRAY_JOB_ID, and %a is the SLURM_ARRAY_TASK_ID

=== Examples ===
==== Change the Size of the Run ====
Array Jobs have a variety of uses, one of the easiest to comprehend is the following:

I have an application, app1 I need to run the exact same way, on the same data set, with only the size of the run changing.

My original script looks like this:

<syntaxhighlight lang="bash">
#!/bin/bash
RUNSIZE=50
#RUNSIZE=100
#RUNSIZE=150
#RUNSIZE=200
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
For every run of that job I have to change the RUNSIZE variable, and submit each script. This gets tedious.

With Array Jobs the script can be written like so:

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=50-200:50
RUNSIZE=$SLURM_ARRAY_TASK_ID
app1 $RUNSIZE dataset.txt
</syntaxhighlight>
I then submit that job, and Slurm understands that it needs to run it 4 times, once for each task. It also knows that it can and should run these tasks in parallel.

==== Choosing a Dataset ====
A slightly more complex use of Array Jobs is the following:

I have an application, app2, that needs to be run against every line of my dataset. Every line changes how app2 runs slightly, but I need to compare the runs against each other.

Originally I had to take each line of my dataset and generate a new submit script and submit the job. This was done with yet another script:

<syntaxhighlight lang="bash">
#!/bin/bash
DATASET=dataset.txt
scriptnum=0
while read LINE
do
echo "app2 $LINE" > ${scriptnum}.sh
sbatch ${scriptnum}.sh
scriptnum=$(( $scriptnum + 1 ))
done < $DATASET
</syntaxhighlight>
Not only is this needlessly complex, it is also slow, as sbatch has to verify each job as it is submitted. This can be done easily with array jobs, as long as you know the number of lines in the dataset. This number can be obtained like so: wc -l dataset.txt in this case lets call it 5000.

<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --array=1-5000
app2 `sed -n "${SLURM_ARRAY_TASK_ID}p" dataset.txt`
</syntaxhighlight>
This uses a subshell via `, and has the sed command print out only the line number $SLURM_ARRAY_TASK_ID out of the file dataset.txt.

Not only is this a smaller script, it is also faster to submit because it is one job instead of 5000, so sbatch doesn't have to verify as many.

To give you an idea about time saved: submitting 1 job takes 1-2 seconds. by extension if you are submitting 5000, that is 5,000-10,000 seconds, or 1.5-3 hours.

== Checkpoint/Restart using DMTCP ==

DMTCP is Distributed Multi-Threaded CheckPoint software that will checkpoint your application without modification, and
can be set up to automatically restart your job from the last checkpoint if for example the node you are running on fails.
This has been tested successfully
on Beocat for some scalar and OpenMP codes, but has failed on all MPI tests so far. We would like to encourage users to
try DMTCP out if their non-MPI jobs run longer than 24 hours. If you want to try this, please contact us first since we are still
experimenting with DMTCP.

The sample job submission script below shows how dmtcp_launch is used to start the application, then dmtcp_restart is used to start from a checkpoint if the job has failed and been rescheduled.
If you are putting this in an array script, then add the Slurm array task ID to the end of the ckeckpoint directory name
like ckptdir=ckpt-$SLURM_ARRAY_TASK_ID.

#!/bin/bash -l
#SBATCH --job-name=gromacs
#SBATCH --mem=50G
#SBATCH --time=24:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module reset
module load GROMACS/2016.4-foss-2017beocatb-hybrid
module load DMTCP
module list

ckptdir=ckpt
mkdir -p $ckptdir
export DMTCP_CHECKPOINT_DIR=$ckptdir

if ! ls -1 $ckptdir | grep -c dmtcp_restart_script > /dev/null
then
echo "Using dmtcp_launch to start the app the first time"
dmtcp_launch --no-coordinator mpirun -np 1 -x OMP_NUM_THREADS=4 gmx_mpi mdrun -nsteps 50000 -ntomp 4 -v -deffnm 1ns -c 1ns.pdb -nice 0
else
echo "Using dmtcp_restart from $ckptdir to continue from a checkpoint"
dmtcp_restart $ckptdir/*.dmtcp
fi

You will need to run several tests to verify that DMTCP is working properly with your application.
First, run a short test without DMTCP and another with DMTCP with the checkpoint interval set to 5 minutes
by adding the line export DMTCP_CHECKPOINT_INTERVAL=300 to your script. Then use kstat -d 1 to
check that the memory in both runs is close to the same. Also use this information to calculate the time
that each checkpoint takes. In most cases I've seen times less than a minute for checkpointing that will normally
be done once each hour. If your application is taking more time, let us know. Sometimes this can be sped up
by simply turning off compression by adding the line export DMTCP_GZIP=0. Make sure to remove the
line where you set the checkpoint interval to 300 seconds so that the default time of once per hour will be used.

After verifying that your code completes using DMTCP and does not take significantly more time or memory, you
will need to start a run then scancel it after the first checkpoint, then resubmit the same script to make
sure that it restarts and runs to completion. If you are working with an array job script, the last is to try a few
array tasks at once to make sure there is no conflict between the jobs.

== Running jobs interactively ==
Some jobs just don't behave like we think they should, or need to be run with somebody sitting at the keyboard and typing in response to the output the computers are generating. Beocat has a facility for this, called 'srun'. srun uses the exact same command-line arguments as sbatch, but you need to add the following arguments at the end: <tt>--pty bash</tt>. If no node is available with your resource requirements, srun will tell you something like the following:
srun --pty bash
srun: Force Terminated job 217
srun: error: CPU count per node can not be satisfied
srun: error: Unable to allocate resources: Requested node configuration is not available
Note that, like sbatch, your interactive job will timeout after your allotted time has passed.

== Connecting to an existing job ==
You can connect to an existing job using srun in the same way that the MonitorNode command
allowed us to in the old cluster. This is essentially like using ssh to get into the node where your job is running which
can be very useful in allowing you to look at files in /tmp/job# or in running htop to view the
activity level for your job.

srun --jobid=# --pty bash where '#' is the job ID number

== Altering Job Requests ==
We generally do not support users to modify job parameters once the job has been submitted. It can be done, but there are numerous catches, and all of the variations can be a bit problematic; it is normally easier to simply delete the job (using '''scancel ''jobid''''') and resubmit it with the right parameters. '''If your job doesn't start after modifying such parameters (after a reasonable amount of time), delete the job and resubmit it.'''

As it is unsupported, this is an excercise left to the reader. A starting point is <tt>man scontrol</tt>
== Killable jobs ==
There are a growing number of machines within Beocat that are owned by a particular person or group. Normally jobs from users that aren't in the group designated by the owner of these machines cannot use them. This is because we have guaranteed that the nodes will be accessible and available to the owner at any given time. We will allow others to use these nodes if they designate their job as "killable." If your job is designated as killable, your job will be able to use these nodes, but can (and will) be killed off at any point in time to make way for the designated owner's jobs. Jobs that are marked killable will be re-queued and may restart on another node.

The way you would designate your job as killable is to add <tt>--gres=killable:1</tt> to the '''<tt>sbatch</tt> or <tt>srun</tt>''' arguments. This could be either on the command-line or in your script file.

''Note: This is a submit-time only request, it cannot be added by a normal user after the job has been submitted.'' If you would like jobs modified to be '''killable''' after the jobs have been submitted (and it is too much work to <tt>scancel</tt> the jobs and re-submit), send an e-mail to the administrators detailing the job ids and what you would like done.

== Scheduling Priority ==
Some users are members of projects that have contributed to Beocat. When those users have contributed nodes, the group gets access to a "partition" giving you priority on those nodes.

In most situations, the scheduler will automatically add those priority partitions to the jobs as submitted. You should not need to include a partition list in your job submission.

There are currently just a few exceptions that we will not automatically add:
* ksu-chem-mri.q
* ksu-gen-gpu.q
* ksu-gen-highmem.q

If you have access to those any of the non-automatic partitions, and have need of the resources in that partition, you can then alter your <tt>#SBATCH</tt> lines to include your new partition:
#SBATCH --partition=ksu-gen-highmem.q

Otherwise, you shouldn't modify the partition line at all unless you really know what you're doing.

== Graphical Applications ==
Some applications are graphical and need to have some graphical input/output. We currently accomplish this with X11 forwarding or [[OpenOnDemand]]
=== OpenOnDemand ===
[[OpenOnDemand]] is likely the easier and more performant way to run a graphical application on the cluster.
# visit [https://ondemand.beocat.ksu.edu/ ondemand] and login with your cluster credentials.
# Check the "Interactive Apps" dropdown. We may have a workflow ready for you. If not choose the desktop.
# Select the resources you need
# Select launch
# A job is now submitted to the cluster and once the job is started you'll see a Connect button
# use the app as needed. If using the desktop, start your graphical application.

=== X11 Forwarding ===
==== Connecting with an X11 client ====
===== Windows =====
If you are running Windows, we recommend MobaXTerm as your file/ssh manager, this is because it is one relatively simple tool to do everything. MobaXTerm also automatically connects with X11 forwarding enabled.
===== Linux/OSX =====
Both Linux and OSX can connect in an X11 forwarding mode. Linux will have all of the tools you need installed already, OSX will need [https://www.xquartz.org/ XQuartz] installed.

Then you will need to change your 'ssh' command slightly:

ssh -Y eid@headnode.beocat.ksu.edu

The '''-Y''' argument tells ssh to setup X11 forwarding.
==== Starting an Graphical job ====
All graphical jobs, by design, must be interactive, so we'll use the srun command. On a headnode, we run the following:
# load an X11 enabled application
module load Octave
# start an X11 job, sbatch arguments are accepted for srun as well, 1 node, 1 hour, 1 gb of memory
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 octave --gui

Because these jobs are interactive, they may not be able to run at all times, depending on how busy the scheduler is at any point in time. '''--pty --x11''' are required arguments setting up the job, and '''octave --gui''' is the command to run inside the job.

== Job Accounting ==
Some people may find it useful to know what their job did during its run. The sacct tool will read Slurm's accounting database and give you summarized or detailed views on jobs that have run within Beocat.
=== sacct ===
This data can usually be used to diagnose two very common job failures.
==== Job debugging ====
It is simplest if you know the job number of the job you are trying to get information on.
<syntaxhighlight lang="bash">
# if you know the jobid, put it here:
sacct -j 1122334455 -l
# if you don't know the job id, you can look at your jobs started since some day:
sacct -S 2017-01-01
</syntaxhighlight>

===== My job didn't do anything when it ran! =====
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|-
|3
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|218||218||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||12||00:00:00||FAILED||2:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=12,mem=1G,node=1||cpu=12,mem=1G,node=1
|-
|218.batch||218.batch||batch||||137940K||dwarf37||0||137940K||1576K||dwarf37||0||1576K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||1.36G||0||0||0||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|-
|218.0||218.0||qqqqstat||||204212K||dwarf37||0||204212K||1420K||dwarf37||0||1420K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||12||00:00:00||FAILED||2:0||196.52M||Unknown||Unknown||Unknown||1Gn||0||0||dwarf37||65534||0||0.00M||dwarf37||0||0.00M||||||||cpu=12,mem=1G,node=1
|}</div> 
If you look at the columns showing Elapsed and State, you can see that they show 00:00:00 and FAILED respectively. This means that the job started and then promptly ended. This points to something being wrong with your submission script. Perhaps there is a typo somewhere in it.

===== My job ran but didn't finish! =====
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|-
|3
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|220||220||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:01:27||TIMEOUT||0:0||||Unknown||Unknown||Unknown||1Gn||||||||||||||||||||||||cpu=1,mem=1G,node=1||cpu=1,mem=1G,node=1
|-
|220.batch||220.batch||batch||||370716K||dwarf37||0||370716K||7060K||dwarf37||0||7060K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:28||CANCELLED||0:15||1.23G||0||0||0||1Gn||0||0.16M||dwarf37||0||0.16M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|-
|220.0||220.0||sleep||||204212K||dwarf37||0||107916K||1000K||dwarf37||0||620K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:01:27||CANCELLED||0:15||1.54G||Unknown||Unknown||Unknown||1Gn||0||0.05M||dwarf37||0||0.05M||0.00M||dwarf37||0||0.00M||||||||cpu=1,mem=1G,node=1
|}</div> 
If you look at the column showing State, we can see some pointers to the issue. The job ran out of time (TIMEOUT) and then was killed (CANCELLED).
{| class="wikitable" style="float:left; margin:0; margin-right:-1px; {{{style|}}}
|-
|  
|-
|1
|-
|2
|}
<div style="overflow-x:auto; white-space:nowrap;">
{| class="wikitable" style="margin:0; {{{style|}}}"
|-
!JobID!!JobIDRaw!!JobName!!Partition!!MaxVMSize!!MaxVMSizeNode!!MaxVMSizeTask!!AveVMSize!!MaxRSS!!MaxRSSNode!!MaxRSSTask!!AveRSS!!MaxPages!!MaxPagesNode!!MaxPagesTask!!AvePages!!MinCPU!!MinCPUNode!!MinCPUTask!!AveCPU!!NTasks!!AllocCPUS!!Elapsed!!State!!ExitCode!!AveCPUFreq!!ReqCPUFreqMin!!ReqCPUFreqMax!!ReqCPUFreqGov!!ReqMem!!ConsumedEnergy!!MaxDiskRead!!MaxDiskReadNode!!MaxDiskReadTask!!AveDiskRead!!MaxDiskWrite!!MaxDiskWriteNode!!MaxDiskWriteTask!!AveDiskWrite!!AllocGRES!!ReqGRES!!ReqTRES!!AllocTRES
|-
|221||221||slurm_simple.sh||batch.q||||||||||||||||||||||||||||||||||||1||00:00:00||CANCELLED by 0||0:0||||Unknown||Unknown||Unknown||1Mn||||||||||||||||||||||||cpu=1,mem=1M,node=1||cpu=1,mem=1M,node=1
|-
|221.batch||221.batch||batch||||137940K||dwarf37||0||137940K||1144K||dwarf37||0||1144K||0||dwarf37||0||0||00:00:00||dwarf37||0||00:00:00||1||1||00:00:01||CANCELLED||0:15||2.62G||0||0||0||1Mn||0||0||dwarf37||65534||0||0||dwarf37||65534||0||||||||cpu=1,mem=1M,node=1
|}</div> 
If you look at the column showing State, we see it was "CANCELLED by 0", then we look at the AllocTRES column to see our allocated resources, and see that 1MB of memory was granted. Combine that with the column "MaxRSS" and we see that the memory granted was less than the memory we tried to use, thus the job was "CANCELLED".

Nautilus

2025-06-23T20:17:09Z

Nathanrwells:

== Nautilus ==
To access the Nautilus namespace, login using K-State SSO at https://portal.nrp-nautilus.io/ . Once you have done so, email beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] and request to be added to the Beocat Nautilus namespace (ksu-nrp-cluster). Once you have received notification that you have been added to the namespace, you can continue with the following steps to get set up to use the cluster resources.
<ol>
<li>SSH into headnode.beocat.ksu.edu</li>
<li>SSH into fiona (fiona hosts the kubectl tool we will use for this)</li>
<li>Once on fiona, use the command ‘cd ~’ to ensure you are in your home directory. If you
are not, this will return you to the top level of your home directory.<li>
<li>From there you will need to create a .kube directory inside of your home directory. Use
the command ‘mkdir ~/.kube’</li>
<li>Login to https://portal.nrp-nautilus.io/ using the same login previously used to create your
account (this will be your K-State EID login)</li>
<li>From here it is MANDATORY to read the cluster policy documentation provided by the
National Research Platform for the Nautilus program. You can find this here.
https://docs.nationalresearchplatform.org/userdocs/start/policies/ </li>
a. This is to ensure we do not break any of the rules put in place by the NRP.
 b. Return to https://portal.nrp-nautilus.io/ and accept the Acceptable Use Policy (AUP)
<li>Next, return to the website specified in step 5, in the top right corner of the page press
the “Get Config” option. </li>
a. This will download a file called ‘config’
<li>You will need to move the file to your ~/.kube directory created in step 4.</li>
a. To do this you can copy and paste the contents through the command line
 b. You can also utilize the OpenOnDemand tool to upload the file through the web
interface. Information for this tool can be found here:
https://support.beocat.ksu.edu/Docs/OpenOnDemand
 c. You can also use other means of moving the contents to the Beocat
headnodes/your home directory, but these are just a few examples.
 d. NOTE: Because we added a period before the directory name it is now a hidden directory,
and the directory will not appear when running a normal ‘ls’, to see the directory you will
need to run “ls -a” or “ls -la”.
<li>Once you have read the required documentation, created the .kube directory in your
home directory, and placed the config file in the '~/.kube' directory, you are now ready to continue!</li>
<li>Below is an example pod that can be used. It does not request much in the way of resources so you will likely need to change some things. Be sure to change the “name:” field
underneath “metadata:”. Change the text “test-pod” to “{eid}-pod” where ‘{eid}’ is your
K-State ID. It will look something like this “dan-pod”.</li>

<syntaxhighlight lang=yaml>
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: mypod
image: ubuntu
resources:
limits:
memory: 400Mi
cpu: 100m
requests:
memory: 100Mi
cpu: 100m
command: ["sh", "-c", "echo 'Im a new pod'"]
</syntaxhighlight>
<li>Place your .yaml file in the same directory created earlier (~/.kube).</li>
<li>If you are not already in the .kube directory enter the command “cd ~/.kube” to change
your current directory.</li>
<li>Now we are going to create our ‘pod’. This will request a ubuntu pc using the
specifications from above.</li>
a. To do this enter the command “kubectl create -f pod1.yaml” NOTE: You must be
in the same directory that you placed the pod1.yaml file in (in this situation, the above pod config was put into a file named pod1.yaml).
 b. If the command is successful you will see an output of “pod/{eid}-pod created”.
<li>You will need to wait until the container for the pod is finished creating. You can check
this by running “kubectl get pods”</li>
a. Once you run this command, it will output all the pods currently running or being
created in the namespace. Look for yours among the list of pods, the name will
be the same name specified in step 10.
 b. Once you locate your pod, check its STATUS. If the pod says Running, then you
are good to proceed. If it says Container Creating, then you will need to wait just a
bit. It should not take long.
<li>You can now execute and enter the pod by running “kubectl exec -it {eid}-pod --
/bin/bash”. Where ‘{eid}-pod’ is the pod created in step 13/the name specified in step 10.</li>
a. Executing this command will open the pod you created and run a bash console
on the pod.
 b. NOTE: If you have trouble logging into the pod, and are met with a “You must be
logged in to the server, you can run “kubectl proxy”, and after a moment, you can
cancel the command with a “crtl+c”. This should remedy the error.
</ol>

Additional documentation for Kubernetes can be found on the Kubernetes website https://kubernetes.io/docs/home

CUDA

2025-06-23T20:16:35Z

Nathanrwells:

== CUDA Overview ==
[[wikipedia:CUDA|CUDA]] is a feature set for programming nVidia [[wikipedia:Graphics_processing_unit|GPUs]]. We have many dwarf nodes that are CUDA-enabled with 1-2 GPUs and most of the Wizard nodes have 4 GPUs each. Most of these are consumer grade [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ nVidia 1080 Ti graphics cards] that are good for accelerating 32-bit calculations. Dwarf36-38 have two [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ nVidia RTX A4000 graphic cards] and dwarf39 has two [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ nVidia 1080 Ti graphics cards] that are available for anybody to use but you'll need to email beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] to request being added to the GPU priority group then you'll need to submit jobs with --partition=ksu-gen-gpu.q. Wizard20 and wizard21 each have two [https://www.nvidia.com/object/quadro-graphics-with-pascal.html nVidia P100 cards] that are much more costly than the consumer grade 1080Ti cards but can accelerate 64-bit calculations much better.

== Training videos ==
CUDA Programming Model Overview: [http://www.youtube.com/watch?v=aveYOlBSe-Y http://www.youtube.com/watch?v=aveYOlBSe-Y]

{{#widget:YouTube|id=aveYOlBSe-Y|width=800|height=600}}

CUDA Programming Basics Part I (Host functions): [http://www.youtube.com/watch?v=79VARRFwQgY http://www.youtube.com/watch?v=79VARRFwQgY]

{{#widget:YouTube|id=79VARRFwQgY|width=800|height=600}}

CUDA Programming Basics Part II (Device functions): [http://www.youtube.com/watch?v=G5-iI1ogDW4 http://www.youtube.com/watch?v=G5-iI1ogDW4]

{{#widget:YouTube|id=G5-iI1ogDW4|width=800|height=600}}
== Compiling CUDA Applications ==
nvcc is the compiler for CUDA applications. When compiling your applications manually you will need to load a CUDA enabled compiler toolchain (e.g. fosscuda):

* module load fosscuda
* '''Do not run your cuda applications on the headnode. I cannot guarantee it will run, and it will give you terrible results if it does run.'''

With those two things in mind, you can compile CUDA applications as follows:

<syntaxhighlight lang="bash">
module load fosscuda
nvcc <source>.cu -o <output>
</syntaxhighlight>

== Example ==
=== Create your Application ===
Copy the following Application into Beocat as vecadd.cu
<syntaxhighlight lang="c">
// Kernel definition, see also section 4.2.3 of Nvidia Cuda Programming Guide
__global__ void vecAdd(float* A, float* B, float* C)
{
// threadIdx.x is a built-in variable provided by CUDA at runtime
int i = threadIdx.x;
A[i]=0;
B[i]=i;
C[i] = A[i] + B[i];
}

#include <stdio.h>
#define SIZE 10
int main()
{
int N=SIZE;
float A[SIZE], B[SIZE], C[SIZE];
float *devPtrA;
float *devPtrB;
float *devPtrC;
int memsize= SIZE * sizeof(float);

cudaMalloc((void**)&devPtrA, memsize);
cudaMalloc((void**)&devPtrB, memsize);
cudaMalloc((void**)&devPtrC, memsize);
cudaMemcpy(devPtrA, A, memsize, cudaMemcpyHostToDevice);
cudaMemcpy(devPtrB, B, memsize, cudaMemcpyHostToDevice);
// __global__ functions are called: Func<<< Dg, Db, Ns >>>(parameter);
vecAdd<<<1, N>>>(devPtrA, devPtrB, devPtrC);
cudaMemcpy(C, devPtrC, memsize, cudaMemcpyDeviceToHost);

for (int i=0; i<SIZE; i++)
printf("C[%d]=%f\n",i,C[i]);

cudaFree(devPtrA);
cudaFree(devPtrA);
cudaFree(devPtrA);

}
</syntaxhighlight>
=== Gain Access to a CUDA-capable Node ===
See our [[AdvancedSlurm|advanced scheduler documentation]]
=== Compile Your Application ===
<syntaxhighlight lang="bash">
module load fosscuda
nvcc vecadd.cu -o vecadd
</syntaxhighlight>
This will create a program with the name 'vecadd' (specified by the '-o' flag).

=== Run Your Application ===
Run the program as you usually would, namely
<syntaxhighlight lang="bash">
./vecadd
</syntaxhighlight>

Assuming you don't want to run the program interactively because this is a large job, you can submit a job via sbatch, just be sure to add '<tt>--gres=gpu:1</tt>' to the '''sbatch''' directive.

Galaxy File Upload

2025-06-23T20:16:06Z

Nathanrwells:

= Large File Uploads to Galaxy =
The Galaxy web UI is sometimes inconsistent with large downloads. As such, we have made user directory imports available on Galaxy. You can upload files to /bulk/galaxy/user-space/, and locating the user space that was created for you on first login. The folder is usually named "eid@ksu.edu" or if you are from another college (VetMed for instance) "eid@vet.k-state.edu". You can use the command "ls /bulk/galaxy/user-space" to find the name of your directory.

From here, we have written some instructions on how to utilize this upload method.
== Upload Files ==
First, you need to transfer the data onto Beocat, this can be done through any number of ways. We have some documentation on uploading data into Beocat that can be found here (for large files, we suggest using Globus, scp or an FTP program): https://support.beocat.ksu.edu/Docs/Main_Page#Transferring_data_to_Beocat

Next, due to how Galaxy handles data exposure to the web UI, we need to move this data to a userspace that was created for you in Galaxy on your first login:

Move the data to the following directory (You should be able to move data here, if you are unable to, please let us know at beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]). Note that the backwards slash is a necessary character to escape the @ symbol from your email address:
/bulk/galaxy/user-space/eid\@vet.k-state.edu/

Next, login to galaxy.beocat.ksu.edu with your eID through KeyCloak.

Then, navigate to the "Shared Data" tab at the top of the page, in the drop-down menu, navigate to "Data Libraries". This should take you to a relatively empty page that allows you to create a library with a "+ Library" in the top left of the web page. For reference, I have included a screenshot of my Data Libraries:

[[File:Data library creation.png|CreatingDataLibrary]]

Create a new Library with any name, and then open it by clicking on its name. In this case, from the photo above, you could click on "Test2" or "Upload".

This takes you to a page that will let you manage the data inside of that library. It should look like this:

[[File:Library explanation.png|ExplainingDataLibrary]]

From here you can either add a folder to help manage your data, upload data to the library, or add the current data in the library to your History so that it can be accessed. We are going to upload data to the library. Clicking "+ Datasets" will expand a drop-down menu, from here, select "from User Directory". This will allow you to upload any data to galaxy from that /bulk directory we moved your data to earlier on Beocat. That process looks something like this:

[[File:Import files.png|ImportToDataLibrary]]

In this, I just uploaded two empty text files. From here, it has to do some auto-magic to get the data to work inside galaxy, with the "state" of the data. I am not exactly sure what this, or how long it might take. I would imagine not long.

From here we finally need to publish the data to our history as a so that we can utilize it. I am going to import this data as a Dataset, though if a collection suits better, use that. In the photo below, I am uploading "iamanothertextfile.txt" to my new history called "IAmANewHistory".

Once that is added a notification should pop up in the bottom right hand corner of the browser. If not, navigate to the homepage and manually open the History your data was added to. From here you should be able to process your data like normal.

ProposalDescription

2025-06-23T20:15:32Z

Nathanrwells: /* Description of Beocat for Proposals or Information */

== Description of Beocat for Proposals or Information ==

Below is a current description of Beocat Compute resources and availability that can be used within proposals or to provide information. If you have any questions regarding this, please reach out to us at beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket].

=== Compute Resources Description (Updated Oct. 1 2024) ===

Beocat, the K-State research computing cluster, is currently the largest academic supercomputer in Kansas. Its hardware includes nearly 400 researcher-funded computers, approximately 3.3PB of storage, ~10,000 processor cores on machines ranging from dual-processor Xeon e5 nodes with 128GB RAM and 100GbE to 128 core AMD nodes with 2TB RAM connected by 40-100 Gbps networks, and a total of 170 different GPUs ranging from GTX 1080ti's to NVIDIA L40S's. Beocat and its staff have provided tours demonstrating the value of K-State research and a high-tech look at our research facilities for over 3,000 participants, including USD383 StarBase, current and prospective students, funding agencies, faculty recruitment, and outreach activities. Classes supported include topics such as bioinformatics, business analytics, cybersecurity, data science, deep learning, economics, chemistry, and genetics. Beocat is supported by many NSF and university grants, and it acts as the central computing resource for multiple departments across campus. Beocat staff includes one full-time system administrators, a full-time applications scientist with a PhD in Physics and 35 years’ experience optimizing parallel programs and assisting researchers, and a part-time director.

Beocat is available to any academic researcher in Kansas and their partners under the statewide KanShare MOU. Under current policy, heavy users are expected to buy in through adding computational or personnel resources for the cluster (condo computing). Their jobs, then, are given guaranteed priority on any contributed machines, and they have access to other resources in the cluster on an as-available basis. Thus, projects can preserve a guaranteed base level of computation while utilizing the larger cluster for major computations. Users can also purchase archival data storage as needed. Dr. Daniel Andresen is the K-State XSEDE Campus Champion in the event national-class computational resources are required.

ProposalDescription

2025-06-23T20:15:08Z

Nathanrwells: /* Compute Resources Description */

== Description of Beocat for Proposals or Information ==

Below is a current description of Beocat Compute resources and availability that can be used within proposals or to provide information. If you have any questions regarding this, please reach out to us at beocat@cs.ksu.edu

=== Compute Resources Description (Updated Oct. 1 2024) ===

Beocat, the K-State research computing cluster, is currently the largest academic supercomputer in Kansas. Its hardware includes nearly 400 researcher-funded computers, approximately 3.3PB of storage, ~10,000 processor cores on machines ranging from dual-processor Xeon e5 nodes with 128GB RAM and 100GbE to 128 core AMD nodes with 2TB RAM connected by 40-100 Gbps networks, and a total of 170 different GPUs ranging from GTX 1080ti's to NVIDIA L40S's. Beocat and its staff have provided tours demonstrating the value of K-State research and a high-tech look at our research facilities for over 3,000 participants, including USD383 StarBase, current and prospective students, funding agencies, faculty recruitment, and outreach activities. Classes supported include topics such as bioinformatics, business analytics, cybersecurity, data science, deep learning, economics, chemistry, and genetics. Beocat is supported by many NSF and university grants, and it acts as the central computing resource for multiple departments across campus. Beocat staff includes one full-time system administrators, a full-time applications scientist with a PhD in Physics and 35 years’ experience optimizing parallel programs and assisting researchers, and a part-time director.

Beocat is available to any academic researcher in Kansas and their partners under the statewide KanShare MOU. Under current policy, heavy users are expected to buy in through adding computational or personnel resources for the cluster (condo computing). Their jobs, then, are given guaranteed priority on any contributed machines, and they have access to other resources in the cluster on an as-available basis. Thus, projects can preserve a guaranteed base level of computation while utilizing the larger cluster for major computations. Users can also purchase archival data storage as needed. Dr. Daniel Andresen is the K-State XSEDE Campus Champion in the event national-class computational resources are required.

Main Page

2025-06-23T20:14:13Z

Nathanrwells:

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

=== Online Documentations ===

* Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/]
* Read about [[Installed software]] and languages
* Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] and download the [[Media:Slurm-quick-reference.pdf|Slurm Quick Reference PDF]]
* Run interactive jobs with [[OpenOnDemand]]
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]]
* Big Data course on Beocat! [[BigDataOnBeocat]]
* Interested in web-based computational biology research? Check out [[GalaxyDocs|Galaxy!]]
* Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]]

=== Training Videos and Slides ===

* [https://www.youtube.com/watch?v=7NOB_HGQE0U Beocat Introduction] and [[Media:Beocat-Beoshock-Intro.pdf|slides]]
* [https://www.youtube.com/watch?v=b_yawpwFRdk Linux and Bash Introduction] and [[Media:Linux-Intro-cheatsheet.pdf|Linux Quick Reference PDF]]
* [https://www.youtube.com/watch?v=vcC-DURbH6c Advanced HPC Usage] and [[Media:HPC-Advanced-Usage.pdf|slides]]
* [https://www.youtube.com/watch?v=inJbYdZacjs HPC Parallel Computing] and [[Media:HPC-Parallel-Computing.pdf|slides]]

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
 With the recent changes to how KState handles DUO authentication, we recommend you use Globus to transfer files in and out of Beocat
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocathelp@ksu.edu beocathelp@ksu.edu] or or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]. ''Please'' send all email to this address or through TDX and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you email us at beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket], please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

GalaxyDocs

2025-06-23T20:12:51Z

Nathanrwells:

== What is Galaxy? ==
[https://galaxyproject.org/ Galaxy] is a scientific workflow, data integration, and data and analysis persistence and publishing platform that aims to make computational biology accessible to research scientists that do not have computer programming or systems administration experience.

== How do I access Galaxy? ==
Access to Beocat's local instance of Galaxy is easy. Simply navigate to [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/] and sign in if prompted using the Keycloak login. This will use your Beocat EID and Password to login, you will also be prompted to authenticate against DUO.

Please note that this utilizes Beocat's /bulk directory. This is a billed directory, and as such, if usage in your respective upload directory for Galaxy exceeds the billing threshold (1T), we will contact you regarding remediation.

== Upload Larger Files to Galaxy ==
Have some larger files that need to be uploaded to Galaxy? We provide some documentation on how to upload files directly to Beocat and then import them to Galaxy for use. It can be found here: [[Galaxy_File_Upload| Galaxy File Upload]]

== How do I use Galaxy? ==

After accessing our local instance of Galaxy you should have access to all installed tools which will be in the tool panel on the left-hand side of your home screen on Galaxy. It should look like this:

[[File:Galaxy Toolbox.png|Tool Panel]]

Each category (Get Data, Send Data, MetaGenomics Tools, etc) is expandable reveal subcategories that help sort out many installed tools. Each tool will be placed under its respective category once it is installed.

From here you can select a tool to use and submit to the Slurm cluster. After opening a tool you will be given options (if available) to configure the tool, including its inputs, what the tool will do (within its constraints) outputs, and what compute resources the tool will submit with. This means that you can specify the resources given to slurm jobs by expanding the drop-down menu "Job Resource Paramters" and selecting "Specify Job resource parameters" which will show the following options to modify.

Processors: Number of processing cores, 'ppn' value (1-128) this is equivalent to slurms "--cpus-per-task". 
Memory: Memory size in gigabytes, 'pmem' value (1-1500). Note that the job scheduler uses --mem-per-cpu to allocate memory for your slurm job. This mean the number given for memory will be multiplied by the Processors count from above. I.e 2 Processors with 5 Memory will be 10GB of memory. 
Priority Queue: If you have access to a priority queue, and would like to use it, enter the partition name here. 
Runtime: How long you want your job to run for in hours.

If you fail to switch your job resource parameters, your job will submit with the default resources. This means it will submit with 5Gb of Memory and 1 CPU with 1 hour of Runtime and no Priority queue.

=== Canceling a running job ===
While Galaxy does submit to slurm, you will not be able to cancel the job in the same way you typically do. With Galaxy, to cancel your upcoming/currently running job simply press the trash can icon next the name of your job.

== Tool Requests ==
If you are missing a specific tool and would like to have it added to Galaxy, please contact beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket] with a link to the tool. Additionally, you can browse through Galaxy's own [https://toolshed.g2.bx.psu.edu/ toolshed] to make a recommendation.

== Data Management ==

Galaxy follows the typical costs for bulk data storage, as Galaxy utilizes /bulk/galaxy for storage. Bulk data storage may be provided at a cost of $45/TB/year billed monthly. Billing starting at 1TB of usage. Users can easily see how much data they are utilizing in galaxy by checking the top right corner of the 'home' page of galaxy. This will say "Using ####MB/GB/TB", above your histories.

[[File:Galaxy_Data_usage_example.png|Usage Data]]

Clicking on this usage will bring you to a storage dashboard where you can easily manage your files and derelict dataset histories.

[[File:Storage_dashboard.png|Storage Dashboard]]

== Requesting Help ==
To request help with [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/], please contact beocathelp@ksu.edu,or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]. 
When requesting help, it is best to give as much information as possible so that we may solve your issue in a timely manner.

== Acknowledgements ==
Beocat's installation of UseGalaxy is funded through K-INBRE with an Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under grant number P20GM103418.

This initiative was started through the Data Science Core group to bring easy to use GUI-based computational biology research to students and researchers at Kansas State University through Beocat.

Additional information on K-INBRE can be found [https://www.k-inbre.org/pages/k-inbre_about_bio-core.html here]

RSICC

2025-06-23T20:11:45Z

Nathanrwells: /* RSICC Codes */

== RSICC Codes ==
RSICC requires administrators of the system to be licensed for every code (and version) a user wishes to be on the system.

Let us know which RSICC software you'd like to run on Beocat. We will have to become licensed for it before it can be put on the system.

If it is an RSICC software that we've already obtained, you *must* provide proof of a license by providing us with an electronic copy of the email message from RSICC’s Request History tool or a copy of the user’s License Agreement. The email generated by RSICC’s Request History tool lists all of the software the individual has a license to use. The Request History link is available on RSICC’s Customer Service webpage, https://rsicc.ornl.gov/CustomerService.aspx Send those e-mails to beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket]

RSICC

2025-06-23T20:11:34Z

Nathanrwells: /* RSICC Codes */

== RSICC Codes ==
RSICC requires administrators of the system to be licensed for every code (and version) a user wishes to be on the system.

Let us know which RSICC software you'd like to run on Beocat. We will have to become licensed for it before it can be put on the system.

If it is an RSICC software that we've already obtained, you *must* provide proof of a license by providing us with an electronic copy of the email message from RSICC’s Request History tool or a copy of the user’s License Agreement. The email generated by RSICC’s Request History tool lists all of the software the individual has a license to use. The Request History link is available on RSICC’s Customer Service webpage, https://rsicc.ornl.gov/CustomerService.aspx Send those e-mails to beocathelp@ksu.edu or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 | TDX Ticket]

RSICC

2025-06-23T20:11:12Z

Nathanrwells: /* RSICC Codes */

== RSICC Codes ==
RSICC requires administrators of the system to be licensed for every code (and version) a user wishes to be on the system.

Let us know which RSICC software you'd like to run on Beocat. We will have to become licensed for it before it can be put on the system.

If it is an RSICC software that we've already obtained, you *must* provide proof of a license by providing us with an electronic copy of the email message from RSICC’s Request History tool or a copy of the user’s License Agreement. The email generated by RSICC’s Request History tool lists all of the software the individual has a license to use. The Request History link is available on RSICC’s Customer Service webpage, https://rsicc.ornl.gov/CustomerService.aspx Send those e-mails to beocathelp@ksu.edu or contact Beocat staff through a TDX Ticket [[https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 | TDXSupport]]

Main Page

2025-06-17T15:36:55Z

Nathanrwells: /* Transferring data to Beocat */

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

=== Online Documentations ===

* Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/]
* Read about [[Installed software]] and languages
* Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] and download the [[Media:Slurm-quick-reference.pdf|Slurm Quick Reference PDF]]
* Run interactive jobs with [[OpenOnDemand]]
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]]
* Big Data course on Beocat! [[BigDataOnBeocat]]
* Interested in web-based computational biology research? Check out [[GalaxyDocs|Galaxy!]]
* Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]]

=== Training Videos and Slides ===

* [https://www.youtube.com/watch?v=7NOB_HGQE0U Beocat Introduction] and [[Media:Beocat-Beoshock-Intro.pdf|slides]]
* [https://www.youtube.com/watch?v=b_yawpwFRdk Linux and Bash Introduction] and [[Media:Linux-Intro-cheatsheet.pdf|Linux Quick Reference PDF]]
* [https://www.youtube.com/watch?v=vcC-DURbH6c Advanced HPC Usage] and [[Media:HPC-Advanced-Usage.pdf|slides]]
* [https://www.youtube.com/watch?v=inJbYdZacjs HPC Parallel Computing] and [[Media:HPC-Parallel-Computing.pdf|slides]]

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
 With the recent changes to how KState handles DUO authentication, we recommend you use Globus to transfer files in and out of Beocat
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you email us at beocat@cs.ksu.edu please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

In the future, Beocat will be moving towards the Kstate Central IT TDX ticketing system. Please expect a change in how we handle user support in Summer/Fall of 2025.

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Main Page

2025-06-17T15:36:24Z

Nathanrwells:

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

=== Online Documentations ===

* Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/]
* Read about [[Installed software]] and languages
* Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] and download the [[Media:Slurm-quick-reference.pdf|Slurm Quick Reference PDF]]
* Run interactive jobs with [[OpenOnDemand]]
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]]
* Big Data course on Beocat! [[BigDataOnBeocat]]
* Interested in web-based computational biology research? Check out [[GalaxyDocs|Galaxy!]]
* Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]]

=== Training Videos and Slides ===

* [https://www.youtube.com/watch?v=7NOB_HGQE0U Beocat Introduction] and [[Media:Beocat-Beoshock-Intro.pdf|slides]]
* [https://www.youtube.com/watch?v=b_yawpwFRdk Linux and Bash Introduction] and [[Media:Linux-Intro-cheatsheet.pdf|Linux Quick Reference PDF]]
* [https://www.youtube.com/watch?v=vcC-DURbH6c Advanced HPC Usage] and [[Media:HPC-Advanced-Usage.pdf|slides]]
* [https://www.youtube.com/watch?v=inJbYdZacjs HPC Parallel Computing] and [[Media:HPC-Parallel-Computing.pdf|slides]]

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
With the recent changes to how KState handles DUO authentication, we recommend you use Globus to transfer files in and out of Beocat
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you email us at beocat@cs.ksu.edu please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

In the future, Beocat will be moving towards the Kstate Central IT TDX ticketing system. Please expect a change in how we handle user support in Summer/Fall of 2025.

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

GalaxyDocs

2025-04-14T17:22:21Z

Nathanrwells: /* How do I access Galaxy? */

== What is Galaxy? ==
[https://galaxyproject.org/ Galaxy] is a scientific workflow, data integration, and data and analysis persistence and publishing platform that aims to make computational biology accessible to research scientists that do not have computer programming or systems administration experience.

== How do I access Galaxy? ==
Access to Beocat's local instance of Galaxy is easy. Simply navigate to [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/] and sign in if prompted using the Keycloak login. This will use your Beocat EID and Password to login, you will also be prompted to authenticate against DUO.

Please note that this utilizes Beocat's /bulk directory. This is a billed directory, and as such, if usage in your respective upload directory for Galaxy exceeds the billing threshold (1T), we will contact you regarding remediation.

== Upload Larger Files to Galaxy ==
Have some larger files that need to be uploaded to Galaxy? We provide some documentation on how to upload files directly to Beocat and then import them to Galaxy for use. It can be found here: [[Galaxy_File_Upload| Galaxy File Upload]]

== How do I use Galaxy? ==

After accessing our local instance of Galaxy you should have access to all installed tools which will be in the tool panel on the left-hand side of your home screen on Galaxy. It should look like this:

[[File:Galaxy Toolbox.png|Tool Panel]]

Each category (Get Data, Send Data, MetaGenomics Tools, etc) is expandable reveal subcategories that help sort out many installed tools. Each tool will be placed under its respective category once it is installed.

From here you can select a tool to use and submit to the Slurm cluster. After opening a tool you will be given options (if available) to configure the tool, including its inputs, what the tool will do (within its constraints) outputs, and what compute resources the tool will submit with. This means that you can specify the resources given to slurm jobs by expanding the drop-down menu "Job Resource Paramters" and selecting "Specify Job resource parameters" which will show the following options to modify.

Processors: Number of processing cores, 'ppn' value (1-128) this is equivalent to slurms "--cpus-per-task". 
Memory: Memory size in gigabytes, 'pmem' value (1-1500). Note that the job scheduler uses --mem-per-cpu to allocate memory for your slurm job. This mean the number given for memory will be multiplied by the Processors count from above. I.e 2 Processors with 5 Memory will be 10GB of memory. 
Priority Queue: If you have access to a priority queue, and would like to use it, enter the partition name here. 
Runtime: How long you want your job to run for in hours.

If you fail to switch your job resource parameters, your job will submit with the default resources. This means it will submit with 5Gb of Memory and 1 CPU with 1 hour of Runtime and no Priority queue.

=== Canceling a running job ===
While Galaxy does submit to slurm, you will not be able to cancel the job in the same way you typically do. With Galaxy, to cancel your upcoming/currently running job simply press the trash can icon next the name of your job.

== Tool Requests ==
If you are missing a specific tool and would like to have it added to Galaxy, please contact beocat@cs.ksu.edu with a link to the tool. Additionally, you can browse through Galaxy's own [https://toolshed.g2.bx.psu.edu/ toolshed] to make a recommendation.

== Data Management ==

Galaxy follows the typical costs for bulk data storage, as Galaxy utilizes /bulk/galaxy for storage. Bulk data storage may be provided at a cost of $45/TB/year billed monthly. Billing starting at 1TB of usage. Users can easily see how much data they are utilizing in galaxy by checking the top right corner of the 'home' page of galaxy. This will say "Using ####MB/GB/TB", above your histories.

[[File:Galaxy_Data_usage_example.png|Usage Data]]

Clicking on this usage will bring you to a storage dashboard where you can easily manage your files and derelict dataset histories.

[[File:Storage_dashboard.png|Storage Dashboard]]

== Requesting Help ==
To request help with [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/], please contact beocat@cs.ksu.edu. 
When requesting help, it is best to give as much information as possible so that we may solve your issue in a timely manner.

== Acknowledgements ==
Beocat's installation of UseGalaxy is funded through K-INBRE with an Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under grant number P20GM103418.

This initiative was started through the Data Science Core group to bring easy to use GUI-based computational biology research to students and researchers at Kansas State University through Beocat.

Additional information on K-INBRE can be found [https://www.k-inbre.org/pages/k-inbre_about_bio-core.html here]

GalaxyDocs

2025-04-14T16:31:56Z

Nathanrwells:

== What is Galaxy? ==
[https://galaxyproject.org/ Galaxy] is a scientific workflow, data integration, and data and analysis persistence and publishing platform that aims to make computational biology accessible to research scientists that do not have computer programming or systems administration experience.

== How do I access Galaxy? ==
Access to Beocat's local instance of Galaxy is easy. Simply navigate to [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/] and sign in if prompted using the Keycloak login. This will use your Beocat EID and Password to login, you will also be prompted to authenticate against DUO.

== Upload Larger Files to Galaxy ==
Have some larger files that need to be uploaded to Galaxy? We provide some documentation on how to upload files directly to Beocat and then import them to Galaxy for use. It can be found here: [[Galaxy_File_Upload| Galaxy File Upload]]

== How do I use Galaxy? ==

After accessing our local instance of Galaxy you should have access to all installed tools which will be in the tool panel on the left-hand side of your home screen on Galaxy. It should look like this:

[[File:Galaxy Toolbox.png|Tool Panel]]

Each category (Get Data, Send Data, MetaGenomics Tools, etc) is expandable reveal subcategories that help sort out many installed tools. Each tool will be placed under its respective category once it is installed.

From here you can select a tool to use and submit to the Slurm cluster. After opening a tool you will be given options (if available) to configure the tool, including its inputs, what the tool will do (within its constraints) outputs, and what compute resources the tool will submit with. This means that you can specify the resources given to slurm jobs by expanding the drop-down menu "Job Resource Paramters" and selecting "Specify Job resource parameters" which will show the following options to modify.

Processors: Number of processing cores, 'ppn' value (1-128) this is equivalent to slurms "--cpus-per-task". 
Memory: Memory size in gigabytes, 'pmem' value (1-1500). Note that the job scheduler uses --mem-per-cpu to allocate memory for your slurm job. This mean the number given for memory will be multiplied by the Processors count from above. I.e 2 Processors with 5 Memory will be 10GB of memory. 
Priority Queue: If you have access to a priority queue, and would like to use it, enter the partition name here. 
Runtime: How long you want your job to run for in hours.

If you fail to switch your job resource parameters, your job will submit with the default resources. This means it will submit with 5Gb of Memory and 1 CPU with 1 hour of Runtime and no Priority queue.

=== Canceling a running job ===
While Galaxy does submit to slurm, you will not be able to cancel the job in the same way you typically do. With Galaxy, to cancel your upcoming/currently running job simply press the trash can icon next the name of your job.

== Tool Requests ==
If you are missing a specific tool and would like to have it added to Galaxy, please contact beocat@cs.ksu.edu with a link to the tool. Additionally, you can browse through Galaxy's own [https://toolshed.g2.bx.psu.edu/ toolshed] to make a recommendation.

== Data Management ==

Galaxy follows the typical costs for bulk data storage, as Galaxy utilizes /bulk/galaxy for storage. Bulk data storage may be provided at a cost of $45/TB/year billed monthly. Billing starting at 1TB of usage. Users can easily see how much data they are utilizing in galaxy by checking the top right corner of the 'home' page of galaxy. This will say "Using ####MB/GB/TB", above your histories.

[[File:Galaxy_Data_usage_example.png|Usage Data]]

Clicking on this usage will bring you to a storage dashboard where you can easily manage your files and derelict dataset histories.

[[File:Storage_dashboard.png|Storage Dashboard]]

== Requesting Help ==
To request help with [https://galaxy.beocat.ksu.edu/ https://galaxy.beocat.ksu.edu/], please contact beocat@cs.ksu.edu. 
When requesting help, it is best to give as much information as possible so that we may solve your issue in a timely manner.

== Acknowledgements ==
Beocat's installation of UseGalaxy is funded through K-INBRE with an Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under grant number P20GM103418.

This initiative was started through the Data Science Core group to bring easy to use GUI-based computational biology research to students and researchers at Kansas State University through Beocat.

Additional information on K-INBRE can be found [https://www.k-inbre.org/pages/k-inbre_about_bio-core.html here]

Galaxy File Upload

2025-04-14T16:30:26Z

Nathanrwells: /* Upload Files */

= Large File Uploads to Galaxy =
The Galaxy web UI is sometimes inconsistent with large downloads. As such, we have made user directory imports available on Galaxy. You can upload files to /bulk/galaxy/user-space/, and locating the user space that was created for you on first login. The folder is usually named "eid@ksu.edu" or if you are from another college (VetMed for instance) "eid@vet.k-state.edu". You can use the command "ls /bulk/galaxy/user-space" to find the name of your directory.

From here, we have written some instructions on how to utilize this upload method.
== Upload Files ==
First, you need to transfer the data onto Beocat, this can be done through any number of ways. We have some documentation on uploading data into Beocat that can be found here (for large files, we suggest using Globus, scp or an FTP program): https://support.beocat.ksu.edu/Docs/Main_Page#Transferring_data_to_Beocat

Next, due to how Galaxy handles data exposure to the web UI, we need to move this data to a userspace that was created for you in Galaxy on your first login:

Move the data to the following directory (You should be able to move data here, if you are unable to, please let us know at beocat@cs.ksu.edu). Note that the backwards slash is a necessary character to escape the @ symbol from your email address:
/bulk/galaxy/user-space/eid\@vet.k-state.edu/

Next, login to galaxy.beocat.ksu.edu with your eID through KeyCloak.

Then, navigate to the "Shared Data" tab at the top of the page, in the drop-down menu, navigate to "Data Libraries". This should take you to a relatively empty page that allows you to create a library with a "+ Library" in the top left of the web page. For reference, I have included a screenshot of my Data Libraries:

[[File:Data library creation.png|CreatingDataLibrary]]

Create a new Library with any name, and then open it by clicking on its name. In this case, from the photo above, you could click on "Test2" or "Upload".

This takes you to a page that will let you manage the data inside of that library. It should look like this:

[[File:Library explanation.png|ExplainingDataLibrary]]

From here you can either add a folder to help manage your data, upload data to the library, or add the current data in the library to your History so that it can be accessed. We are going to upload data to the library. Clicking "+ Datasets" will expand a drop-down menu, from here, select "from User Directory". This will allow you to upload any data to galaxy from that /bulk directory we moved your data to earlier on Beocat. That process looks something like this:

[[File:Import files.png|ImportToDataLibrary]]

In this, I just uploaded two empty text files. From here, it has to do some auto-magic to get the data to work inside galaxy, with the "state" of the data. I am not exactly sure what this, or how long it might take. I would imagine not long.

From here we finally need to publish the data to our history as a so that we can utilize it. I am going to import this data as a Dataset, though if a collection suits better, use that. In the photo below, I am uploading "iamanothertextfile.txt" to my new history called "IAmANewHistory".

Once that is added a notification should pop up in the bottom right hand corner of the browser. If not, navigate to the homepage and manually open the History your data was added to. From here you should be able to process your data like normal.

Galaxy File Upload

2025-04-14T16:28:35Z

Nathanrwells:

= Large File Uploads to Galaxy =
The Galaxy web UI is sometimes inconsistent with large downloads. As such, we have made user directory imports available on Galaxy. You can upload files to /bulk/galaxy/user-space/, and locating the user space that was created for you on first login. The folder is usually named "eid@ksu.edu" or if you are from another college (VetMed for instance) "eid@vet.k-state.edu". You can use the command "ls /bulk/galaxy/user-space" to find the name of your directory.

From here, we have written some instructions on how to utilize this upload method.
== Upload Files ==
First, you need to transfer the data onto Beocat, this can be done through any number of ways. We have some documentation on uploading data into Beocat that can be found here (for large files, we suggest using Globus, scp or an FTP program): https://support.beocat.ksu.edu/Docs/Main_Page#Transferring_data_to_Beocat

Next, due to how Galaxy handles data exposure to the web UI, we need to move this data to a userspace that was created for you in Galaxy on your first login:

Move the data to the following directory (You should be able to move data here, if you are unable to, please let me know). Note that the backwards slash is a necessary character to escape the @ symbol from your email address:
/bulk/galaxy/user-space/sghimire1\@vet.k-state.edu/

Next, login to galaxy.beocat.ksu.edu with your eID through KeyCloak.

Then, navigate to the "Shared Data" tab at the top of the page, in the drop-down menu, navigate to "Data Libraries". This should take you to a relatively empty page that allows you to create a library with a "+ Library" in the top left of the web page. For reference, I have included a screenshot of my Data Libraries:

[[File:Data library creation.png|CreatingDataLibrary]]

Create a new Library with any name, and then open it by clicking on its name. In this case, from the photo above, you could click on "Test2" or "Upload".

This takes you to a page that will let you manage the data inside of that library. It should look like this:

[[File:Library explanation.png|ExplainingDataLibrary]]

From here you can either add a folder to help manage your data, upload data to the library, or add the current data in the library to your History so that it can be accessed. We are going to upload data to the library. Clicking "+ Datasets" will expand a drop-down menu, from here, select "from User Directory". This will allow you to upload any data to galaxy from that /bulk directory we moved your data to earlier on Beocat. That process looks something like this:

[[File:Import files.png|ImportToDataLibrary]]

In this, I just uploaded two empty text files. From here, it has to do some auto-magic to get the data to work inside galaxy, with the "state" of the data. I am not exactly sure what this, or how long it might take. I would imagine not long.

From here we finally need to publish the data to our history as a so that we can utilize it. I am going to import this data as a Dataset, though if a collection suits better, use that. In the photo below, I am uploading "iamanothertextfile.txt" to my new history called "IAmANewHistory".

Once that is added a little notification should pop up in the bottom right hand corner of the browser. If not, navigate to the homepage and manually open the History your data was added to. From here you should be able to process your data like normal.

Galaxy File Upload

2025-04-14T16:27:39Z

Nathanrwells: /* Upload Files */

= Large File Uploads to Galaxy =
The Galaxy web UI is sometimes inconsistent with large downloads. As such, we have made user directory imports available on Galaxy. You can upload files to /bulk/galaxy/user-space/, and locating the user space that was created for you on first login. The folder is usually named "eid@ksu.edu" or if you are from another college (VetMed for instance) "eid@vet.k-state.edu". You can use the command "ls /bulk/galaxy/user-space" to find the name of your directory.

From here, we have written some instructions on how to utilize this upload method.
== Upload Files ==
First, you need to transfer the data onto Beocat, this can be done through any number of ways. We have some documentation on uploading data into Beocat that can be found here (for large files, we suggest using Globus, scp or an FTP program): https://support.beocat.ksu.edu/Docs/Main_Page#Transferring_data_to_Beocat

Next, due to how Galaxy handles data exposure to the web UI, we need to move this data to a userspace that was created for you in Galaxy on your first login:

Move the data to the following directory (You should be able to move data here, if you are unable to, please let me know). Note that the backwards slash is a necessary character to escape the @ symbol from your email address:
/bulk/galaxy/user-space/sghimire1\@vet.k-state.edu/

Next, login to galaxy.beocat.ksu.edu with your eID through KeyCloak.

Then, navigate to the "Shared Data" tab at the top of the page, in the drop-down menu, navigate to "Data Libraries". This should take you to a relatively empty page that allows you to create a library with a "+ Library" in the top left of the web page. For reference, I have included a screenshot of my Data Libraries:

[[File:Data library creation.png|thumb]]

Create a new Library with any name, and then open it by clicking on its name. In this case, from the photo above, you could click on "Test2" or "Upload".

This takes you to a page that will let you manage the data inside of that library. It should look like this:

[[File:Library explanation.png|thumb]]

From here you can either add a folder to help manage your data, upload data to the library, or add the current data in the library to your History so that it can be accessed. We are going to upload data to the library. Clicking "+ Datasets" will expand a drop-down menu, from here, select "from User Directory". This will allow you to upload any data to galaxy from that /bulk directory we moved your data to earlier on Beocat. That process looks something like this:

[[File:Import files.png|thumb]]

In this, I just uploaded two empty text files. From here, it has to do some auto-magic to get the data to work inside galaxy, with the "state" of the data. I am not exactly sure what this, or how long it might take. I would imagine not long.

From here we finally need to publish the data to our history as a so that we can utilize it. I am going to import this data as a Dataset, though if a collection suits better, use that. In the photo below, I am uploading "iamanothertextfile.txt" to my new history called "IAmANewHistory".

Once that is added a little notification should pop up in the bottom right hand corner of the browser. If not, navigate to the homepage and manually open the History your data was added to. From here you should be able to process your data like normal.

Galaxy File Upload

2025-04-14T16:27:00Z

Nathanrwells: Created page with "= Large File Uploads to Galaxy = The Galaxy web UI is sometimes inconsistent with large downloads. As such, we have made user directory imports available on Galaxy. You can upload files to /bulk/galaxy/user-space/, and locating the user space that was created for you on first login. The folder is usually named "eid@ksu.edu" or if you are from another college (VetMed for instance) "eid@vet.k-state.edu". You can use the command "ls /bulk/galaxy/user-space" to find the name..."

= Large File Uploads to Galaxy =
The Galaxy web UI is sometimes inconsistent with large downloads. As such, we have made user directory imports available on Galaxy. You can upload files to /bulk/galaxy/user-space/, and locating the user space that was created for you on first login. The folder is usually named "eid@ksu.edu" or if you are from another college (VetMed for instance) "eid@vet.k-state.edu". You can use the command "ls /bulk/galaxy/user-space" to find the name of your directory.

From here, we have written some instructions on how to utilize this upload method.
== Upload Files ==
First, you need to transfer the data onto Beocat, this can be done through any number of ways. We have some documentation on uploading data into Beocat that can be found here (for large files, we suggest using Globus, scp or an FTP program): https://support.beocat.ksu.edu/Docs/Main_Page#Transferring_data_to_Beocat

Next, due to how Galaxy handles data exposure to the web UI, we need to move this data to a userspace that was created for you in Galaxy on your first login:

Move the data to the following directory (You should be able to move data here, if you are unable to, please let me know). Note that the backwards slash is a necessary character to escape the @ symbol from your email address:
/bulk/galaxy/user-space/sghimire1\@vet.k-state.edu/

Next, login to galaxy.beocat.ksu.edu with your eID through KeyCloak.

Then, navigate to the "Shared Data" tab at the top of the page, in the drop-down menu, navigate to "Data Libraries". This should take you to a relatively empty page that allows you to create a library with a "+ Library" in the top left of the web page. For reference, I have included a screenshot of my Data Libraries:
[[File:Data library creation.png|thumb]]
Create a new Library with any name, and then open it by clicking on its name. In this case, from the photo above, you could click on "Test2" or "Upload".

This takes you to a page that will let you manage the data inside of that library. It should look like this:
[[File:Library explanation.png|thumb]]
From here you can either add a folder to help manage your data, upload data to the library, or add the current data in the library to your History so that it can be accessed. We are going to upload data to the library. Clicking "+ Datasets" will expand a drop-down menu, from here, select "from User Directory". This will allow you to upload any data to galaxy from that /bulk directory we moved your data to earlier on Beocat. That process looks something like this:
[[File:Import files.png|thumb]]
In this, I just uploaded two empty text files. From here, it has to do some auto-magic to get the data to work inside galaxy, with the "state" of the data. I am not exactly sure what this, or how long it might take. I would imagine not long.

From here we finally need to publish the data to our history as a so that we can utilize it. I am going to import this data as a Dataset, though if a collection suits better, use that. In the photo below, I am uploading "iamanothertextfile.txt" to my new history called "IAmANewHistory".

Once that is added a little notification should pop up in the bottom right hand corner of the browser. If not, navigate to the homepage and manually open the History your data was added to. From here you should be able to process your data like normal.

File:Import files.png

2025-04-14T16:26:53Z

Nathanrwells:

import files from Beocat via /bulk

File:Library explanation.png

2025-04-14T16:26:25Z

Nathanrwells:

Explanation of what to do when inside of Data Library.

File:Data library creation.png

2025-04-14T16:25:54Z

Nathanrwells:

creation of Data Library on Galaxy

Main Page

2025-04-10T18:46:56Z

Nathanrwells: /* How do I get help? */

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

==== Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] ====
==== Read about [[Installed software]] and languages ====
==== Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] ====
==== Run Interactive Jobs! [[OpenOnDemand]] ====
==== [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] ====

==== Big Data course on Beocat! [[BigDataOnBeocat]] ====
==== Interested in Web-Based computational biology research? Check out [[GalaxyDocs|Galaxy!]] ====
==== Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]] ====

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you email us at beocat@cs.ksu.edu please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

In the future, Beocat will be moving towards the Kstate Central IT TDX ticketing system. Please expect a change in how we handle user support in Summer/Fall of 2025.

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Main Page

2025-04-10T18:46:41Z

Nathanrwells: /* How do I get help? */

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

==== Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] ====
==== Read about [[Installed software]] and languages ====
==== Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] ====
==== Run Interactive Jobs! [[OpenOnDemand]] ====
==== [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] ====

==== Big Data course on Beocat! [[BigDataOnBeocat]] ====
==== Interested in Web-Based computational biology research? Check out [[GalaxyDocs|Galaxy!]] ====
==== Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]] ====

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you email us at beocat@cs.ksu.edu please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

In the future, Beocat will be moving towards the Kstate Central IT TDX ticketing system. Please expect a change in how we handle user support in summer/fall of 2025.

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Main Page

2025-04-10T18:45:53Z

Nathanrwells: /* How do I get help? */

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

==== Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] ====
==== Read about [[Installed software]] and languages ====
==== Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] ====
==== Run Interactive Jobs! [[OpenOnDemand]] ====
==== [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] ====

==== Big Data course on Beocat! [[BigDataOnBeocat]] ====
==== Interested in Web-Based computational biology research? Check out [[GalaxyDocs|Galaxy!]] ====
==== Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]] ====

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you email us at beocat@cs.ksu.edu please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

In the future, Beocat will be moving towards the Kstate Central IT TDX ticketing system, please expect a change in how we handle user support in summer/fall of 2025.

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Main Page

2025-04-10T18:45:27Z

Nathanrwells: /* How do I get help? */

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

==== Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] ====
==== Read about [[Installed software]] and languages ====
==== Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] ====
==== Run Interactive Jobs! [[OpenOnDemand]] ====
==== [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] ====

==== Big Data course on Beocat! [[BigDataOnBeocat]] ====
==== Interested in Web-Based computational biology research? Check out [[GalaxyDocs|Galaxy!]] ====
==== Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]] ====

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you email us at beocat@cs.ksu.edu please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

In the future, Beocat will be moving towards Kstates central TDX ticketing system, please expect a change in how we handle user support in summer/fall of 2025.

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Main Page

2025-04-10T18:41:08Z

Nathanrwells: /* How do I get help? */

== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of RHEL Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]

== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

==== Get an account at [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] ====
==== Read about [[Installed software]] and languages ====
==== Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] ====
==== Run Interactive Jobs! [[OpenOnDemand]] ====
==== [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] ====

==== Big Data course on Beocat! [[BigDataOnBeocat]] ====
==== Interested in Web-Based computational biology research? Check out [[GalaxyDocs|Galaxy!]] ====
==== Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]] ====

== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive

== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]

== Writing and Installing Software on Beocat ==
* If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our [[Installed software]] page to see what we have available and any usage guidelines we have posted there.
* If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our [[FAQ]] for more details.
* In either case, we suggest you head to our [[Tips and Tricks]] page for helpful hints.
* If you wish to install software in your home directory, we have a [[Training Videos#Installing_files_in_your_Home_Directory|video]] showing how to do this.

== How do I get help? ==
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.

If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

<pre style="font-weight: bold;">
Again, when you email us at beocat@cs.ksu.edu please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>

== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

== External Computing Resources ==

We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses. Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project. Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes. For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].

== Policies ==
You can find our policies [[Policy|here]]

== Credits and Accolades ==
See the published credits and other accolades received by Beocat [[Credits|here]]

== Upcoming Events ==
{{#widget:Google Calendar
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|color=711616
|view=AGENDA
}}

Installed software

2025-04-09T18:51:57Z

Nathanrwells: /* Java */

== Module Availability ==
Most people will be just fine running 'module avail' to see a list of modules available on Beocat. There are a couple software packages that are only available on particular node types. For those cases, check [https://modules.beocat.ksu.edu/ our modules website.] If you are used to OpenScienceGrid computing, you may wish to take a look at how to use [[OpenScienceGrid#Using_OpenScienceGrid_modules_on_Beocat|their modules.]]

== Toolchains ==
A toolchain is a set of compilers, libraries and applications that are needed to build software. Some software functions better when using specific toolchains.

We provide a good number of toolchains and versions of toolchains make sure your applications will compile and/or run correctly.

These toolchains include (you can run 'module keyword toolchain'):
; foss: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
; gompi: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.
; iomkl: Intel Cluster Toolchain Compiler Edition provides Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.
; intel: Intel Compiler Suite, providing Intel C/C++ and Fortran compilers, Intel MKL & Intel MPI. Recently made free by Intel, we have less experience with Intel MPI than OpenMPI.

You can run 'module spider $toolchain/' to see the versions we have:
$ module spider iomkl/
* iomkl/2017a
* iomkl/2017b
* iomkl/2017beocatb

If you load one of those (module load iomkl/2017b), you can see the other modules and versions of software that it loaded with the 'module list':
$ module list
Currently Loaded Modules:
1) icc/2017.4.196-GCC-6.4.0-2.28
2) binutils/2.28-GCCcore-6.4.0
3) ifort/2017.4.196-GCC-6.4.0-2.28
4) iccifort/2017.4.196-GCC-6.4.0-2.28
5) GCCcore/6.4.0
6) numactl/2.0.11-GCCcore-6.4.0
7) hwloc/1.11.7-GCCcore-6.4.0
8) OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
9) iompi/2017b
10) imkl/2017.3.196-iompi-2017b
11) iomkl/2017b

As you can see, toolchains can depend on each other. For instance, the iomkl toolchain, depends on iompi, which depends on iccifort, which depend on icc and ifort, which depend on GCCcore which depend on GCC. Hence it is very important that the correct versions of all related software are loaded.

With software we provide, the toolchain used to compile is always specified in the "version" of the software that you want to load.

If you mix toolchains, inconsistent things may happen.

== Most Commonly Used Software ==
Check our [https://modules.beocat.ksu.edu/ modules website] for the most up to date software availability.

The versions mentioned below are representations of what was available at the time of writing, not necessarily what is currently available.
=== [http://www.open-mpi.org/ OpenMPI] ===
We provide lots of versions, you are most likely better off directly loading a toolchain or application to make sure you get the right version, but you can see the versions we have with 'module avail OpenMPI/'

The first step to run an MPI application is to load one of the compiler toolchains that include OpenMPI. You normally will just need to load the default version as below. If your code needs access to nVidia GPUs you'll need the cuda version above. Otherwise some codes are picky about what versions of the underlying GNU or Intel compilers that are needed.

module load foss

If you are working with your own MPI code you will need to start by compiling it. MPI offers mpicc for compiling codes written in C, mpic++ for compiling C++ code, and mpifort for compiling Fortran code. You can get a complete listing of parameters to use by running them with the --help parameter. Below are some examples of compiling with each.

mpicc --help
mpicc -o my_code.x my_code.c
mpic++ -o my_code.x my_code.cc
mpifort -o my_code.x my_code.f

In each case above, you can name the executable file whatever you want (I chose <T>my_code.x). It is common to use different optimization levels, for example, but those may depend on which compiler toolchain you choose. Some are based on the Intel compilers so you'd need to use optimizations for the underlying icc or ifort compilers they call, and some are GNU based so you'd use compiler optimizations for gcc or gfortran.

We have many MPI codes in our modules that you simply need to load before using. Below is an example of loading and running Gromacs which is an MPI based code to simulate large numbers of atoms classically.

module load GROMACS

This loads the Gromacs modules and sets all the paths so you can run the scalar version gmx or the MPI version gmx_mpi. Below is a sample job script for running a complete Gromacs simulation.

#!/bin/bash -l
#SBATCH --mem=120G
#SBATCH --time=24:00:00
#SBATCH --job-name=gromacs
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module reset
module load GROMACS

echo "Running Gromacs on $HOSTNAME"

export OMP_NUM_THREADS=1
time mpirun -x OMP_NUM_THREADS=1 gmx_mpi mdrun -nsteps 500000 -ntomp 1 -v -deffnm 1ns -c 1ns.pdb -nice 0

echo "Finished run on $SLURM_NTASKS $HOSTNAME cores"

mpirun will run your job on all cores requested which in this case is 4 cores on a single node. You will often just need to guess at the memory size for your code, then check on the memory usage with kstat --me and adjust the memory in future jobs.

I prefer to put a module reset in my scripts then manually load the modules needed to insure each run is using the modules it needs. If you don't do this when you submit a job script it will simply use the modules you currently have loaded which is fine too.

I also like to put a time command in front of each part of the script that can use significant amounts of time. This way I can track the amount of time used in each section of the job script. This can prove very useful if your job script copies large data files around at the start, for example, allowing you to see how much time was used for each stage of the job if it runs longer than expected.

The OMP_NUM_THREADS environment variable is set to 1 and passed to the MPI system to insure that each MPI task only uses 1 thread. There are some MPI codes that are also multi-threaded, so this insures that this particular code uses the cores allocated to it in the manner we want.

Once you have your job script ready, submit it using the sbatch command as below where the job script is in the file sb.gromacs.

sbatch sb.gromacs

You should then monitor your job as it goes through the queue and starts running using kstat --me. You code will also generate an output file, usually of the form slurm-#######.out where the 7 # signs are the 7 digit job ID number. If you need to cancel your job use scancel with the 7 digit job ID number.

scancel #######

=== [http://www.r-project.org/ R] ===
You can see what versions of R we provide with 'module avail R/'

==== Packages ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own R Packages ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
module load R
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="R">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="R">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. '<tt>sbatch myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.sbatch

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --mem-per-cpu=4G
# Now we tell Slurm how long we expect our work to take: 15 minutes (D-HH:MM:SS)
#SBATCH --time=0-00:15:00

# Now lets do some actual work. This starts R and loads the file myscript.R
module reset
module load R
R --no-save -q < myscript.R
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
sbatch submit-R.sbatch
</syntaxhighlight>

You can monitor your jobs using kstat --me. The output of your job will be in a slurm-#.out file where '#' is the 7 digit job ID number for your job.

=== [http://www.java.com/ Java] ===
You can see what versions of Java we support with 'module avail Java'

You can load the default version of Java that we offer with "module load Java".

Once you have loaded a Java module, you can use this module to interact with Java how you would normally on your own machine.

Below Is an quick example on how to load the Java module, then compile and run a quick java program that will print input from execution (which in this case is the current working directory).

For reference, here is our java "Main.java" file. Your java filename must match the class name (meaning your file does not have to be called "Main", just make sure these names match).
<syntaxhighlight lang="bash">
public class Main {
static void main (String[] args) {
System.out.println(args[0]);
}
}
</syntaxhighlight>
First we need to load the Java module. At the time of writing, the default Java module is "Java/11.0.20". So we can load that like this:
<syntaxhighlight lang="bash">
module load Java
#or we can load it like this:
module load Java/11.0.20
</syntaxhighlight>
Now we need to compile our "Main.java" file into a class file.
<syntaxhighlight lang="bash">
javac Main.java
</syntaxhighlight>
This will produce a file called "Main.class". Note that "Main" will be whatever you named your file.
Now, we can execute the file and give it something to print. In this case, I am going to print the working directory.
<syntaxhighlight lang="bash">
java Main $PWD
</syntaxhighlight>
Which has an output of wherever I ran this from in the terminal.
<syntaxhighlight lang="bash">
/homes/nathanrwells
</syntaxhighlight>

From here you can put this command inside of a slurm submit script to send off to the compute cluster just like you would with any other bash command. Note that you will need to recompile your "$filename.java" each time you make changes to it, otherwise when you execute the program, nothing will change.

Optionally, you can do all of this compilation and execution inside of your slurm submit script since the files need to have the same name.
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem=4G
#SBATCH --ntasks-per-node=1
#SBATCH --time=0:10:00

module load Java
javac $filename.java
java Main $PWD
</syntaxhighlight>
Making sure to replace $filename with the name of your file.

=== [http://www.python.org/about/ Python] ===
You can see what versions of Python we support with 'module avail Python/'. Note: Running this does not load a Python module, it just shows you a list of the ones that are available.

If you need libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

==== Setting up your virtual environment ====
<syntaxhighlight lang="bash">
# Load Python (pick a version from the 'module avail Python/' list)
module load Python/SOME_VERSION_THAT_YOU_PICKED_FROM_THE_LIST
</syntaxhighlight>
(After running this command Python is loaded. After you logoff and then logon again Python will not be loaded so you must rerun this command every time you logon.)
* Create a location for your virtual environments (optional, but helps keep things organized)
<syntaxhighlight lang="bash">
mkdir ~/virtualenvs
cd ~/virtualenvs
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test'. Note that their [https://docs.python.org/3/library/venv.html documentation] has many more useful options.
<syntaxhighlight lang="bash">
python -m venv --system-site-packages test
# or you could use 'python -m venv test'
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, TensorFlow, or Jupyter
# if you don't use '--system-site-packages' then the virtual environment is completely isolated from our other provided packages and everything it needs it will have to build and install within itself.
</syntaxhighlight>
* Lets look at our virtual environments (the virtual environment name should be in the output):
<syntaxhighlight lang="bash">
ls ~/virtualenvs
</syntaxhighlight>
* Activate one of these
<syntaxhighlight lang="bash">
source ~/virtualenvs/test/bin/activate
</syntaxhighlight>
(After running this command your virtual environment is activated. After you logoff and then logon again your virtual environment will not be loaded so you must rerun this command every time you logon.)
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>

==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment test
<syntaxhighlight lang="bash">
#!/bin/bash
module load Python/THE_SAME_VERSION_YOU_USED_TO_CREATE_YOUR_ENVIRONMENT_ABOVE
source ~/virtualenvs/test/bin/activate
export PYTHONDONTWRITEBYTECODE=1
python ~/path/to/your/python/script.py
</syntaxhighlight>

==== Using MPI with Python within a job ====

We're going to load the SciPy-bundle module, as that has mpi4py available within it.

You check the available versions and load one that uses the python version you would like.
module avail SciPy-bundle

Here is a simple job script using MPI with Python

<syntaxhighlight lang="bash">
#!/bin/bash
module load SciPy-bundle

export PYTHONDONTWRITEBYTECODE=1
mpirun python ~/path/to/your/mpi/python/script.py
</syntaxhighlight>

=== [https://www.tensorflow.org/ TensorFlow] ===
TensorFlow provided by pip is often completely broken on any system that is not running a recent version of Ubuntu. Beocat (and most HPC systems) does not use Ubuntu. As such, we provide TensorFlow modules for you to load.

You can see what versions of TensorFlow we support with 'module avail TensorFlow/'. Note: Running this does not load a TensorFlow module, it just shows you a list of the ones that are available.

If you need other python libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

We document creating a virtual environment [[#Setting up your virtual environment|above]]. You can skip loading the python module, as loading TensorFlow will load the correct version of python module behind the scenes. The singular change you need to make is to use the '--system-site-packages' when creating the virtual environment.
<syntaxhighlight lang=bash>
python -m venv --system-site-packages test
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, or TensorFlow
</syntaxhighlight>

=== Jupyter ===
[https://jupyter.org/ Jupyter] is a framework for creating and running reusable "notebooks" for scientific computing. It runs Python code by default. Normally, it is meant to be used in an interactive manner. Interactive codes can be limiting and/or problematic when used in a cluster environment. We have an example submit script available [https://gitlab.beocat.ksu.edu/Admin-Public/ondemand/job_templates/-/tree/master/Jupyter_Notebook here] to help you transition from an OpenOnDemand interactive job using Jupyter to a non-interactive job.

=== [http://spark.apache.org/ Spark] ===

Spark is a programming language for large scale data processing.
It can be used in conjunction with Python, R, Scala, Java, and SQL.
Spark can be run on Beocat interactively or through the Slurm queue.

To run interactively, you must first request a node or nodes from the Slurm queue.
The line below requests 1 node and 1 core for 24 hours and if available will drop
you into the bash shell on that node.
<syntaxhighlight lang=bash>
srun -J srun -N 1 -n 1 -t 24:00:00 --mem=10G --pty bash
</syntaxhighlight>
We have some sample python based Spark code you can try out that came from the
exercises and homework from the PSC Spark workshop.
<syntaxhighlight lang=bash>
mkdir spark-test
cd spark-test
cp -rp /homes/daveturner/projects/PSC-BigData-Workshop/Shakespeare/* .
</syntaxhighlight>
You will need to set up a python virtual environment and load the nltk package
before you run the first time.
<syntaxhighlight lang=bash>
module load Spark
mkdir -p ~/virtualenvs
cd ~/virtualenvs
python -m venv --system-site-packages spark-test
source ~/virtualenvs/spark-test/bin/activate
pip install nltk
deactivate
</syntaxhighlight>
To run the sample code interactively, load the Python and Spark modules,
source your python virtual environment, change to the sample directory, fire up pyspark,
then execute the sample code.
<syntaxhighlight lang=bash>
module load Spark
source ~/virtualenvs/spark-test/bin/activate
cd ~/spark-test
pyspark
</syntaxhighlight>
<syntaxhighlight lang=python>
exec(open("shakespeare.py").read())
</syntaxhighlight>
You can work interactively from the pyspark prompt (>>>) in addition to running scripts as above.

The Shakespeare directory also contains a sample sbatch submit script that will run the
same shakespeare.py code through the Slurm batch queue.
<syntaxhighlight lang=bash>
#!/bin/bash -l
#SBATCH --job-name=shakespeare
#SBATCH --mem=10G
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

# Load Spark and Python (version 3 here)
module load Spark
source ~/virtualenvs/spark-test/bin/activate

spark-submit shakespeare.py
</syntaxhighlight>
When you run interactively, pyspark initializes your spark context sc.
You will need to do this manually as in the sample python code when you want
to submit jobs through the Slurm queue.
<syntaxhighlight lang=python>
# If there is no Spark Context (not running interactive from pyspark), create it
try:
sc
except NameError:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("App")
sc = SparkContext(conf = conf)
</syntaxhighlight>

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.

To use perl with threads, out a newer version, you can load it with the module command. To see what versions of perl we provide, you can use 'module avail Perl/'

==== Installing Perl Modules ====

The easiest way to install Perl modules is by using cpanm.
Below is an example of installing the Perl module Term::ANSIColor.

<syntaxhighlight lang=bash>
module load Perl
cpanm -i Term::ANSIColor
</syntaxhighlight>

CPAN: LWP::UserAgent loaded ok (v6.39)
Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
CPAN: YAML loaded ok (v1.29)
Reading '/homes/mozes/.cpan/sources/authors/01mailrc.txt.gz'
CPAN: Compress::Zlib loaded ok (v2.084)
............................................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/02packages.details.txt.gz
Reading '/homes/mozes/.cpan/sources/modules/02packages.details.txt.gz'
Database was generated on Mon, 09 Mar 2020 20:41:03 GMT
.............
New CPAN.pm version (v2.27) available.
[Currently running version is v2.22]
You might want to try
install CPAN
reload cpan
to both upgrade CPAN.pm and run the new version without leaving
the current session.
...............................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/03modlist.data.gz
Reading '/homes/mozes/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /homes/mozes/.cpan/Metadata
Running install for module 'Term::ANSIColor'
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz
CPAN: Digest::SHA loaded ok (v6.02)
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/CHECKSUMS
Checksum for /homes/mozes/.cpan/sources/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz ok
CPAN: CPAN::Meta::Requirements loaded ok (v2.140)
CPAN: Parse::CPAN::Meta loaded ok (v2.150010)
CPAN: CPAN::Meta loaded ok (v2.150010)
CPAN: Module::CoreList loaded ok (v5.20190522)
Configuring R/RR/RRA/Term-ANSIColor-5.01.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Term::ANSIColor
Writing MYMETA.yml and MYMETA.json
RRA/Term-ANSIColor-5.01.tar.gz
/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl Makefile.PL -- OK
Running make for R/RR/RRA/Term-ANSIColor-5.01.tar.gz
cp lib/Term/ANSIColor.pm blib/lib/Term/ANSIColor.pm
Manifying 1 pod document
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make -- OK
Running make test for RRA/Term-ANSIColor-5.01.tar.gz
PERL_DL_NONLAZY=1 "/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*/*.t
t/docs/pod-coverage.t ....... skipped: POD coverage tests normally skipped
t/docs/pod-spelling.t ....... skipped: Spelling tests only run for author
t/docs/pod.t ................ skipped: POD syntax tests normally skipped
t/docs/spdx-license.t ....... skipped: SPDX identifier tests normally skipped
t/docs/synopsis.t ........... skipped: Synopsis syntax tests normally skipped
t/module/aliases-env.t ...... ok
t/module/aliases-func.t ..... ok
t/module/basic.t ............ ok
t/module/basic256.t ......... ok
t/module/eval.t ............. ok
t/module/stringify.t ........ ok
t/module/true-color.t ....... ok
t/style/coverage.t .......... skipped: Coverage tests only run for author
t/style/critic.t ............ skipped: Coding style tests only run for author
t/style/minimum-version.t ... skipped: Minimum version tests normally skipped
t/style/obsolete-strings.t .. skipped: Obsolete strings tests normally skipped
t/style/strict.t ............ skipped: Strictness tests normally skipped
t/taint/basic.t ............. ok
All tests successful.
Files=18, Tests=430, 7 wallclock secs ( 0.21 usr 0.08 sys + 3.41 cusr 1.15 csys = 4.85 CPU)
Result: PASS
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make test -- OK
Running make install for RRA/Term-ANSIColor-5.01.tar.gz
Manifying 1 pod document
Installing /homes/mozes/perl5/lib/perl5/Term/ANSIColor.pm
Installing /homes/mozes/perl5/man/man3/Term::ANSIColor.3
Appending installation info to /homes/mozes/perl5/lib/perl5/x86_64-linux-thread-multi/perllocal.pod
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make install -- OK

===== When things go wrong =====
Some perl modules fail to realize they shouldn't be installed globally. Usually, you'll notice this when they try to run 'sudo' something. Unfortunately we do not grant sudo access to anyone other then Beocat system administrators. Usually, this can be worked around by putting the following in your <tt>~/.bashrc</tt> file (at the bottom). Once this is in place, you should log out and log back in.
<syntaxhighlight lang="bash">
PATH="/homes/${USER}/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/homes/${USER}/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}";
export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/homes/${USER}/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}";
export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/homes/${USER}/perl5\""; export PERL_MB_OPT;
</syntaxhighlight>

==== Submitting a job with Perl ====
Much like R (above), you cannot simply '<tt>sbatch myProgram.pl</tt>', but you must create a [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|submit script]] which will call perl. Here is an example:
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell sbatch how long we expect our work to take: 15 minutes (H:MM:SS)
#SBATCH --time=0-0:15:00
# Now lets do some actual work.
module load Perl
perl /path/to/myProgram.pl
</syntaxhighlight>

=== Octave for MatLab codes ===

'module avail Octave/'

The 64-bit version of Octave can be loaded using the command above. Octave can then be used
to work with MatLab codes on the head node and to submit jobs to the compute nodes through the
sbatch scheduler. Octave is made to run MatLab code, but it does have limitations and does not support
everything that MatLab itself does.

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=octave
#SBATCH --output=octave.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load Octave/4.2.1-foss-2017beocatb-enable64

octave < matlab_code.m
</syntaxhighlight>

=== MatLab compiler ===

Beocat also has a single floating user license for the MatLab compiler and the most common toolboxes
including the Parallel Computing Toolbox, Optimization Toolbox, Statistics and Machine Learning Toolbox,
Image Processing Toolbox, Curve Fitting Toolbox, Neural Network Toolbox, Symbolic Math Toolbox,
Global Optimization Toolbox, and the Bioinformatics Toolbox.

Since we only have a single floating user license, this means that you will be expected to develop your MatLab code
with Octave or elsewhere on a laptop or departmental server. Once you're ready to do large runs, then you
move your code to Beocat, compile the MatLab code into an executable, and you can submit as many jobs as
you want to the scheduler. To use the MatLab compiler, you need to load the MATLAB module to compile code and
load the mcr module to run the resulting MatLab executable.

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -o matlab_executable_name
</syntaxhighlight>

If you have addpath() commands in your code, you will need to wrap them in an "if ~deployed" block and tell the
compiler to include that path via the -I flag.

<syntaxhighlight lang="MATLAB">
% wrap addpath() calls like so:
if ~deployed
addpath('./another/folder/with/code/')
end
</syntaxhighlight>

NOTE: The license manager checks the mcc compiler out for a minimum of 30 minutes, so if another user compiles a code
you unfortunately may need to wait for up to 30 minutes to compile your own code.

Compiling with additional paths:

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -I ./another/folder/with/code/ -o matlab_executable_name
</syntaxhighlight>

Any directories added with addpath() will need to be added to the list of compile options as -I arguments. You
can have multiple -I arguments in your compile command.

Here is an example job submission script. Modify time, memory, tasks-per-node, and job name as you see fit:

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=matlab
#SBATCH --output=matlab.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load mcr

./matlab_executable_name
</syntaxhighlight>

For those who make use of mex files - compiled C and C++ code with matlab bindings - you will need to add these
files to the compiled archive via the -a flag. See the behavior of this flag in the [https://www.mathworks.com/help/compiler/mcc.html compiler documentation]. You can either target specific .mex files or entire directories.

Because codes often require adding several directories to the Matlab path as well as mex files from several locations,
we recommend writing a script to preserve and help document the steps to compile your Matlab code. Here is an
abbreviated example from a current user:

<syntaxhighlight lang="bash">
#!/bin/bash -l

module load MATLAB

cd matlabPyrTools/MEX/

# compile mex files
mex upConv.c convolve.c wrap.c edges.c
mex corrDn.c convolve.c wrap.c edges.c
mex histo.c
mex innerProd.c

cd ../..

mcc -m mongrel_creation.m \
-I ./matlabPyrTools/MEX/ \
-I ./matlabPyrTools/ \
-I ./FastICA/ \
-a ./matlabPyrTools/MEX/ \
-a ./texturesynth/ \
-o mongrel_creation_binary
</syntaxhighlight>

Again, we only have a single floating user license for MatLab so the model is to develop and debug your MatLab code
elsewhere or using Octave on Beocat, then you can compile the MatLab code into an executable and run it without
limits on Beocat.

For more info on the mcc compiler see: https://www.mathworks.com/help/compiler/mcc.html

=== COMSOL ===
Beocat has no license for COMSOL. If you want to use it, you must provide your own.

module spider COMSOL/
----------------------------------------------------------------------------
COMSOL: COMSOL/5.3
----------------------------------------------------------------------------
Description:
COMSOL Multiphysics software, an interactive environment for modeling
and simulating scientific and engineering problems

This module can be loaded directly: module load COMSOL/5.3

Help:

Description
===========
COMSOL Multiphysics software, an interactive environment for modeling and
simulating scientific and engineering problems
You must provide your own license.
export LM_LICENSE_FILE=/the/path/to/your/license/file
*OR*
export LM_LICENSE_FILE=$LICENSE_SERVER_PORT@$LICENSE_SERVER_HOSTNAME
e.g. export LM_LICENSE_FILE=1719@some.flexlm.server.ksu.edu

More information
================
- Homepage: https://www.comsol.com/
==== Graphical COMSOL ====
Running COMSOL in graphical mode on a cluster is generally a bad idea. If you choose to run it in graphical mode on a compute node, you will need to do something like the following:
<syntaxhighlight lang="bash">
# Connect to the cluster with X11 forwarding (ssh -Y or mobaxterm)
# load the comsol module on the headnode
module load COMSOL
# export your comsol license as mentioned above, and tell the scheduler to run the software
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 comsol -3drend sw
</syntaxhighlight>

=== .NET Core ===
==== Load .NET ====
mozes@[eunomia] ~ $ module load dotNET-Core-SDK
==== create an application ====
Following instructions from [https://docs.microsoft.com/en-us/dotnet/core/tutorials/using-with-xplat-cli here], we'll create a simple 'Hello World' application
mozes@[eunomia] ~ $ mkdir Hello

mozes@[eunomia] ~ $ cd Hello

mozes@[eunomia] ~/Hello $ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true

mozes@[eunomia] ~/Hello $ dotnet new console
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /homes/mozes/Hello/Hello.csproj...
Restoring packages for /homes/mozes/Hello/Hello.csproj...
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.props.
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.targets.
Restore completed in 358.43 ms for /homes/mozes/Hello/Hello.csproj.

Restore succeeded.

==== Edit your program ====
mozes@[eunomia] ~/Hello $ vi Program.cs
==== Run your .NET application ====
mozes@[eunomia] ~/Hello $ dotnet run
Hello World!
==== Build and run the built application ====
mozes@[eunomia] ~/Hello $ dotnet build
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

Restore completed in 106.12 ms for /homes/mozes/Hello/Hello.csproj.
Hello -> /homes/mozes/Hello/bin/Debug/netcoreapp2.1/Hello.dll

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:02.86

mozes@[eunomia] ~/Hello $ dotnet bin/Debug/netcoreapp2.1/Hello.dll
Hello World!

== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

== Loading multiple modules ==
modules, when loaded, will stay loaded for the duration of your session until they are unloaded.

; You can load multiple pieces of software with one module load command. : module load iompi iomkl

; You can unload all software : module reset

; If you see output from a module load command that looks like ''"The following have been reloaded with a version change"'' you likely have tried to load two pieces of software that have not been tested together. There may be serious issues with using either pieces of software while you're in this state. Libraries missing, applications non-functional. If you encounter issues, you will want to unload all software before switching modules. : 'module reset' and then 'module load'

== Containers ==
More and more science is being done within containers, these days. Sometimes referred to Docker or Kubernetes, containers allow you to package an entire software runtime platform and run that software on another computer or site with minimal fuss.

Unfortunately, Docker and Kubernetes are not particularly well suited to multi-user HPC environments, but that's not to say that you can't make use of these containers on Beocat.

=== Apptainer ===
[https://apptainer.org/docs/user/1.2/index.html Apptainer] is a container runtime that is designed for HPC environments. It can convert docker containers to its own format, and can be used within a job on Beocat. It is a very broad topic and we've made the decision to point you to the upstream documentation, as it is much more likely that they'll have up to date and functional instructions to help you utilize containers. If you need additional assistance, please don't hesitate to reach out to us.

Installed software

2025-04-03T20:45:12Z

Nathanrwells: /* Java */

== Module Availability ==
Most people will be just fine running 'module avail' to see a list of modules available on Beocat. There are a couple software packages that are only available on particular node types. For those cases, check [https://modules.beocat.ksu.edu/ our modules website.] If you are used to OpenScienceGrid computing, you may wish to take a look at how to use [[OpenScienceGrid#Using_OpenScienceGrid_modules_on_Beocat|their modules.]]

== Toolchains ==
A toolchain is a set of compilers, libraries and applications that are needed to build software. Some software functions better when using specific toolchains.

We provide a good number of toolchains and versions of toolchains make sure your applications will compile and/or run correctly.

These toolchains include (you can run 'module keyword toolchain'):
; foss: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
; gompi: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.
; iomkl: Intel Cluster Toolchain Compiler Edition provides Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.
; intel: Intel Compiler Suite, providing Intel C/C++ and Fortran compilers, Intel MKL & Intel MPI. Recently made free by Intel, we have less experience with Intel MPI than OpenMPI.

You can run 'module spider $toolchain/' to see the versions we have:
$ module spider iomkl/
* iomkl/2017a
* iomkl/2017b
* iomkl/2017beocatb

If you load one of those (module load iomkl/2017b), you can see the other modules and versions of software that it loaded with the 'module list':
$ module list
Currently Loaded Modules:
1) icc/2017.4.196-GCC-6.4.0-2.28
2) binutils/2.28-GCCcore-6.4.0
3) ifort/2017.4.196-GCC-6.4.0-2.28
4) iccifort/2017.4.196-GCC-6.4.0-2.28
5) GCCcore/6.4.0
6) numactl/2.0.11-GCCcore-6.4.0
7) hwloc/1.11.7-GCCcore-6.4.0
8) OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
9) iompi/2017b
10) imkl/2017.3.196-iompi-2017b
11) iomkl/2017b

As you can see, toolchains can depend on each other. For instance, the iomkl toolchain, depends on iompi, which depends on iccifort, which depend on icc and ifort, which depend on GCCcore which depend on GCC. Hence it is very important that the correct versions of all related software are loaded.

With software we provide, the toolchain used to compile is always specified in the "version" of the software that you want to load.

If you mix toolchains, inconsistent things may happen.

== Most Commonly Used Software ==
Check our [https://modules.beocat.ksu.edu/ modules website] for the most up to date software availability.

The versions mentioned below are representations of what was available at the time of writing, not necessarily what is currently available.
=== [http://www.open-mpi.org/ OpenMPI] ===
We provide lots of versions, you are most likely better off directly loading a toolchain or application to make sure you get the right version, but you can see the versions we have with 'module avail OpenMPI/'

The first step to run an MPI application is to load one of the compiler toolchains that include OpenMPI. You normally will just need to load the default version as below. If your code needs access to nVidia GPUs you'll need the cuda version above. Otherwise some codes are picky about what versions of the underlying GNU or Intel compilers that are needed.

module load foss

If you are working with your own MPI code you will need to start by compiling it. MPI offers mpicc for compiling codes written in C, mpic++ for compiling C++ code, and mpifort for compiling Fortran code. You can get a complete listing of parameters to use by running them with the --help parameter. Below are some examples of compiling with each.

mpicc --help
mpicc -o my_code.x my_code.c
mpic++ -o my_code.x my_code.cc
mpifort -o my_code.x my_code.f

In each case above, you can name the executable file whatever you want (I chose <T>my_code.x). It is common to use different optimization levels, for example, but those may depend on which compiler toolchain you choose. Some are based on the Intel compilers so you'd need to use optimizations for the underlying icc or ifort compilers they call, and some are GNU based so you'd use compiler optimizations for gcc or gfortran.

We have many MPI codes in our modules that you simply need to load before using. Below is an example of loading and running Gromacs which is an MPI based code to simulate large numbers of atoms classically.

module load GROMACS

This loads the Gromacs modules and sets all the paths so you can run the scalar version gmx or the MPI version gmx_mpi. Below is a sample job script for running a complete Gromacs simulation.

#!/bin/bash -l
#SBATCH --mem=120G
#SBATCH --time=24:00:00
#SBATCH --job-name=gromacs
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module reset
module load GROMACS

echo "Running Gromacs on $HOSTNAME"

export OMP_NUM_THREADS=1
time mpirun -x OMP_NUM_THREADS=1 gmx_mpi mdrun -nsteps 500000 -ntomp 1 -v -deffnm 1ns -c 1ns.pdb -nice 0

echo "Finished run on $SLURM_NTASKS $HOSTNAME cores"

mpirun will run your job on all cores requested which in this case is 4 cores on a single node. You will often just need to guess at the memory size for your code, then check on the memory usage with kstat --me and adjust the memory in future jobs.

I prefer to put a module reset in my scripts then manually load the modules needed to insure each run is using the modules it needs. If you don't do this when you submit a job script it will simply use the modules you currently have loaded which is fine too.

I also like to put a time command in front of each part of the script that can use significant amounts of time. This way I can track the amount of time used in each section of the job script. This can prove very useful if your job script copies large data files around at the start, for example, allowing you to see how much time was used for each stage of the job if it runs longer than expected.

The OMP_NUM_THREADS environment variable is set to 1 and passed to the MPI system to insure that each MPI task only uses 1 thread. There are some MPI codes that are also multi-threaded, so this insures that this particular code uses the cores allocated to it in the manner we want.

Once you have your job script ready, submit it using the sbatch command as below where the job script is in the file sb.gromacs.

sbatch sb.gromacs

You should then monitor your job as it goes through the queue and starts running using kstat --me. You code will also generate an output file, usually of the form slurm-#######.out where the 7 # signs are the 7 digit job ID number. If you need to cancel your job use scancel with the 7 digit job ID number.

scancel #######

=== [http://www.r-project.org/ R] ===
You can see what versions of R we provide with 'module avail R/'

==== Packages ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own R Packages ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
module load R
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="R">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="R">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. '<tt>sbatch myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.sbatch

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --mem-per-cpu=4G
# Now we tell Slurm how long we expect our work to take: 15 minutes (D-HH:MM:SS)
#SBATCH --time=0-00:15:00

# Now lets do some actual work. This starts R and loads the file myscript.R
module reset
module load R
R --no-save -q < myscript.R
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
sbatch submit-R.sbatch
</syntaxhighlight>

You can monitor your jobs using kstat --me. The output of your job will be in a slurm-#.out file where '#' is the 7 digit job ID number for your job.

=== [http://www.java.com/ Java] ===
You can see what versions of Java we support with 'module avail Java'

You can load the default version of Java that we offer with "module load Java".

Once you have loaded a Java module, you can use this module to interact with Java how you would normally on your own machine.

Below Is an quick example on how to load the Java module, then compile and run a quick java program that will print input from execution (which in this case is the current working directory).

For reference, here is our java "Main.java" file. Your java filename must match the class name (meaning your file does not have to be called "Main", just make sure these names match).
<syntaxhighlight lang="bash">
public class Main {
static void main (String{} args) {
System.out.println(args[0]);
}
}
</syntaxhighlight>
First we need to load the Java module. At the time of writing, the default Java module is "Java/11.0.20". So we can load that like this:
<syntaxhighlight lang="bash">
module load Java
#or we can load it like this:
module load Java/11.0.20
</syntaxhighlight>
Now we need to compile our "Main.java" file into a class file.
<syntaxhighlight lang="bash">
javac Main.java
</syntaxhighlight>
This will produce a file called "Main.class". Note that "Main" will be whatever you named your file.
Now, we can execute the file and give it something to print. In this case, I am going to print the working directory.
<syntaxhighlight lang="bash">
java Main $PWD
</syntaxhighlight>
Which has an output of wherever I ran this from in the terminal.
<syntaxhighlight lang="bash">
/homes/nathanrwells
</syntaxhighlight>

From here you can put this command inside of a slurm submit script to send off to the compute cluster just like you would with any other bash command. Note that you will need to recompile your "$filename.java" each time you make changes to it, otherwise when you execute the program, nothing will change.

Optionally, you can do all of this compilation and execution inside of your slurm submit script since the files need to have the same name.
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem=4G
#SBATCH --ntasks-per-node=1
#SBATCH --time=0:10:00

module load Java
javac $filename.java
java Main $PWD
</syntaxhighlight>
Making sure to replace $filename with the name of your file.

=== [http://www.python.org/about/ Python] ===
You can see what versions of Python we support with 'module avail Python/'. Note: Running this does not load a Python module, it just shows you a list of the ones that are available.

If you need libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

==== Setting up your virtual environment ====
<syntaxhighlight lang="bash">
# Load Python (pick a version from the 'module avail Python/' list)
module load Python/SOME_VERSION_THAT_YOU_PICKED_FROM_THE_LIST
</syntaxhighlight>
(After running this command Python is loaded. After you logoff and then logon again Python will not be loaded so you must rerun this command every time you logon.)
* Create a location for your virtual environments (optional, but helps keep things organized)
<syntaxhighlight lang="bash">
mkdir ~/virtualenvs
cd ~/virtualenvs
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test'. Note that their [https://docs.python.org/3/library/venv.html documentation] has many more useful options.
<syntaxhighlight lang="bash">
python -m venv --system-site-packages test
# or you could use 'python -m venv test'
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, TensorFlow, or Jupyter
# if you don't use '--system-site-packages' then the virtual environment is completely isolated from our other provided packages and everything it needs it will have to build and install within itself.
</syntaxhighlight>
* Lets look at our virtual environments (the virtual environment name should be in the output):
<syntaxhighlight lang="bash">
ls ~/virtualenvs
</syntaxhighlight>
* Activate one of these
<syntaxhighlight lang="bash">
source ~/virtualenvs/test/bin/activate
</syntaxhighlight>
(After running this command your virtual environment is activated. After you logoff and then logon again your virtual environment will not be loaded so you must rerun this command every time you logon.)
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>

==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment test
<syntaxhighlight lang="bash">
#!/bin/bash
module load Python/THE_SAME_VERSION_YOU_USED_TO_CREATE_YOUR_ENVIRONMENT_ABOVE
source ~/virtualenvs/test/bin/activate
export PYTHONDONTWRITEBYTECODE=1
python ~/path/to/your/python/script.py
</syntaxhighlight>

==== Using MPI with Python within a job ====

We're going to load the SciPy-bundle module, as that has mpi4py available within it.

You check the available versions and load one that uses the python version you would like.
module avail SciPy-bundle

Here is a simple job script using MPI with Python

<syntaxhighlight lang="bash">
#!/bin/bash
module load SciPy-bundle

export PYTHONDONTWRITEBYTECODE=1
mpirun python ~/path/to/your/mpi/python/script.py
</syntaxhighlight>

=== [https://www.tensorflow.org/ TensorFlow] ===
TensorFlow provided by pip is often completely broken on any system that is not running a recent version of Ubuntu. Beocat (and most HPC systems) does not use Ubuntu. As such, we provide TensorFlow modules for you to load.

You can see what versions of TensorFlow we support with 'module avail TensorFlow/'. Note: Running this does not load a TensorFlow module, it just shows you a list of the ones that are available.

If you need other python libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

We document creating a virtual environment [[#Setting up your virtual environment|above]]. You can skip loading the python module, as loading TensorFlow will load the correct version of python module behind the scenes. The singular change you need to make is to use the '--system-site-packages' when creating the virtual environment.
<syntaxhighlight lang=bash>
python -m venv --system-site-packages test
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, or TensorFlow
</syntaxhighlight>

=== Jupyter ===
[https://jupyter.org/ Jupyter] is a framework for creating and running reusable "notebooks" for scientific computing. It runs Python code by default. Normally, it is meant to be used in an interactive manner. Interactive codes can be limiting and/or problematic when used in a cluster environment. We have an example submit script available [https://gitlab.beocat.ksu.edu/Admin-Public/ondemand/job_templates/-/tree/master/Jupyter_Notebook here] to help you transition from an OpenOnDemand interactive job using Jupyter to a non-interactive job.

=== [http://spark.apache.org/ Spark] ===

Spark is a programming language for large scale data processing.
It can be used in conjunction with Python, R, Scala, Java, and SQL.
Spark can be run on Beocat interactively or through the Slurm queue.

To run interactively, you must first request a node or nodes from the Slurm queue.
The line below requests 1 node and 1 core for 24 hours and if available will drop
you into the bash shell on that node.
<syntaxhighlight lang=bash>
srun -J srun -N 1 -n 1 -t 24:00:00 --mem=10G --pty bash
</syntaxhighlight>
We have some sample python based Spark code you can try out that came from the
exercises and homework from the PSC Spark workshop.
<syntaxhighlight lang=bash>
mkdir spark-test
cd spark-test
cp -rp /homes/daveturner/projects/PSC-BigData-Workshop/Shakespeare/* .
</syntaxhighlight>
You will need to set up a python virtual environment and load the nltk package
before you run the first time.
<syntaxhighlight lang=bash>
module load Spark
mkdir -p ~/virtualenvs
cd ~/virtualenvs
python -m venv --system-site-packages spark-test
source ~/virtualenvs/spark-test/bin/activate
pip install nltk
deactivate
</syntaxhighlight>
To run the sample code interactively, load the Python and Spark modules,
source your python virtual environment, change to the sample directory, fire up pyspark,
then execute the sample code.
<syntaxhighlight lang=bash>
module load Spark
source ~/virtualenvs/spark-test/bin/activate
cd ~/spark-test
pyspark
</syntaxhighlight>
<syntaxhighlight lang=python>
exec(open("shakespeare.py").read())
</syntaxhighlight>
You can work interactively from the pyspark prompt (>>>) in addition to running scripts as above.

The Shakespeare directory also contains a sample sbatch submit script that will run the
same shakespeare.py code through the Slurm batch queue.
<syntaxhighlight lang=bash>
#!/bin/bash -l
#SBATCH --job-name=shakespeare
#SBATCH --mem=10G
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

# Load Spark and Python (version 3 here)
module load Spark
source ~/virtualenvs/spark-test/bin/activate

spark-submit shakespeare.py
</syntaxhighlight>
When you run interactively, pyspark initializes your spark context sc.
You will need to do this manually as in the sample python code when you want
to submit jobs through the Slurm queue.
<syntaxhighlight lang=python>
# If there is no Spark Context (not running interactive from pyspark), create it
try:
sc
except NameError:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("App")
sc = SparkContext(conf = conf)
</syntaxhighlight>

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.

To use perl with threads, out a newer version, you can load it with the module command. To see what versions of perl we provide, you can use 'module avail Perl/'

==== Installing Perl Modules ====

The easiest way to install Perl modules is by using cpanm.
Below is an example of installing the Perl module Term::ANSIColor.

<syntaxhighlight lang=bash>
module load Perl
cpanm -i Term::ANSIColor
</syntaxhighlight>

CPAN: LWP::UserAgent loaded ok (v6.39)
Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
CPAN: YAML loaded ok (v1.29)
Reading '/homes/mozes/.cpan/sources/authors/01mailrc.txt.gz'
CPAN: Compress::Zlib loaded ok (v2.084)
............................................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/02packages.details.txt.gz
Reading '/homes/mozes/.cpan/sources/modules/02packages.details.txt.gz'
Database was generated on Mon, 09 Mar 2020 20:41:03 GMT
.............
New CPAN.pm version (v2.27) available.
[Currently running version is v2.22]
You might want to try
install CPAN
reload cpan
to both upgrade CPAN.pm and run the new version without leaving
the current session.
...............................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/03modlist.data.gz
Reading '/homes/mozes/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /homes/mozes/.cpan/Metadata
Running install for module 'Term::ANSIColor'
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz
CPAN: Digest::SHA loaded ok (v6.02)
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/CHECKSUMS
Checksum for /homes/mozes/.cpan/sources/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz ok
CPAN: CPAN::Meta::Requirements loaded ok (v2.140)
CPAN: Parse::CPAN::Meta loaded ok (v2.150010)
CPAN: CPAN::Meta loaded ok (v2.150010)
CPAN: Module::CoreList loaded ok (v5.20190522)
Configuring R/RR/RRA/Term-ANSIColor-5.01.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Term::ANSIColor
Writing MYMETA.yml and MYMETA.json
RRA/Term-ANSIColor-5.01.tar.gz
/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl Makefile.PL -- OK
Running make for R/RR/RRA/Term-ANSIColor-5.01.tar.gz
cp lib/Term/ANSIColor.pm blib/lib/Term/ANSIColor.pm
Manifying 1 pod document
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make -- OK
Running make test for RRA/Term-ANSIColor-5.01.tar.gz
PERL_DL_NONLAZY=1 "/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*/*.t
t/docs/pod-coverage.t ....... skipped: POD coverage tests normally skipped
t/docs/pod-spelling.t ....... skipped: Spelling tests only run for author
t/docs/pod.t ................ skipped: POD syntax tests normally skipped
t/docs/spdx-license.t ....... skipped: SPDX identifier tests normally skipped
t/docs/synopsis.t ........... skipped: Synopsis syntax tests normally skipped
t/module/aliases-env.t ...... ok
t/module/aliases-func.t ..... ok
t/module/basic.t ............ ok
t/module/basic256.t ......... ok
t/module/eval.t ............. ok
t/module/stringify.t ........ ok
t/module/true-color.t ....... ok
t/style/coverage.t .......... skipped: Coverage tests only run for author
t/style/critic.t ............ skipped: Coding style tests only run for author
t/style/minimum-version.t ... skipped: Minimum version tests normally skipped
t/style/obsolete-strings.t .. skipped: Obsolete strings tests normally skipped
t/style/strict.t ............ skipped: Strictness tests normally skipped
t/taint/basic.t ............. ok
All tests successful.
Files=18, Tests=430, 7 wallclock secs ( 0.21 usr 0.08 sys + 3.41 cusr 1.15 csys = 4.85 CPU)
Result: PASS
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make test -- OK
Running make install for RRA/Term-ANSIColor-5.01.tar.gz
Manifying 1 pod document
Installing /homes/mozes/perl5/lib/perl5/Term/ANSIColor.pm
Installing /homes/mozes/perl5/man/man3/Term::ANSIColor.3
Appending installation info to /homes/mozes/perl5/lib/perl5/x86_64-linux-thread-multi/perllocal.pod
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make install -- OK

===== When things go wrong =====
Some perl modules fail to realize they shouldn't be installed globally. Usually, you'll notice this when they try to run 'sudo' something. Unfortunately we do not grant sudo access to anyone other then Beocat system administrators. Usually, this can be worked around by putting the following in your <tt>~/.bashrc</tt> file (at the bottom). Once this is in place, you should log out and log back in.
<syntaxhighlight lang="bash">
PATH="/homes/${USER}/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/homes/${USER}/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}";
export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/homes/${USER}/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}";
export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/homes/${USER}/perl5\""; export PERL_MB_OPT;
</syntaxhighlight>

==== Submitting a job with Perl ====
Much like R (above), you cannot simply '<tt>sbatch myProgram.pl</tt>', but you must create a [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|submit script]] which will call perl. Here is an example:
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell sbatch how long we expect our work to take: 15 minutes (H:MM:SS)
#SBATCH --time=0-0:15:00
# Now lets do some actual work.
module load Perl
perl /path/to/myProgram.pl
</syntaxhighlight>

=== Octave for MatLab codes ===

'module avail Octave/'

The 64-bit version of Octave can be loaded using the command above. Octave can then be used
to work with MatLab codes on the head node and to submit jobs to the compute nodes through the
sbatch scheduler. Octave is made to run MatLab code, but it does have limitations and does not support
everything that MatLab itself does.

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=octave
#SBATCH --output=octave.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load Octave/4.2.1-foss-2017beocatb-enable64

octave < matlab_code.m
</syntaxhighlight>

=== MatLab compiler ===

Beocat also has a single floating user license for the MatLab compiler and the most common toolboxes
including the Parallel Computing Toolbox, Optimization Toolbox, Statistics and Machine Learning Toolbox,
Image Processing Toolbox, Curve Fitting Toolbox, Neural Network Toolbox, Symbolic Math Toolbox,
Global Optimization Toolbox, and the Bioinformatics Toolbox.

Since we only have a single floating user license, this means that you will be expected to develop your MatLab code
with Octave or elsewhere on a laptop or departmental server. Once you're ready to do large runs, then you
move your code to Beocat, compile the MatLab code into an executable, and you can submit as many jobs as
you want to the scheduler. To use the MatLab compiler, you need to load the MATLAB module to compile code and
load the mcr module to run the resulting MatLab executable.

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -o matlab_executable_name
</syntaxhighlight>

If you have addpath() commands in your code, you will need to wrap them in an "if ~deployed" block and tell the
compiler to include that path via the -I flag.

<syntaxhighlight lang="MATLAB">
% wrap addpath() calls like so:
if ~deployed
addpath('./another/folder/with/code/')
end
</syntaxhighlight>

NOTE: The license manager checks the mcc compiler out for a minimum of 30 minutes, so if another user compiles a code
you unfortunately may need to wait for up to 30 minutes to compile your own code.

Compiling with additional paths:

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -I ./another/folder/with/code/ -o matlab_executable_name
</syntaxhighlight>

Any directories added with addpath() will need to be added to the list of compile options as -I arguments. You
can have multiple -I arguments in your compile command.

Here is an example job submission script. Modify time, memory, tasks-per-node, and job name as you see fit:

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=matlab
#SBATCH --output=matlab.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load mcr

./matlab_executable_name
</syntaxhighlight>

For those who make use of mex files - compiled C and C++ code with matlab bindings - you will need to add these
files to the compiled archive via the -a flag. See the behavior of this flag in the [https://www.mathworks.com/help/compiler/mcc.html compiler documentation]. You can either target specific .mex files or entire directories.

Because codes often require adding several directories to the Matlab path as well as mex files from several locations,
we recommend writing a script to preserve and help document the steps to compile your Matlab code. Here is an
abbreviated example from a current user:

<syntaxhighlight lang="bash">
#!/bin/bash -l

module load MATLAB

cd matlabPyrTools/MEX/

# compile mex files
mex upConv.c convolve.c wrap.c edges.c
mex corrDn.c convolve.c wrap.c edges.c
mex histo.c
mex innerProd.c

cd ../..

mcc -m mongrel_creation.m \
-I ./matlabPyrTools/MEX/ \
-I ./matlabPyrTools/ \
-I ./FastICA/ \
-a ./matlabPyrTools/MEX/ \
-a ./texturesynth/ \
-o mongrel_creation_binary
</syntaxhighlight>

Again, we only have a single floating user license for MatLab so the model is to develop and debug your MatLab code
elsewhere or using Octave on Beocat, then you can compile the MatLab code into an executable and run it without
limits on Beocat.

For more info on the mcc compiler see: https://www.mathworks.com/help/compiler/mcc.html

=== COMSOL ===
Beocat has no license for COMSOL. If you want to use it, you must provide your own.

module spider COMSOL/
----------------------------------------------------------------------------
COMSOL: COMSOL/5.3
----------------------------------------------------------------------------
Description:
COMSOL Multiphysics software, an interactive environment for modeling
and simulating scientific and engineering problems

This module can be loaded directly: module load COMSOL/5.3

Help:

Description
===========
COMSOL Multiphysics software, an interactive environment for modeling and
simulating scientific and engineering problems
You must provide your own license.
export LM_LICENSE_FILE=/the/path/to/your/license/file
*OR*
export LM_LICENSE_FILE=$LICENSE_SERVER_PORT@$LICENSE_SERVER_HOSTNAME
e.g. export LM_LICENSE_FILE=1719@some.flexlm.server.ksu.edu

More information
================
- Homepage: https://www.comsol.com/
==== Graphical COMSOL ====
Running COMSOL in graphical mode on a cluster is generally a bad idea. If you choose to run it in graphical mode on a compute node, you will need to do something like the following:
<syntaxhighlight lang="bash">
# Connect to the cluster with X11 forwarding (ssh -Y or mobaxterm)
# load the comsol module on the headnode
module load COMSOL
# export your comsol license as mentioned above, and tell the scheduler to run the software
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 comsol -3drend sw
</syntaxhighlight>

=== .NET Core ===
==== Load .NET ====
mozes@[eunomia] ~ $ module load dotNET-Core-SDK
==== create an application ====
Following instructions from [https://docs.microsoft.com/en-us/dotnet/core/tutorials/using-with-xplat-cli here], we'll create a simple 'Hello World' application
mozes@[eunomia] ~ $ mkdir Hello

mozes@[eunomia] ~ $ cd Hello

mozes@[eunomia] ~/Hello $ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true

mozes@[eunomia] ~/Hello $ dotnet new console
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /homes/mozes/Hello/Hello.csproj...
Restoring packages for /homes/mozes/Hello/Hello.csproj...
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.props.
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.targets.
Restore completed in 358.43 ms for /homes/mozes/Hello/Hello.csproj.

Restore succeeded.

==== Edit your program ====
mozes@[eunomia] ~/Hello $ vi Program.cs
==== Run your .NET application ====
mozes@[eunomia] ~/Hello $ dotnet run
Hello World!
==== Build and run the built application ====
mozes@[eunomia] ~/Hello $ dotnet build
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

Restore completed in 106.12 ms for /homes/mozes/Hello/Hello.csproj.
Hello -> /homes/mozes/Hello/bin/Debug/netcoreapp2.1/Hello.dll

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:02.86

mozes@[eunomia] ~/Hello $ dotnet bin/Debug/netcoreapp2.1/Hello.dll
Hello World!

== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

== Loading multiple modules ==
modules, when loaded, will stay loaded for the duration of your session until they are unloaded.

; You can load multiple pieces of software with one module load command. : module load iompi iomkl

; You can unload all software : module reset

; If you see output from a module load command that looks like ''"The following have been reloaded with a version change"'' you likely have tried to load two pieces of software that have not been tested together. There may be serious issues with using either pieces of software while you're in this state. Libraries missing, applications non-functional. If you encounter issues, you will want to unload all software before switching modules. : 'module reset' and then 'module load'

== Containers ==
More and more science is being done within containers, these days. Sometimes referred to Docker or Kubernetes, containers allow you to package an entire software runtime platform and run that software on another computer or site with minimal fuss.

Unfortunately, Docker and Kubernetes are not particularly well suited to multi-user HPC environments, but that's not to say that you can't make use of these containers on Beocat.

=== Apptainer ===
[https://apptainer.org/docs/user/1.2/index.html Apptainer] is a container runtime that is designed for HPC environments. It can convert docker containers to its own format, and can be used within a job on Beocat. It is a very broad topic and we've made the decision to point you to the upstream documentation, as it is much more likely that they'll have up to date and functional instructions to help you utilize containers. If you need additional assistance, please don't hesitate to reach out to us.

Installed software

2025-04-03T20:43:00Z

Nathanrwells: /* Java */

== Module Availability ==
Most people will be just fine running 'module avail' to see a list of modules available on Beocat. There are a couple software packages that are only available on particular node types. For those cases, check [https://modules.beocat.ksu.edu/ our modules website.] If you are used to OpenScienceGrid computing, you may wish to take a look at how to use [[OpenScienceGrid#Using_OpenScienceGrid_modules_on_Beocat|their modules.]]

== Toolchains ==
A toolchain is a set of compilers, libraries and applications that are needed to build software. Some software functions better when using specific toolchains.

We provide a good number of toolchains and versions of toolchains make sure your applications will compile and/or run correctly.

These toolchains include (you can run 'module keyword toolchain'):
; foss: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
; gompi: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.
; iomkl: Intel Cluster Toolchain Compiler Edition provides Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.
; intel: Intel Compiler Suite, providing Intel C/C++ and Fortran compilers, Intel MKL & Intel MPI. Recently made free by Intel, we have less experience with Intel MPI than OpenMPI.

You can run 'module spider $toolchain/' to see the versions we have:
$ module spider iomkl/
* iomkl/2017a
* iomkl/2017b
* iomkl/2017beocatb

If you load one of those (module load iomkl/2017b), you can see the other modules and versions of software that it loaded with the 'module list':
$ module list
Currently Loaded Modules:
1) icc/2017.4.196-GCC-6.4.0-2.28
2) binutils/2.28-GCCcore-6.4.0
3) ifort/2017.4.196-GCC-6.4.0-2.28
4) iccifort/2017.4.196-GCC-6.4.0-2.28
5) GCCcore/6.4.0
6) numactl/2.0.11-GCCcore-6.4.0
7) hwloc/1.11.7-GCCcore-6.4.0
8) OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
9) iompi/2017b
10) imkl/2017.3.196-iompi-2017b
11) iomkl/2017b

As you can see, toolchains can depend on each other. For instance, the iomkl toolchain, depends on iompi, which depends on iccifort, which depend on icc and ifort, which depend on GCCcore which depend on GCC. Hence it is very important that the correct versions of all related software are loaded.

With software we provide, the toolchain used to compile is always specified in the "version" of the software that you want to load.

If you mix toolchains, inconsistent things may happen.

== Most Commonly Used Software ==
Check our [https://modules.beocat.ksu.edu/ modules website] for the most up to date software availability.

The versions mentioned below are representations of what was available at the time of writing, not necessarily what is currently available.
=== [http://www.open-mpi.org/ OpenMPI] ===
We provide lots of versions, you are most likely better off directly loading a toolchain or application to make sure you get the right version, but you can see the versions we have with 'module avail OpenMPI/'

The first step to run an MPI application is to load one of the compiler toolchains that include OpenMPI. You normally will just need to load the default version as below. If your code needs access to nVidia GPUs you'll need the cuda version above. Otherwise some codes are picky about what versions of the underlying GNU or Intel compilers that are needed.

module load foss

If you are working with your own MPI code you will need to start by compiling it. MPI offers mpicc for compiling codes written in C, mpic++ for compiling C++ code, and mpifort for compiling Fortran code. You can get a complete listing of parameters to use by running them with the --help parameter. Below are some examples of compiling with each.

mpicc --help
mpicc -o my_code.x my_code.c
mpic++ -o my_code.x my_code.cc
mpifort -o my_code.x my_code.f

In each case above, you can name the executable file whatever you want (I chose <T>my_code.x). It is common to use different optimization levels, for example, but those may depend on which compiler toolchain you choose. Some are based on the Intel compilers so you'd need to use optimizations for the underlying icc or ifort compilers they call, and some are GNU based so you'd use compiler optimizations for gcc or gfortran.

We have many MPI codes in our modules that you simply need to load before using. Below is an example of loading and running Gromacs which is an MPI based code to simulate large numbers of atoms classically.

module load GROMACS

This loads the Gromacs modules and sets all the paths so you can run the scalar version gmx or the MPI version gmx_mpi. Below is a sample job script for running a complete Gromacs simulation.

#!/bin/bash -l
#SBATCH --mem=120G
#SBATCH --time=24:00:00
#SBATCH --job-name=gromacs
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module reset
module load GROMACS

echo "Running Gromacs on $HOSTNAME"

export OMP_NUM_THREADS=1
time mpirun -x OMP_NUM_THREADS=1 gmx_mpi mdrun -nsteps 500000 -ntomp 1 -v -deffnm 1ns -c 1ns.pdb -nice 0

echo "Finished run on $SLURM_NTASKS $HOSTNAME cores"

mpirun will run your job on all cores requested which in this case is 4 cores on a single node. You will often just need to guess at the memory size for your code, then check on the memory usage with kstat --me and adjust the memory in future jobs.

I prefer to put a module reset in my scripts then manually load the modules needed to insure each run is using the modules it needs. If you don't do this when you submit a job script it will simply use the modules you currently have loaded which is fine too.

I also like to put a time command in front of each part of the script that can use significant amounts of time. This way I can track the amount of time used in each section of the job script. This can prove very useful if your job script copies large data files around at the start, for example, allowing you to see how much time was used for each stage of the job if it runs longer than expected.

The OMP_NUM_THREADS environment variable is set to 1 and passed to the MPI system to insure that each MPI task only uses 1 thread. There are some MPI codes that are also multi-threaded, so this insures that this particular code uses the cores allocated to it in the manner we want.

Once you have your job script ready, submit it using the sbatch command as below where the job script is in the file sb.gromacs.

sbatch sb.gromacs

You should then monitor your job as it goes through the queue and starts running using kstat --me. You code will also generate an output file, usually of the form slurm-#######.out where the 7 # signs are the 7 digit job ID number. If you need to cancel your job use scancel with the 7 digit job ID number.

scancel #######

=== [http://www.r-project.org/ R] ===
You can see what versions of R we provide with 'module avail R/'

==== Packages ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own R Packages ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
module load R
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="R">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="R">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. '<tt>sbatch myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.sbatch

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --mem-per-cpu=4G
# Now we tell Slurm how long we expect our work to take: 15 minutes (D-HH:MM:SS)
#SBATCH --time=0-00:15:00

# Now lets do some actual work. This starts R and loads the file myscript.R
module reset
module load R
R --no-save -q < myscript.R
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
sbatch submit-R.sbatch
</syntaxhighlight>

You can monitor your jobs using kstat --me. The output of your job will be in a slurm-#.out file where '#' is the 7 digit job ID number for your job.

=== [http://www.java.com/ Java] ===
You can see what versions of Java we support with 'module avail Java'

You can load the default version of Java that we offer with "module load Java".

Once you have loaded a Java module, you can use this module to interact with Java how you would normally on your own machine.

Below Is an quick example on how to load the Java module, then compile and run a quick java program that will print input from execution (which in this case is the current working directory).

For reference, here is our java "Main.java" file. Your java filename must match the class name (meaning your file does not have to be called "Main", just make sure these names match).
<syntaxhighlight lang="bash">
public class Main {
static void main (String{} args) {
System.out.println(args[0]);
}
}
</syntaxhighlight>
First we need to load the Java module. At the time of writing, the default Java module is "Java/11.0.20". So we can load that like this:
<syntaxhighlight lang="bash">
module load Java
#or we can load it like this:
module load Java/11.0.20
</syntaxhighlight>
Now we need to compile our "Main.java" file into a class file.
<syntaxhighlight lang="bash">
javac Main.java
</syntaxhighlight>
This will produce a file called "Main.class". Note that "Main" will be whatever you named your file.
Now, we can execute the file and give it something to print. In this case, I am going to print the working directory.
<syntaxhighlight lang="bash">
java Main $PWD
</syntaxhighlight>
Which has an output of wherever I ran this from in the terminal.
<syntaxhighlight lang="bash">
/homes/nathanrwells
</syntaxhighlight>

From here you can put this command inside of a slurm submit script to send off to the compute cluster just like you would with any other bash command. Note that you will need to recompile your "%filename.java" each time you make changes to it, otherwise when you execute the program, nothing will change.

Optionally, you can do all of this compilation and execution inside of your slurm submit script since the files need to have the same name.
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem=4G
#SBATCH --ntasks-per-node=1
#SBATCH --time=0:10:00

module load Java
javac $filename.java
java Main $PWD
</syntaxhighlight>
Making sure to replace $filename with the name of your file.

=== [http://www.python.org/about/ Python] ===
You can see what versions of Python we support with 'module avail Python/'. Note: Running this does not load a Python module, it just shows you a list of the ones that are available.

If you need libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

==== Setting up your virtual environment ====
<syntaxhighlight lang="bash">
# Load Python (pick a version from the 'module avail Python/' list)
module load Python/SOME_VERSION_THAT_YOU_PICKED_FROM_THE_LIST
</syntaxhighlight>
(After running this command Python is loaded. After you logoff and then logon again Python will not be loaded so you must rerun this command every time you logon.)
* Create a location for your virtual environments (optional, but helps keep things organized)
<syntaxhighlight lang="bash">
mkdir ~/virtualenvs
cd ~/virtualenvs
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test'. Note that their [https://docs.python.org/3/library/venv.html documentation] has many more useful options.
<syntaxhighlight lang="bash">
python -m venv --system-site-packages test
# or you could use 'python -m venv test'
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, TensorFlow, or Jupyter
# if you don't use '--system-site-packages' then the virtual environment is completely isolated from our other provided packages and everything it needs it will have to build and install within itself.
</syntaxhighlight>
* Lets look at our virtual environments (the virtual environment name should be in the output):
<syntaxhighlight lang="bash">
ls ~/virtualenvs
</syntaxhighlight>
* Activate one of these
<syntaxhighlight lang="bash">
source ~/virtualenvs/test/bin/activate
</syntaxhighlight>
(After running this command your virtual environment is activated. After you logoff and then logon again your virtual environment will not be loaded so you must rerun this command every time you logon.)
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>

==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment test
<syntaxhighlight lang="bash">
#!/bin/bash
module load Python/THE_SAME_VERSION_YOU_USED_TO_CREATE_YOUR_ENVIRONMENT_ABOVE
source ~/virtualenvs/test/bin/activate
export PYTHONDONTWRITEBYTECODE=1
python ~/path/to/your/python/script.py
</syntaxhighlight>

==== Using MPI with Python within a job ====

We're going to load the SciPy-bundle module, as that has mpi4py available within it.

You check the available versions and load one that uses the python version you would like.
module avail SciPy-bundle

Here is a simple job script using MPI with Python

<syntaxhighlight lang="bash">
#!/bin/bash
module load SciPy-bundle

export PYTHONDONTWRITEBYTECODE=1
mpirun python ~/path/to/your/mpi/python/script.py
</syntaxhighlight>

=== [https://www.tensorflow.org/ TensorFlow] ===
TensorFlow provided by pip is often completely broken on any system that is not running a recent version of Ubuntu. Beocat (and most HPC systems) does not use Ubuntu. As such, we provide TensorFlow modules for you to load.

You can see what versions of TensorFlow we support with 'module avail TensorFlow/'. Note: Running this does not load a TensorFlow module, it just shows you a list of the ones that are available.

If you need other python libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

We document creating a virtual environment [[#Setting up your virtual environment|above]]. You can skip loading the python module, as loading TensorFlow will load the correct version of python module behind the scenes. The singular change you need to make is to use the '--system-site-packages' when creating the virtual environment.
<syntaxhighlight lang=bash>
python -m venv --system-site-packages test
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, or TensorFlow
</syntaxhighlight>

=== Jupyter ===
[https://jupyter.org/ Jupyter] is a framework for creating and running reusable "notebooks" for scientific computing. It runs Python code by default. Normally, it is meant to be used in an interactive manner. Interactive codes can be limiting and/or problematic when used in a cluster environment. We have an example submit script available [https://gitlab.beocat.ksu.edu/Admin-Public/ondemand/job_templates/-/tree/master/Jupyter_Notebook here] to help you transition from an OpenOnDemand interactive job using Jupyter to a non-interactive job.

=== [http://spark.apache.org/ Spark] ===

Spark is a programming language for large scale data processing.
It can be used in conjunction with Python, R, Scala, Java, and SQL.
Spark can be run on Beocat interactively or through the Slurm queue.

To run interactively, you must first request a node or nodes from the Slurm queue.
The line below requests 1 node and 1 core for 24 hours and if available will drop
you into the bash shell on that node.
<syntaxhighlight lang=bash>
srun -J srun -N 1 -n 1 -t 24:00:00 --mem=10G --pty bash
</syntaxhighlight>
We have some sample python based Spark code you can try out that came from the
exercises and homework from the PSC Spark workshop.
<syntaxhighlight lang=bash>
mkdir spark-test
cd spark-test
cp -rp /homes/daveturner/projects/PSC-BigData-Workshop/Shakespeare/* .
</syntaxhighlight>
You will need to set up a python virtual environment and load the nltk package
before you run the first time.
<syntaxhighlight lang=bash>
module load Spark
mkdir -p ~/virtualenvs
cd ~/virtualenvs
python -m venv --system-site-packages spark-test
source ~/virtualenvs/spark-test/bin/activate
pip install nltk
deactivate
</syntaxhighlight>
To run the sample code interactively, load the Python and Spark modules,
source your python virtual environment, change to the sample directory, fire up pyspark,
then execute the sample code.
<syntaxhighlight lang=bash>
module load Spark
source ~/virtualenvs/spark-test/bin/activate
cd ~/spark-test
pyspark
</syntaxhighlight>
<syntaxhighlight lang=python>
exec(open("shakespeare.py").read())
</syntaxhighlight>
You can work interactively from the pyspark prompt (>>>) in addition to running scripts as above.

The Shakespeare directory also contains a sample sbatch submit script that will run the
same shakespeare.py code through the Slurm batch queue.
<syntaxhighlight lang=bash>
#!/bin/bash -l
#SBATCH --job-name=shakespeare
#SBATCH --mem=10G
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

# Load Spark and Python (version 3 here)
module load Spark
source ~/virtualenvs/spark-test/bin/activate

spark-submit shakespeare.py
</syntaxhighlight>
When you run interactively, pyspark initializes your spark context sc.
You will need to do this manually as in the sample python code when you want
to submit jobs through the Slurm queue.
<syntaxhighlight lang=python>
# If there is no Spark Context (not running interactive from pyspark), create it
try:
sc
except NameError:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("App")
sc = SparkContext(conf = conf)
</syntaxhighlight>

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.

To use perl with threads, out a newer version, you can load it with the module command. To see what versions of perl we provide, you can use 'module avail Perl/'

==== Installing Perl Modules ====

The easiest way to install Perl modules is by using cpanm.
Below is an example of installing the Perl module Term::ANSIColor.

<syntaxhighlight lang=bash>
module load Perl
cpanm -i Term::ANSIColor
</syntaxhighlight>

CPAN: LWP::UserAgent loaded ok (v6.39)
Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
CPAN: YAML loaded ok (v1.29)
Reading '/homes/mozes/.cpan/sources/authors/01mailrc.txt.gz'
CPAN: Compress::Zlib loaded ok (v2.084)
............................................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/02packages.details.txt.gz
Reading '/homes/mozes/.cpan/sources/modules/02packages.details.txt.gz'
Database was generated on Mon, 09 Mar 2020 20:41:03 GMT
.............
New CPAN.pm version (v2.27) available.
[Currently running version is v2.22]
You might want to try
install CPAN
reload cpan
to both upgrade CPAN.pm and run the new version without leaving
the current session.
...............................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/03modlist.data.gz
Reading '/homes/mozes/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /homes/mozes/.cpan/Metadata
Running install for module 'Term::ANSIColor'
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz
CPAN: Digest::SHA loaded ok (v6.02)
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/CHECKSUMS
Checksum for /homes/mozes/.cpan/sources/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz ok
CPAN: CPAN::Meta::Requirements loaded ok (v2.140)
CPAN: Parse::CPAN::Meta loaded ok (v2.150010)
CPAN: CPAN::Meta loaded ok (v2.150010)
CPAN: Module::CoreList loaded ok (v5.20190522)
Configuring R/RR/RRA/Term-ANSIColor-5.01.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Term::ANSIColor
Writing MYMETA.yml and MYMETA.json
RRA/Term-ANSIColor-5.01.tar.gz
/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl Makefile.PL -- OK
Running make for R/RR/RRA/Term-ANSIColor-5.01.tar.gz
cp lib/Term/ANSIColor.pm blib/lib/Term/ANSIColor.pm
Manifying 1 pod document
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make -- OK
Running make test for RRA/Term-ANSIColor-5.01.tar.gz
PERL_DL_NONLAZY=1 "/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*/*.t
t/docs/pod-coverage.t ....... skipped: POD coverage tests normally skipped
t/docs/pod-spelling.t ....... skipped: Spelling tests only run for author
t/docs/pod.t ................ skipped: POD syntax tests normally skipped
t/docs/spdx-license.t ....... skipped: SPDX identifier tests normally skipped
t/docs/synopsis.t ........... skipped: Synopsis syntax tests normally skipped
t/module/aliases-env.t ...... ok
t/module/aliases-func.t ..... ok
t/module/basic.t ............ ok
t/module/basic256.t ......... ok
t/module/eval.t ............. ok
t/module/stringify.t ........ ok
t/module/true-color.t ....... ok
t/style/coverage.t .......... skipped: Coverage tests only run for author
t/style/critic.t ............ skipped: Coding style tests only run for author
t/style/minimum-version.t ... skipped: Minimum version tests normally skipped
t/style/obsolete-strings.t .. skipped: Obsolete strings tests normally skipped
t/style/strict.t ............ skipped: Strictness tests normally skipped
t/taint/basic.t ............. ok
All tests successful.
Files=18, Tests=430, 7 wallclock secs ( 0.21 usr 0.08 sys + 3.41 cusr 1.15 csys = 4.85 CPU)
Result: PASS
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make test -- OK
Running make install for RRA/Term-ANSIColor-5.01.tar.gz
Manifying 1 pod document
Installing /homes/mozes/perl5/lib/perl5/Term/ANSIColor.pm
Installing /homes/mozes/perl5/man/man3/Term::ANSIColor.3
Appending installation info to /homes/mozes/perl5/lib/perl5/x86_64-linux-thread-multi/perllocal.pod
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make install -- OK

===== When things go wrong =====
Some perl modules fail to realize they shouldn't be installed globally. Usually, you'll notice this when they try to run 'sudo' something. Unfortunately we do not grant sudo access to anyone other then Beocat system administrators. Usually, this can be worked around by putting the following in your <tt>~/.bashrc</tt> file (at the bottom). Once this is in place, you should log out and log back in.
<syntaxhighlight lang="bash">
PATH="/homes/${USER}/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/homes/${USER}/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}";
export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/homes/${USER}/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}";
export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/homes/${USER}/perl5\""; export PERL_MB_OPT;
</syntaxhighlight>

==== Submitting a job with Perl ====
Much like R (above), you cannot simply '<tt>sbatch myProgram.pl</tt>', but you must create a [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|submit script]] which will call perl. Here is an example:
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell sbatch how long we expect our work to take: 15 minutes (H:MM:SS)
#SBATCH --time=0-0:15:00
# Now lets do some actual work.
module load Perl
perl /path/to/myProgram.pl
</syntaxhighlight>

=== Octave for MatLab codes ===

'module avail Octave/'

The 64-bit version of Octave can be loaded using the command above. Octave can then be used
to work with MatLab codes on the head node and to submit jobs to the compute nodes through the
sbatch scheduler. Octave is made to run MatLab code, but it does have limitations and does not support
everything that MatLab itself does.

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=octave
#SBATCH --output=octave.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load Octave/4.2.1-foss-2017beocatb-enable64

octave < matlab_code.m
</syntaxhighlight>

=== MatLab compiler ===

Beocat also has a single floating user license for the MatLab compiler and the most common toolboxes
including the Parallel Computing Toolbox, Optimization Toolbox, Statistics and Machine Learning Toolbox,
Image Processing Toolbox, Curve Fitting Toolbox, Neural Network Toolbox, Symbolic Math Toolbox,
Global Optimization Toolbox, and the Bioinformatics Toolbox.

Since we only have a single floating user license, this means that you will be expected to develop your MatLab code
with Octave or elsewhere on a laptop or departmental server. Once you're ready to do large runs, then you
move your code to Beocat, compile the MatLab code into an executable, and you can submit as many jobs as
you want to the scheduler. To use the MatLab compiler, you need to load the MATLAB module to compile code and
load the mcr module to run the resulting MatLab executable.

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -o matlab_executable_name
</syntaxhighlight>

If you have addpath() commands in your code, you will need to wrap them in an "if ~deployed" block and tell the
compiler to include that path via the -I flag.

<syntaxhighlight lang="MATLAB">
% wrap addpath() calls like so:
if ~deployed
addpath('./another/folder/with/code/')
end
</syntaxhighlight>

NOTE: The license manager checks the mcc compiler out for a minimum of 30 minutes, so if another user compiles a code
you unfortunately may need to wait for up to 30 minutes to compile your own code.

Compiling with additional paths:

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -I ./another/folder/with/code/ -o matlab_executable_name
</syntaxhighlight>

Any directories added with addpath() will need to be added to the list of compile options as -I arguments. You
can have multiple -I arguments in your compile command.

Here is an example job submission script. Modify time, memory, tasks-per-node, and job name as you see fit:

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=matlab
#SBATCH --output=matlab.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load mcr

./matlab_executable_name
</syntaxhighlight>

For those who make use of mex files - compiled C and C++ code with matlab bindings - you will need to add these
files to the compiled archive via the -a flag. See the behavior of this flag in the [https://www.mathworks.com/help/compiler/mcc.html compiler documentation]. You can either target specific .mex files or entire directories.

Because codes often require adding several directories to the Matlab path as well as mex files from several locations,
we recommend writing a script to preserve and help document the steps to compile your Matlab code. Here is an
abbreviated example from a current user:

<syntaxhighlight lang="bash">
#!/bin/bash -l

module load MATLAB

cd matlabPyrTools/MEX/

# compile mex files
mex upConv.c convolve.c wrap.c edges.c
mex corrDn.c convolve.c wrap.c edges.c
mex histo.c
mex innerProd.c

cd ../..

mcc -m mongrel_creation.m \
-I ./matlabPyrTools/MEX/ \
-I ./matlabPyrTools/ \
-I ./FastICA/ \
-a ./matlabPyrTools/MEX/ \
-a ./texturesynth/ \
-o mongrel_creation_binary
</syntaxhighlight>

Again, we only have a single floating user license for MatLab so the model is to develop and debug your MatLab code
elsewhere or using Octave on Beocat, then you can compile the MatLab code into an executable and run it without
limits on Beocat.

For more info on the mcc compiler see: https://www.mathworks.com/help/compiler/mcc.html

=== COMSOL ===
Beocat has no license for COMSOL. If you want to use it, you must provide your own.

module spider COMSOL/
----------------------------------------------------------------------------
COMSOL: COMSOL/5.3
----------------------------------------------------------------------------
Description:
COMSOL Multiphysics software, an interactive environment for modeling
and simulating scientific and engineering problems

This module can be loaded directly: module load COMSOL/5.3

Help:

Description
===========
COMSOL Multiphysics software, an interactive environment for modeling and
simulating scientific and engineering problems
You must provide your own license.
export LM_LICENSE_FILE=/the/path/to/your/license/file
*OR*
export LM_LICENSE_FILE=$LICENSE_SERVER_PORT@$LICENSE_SERVER_HOSTNAME
e.g. export LM_LICENSE_FILE=1719@some.flexlm.server.ksu.edu

More information
================
- Homepage: https://www.comsol.com/
==== Graphical COMSOL ====
Running COMSOL in graphical mode on a cluster is generally a bad idea. If you choose to run it in graphical mode on a compute node, you will need to do something like the following:
<syntaxhighlight lang="bash">
# Connect to the cluster with X11 forwarding (ssh -Y or mobaxterm)
# load the comsol module on the headnode
module load COMSOL
# export your comsol license as mentioned above, and tell the scheduler to run the software
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 comsol -3drend sw
</syntaxhighlight>

=== .NET Core ===
==== Load .NET ====
mozes@[eunomia] ~ $ module load dotNET-Core-SDK
==== create an application ====
Following instructions from [https://docs.microsoft.com/en-us/dotnet/core/tutorials/using-with-xplat-cli here], we'll create a simple 'Hello World' application
mozes@[eunomia] ~ $ mkdir Hello

mozes@[eunomia] ~ $ cd Hello

mozes@[eunomia] ~/Hello $ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true

mozes@[eunomia] ~/Hello $ dotnet new console
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /homes/mozes/Hello/Hello.csproj...
Restoring packages for /homes/mozes/Hello/Hello.csproj...
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.props.
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.targets.
Restore completed in 358.43 ms for /homes/mozes/Hello/Hello.csproj.

Restore succeeded.

==== Edit your program ====
mozes@[eunomia] ~/Hello $ vi Program.cs
==== Run your .NET application ====
mozes@[eunomia] ~/Hello $ dotnet run
Hello World!
==== Build and run the built application ====
mozes@[eunomia] ~/Hello $ dotnet build
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

Restore completed in 106.12 ms for /homes/mozes/Hello/Hello.csproj.
Hello -> /homes/mozes/Hello/bin/Debug/netcoreapp2.1/Hello.dll

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:02.86

mozes@[eunomia] ~/Hello $ dotnet bin/Debug/netcoreapp2.1/Hello.dll
Hello World!

== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

== Loading multiple modules ==
modules, when loaded, will stay loaded for the duration of your session until they are unloaded.

; You can load multiple pieces of software with one module load command. : module load iompi iomkl

; You can unload all software : module reset

; If you see output from a module load command that looks like ''"The following have been reloaded with a version change"'' you likely have tried to load two pieces of software that have not been tested together. There may be serious issues with using either pieces of software while you're in this state. Libraries missing, applications non-functional. If you encounter issues, you will want to unload all software before switching modules. : 'module reset' and then 'module load'

== Containers ==
More and more science is being done within containers, these days. Sometimes referred to Docker or Kubernetes, containers allow you to package an entire software runtime platform and run that software on another computer or site with minimal fuss.

Unfortunately, Docker and Kubernetes are not particularly well suited to multi-user HPC environments, but that's not to say that you can't make use of these containers on Beocat.

=== Apptainer ===
[https://apptainer.org/docs/user/1.2/index.html Apptainer] is a container runtime that is designed for HPC environments. It can convert docker containers to its own format, and can be used within a job on Beocat. It is a very broad topic and we've made the decision to point you to the upstream documentation, as it is much more likely that they'll have up to date and functional instructions to help you utilize containers. If you need additional assistance, please don't hesitate to reach out to us.

Installed software

2025-04-03T20:42:02Z

Nathanrwells: /* Java */

== Module Availability ==
Most people will be just fine running 'module avail' to see a list of modules available on Beocat. There are a couple software packages that are only available on particular node types. For those cases, check [https://modules.beocat.ksu.edu/ our modules website.] If you are used to OpenScienceGrid computing, you may wish to take a look at how to use [[OpenScienceGrid#Using_OpenScienceGrid_modules_on_Beocat|their modules.]]

== Toolchains ==
A toolchain is a set of compilers, libraries and applications that are needed to build software. Some software functions better when using specific toolchains.

We provide a good number of toolchains and versions of toolchains make sure your applications will compile and/or run correctly.

These toolchains include (you can run 'module keyword toolchain'):
; foss: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
; gompi: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.
; iomkl: Intel Cluster Toolchain Compiler Edition provides Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.
; intel: Intel Compiler Suite, providing Intel C/C++ and Fortran compilers, Intel MKL & Intel MPI. Recently made free by Intel, we have less experience with Intel MPI than OpenMPI.

You can run 'module spider $toolchain/' to see the versions we have:
$ module spider iomkl/
* iomkl/2017a
* iomkl/2017b
* iomkl/2017beocatb

If you load one of those (module load iomkl/2017b), you can see the other modules and versions of software that it loaded with the 'module list':
$ module list
Currently Loaded Modules:
1) icc/2017.4.196-GCC-6.4.0-2.28
2) binutils/2.28-GCCcore-6.4.0
3) ifort/2017.4.196-GCC-6.4.0-2.28
4) iccifort/2017.4.196-GCC-6.4.0-2.28
5) GCCcore/6.4.0
6) numactl/2.0.11-GCCcore-6.4.0
7) hwloc/1.11.7-GCCcore-6.4.0
8) OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
9) iompi/2017b
10) imkl/2017.3.196-iompi-2017b
11) iomkl/2017b

As you can see, toolchains can depend on each other. For instance, the iomkl toolchain, depends on iompi, which depends on iccifort, which depend on icc and ifort, which depend on GCCcore which depend on GCC. Hence it is very important that the correct versions of all related software are loaded.

With software we provide, the toolchain used to compile is always specified in the "version" of the software that you want to load.

If you mix toolchains, inconsistent things may happen.

== Most Commonly Used Software ==
Check our [https://modules.beocat.ksu.edu/ modules website] for the most up to date software availability.

The versions mentioned below are representations of what was available at the time of writing, not necessarily what is currently available.
=== [http://www.open-mpi.org/ OpenMPI] ===
We provide lots of versions, you are most likely better off directly loading a toolchain or application to make sure you get the right version, but you can see the versions we have with 'module avail OpenMPI/'

The first step to run an MPI application is to load one of the compiler toolchains that include OpenMPI. You normally will just need to load the default version as below. If your code needs access to nVidia GPUs you'll need the cuda version above. Otherwise some codes are picky about what versions of the underlying GNU or Intel compilers that are needed.

module load foss

If you are working with your own MPI code you will need to start by compiling it. MPI offers mpicc for compiling codes written in C, mpic++ for compiling C++ code, and mpifort for compiling Fortran code. You can get a complete listing of parameters to use by running them with the --help parameter. Below are some examples of compiling with each.

mpicc --help
mpicc -o my_code.x my_code.c
mpic++ -o my_code.x my_code.cc
mpifort -o my_code.x my_code.f

In each case above, you can name the executable file whatever you want (I chose <T>my_code.x). It is common to use different optimization levels, for example, but those may depend on which compiler toolchain you choose. Some are based on the Intel compilers so you'd need to use optimizations for the underlying icc or ifort compilers they call, and some are GNU based so you'd use compiler optimizations for gcc or gfortran.

We have many MPI codes in our modules that you simply need to load before using. Below is an example of loading and running Gromacs which is an MPI based code to simulate large numbers of atoms classically.

module load GROMACS

This loads the Gromacs modules and sets all the paths so you can run the scalar version gmx or the MPI version gmx_mpi. Below is a sample job script for running a complete Gromacs simulation.

#!/bin/bash -l
#SBATCH --mem=120G
#SBATCH --time=24:00:00
#SBATCH --job-name=gromacs
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module reset
module load GROMACS

echo "Running Gromacs on $HOSTNAME"

export OMP_NUM_THREADS=1
time mpirun -x OMP_NUM_THREADS=1 gmx_mpi mdrun -nsteps 500000 -ntomp 1 -v -deffnm 1ns -c 1ns.pdb -nice 0

echo "Finished run on $SLURM_NTASKS $HOSTNAME cores"

mpirun will run your job on all cores requested which in this case is 4 cores on a single node. You will often just need to guess at the memory size for your code, then check on the memory usage with kstat --me and adjust the memory in future jobs.

I prefer to put a module reset in my scripts then manually load the modules needed to insure each run is using the modules it needs. If you don't do this when you submit a job script it will simply use the modules you currently have loaded which is fine too.

I also like to put a time command in front of each part of the script that can use significant amounts of time. This way I can track the amount of time used in each section of the job script. This can prove very useful if your job script copies large data files around at the start, for example, allowing you to see how much time was used for each stage of the job if it runs longer than expected.

The OMP_NUM_THREADS environment variable is set to 1 and passed to the MPI system to insure that each MPI task only uses 1 thread. There are some MPI codes that are also multi-threaded, so this insures that this particular code uses the cores allocated to it in the manner we want.

Once you have your job script ready, submit it using the sbatch command as below where the job script is in the file sb.gromacs.

sbatch sb.gromacs

You should then monitor your job as it goes through the queue and starts running using kstat --me. You code will also generate an output file, usually of the form slurm-#######.out where the 7 # signs are the 7 digit job ID number. If you need to cancel your job use scancel with the 7 digit job ID number.

scancel #######

=== [http://www.r-project.org/ R] ===
You can see what versions of R we provide with 'module avail R/'

==== Packages ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own R Packages ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
module load R
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="R">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="R">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. '<tt>sbatch myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.sbatch

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --mem-per-cpu=4G
# Now we tell Slurm how long we expect our work to take: 15 minutes (D-HH:MM:SS)
#SBATCH --time=0-00:15:00

# Now lets do some actual work. This starts R and loads the file myscript.R
module reset
module load R
R --no-save -q < myscript.R
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
sbatch submit-R.sbatch
</syntaxhighlight>

You can monitor your jobs using kstat --me. The output of your job will be in a slurm-#.out file where '#' is the 7 digit job ID number for your job.

=== [http://www.java.com/ Java] ===
You can see what versions of Java we support with 'module avail Java'

You can load the default version of Java that we offer with "module load Java".

Once you have loaded a Java module, you can use this module to interact with Java how you would normally on your own machine.

Below Is an quick example on how to load the Java module, then compile and run a quick java program that will print input from execution (which in this case is the current working directory).

For reference, here is our java "Main.java" file. Your java filename must match the class name (meaning your file does not have to be called "Main", just make sure these names match).
<syntaxhighlight lang="bash">
public class Main {
static void main (String{} args) {
System.out.println(args[0]);
}
}
</syntaxhighlight>
First we need to load the Java module. At the time of writing, the default Java module is "Java/11.0.20". So we can load that like this:
<syntaxhighlight lang="bash">
module load Java
#or we can load it like this:
module load Java/11.0.20
</syntaxhighlight>
Now we need to compile our "Main.java" file into a class file.
<syntaxhighlight lang="bash">
javac Main.java
</syntaxhighlight>
This will produce a file called "Main.class". Note that "Main" will be whatever you named your file.
Now, we can execute the file and give it something to print. In this case, I am going to print the working directory.
<syntaxhighlight lang="bash">
java Main $PWD
</syntaxhighlight>
Which has an output of wherever I ran this from in the terminal.
<syntaxhighlight lang="bash">
/homes/nathanrwells
</syntaxhighlight>

From here you can put this command inside of a slurm submit script to send off to the compute cluster just like you would with any other bash command. Note that you will need to recompile your "%filename.java" each time you make changes to it, otherwise when you execute the program, nothing will change.

Optionally, you can do all of this compilation and execution inside of your slurm submit script since the files need to have the same name.
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem=4G
#SBATCH --ntasks-per-node=1
#SBATCH --time=0:10:00

module load Java
javac $filename.java
java Main $PWD
</syntaxhighlight>

=== [http://www.python.org/about/ Python] ===
You can see what versions of Python we support with 'module avail Python/'. Note: Running this does not load a Python module, it just shows you a list of the ones that are available.

If you need libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

==== Setting up your virtual environment ====
<syntaxhighlight lang="bash">
# Load Python (pick a version from the 'module avail Python/' list)
module load Python/SOME_VERSION_THAT_YOU_PICKED_FROM_THE_LIST
</syntaxhighlight>
(After running this command Python is loaded. After you logoff and then logon again Python will not be loaded so you must rerun this command every time you logon.)
* Create a location for your virtual environments (optional, but helps keep things organized)
<syntaxhighlight lang="bash">
mkdir ~/virtualenvs
cd ~/virtualenvs
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test'. Note that their [https://docs.python.org/3/library/venv.html documentation] has many more useful options.
<syntaxhighlight lang="bash">
python -m venv --system-site-packages test
# or you could use 'python -m venv test'
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, TensorFlow, or Jupyter
# if you don't use '--system-site-packages' then the virtual environment is completely isolated from our other provided packages and everything it needs it will have to build and install within itself.
</syntaxhighlight>
* Lets look at our virtual environments (the virtual environment name should be in the output):
<syntaxhighlight lang="bash">
ls ~/virtualenvs
</syntaxhighlight>
* Activate one of these
<syntaxhighlight lang="bash">
source ~/virtualenvs/test/bin/activate
</syntaxhighlight>
(After running this command your virtual environment is activated. After you logoff and then logon again your virtual environment will not be loaded so you must rerun this command every time you logon.)
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>

==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment test
<syntaxhighlight lang="bash">
#!/bin/bash
module load Python/THE_SAME_VERSION_YOU_USED_TO_CREATE_YOUR_ENVIRONMENT_ABOVE
source ~/virtualenvs/test/bin/activate
export PYTHONDONTWRITEBYTECODE=1
python ~/path/to/your/python/script.py
</syntaxhighlight>

==== Using MPI with Python within a job ====

We're going to load the SciPy-bundle module, as that has mpi4py available within it.

You check the available versions and load one that uses the python version you would like.
module avail SciPy-bundle

Here is a simple job script using MPI with Python

<syntaxhighlight lang="bash">
#!/bin/bash
module load SciPy-bundle

export PYTHONDONTWRITEBYTECODE=1
mpirun python ~/path/to/your/mpi/python/script.py
</syntaxhighlight>

=== [https://www.tensorflow.org/ TensorFlow] ===
TensorFlow provided by pip is often completely broken on any system that is not running a recent version of Ubuntu. Beocat (and most HPC systems) does not use Ubuntu. As such, we provide TensorFlow modules for you to load.

You can see what versions of TensorFlow we support with 'module avail TensorFlow/'. Note: Running this does not load a TensorFlow module, it just shows you a list of the ones that are available.

If you need other python libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

We document creating a virtual environment [[#Setting up your virtual environment|above]]. You can skip loading the python module, as loading TensorFlow will load the correct version of python module behind the scenes. The singular change you need to make is to use the '--system-site-packages' when creating the virtual environment.
<syntaxhighlight lang=bash>
python -m venv --system-site-packages test
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, or TensorFlow
</syntaxhighlight>

=== Jupyter ===
[https://jupyter.org/ Jupyter] is a framework for creating and running reusable "notebooks" for scientific computing. It runs Python code by default. Normally, it is meant to be used in an interactive manner. Interactive codes can be limiting and/or problematic when used in a cluster environment. We have an example submit script available [https://gitlab.beocat.ksu.edu/Admin-Public/ondemand/job_templates/-/tree/master/Jupyter_Notebook here] to help you transition from an OpenOnDemand interactive job using Jupyter to a non-interactive job.

=== [http://spark.apache.org/ Spark] ===

Spark is a programming language for large scale data processing.
It can be used in conjunction with Python, R, Scala, Java, and SQL.
Spark can be run on Beocat interactively or through the Slurm queue.

To run interactively, you must first request a node or nodes from the Slurm queue.
The line below requests 1 node and 1 core for 24 hours and if available will drop
you into the bash shell on that node.
<syntaxhighlight lang=bash>
srun -J srun -N 1 -n 1 -t 24:00:00 --mem=10G --pty bash
</syntaxhighlight>
We have some sample python based Spark code you can try out that came from the
exercises and homework from the PSC Spark workshop.
<syntaxhighlight lang=bash>
mkdir spark-test
cd spark-test
cp -rp /homes/daveturner/projects/PSC-BigData-Workshop/Shakespeare/* .
</syntaxhighlight>
You will need to set up a python virtual environment and load the nltk package
before you run the first time.
<syntaxhighlight lang=bash>
module load Spark
mkdir -p ~/virtualenvs
cd ~/virtualenvs
python -m venv --system-site-packages spark-test
source ~/virtualenvs/spark-test/bin/activate
pip install nltk
deactivate
</syntaxhighlight>
To run the sample code interactively, load the Python and Spark modules,
source your python virtual environment, change to the sample directory, fire up pyspark,
then execute the sample code.
<syntaxhighlight lang=bash>
module load Spark
source ~/virtualenvs/spark-test/bin/activate
cd ~/spark-test
pyspark
</syntaxhighlight>
<syntaxhighlight lang=python>
exec(open("shakespeare.py").read())
</syntaxhighlight>
You can work interactively from the pyspark prompt (>>>) in addition to running scripts as above.

The Shakespeare directory also contains a sample sbatch submit script that will run the
same shakespeare.py code through the Slurm batch queue.
<syntaxhighlight lang=bash>
#!/bin/bash -l
#SBATCH --job-name=shakespeare
#SBATCH --mem=10G
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

# Load Spark and Python (version 3 here)
module load Spark
source ~/virtualenvs/spark-test/bin/activate

spark-submit shakespeare.py
</syntaxhighlight>
When you run interactively, pyspark initializes your spark context sc.
You will need to do this manually as in the sample python code when you want
to submit jobs through the Slurm queue.
<syntaxhighlight lang=python>
# If there is no Spark Context (not running interactive from pyspark), create it
try:
sc
except NameError:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("App")
sc = SparkContext(conf = conf)
</syntaxhighlight>

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.

To use perl with threads, out a newer version, you can load it with the module command. To see what versions of perl we provide, you can use 'module avail Perl/'

==== Installing Perl Modules ====

The easiest way to install Perl modules is by using cpanm.
Below is an example of installing the Perl module Term::ANSIColor.

<syntaxhighlight lang=bash>
module load Perl
cpanm -i Term::ANSIColor
</syntaxhighlight>

CPAN: LWP::UserAgent loaded ok (v6.39)
Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
CPAN: YAML loaded ok (v1.29)
Reading '/homes/mozes/.cpan/sources/authors/01mailrc.txt.gz'
CPAN: Compress::Zlib loaded ok (v2.084)
............................................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/02packages.details.txt.gz
Reading '/homes/mozes/.cpan/sources/modules/02packages.details.txt.gz'
Database was generated on Mon, 09 Mar 2020 20:41:03 GMT
.............
New CPAN.pm version (v2.27) available.
[Currently running version is v2.22]
You might want to try
install CPAN
reload cpan
to both upgrade CPAN.pm and run the new version without leaving
the current session.
...............................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/03modlist.data.gz
Reading '/homes/mozes/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /homes/mozes/.cpan/Metadata
Running install for module 'Term::ANSIColor'
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz
CPAN: Digest::SHA loaded ok (v6.02)
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/CHECKSUMS
Checksum for /homes/mozes/.cpan/sources/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz ok
CPAN: CPAN::Meta::Requirements loaded ok (v2.140)
CPAN: Parse::CPAN::Meta loaded ok (v2.150010)
CPAN: CPAN::Meta loaded ok (v2.150010)
CPAN: Module::CoreList loaded ok (v5.20190522)
Configuring R/RR/RRA/Term-ANSIColor-5.01.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Term::ANSIColor
Writing MYMETA.yml and MYMETA.json
RRA/Term-ANSIColor-5.01.tar.gz
/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl Makefile.PL -- OK
Running make for R/RR/RRA/Term-ANSIColor-5.01.tar.gz
cp lib/Term/ANSIColor.pm blib/lib/Term/ANSIColor.pm
Manifying 1 pod document
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make -- OK
Running make test for RRA/Term-ANSIColor-5.01.tar.gz
PERL_DL_NONLAZY=1 "/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*/*.t
t/docs/pod-coverage.t ....... skipped: POD coverage tests normally skipped
t/docs/pod-spelling.t ....... skipped: Spelling tests only run for author
t/docs/pod.t ................ skipped: POD syntax tests normally skipped
t/docs/spdx-license.t ....... skipped: SPDX identifier tests normally skipped
t/docs/synopsis.t ........... skipped: Synopsis syntax tests normally skipped
t/module/aliases-env.t ...... ok
t/module/aliases-func.t ..... ok
t/module/basic.t ............ ok
t/module/basic256.t ......... ok
t/module/eval.t ............. ok
t/module/stringify.t ........ ok
t/module/true-color.t ....... ok
t/style/coverage.t .......... skipped: Coverage tests only run for author
t/style/critic.t ............ skipped: Coding style tests only run for author
t/style/minimum-version.t ... skipped: Minimum version tests normally skipped
t/style/obsolete-strings.t .. skipped: Obsolete strings tests normally skipped
t/style/strict.t ............ skipped: Strictness tests normally skipped
t/taint/basic.t ............. ok
All tests successful.
Files=18, Tests=430, 7 wallclock secs ( 0.21 usr 0.08 sys + 3.41 cusr 1.15 csys = 4.85 CPU)
Result: PASS
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make test -- OK
Running make install for RRA/Term-ANSIColor-5.01.tar.gz
Manifying 1 pod document
Installing /homes/mozes/perl5/lib/perl5/Term/ANSIColor.pm
Installing /homes/mozes/perl5/man/man3/Term::ANSIColor.3
Appending installation info to /homes/mozes/perl5/lib/perl5/x86_64-linux-thread-multi/perllocal.pod
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make install -- OK

===== When things go wrong =====
Some perl modules fail to realize they shouldn't be installed globally. Usually, you'll notice this when they try to run 'sudo' something. Unfortunately we do not grant sudo access to anyone other then Beocat system administrators. Usually, this can be worked around by putting the following in your <tt>~/.bashrc</tt> file (at the bottom). Once this is in place, you should log out and log back in.
<syntaxhighlight lang="bash">
PATH="/homes/${USER}/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/homes/${USER}/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}";
export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/homes/${USER}/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}";
export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/homes/${USER}/perl5\""; export PERL_MB_OPT;
</syntaxhighlight>

==== Submitting a job with Perl ====
Much like R (above), you cannot simply '<tt>sbatch myProgram.pl</tt>', but you must create a [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|submit script]] which will call perl. Here is an example:
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell sbatch how long we expect our work to take: 15 minutes (H:MM:SS)
#SBATCH --time=0-0:15:00
# Now lets do some actual work.
module load Perl
perl /path/to/myProgram.pl
</syntaxhighlight>

=== Octave for MatLab codes ===

'module avail Octave/'

The 64-bit version of Octave can be loaded using the command above. Octave can then be used
to work with MatLab codes on the head node and to submit jobs to the compute nodes through the
sbatch scheduler. Octave is made to run MatLab code, but it does have limitations and does not support
everything that MatLab itself does.

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=octave
#SBATCH --output=octave.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load Octave/4.2.1-foss-2017beocatb-enable64

octave < matlab_code.m
</syntaxhighlight>

=== MatLab compiler ===

Beocat also has a single floating user license for the MatLab compiler and the most common toolboxes
including the Parallel Computing Toolbox, Optimization Toolbox, Statistics and Machine Learning Toolbox,
Image Processing Toolbox, Curve Fitting Toolbox, Neural Network Toolbox, Symbolic Math Toolbox,
Global Optimization Toolbox, and the Bioinformatics Toolbox.

Since we only have a single floating user license, this means that you will be expected to develop your MatLab code
with Octave or elsewhere on a laptop or departmental server. Once you're ready to do large runs, then you
move your code to Beocat, compile the MatLab code into an executable, and you can submit as many jobs as
you want to the scheduler. To use the MatLab compiler, you need to load the MATLAB module to compile code and
load the mcr module to run the resulting MatLab executable.

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -o matlab_executable_name
</syntaxhighlight>

If you have addpath() commands in your code, you will need to wrap them in an "if ~deployed" block and tell the
compiler to include that path via the -I flag.

<syntaxhighlight lang="MATLAB">
% wrap addpath() calls like so:
if ~deployed
addpath('./another/folder/with/code/')
end
</syntaxhighlight>

NOTE: The license manager checks the mcc compiler out for a minimum of 30 minutes, so if another user compiles a code
you unfortunately may need to wait for up to 30 minutes to compile your own code.

Compiling with additional paths:

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -I ./another/folder/with/code/ -o matlab_executable_name
</syntaxhighlight>

Any directories added with addpath() will need to be added to the list of compile options as -I arguments. You
can have multiple -I arguments in your compile command.

Here is an example job submission script. Modify time, memory, tasks-per-node, and job name as you see fit:

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=matlab
#SBATCH --output=matlab.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load mcr

./matlab_executable_name
</syntaxhighlight>

For those who make use of mex files - compiled C and C++ code with matlab bindings - you will need to add these
files to the compiled archive via the -a flag. See the behavior of this flag in the [https://www.mathworks.com/help/compiler/mcc.html compiler documentation]. You can either target specific .mex files or entire directories.

Because codes often require adding several directories to the Matlab path as well as mex files from several locations,
we recommend writing a script to preserve and help document the steps to compile your Matlab code. Here is an
abbreviated example from a current user:

<syntaxhighlight lang="bash">
#!/bin/bash -l

module load MATLAB

cd matlabPyrTools/MEX/

# compile mex files
mex upConv.c convolve.c wrap.c edges.c
mex corrDn.c convolve.c wrap.c edges.c
mex histo.c
mex innerProd.c

cd ../..

mcc -m mongrel_creation.m \
-I ./matlabPyrTools/MEX/ \
-I ./matlabPyrTools/ \
-I ./FastICA/ \
-a ./matlabPyrTools/MEX/ \
-a ./texturesynth/ \
-o mongrel_creation_binary
</syntaxhighlight>

Again, we only have a single floating user license for MatLab so the model is to develop and debug your MatLab code
elsewhere or using Octave on Beocat, then you can compile the MatLab code into an executable and run it without
limits on Beocat.

For more info on the mcc compiler see: https://www.mathworks.com/help/compiler/mcc.html

=== COMSOL ===
Beocat has no license for COMSOL. If you want to use it, you must provide your own.

module spider COMSOL/
----------------------------------------------------------------------------
COMSOL: COMSOL/5.3
----------------------------------------------------------------------------
Description:
COMSOL Multiphysics software, an interactive environment for modeling
and simulating scientific and engineering problems

This module can be loaded directly: module load COMSOL/5.3

Help:

Description
===========
COMSOL Multiphysics software, an interactive environment for modeling and
simulating scientific and engineering problems
You must provide your own license.
export LM_LICENSE_FILE=/the/path/to/your/license/file
*OR*
export LM_LICENSE_FILE=$LICENSE_SERVER_PORT@$LICENSE_SERVER_HOSTNAME
e.g. export LM_LICENSE_FILE=1719@some.flexlm.server.ksu.edu

More information
================
- Homepage: https://www.comsol.com/
==== Graphical COMSOL ====
Running COMSOL in graphical mode on a cluster is generally a bad idea. If you choose to run it in graphical mode on a compute node, you will need to do something like the following:
<syntaxhighlight lang="bash">
# Connect to the cluster with X11 forwarding (ssh -Y or mobaxterm)
# load the comsol module on the headnode
module load COMSOL
# export your comsol license as mentioned above, and tell the scheduler to run the software
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 comsol -3drend sw
</syntaxhighlight>

=== .NET Core ===
==== Load .NET ====
mozes@[eunomia] ~ $ module load dotNET-Core-SDK
==== create an application ====
Following instructions from [https://docs.microsoft.com/en-us/dotnet/core/tutorials/using-with-xplat-cli here], we'll create a simple 'Hello World' application
mozes@[eunomia] ~ $ mkdir Hello

mozes@[eunomia] ~ $ cd Hello

mozes@[eunomia] ~/Hello $ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true

mozes@[eunomia] ~/Hello $ dotnet new console
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /homes/mozes/Hello/Hello.csproj...
Restoring packages for /homes/mozes/Hello/Hello.csproj...
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.props.
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.targets.
Restore completed in 358.43 ms for /homes/mozes/Hello/Hello.csproj.

Restore succeeded.

==== Edit your program ====
mozes@[eunomia] ~/Hello $ vi Program.cs
==== Run your .NET application ====
mozes@[eunomia] ~/Hello $ dotnet run
Hello World!
==== Build and run the built application ====
mozes@[eunomia] ~/Hello $ dotnet build
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

Restore completed in 106.12 ms for /homes/mozes/Hello/Hello.csproj.
Hello -> /homes/mozes/Hello/bin/Debug/netcoreapp2.1/Hello.dll

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:02.86

mozes@[eunomia] ~/Hello $ dotnet bin/Debug/netcoreapp2.1/Hello.dll
Hello World!

== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

== Loading multiple modules ==
modules, when loaded, will stay loaded for the duration of your session until they are unloaded.

; You can load multiple pieces of software with one module load command. : module load iompi iomkl

; You can unload all software : module reset

; If you see output from a module load command that looks like ''"The following have been reloaded with a version change"'' you likely have tried to load two pieces of software that have not been tested together. There may be serious issues with using either pieces of software while you're in this state. Libraries missing, applications non-functional. If you encounter issues, you will want to unload all software before switching modules. : 'module reset' and then 'module load'

== Containers ==
More and more science is being done within containers, these days. Sometimes referred to Docker or Kubernetes, containers allow you to package an entire software runtime platform and run that software on another computer or site with minimal fuss.

Unfortunately, Docker and Kubernetes are not particularly well suited to multi-user HPC environments, but that's not to say that you can't make use of these containers on Beocat.

=== Apptainer ===
[https://apptainer.org/docs/user/1.2/index.html Apptainer] is a container runtime that is designed for HPC environments. It can convert docker containers to its own format, and can be used within a job on Beocat. It is a very broad topic and we've made the decision to point you to the upstream documentation, as it is much more likely that they'll have up to date and functional instructions to help you utilize containers. If you need additional assistance, please don't hesitate to reach out to us.

Installed software

2025-04-03T20:32:57Z

Nathanrwells: /* Java */

== Module Availability ==
Most people will be just fine running 'module avail' to see a list of modules available on Beocat. There are a couple software packages that are only available on particular node types. For those cases, check [https://modules.beocat.ksu.edu/ our modules website.] If you are used to OpenScienceGrid computing, you may wish to take a look at how to use [[OpenScienceGrid#Using_OpenScienceGrid_modules_on_Beocat|their modules.]]

== Toolchains ==
A toolchain is a set of compilers, libraries and applications that are needed to build software. Some software functions better when using specific toolchains.

We provide a good number of toolchains and versions of toolchains make sure your applications will compile and/or run correctly.

These toolchains include (you can run 'module keyword toolchain'):
; foss: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK.
; gompi: GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support.
; iomkl: Intel Cluster Toolchain Compiler Edition provides Intel C/C++ and Fortran compilers, Intel MKL & OpenMPI.
; intel: Intel Compiler Suite, providing Intel C/C++ and Fortran compilers, Intel MKL & Intel MPI. Recently made free by Intel, we have less experience with Intel MPI than OpenMPI.

You can run 'module spider $toolchain/' to see the versions we have:
$ module spider iomkl/
* iomkl/2017a
* iomkl/2017b
* iomkl/2017beocatb

If you load one of those (module load iomkl/2017b), you can see the other modules and versions of software that it loaded with the 'module list':
$ module list
Currently Loaded Modules:
1) icc/2017.4.196-GCC-6.4.0-2.28
2) binutils/2.28-GCCcore-6.4.0
3) ifort/2017.4.196-GCC-6.4.0-2.28
4) iccifort/2017.4.196-GCC-6.4.0-2.28
5) GCCcore/6.4.0
6) numactl/2.0.11-GCCcore-6.4.0
7) hwloc/1.11.7-GCCcore-6.4.0
8) OpenMPI/2.1.1-iccifort-2017.4.196-GCC-6.4.0-2.28
9) iompi/2017b
10) imkl/2017.3.196-iompi-2017b
11) iomkl/2017b

As you can see, toolchains can depend on each other. For instance, the iomkl toolchain, depends on iompi, which depends on iccifort, which depend on icc and ifort, which depend on GCCcore which depend on GCC. Hence it is very important that the correct versions of all related software are loaded.

With software we provide, the toolchain used to compile is always specified in the "version" of the software that you want to load.

If you mix toolchains, inconsistent things may happen.

== Most Commonly Used Software ==
Check our [https://modules.beocat.ksu.edu/ modules website] for the most up to date software availability.

The versions mentioned below are representations of what was available at the time of writing, not necessarily what is currently available.
=== [http://www.open-mpi.org/ OpenMPI] ===
We provide lots of versions, you are most likely better off directly loading a toolchain or application to make sure you get the right version, but you can see the versions we have with 'module avail OpenMPI/'

The first step to run an MPI application is to load one of the compiler toolchains that include OpenMPI. You normally will just need to load the default version as below. If your code needs access to nVidia GPUs you'll need the cuda version above. Otherwise some codes are picky about what versions of the underlying GNU or Intel compilers that are needed.

module load foss

If you are working with your own MPI code you will need to start by compiling it. MPI offers mpicc for compiling codes written in C, mpic++ for compiling C++ code, and mpifort for compiling Fortran code. You can get a complete listing of parameters to use by running them with the --help parameter. Below are some examples of compiling with each.

mpicc --help
mpicc -o my_code.x my_code.c
mpic++ -o my_code.x my_code.cc
mpifort -o my_code.x my_code.f

In each case above, you can name the executable file whatever you want (I chose <T>my_code.x). It is common to use different optimization levels, for example, but those may depend on which compiler toolchain you choose. Some are based on the Intel compilers so you'd need to use optimizations for the underlying icc or ifort compilers they call, and some are GNU based so you'd use compiler optimizations for gcc or gfortran.

We have many MPI codes in our modules that you simply need to load before using. Below is an example of loading and running Gromacs which is an MPI based code to simulate large numbers of atoms classically.

module load GROMACS

This loads the Gromacs modules and sets all the paths so you can run the scalar version gmx or the MPI version gmx_mpi. Below is a sample job script for running a complete Gromacs simulation.

#!/bin/bash -l
#SBATCH --mem=120G
#SBATCH --time=24:00:00
#SBATCH --job-name=gromacs
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4

module reset
module load GROMACS

echo "Running Gromacs on $HOSTNAME"

export OMP_NUM_THREADS=1
time mpirun -x OMP_NUM_THREADS=1 gmx_mpi mdrun -nsteps 500000 -ntomp 1 -v -deffnm 1ns -c 1ns.pdb -nice 0

echo "Finished run on $SLURM_NTASKS $HOSTNAME cores"

mpirun will run your job on all cores requested which in this case is 4 cores on a single node. You will often just need to guess at the memory size for your code, then check on the memory usage with kstat --me and adjust the memory in future jobs.

I prefer to put a module reset in my scripts then manually load the modules needed to insure each run is using the modules it needs. If you don't do this when you submit a job script it will simply use the modules you currently have loaded which is fine too.

I also like to put a time command in front of each part of the script that can use significant amounts of time. This way I can track the amount of time used in each section of the job script. This can prove very useful if your job script copies large data files around at the start, for example, allowing you to see how much time was used for each stage of the job if it runs longer than expected.

The OMP_NUM_THREADS environment variable is set to 1 and passed to the MPI system to insure that each MPI task only uses 1 thread. There are some MPI codes that are also multi-threaded, so this insures that this particular code uses the cores allocated to it in the manner we want.

Once you have your job script ready, submit it using the sbatch command as below where the job script is in the file sb.gromacs.

sbatch sb.gromacs

You should then monitor your job as it goes through the queue and starts running using kstat --me. You code will also generate an output file, usually of the form slurm-#######.out where the 7 # signs are the 7 digit job ID number. If you need to cancel your job use scancel with the 7 digit job ID number.

scancel #######

=== [http://www.r-project.org/ R] ===
You can see what versions of R we provide with 'module avail R/'

==== Packages ====
We provide a small number of R modules installed by default, these are generally modules that are needed by more than one person.

==== Installing your own R Packages ====
To install your own module, login to Beocat and start R interactively
<syntaxhighlight lang="bash">
module load R
R
</syntaxhighlight>
Then install the package using
<syntaxhighlight lang="R">
install.packages("PACKAGENAME")
</syntaxhighlight>
Follow the prompts. Note that there is a CRAN mirror at KU - it will be listed as "USA (KS)".

After installing you can test before leaving interactive mode by issuing the command
<syntaxhighlight lang="R">
library("PACKAGENAME")
</syntaxhighlight>
==== Running R Jobs ====

You cannot submit an R script directly. '<tt>sbatch myscript.R</tt>' will result in an error. Instead, you need to make a bash [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|script]] that will call R appropriately. Here is a minimal example. We'll save this as submit-R.sbatch

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --mem-per-cpu=4G
# Now we tell Slurm how long we expect our work to take: 15 minutes (D-HH:MM:SS)
#SBATCH --time=0-00:15:00

# Now lets do some actual work. This starts R and loads the file myscript.R
module reset
module load R
R --no-save -q < myscript.R
</syntaxhighlight>

Now, to submit your R job, you would type
<syntaxhighlight lang="bash">
sbatch submit-R.sbatch
</syntaxhighlight>

You can monitor your jobs using kstat --me. The output of your job will be in a slurm-#.out file where '#' is the 7 digit job ID number for your job.

=== [http://www.java.com/ Java] ===
You can see what versions of Java we support with 'module avail Java'

You can load the default version of Java that we offer with "module load Java".

Once you have loaded a Java module, you can use this module to interact with Java how you would normally on your own machine.

Below Is an quick exmaple on how to load the Java module, then compile and run a quick java program that will print input from the execution.

For reference, here is our java "Main.java" file. Your java filename must match the class name (meaning your file does not have to be called "Main", just make sure these names match).
<syntaxhighlight lang="bash">
public class Main {
static void main (String{} args) {
System.out.println(args[0]);
}
}
</syntaxhighlight>
First we need to load the Java module. At the time of writing, the default Java module is "Java/11.0.20". So we can load that like this:
<syntaxhighlight lang="bash">
module load Java
#or we can load it like this:
module load Java/11.0.20
</syntaxhighlight>
Now we need to compile our "Main.java" file into a class file.
<syntaxhighlight lang="bash">
javac Main.java
</syntaxhighlight>
This will produce a file called "Main.class". Note that "Main" will be whatever you named your file.
Now, we can execute the file and give it something to print. In this case, I am going to print the working directory.
<syntaxhighlight lang="bash">
java Main $PWD
</syntaxhighlight>
Which has an output of wherever I ran this from in the terminal.
<syntaxhighlight lang="bash">
/homes/nathanrwells
</syntaxhighlight>

=== [http://www.python.org/about/ Python] ===
You can see what versions of Python we support with 'module avail Python/'. Note: Running this does not load a Python module, it just shows you a list of the ones that are available.

If you need libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

==== Setting up your virtual environment ====
<syntaxhighlight lang="bash">
# Load Python (pick a version from the 'module avail Python/' list)
module load Python/SOME_VERSION_THAT_YOU_PICKED_FROM_THE_LIST
</syntaxhighlight>
(After running this command Python is loaded. After you logoff and then logon again Python will not be loaded so you must rerun this command every time you logon.)
* Create a location for your virtual environments (optional, but helps keep things organized)
<syntaxhighlight lang="bash">
mkdir ~/virtualenvs
cd ~/virtualenvs
</syntaxhighlight>
* Create a virtual environment. Here I will create a default virtual environment called 'test'. Note that their [https://docs.python.org/3/library/venv.html documentation] has many more useful options.
<syntaxhighlight lang="bash">
python -m venv --system-site-packages test
# or you could use 'python -m venv test'
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, TensorFlow, or Jupyter
# if you don't use '--system-site-packages' then the virtual environment is completely isolated from our other provided packages and everything it needs it will have to build and install within itself.
</syntaxhighlight>
* Lets look at our virtual environments (the virtual environment name should be in the output):
<syntaxhighlight lang="bash">
ls ~/virtualenvs
</syntaxhighlight>
* Activate one of these
<syntaxhighlight lang="bash">
source ~/virtualenvs/test/bin/activate
</syntaxhighlight>
(After running this command your virtual environment is activated. After you logoff and then logon again your virtual environment will not be loaded so you must rerun this command every time you logon.)
* You can now install the python modules you want. This can be done using <tt>pip</tt>.
<syntaxhighlight lang="bash">
pip install numpy biopython
</syntaxhighlight>

==== Using your virtual environment within a job ====
Here is a simple job script using the virtual environment test
<syntaxhighlight lang="bash">
#!/bin/bash
module load Python/THE_SAME_VERSION_YOU_USED_TO_CREATE_YOUR_ENVIRONMENT_ABOVE
source ~/virtualenvs/test/bin/activate
export PYTHONDONTWRITEBYTECODE=1
python ~/path/to/your/python/script.py
</syntaxhighlight>

==== Using MPI with Python within a job ====

We're going to load the SciPy-bundle module, as that has mpi4py available within it.

You check the available versions and load one that uses the python version you would like.
module avail SciPy-bundle

Here is a simple job script using MPI with Python

<syntaxhighlight lang="bash">
#!/bin/bash
module load SciPy-bundle

export PYTHONDONTWRITEBYTECODE=1
mpirun python ~/path/to/your/mpi/python/script.py
</syntaxhighlight>

=== [https://www.tensorflow.org/ TensorFlow] ===
TensorFlow provided by pip is often completely broken on any system that is not running a recent version of Ubuntu. Beocat (and most HPC systems) does not use Ubuntu. As such, we provide TensorFlow modules for you to load.

You can see what versions of TensorFlow we support with 'module avail TensorFlow/'. Note: Running this does not load a TensorFlow module, it just shows you a list of the ones that are available.

If you need other python libraries that we do not have installed, you should use [https://docs.python.org/3/library/venv.html python -m venv] to setup a virtual python environment in your home directory. This will let you install python libraries as you please.

We document creating a virtual environment [[#Setting up your virtual environment|above]]. You can skip loading the python module, as loading TensorFlow will load the correct version of python module behind the scenes. The singular change you need to make is to use the '--system-site-packages' when creating the virtual environment.
<syntaxhighlight lang=bash>
python -m venv --system-site-packages test
# using the '--system-site-packages' allows the virtual environment to make use of python libraries we have already installed
# particularly useful if you're going to use our SciPy-Bundle, or TensorFlow
</syntaxhighlight>

=== Jupyter ===
[https://jupyter.org/ Jupyter] is a framework for creating and running reusable "notebooks" for scientific computing. It runs Python code by default. Normally, it is meant to be used in an interactive manner. Interactive codes can be limiting and/or problematic when used in a cluster environment. We have an example submit script available [https://gitlab.beocat.ksu.edu/Admin-Public/ondemand/job_templates/-/tree/master/Jupyter_Notebook here] to help you transition from an OpenOnDemand interactive job using Jupyter to a non-interactive job.

=== [http://spark.apache.org/ Spark] ===

Spark is a programming language for large scale data processing.
It can be used in conjunction with Python, R, Scala, Java, and SQL.
Spark can be run on Beocat interactively or through the Slurm queue.

To run interactively, you must first request a node or nodes from the Slurm queue.
The line below requests 1 node and 1 core for 24 hours and if available will drop
you into the bash shell on that node.
<syntaxhighlight lang=bash>
srun -J srun -N 1 -n 1 -t 24:00:00 --mem=10G --pty bash
</syntaxhighlight>
We have some sample python based Spark code you can try out that came from the
exercises and homework from the PSC Spark workshop.
<syntaxhighlight lang=bash>
mkdir spark-test
cd spark-test
cp -rp /homes/daveturner/projects/PSC-BigData-Workshop/Shakespeare/* .
</syntaxhighlight>
You will need to set up a python virtual environment and load the nltk package
before you run the first time.
<syntaxhighlight lang=bash>
module load Spark
mkdir -p ~/virtualenvs
cd ~/virtualenvs
python -m venv --system-site-packages spark-test
source ~/virtualenvs/spark-test/bin/activate
pip install nltk
deactivate
</syntaxhighlight>
To run the sample code interactively, load the Python and Spark modules,
source your python virtual environment, change to the sample directory, fire up pyspark,
then execute the sample code.
<syntaxhighlight lang=bash>
module load Spark
source ~/virtualenvs/spark-test/bin/activate
cd ~/spark-test
pyspark
</syntaxhighlight>
<syntaxhighlight lang=python>
exec(open("shakespeare.py").read())
</syntaxhighlight>
You can work interactively from the pyspark prompt (>>>) in addition to running scripts as above.

The Shakespeare directory also contains a sample sbatch submit script that will run the
same shakespeare.py code through the Slurm batch queue.
<syntaxhighlight lang=bash>
#!/bin/bash -l
#SBATCH --job-name=shakespeare
#SBATCH --mem=10G
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

# Load Spark and Python (version 3 here)
module load Spark
source ~/virtualenvs/spark-test/bin/activate

spark-submit shakespeare.py
</syntaxhighlight>
When you run interactively, pyspark initializes your spark context sc.
You will need to do this manually as in the sample python code when you want
to submit jobs through the Slurm queue.
<syntaxhighlight lang=python>
# If there is no Spark Context (not running interactive from pyspark), create it
try:
sc
except NameError:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("App")
sc = SparkContext(conf = conf)
</syntaxhighlight>

=== [http://www.perl.org/ Perl] ===
The system-wide version of perl is tracking the stable releases of perl. Unfortunately there are some features that we do not include in the system distribution of perl, namely threads.

To use perl with threads, out a newer version, you can load it with the module command. To see what versions of perl we provide, you can use 'module avail Perl/'

==== Installing Perl Modules ====

The easiest way to install Perl modules is by using cpanm.
Below is an example of installing the Perl module Term::ANSIColor.

<syntaxhighlight lang=bash>
module load Perl
cpanm -i Term::ANSIColor
</syntaxhighlight>

CPAN: LWP::UserAgent loaded ok (v6.39)
Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
CPAN: YAML loaded ok (v1.29)
Reading '/homes/mozes/.cpan/sources/authors/01mailrc.txt.gz'
CPAN: Compress::Zlib loaded ok (v2.084)
............................................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/02packages.details.txt.gz
Reading '/homes/mozes/.cpan/sources/modules/02packages.details.txt.gz'
Database was generated on Mon, 09 Mar 2020 20:41:03 GMT
.............
New CPAN.pm version (v2.27) available.
[Currently running version is v2.22]
You might want to try
install CPAN
reload cpan
to both upgrade CPAN.pm and run the new version without leaving
the current session.
...............................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/03modlist.data.gz
Reading '/homes/mozes/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /homes/mozes/.cpan/Metadata
Running install for module 'Term::ANSIColor'
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz
CPAN: Digest::SHA loaded ok (v6.02)
Fetching with LWP:
http://www.cpan.org/authors/id/R/RR/RRA/CHECKSUMS
Checksum for /homes/mozes/.cpan/sources/authors/id/R/RR/RRA/Term-ANSIColor-5.01.tar.gz ok
CPAN: CPAN::Meta::Requirements loaded ok (v2.140)
CPAN: Parse::CPAN::Meta loaded ok (v2.150010)
CPAN: CPAN::Meta loaded ok (v2.150010)
CPAN: Module::CoreList loaded ok (v5.20190522)
Configuring R/RR/RRA/Term-ANSIColor-5.01.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Term::ANSIColor
Writing MYMETA.yml and MYMETA.json
RRA/Term-ANSIColor-5.01.tar.gz
/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl Makefile.PL -- OK
Running make for R/RR/RRA/Term-ANSIColor-5.01.tar.gz
cp lib/Term/ANSIColor.pm blib/lib/Term/ANSIColor.pm
Manifying 1 pod document
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make -- OK
Running make test for RRA/Term-ANSIColor-5.01.tar.gz
PERL_DL_NONLAZY=1 "/opt/software/software/Perl/5.30.0-GCCcore-8.3.0/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*/*.t
t/docs/pod-coverage.t ....... skipped: POD coverage tests normally skipped
t/docs/pod-spelling.t ....... skipped: Spelling tests only run for author
t/docs/pod.t ................ skipped: POD syntax tests normally skipped
t/docs/spdx-license.t ....... skipped: SPDX identifier tests normally skipped
t/docs/synopsis.t ........... skipped: Synopsis syntax tests normally skipped
t/module/aliases-env.t ...... ok
t/module/aliases-func.t ..... ok
t/module/basic.t ............ ok
t/module/basic256.t ......... ok
t/module/eval.t ............. ok
t/module/stringify.t ........ ok
t/module/true-color.t ....... ok
t/style/coverage.t .......... skipped: Coverage tests only run for author
t/style/critic.t ............ skipped: Coding style tests only run for author
t/style/minimum-version.t ... skipped: Minimum version tests normally skipped
t/style/obsolete-strings.t .. skipped: Obsolete strings tests normally skipped
t/style/strict.t ............ skipped: Strictness tests normally skipped
t/taint/basic.t ............. ok
All tests successful.
Files=18, Tests=430, 7 wallclock secs ( 0.21 usr 0.08 sys + 3.41 cusr 1.15 csys = 4.85 CPU)
Result: PASS
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make test -- OK
Running make install for RRA/Term-ANSIColor-5.01.tar.gz
Manifying 1 pod document
Installing /homes/mozes/perl5/lib/perl5/Term/ANSIColor.pm
Installing /homes/mozes/perl5/man/man3/Term::ANSIColor.3
Appending installation info to /homes/mozes/perl5/lib/perl5/x86_64-linux-thread-multi/perllocal.pod
RRA/Term-ANSIColor-5.01.tar.gz
/usr/bin/make install -- OK

===== When things go wrong =====
Some perl modules fail to realize they shouldn't be installed globally. Usually, you'll notice this when they try to run 'sudo' something. Unfortunately we do not grant sudo access to anyone other then Beocat system administrators. Usually, this can be worked around by putting the following in your <tt>~/.bashrc</tt> file (at the bottom). Once this is in place, you should log out and log back in.
<syntaxhighlight lang="bash">
PATH="/homes/${USER}/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/homes/${USER}/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}";
export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/homes/${USER}/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}";
export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/homes/${USER}/perl5\""; export PERL_MB_OPT;
</syntaxhighlight>

==== Submitting a job with Perl ====
Much like R (above), you cannot simply '<tt>sbatch myProgram.pl</tt>', but you must create a [[AdvancedSlurm#Running_from_a_sbatch_Submit_Script|submit script]] which will call perl. Here is an example:
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --mem-per-cpu=1G
# Now we tell sbatch how long we expect our work to take: 15 minutes (H:MM:SS)
#SBATCH --time=0-0:15:00
# Now lets do some actual work.
module load Perl
perl /path/to/myProgram.pl
</syntaxhighlight>

=== Octave for MatLab codes ===

'module avail Octave/'

The 64-bit version of Octave can be loaded using the command above. Octave can then be used
to work with MatLab codes on the head node and to submit jobs to the compute nodes through the
sbatch scheduler. Octave is made to run MatLab code, but it does have limitations and does not support
everything that MatLab itself does.

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=octave
#SBATCH --output=octave.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load Octave/4.2.1-foss-2017beocatb-enable64

octave < matlab_code.m
</syntaxhighlight>

=== MatLab compiler ===

Beocat also has a single floating user license for the MatLab compiler and the most common toolboxes
including the Parallel Computing Toolbox, Optimization Toolbox, Statistics and Machine Learning Toolbox,
Image Processing Toolbox, Curve Fitting Toolbox, Neural Network Toolbox, Symbolic Math Toolbox,
Global Optimization Toolbox, and the Bioinformatics Toolbox.

Since we only have a single floating user license, this means that you will be expected to develop your MatLab code
with Octave or elsewhere on a laptop or departmental server. Once you're ready to do large runs, then you
move your code to Beocat, compile the MatLab code into an executable, and you can submit as many jobs as
you want to the scheduler. To use the MatLab compiler, you need to load the MATLAB module to compile code and
load the mcr module to run the resulting MatLab executable.

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -o matlab_executable_name
</syntaxhighlight>

If you have addpath() commands in your code, you will need to wrap them in an "if ~deployed" block and tell the
compiler to include that path via the -I flag.

<syntaxhighlight lang="MATLAB">
% wrap addpath() calls like so:
if ~deployed
addpath('./another/folder/with/code/')
end
</syntaxhighlight>

NOTE: The license manager checks the mcc compiler out for a minimum of 30 minutes, so if another user compiles a code
you unfortunately may need to wait for up to 30 minutes to compile your own code.

Compiling with additional paths:

<syntaxhighlight lang="bash">
module load MATLAB
mcc -m matlab_main_code.m -I ./another/folder/with/code/ -o matlab_executable_name
</syntaxhighlight>

Any directories added with addpath() will need to be added to the list of compile options as -I arguments. You
can have multiple -I arguments in your compile command.

Here is an example job submission script. Modify time, memory, tasks-per-node, and job name as you see fit:

<syntaxhighlight lang="bash">
#!/bin/bash -l
#SBATCH --job-name=matlab
#SBATCH --output=matlab.o%j
#SBATCH --time=1:00:00
#SBATCH --mem=4G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

module reset
module load mcr

./matlab_executable_name
</syntaxhighlight>

For those who make use of mex files - compiled C and C++ code with matlab bindings - you will need to add these
files to the compiled archive via the -a flag. See the behavior of this flag in the [https://www.mathworks.com/help/compiler/mcc.html compiler documentation]. You can either target specific .mex files or entire directories.

Because codes often require adding several directories to the Matlab path as well as mex files from several locations,
we recommend writing a script to preserve and help document the steps to compile your Matlab code. Here is an
abbreviated example from a current user:

<syntaxhighlight lang="bash">
#!/bin/bash -l

module load MATLAB

cd matlabPyrTools/MEX/

# compile mex files
mex upConv.c convolve.c wrap.c edges.c
mex corrDn.c convolve.c wrap.c edges.c
mex histo.c
mex innerProd.c

cd ../..

mcc -m mongrel_creation.m \
-I ./matlabPyrTools/MEX/ \
-I ./matlabPyrTools/ \
-I ./FastICA/ \
-a ./matlabPyrTools/MEX/ \
-a ./texturesynth/ \
-o mongrel_creation_binary
</syntaxhighlight>

Again, we only have a single floating user license for MatLab so the model is to develop and debug your MatLab code
elsewhere or using Octave on Beocat, then you can compile the MatLab code into an executable and run it without
limits on Beocat.

For more info on the mcc compiler see: https://www.mathworks.com/help/compiler/mcc.html

=== COMSOL ===
Beocat has no license for COMSOL. If you want to use it, you must provide your own.

module spider COMSOL/
----------------------------------------------------------------------------
COMSOL: COMSOL/5.3
----------------------------------------------------------------------------
Description:
COMSOL Multiphysics software, an interactive environment for modeling
and simulating scientific and engineering problems

This module can be loaded directly: module load COMSOL/5.3

Help:

Description
===========
COMSOL Multiphysics software, an interactive environment for modeling and
simulating scientific and engineering problems
You must provide your own license.
export LM_LICENSE_FILE=/the/path/to/your/license/file
*OR*
export LM_LICENSE_FILE=$LICENSE_SERVER_PORT@$LICENSE_SERVER_HOSTNAME
e.g. export LM_LICENSE_FILE=1719@some.flexlm.server.ksu.edu

More information
================
- Homepage: https://www.comsol.com/
==== Graphical COMSOL ====
Running COMSOL in graphical mode on a cluster is generally a bad idea. If you choose to run it in graphical mode on a compute node, you will need to do something like the following:
<syntaxhighlight lang="bash">
# Connect to the cluster with X11 forwarding (ssh -Y or mobaxterm)
# load the comsol module on the headnode
module load COMSOL
# export your comsol license as mentioned above, and tell the scheduler to run the software
srun --nodes=1 --time=1:00:00 --mem=1G --pty --x11 comsol -3drend sw
</syntaxhighlight>

=== .NET Core ===
==== Load .NET ====
mozes@[eunomia] ~ $ module load dotNET-Core-SDK
==== create an application ====
Following instructions from [https://docs.microsoft.com/en-us/dotnet/core/tutorials/using-with-xplat-cli here], we'll create a simple 'Hello World' application
mozes@[eunomia] ~ $ mkdir Hello

mozes@[eunomia] ~ $ cd Hello

mozes@[eunomia] ~/Hello $ export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true

mozes@[eunomia] ~/Hello $ dotnet new console
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /homes/mozes/Hello/Hello.csproj...
Restoring packages for /homes/mozes/Hello/Hello.csproj...
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.props.
Generating MSBuild file /homes/mozes/Hello/obj/Hello.csproj.nuget.g.targets.
Restore completed in 358.43 ms for /homes/mozes/Hello/Hello.csproj.

Restore succeeded.

==== Edit your program ====
mozes@[eunomia] ~/Hello $ vi Program.cs
==== Run your .NET application ====
mozes@[eunomia] ~/Hello $ dotnet run
Hello World!
==== Build and run the built application ====
mozes@[eunomia] ~/Hello $ dotnet build
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

Restore completed in 106.12 ms for /homes/mozes/Hello/Hello.csproj.
Hello -> /homes/mozes/Hello/bin/Debug/netcoreapp2.1/Hello.dll

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:02.86

mozes@[eunomia] ~/Hello $ dotnet bin/Debug/netcoreapp2.1/Hello.dll
Hello World!

== Installing my own software ==
Installing and maintaining software for the many different users of Beocat would be very difficult, if not impossible. For this reason, we don't generally install user-run software on our cluster. Instead, we ask that you install it into your home directories.

In many cases, the software vendor or support site will incorrectly assume that you are installing the software system-wide or that you need 'sudo' access.

As a quick example of installing software in your home directory, we have a sample video on our [[Training Videos]] page. If you're still having problems or questions, please contact support as mentioned on our [[Main Page]].

== Loading multiple modules ==
modules, when loaded, will stay loaded for the duration of your session until they are unloaded.

; You can load multiple pieces of software with one module load command. : module load iompi iomkl

; You can unload all software : module reset

; If you see output from a module load command that looks like ''"The following have been reloaded with a version change"'' you likely have tried to load two pieces of software that have not been tested together. There may be serious issues with using either pieces of software while you're in this state. Libraries missing, applications non-functional. If you encounter issues, you will want to unload all software before switching modules. : 'module reset' and then 'module load'

== Containers ==
More and more science is being done within containers, these days. Sometimes referred to Docker or Kubernetes, containers allow you to package an entire software runtime platform and run that software on another computer or site with minimal fuss.

Unfortunately, Docker and Kubernetes are not particularly well suited to multi-user HPC environments, but that's not to say that you can't make use of these containers on Beocat.

=== Apptainer ===
[https://apptainer.org/docs/user/1.2/index.html Apptainer] is a container runtime that is designed for HPC environments. It can convert docker containers to its own format, and can be used within a job on Beocat. It is a very broad topic and we've made the decision to point you to the upstream documentation, as it is much more likely that they'll have up to date and functional instructions to help you utilize containers. If you need additional assistance, please don't hesitate to reach out to us.