From Beocat
Jump to: navigation, search
No edit summary
 
(61 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== What is Beocat? ==
== What is Beocat? ==
Beocat is the [[wikipedia:High-performance_computing|HPC]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the [http://www.cis.ksu.edu/ Computing and Information Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.
Beocat is the [[wikipedia:High-performance_computing|High-Performance Computing (HPC)]] cluster at [http://www.ksu.edu Kansas State University]. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the [http://www.cs.ksu.edu/ Computer Science] department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.


Beocat is actually comprised of several different cluster computing systems
Beocat is actually comprised of several different cluster computing systems
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of Linux servers coordinated by the [https://arc.liv.ac.uk/trac/SGE SGE] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.cis.ksu.edu/ http://ganglia.beocat.cis.ksu.edu/].
* "Beocat", as used by most people is a [[wikipedia:Beowulf cluster|Beowulf cluster]] of CentOS Linux servers coordinated by the [https://slurm.schedmd.com/ Slurm] job submission and scheduling system. Our [[Compute Nodes]] (hardware) and [[installed software]] have separate pages on this wiki. The current status of this cluster can be monitored by visiting [http://ganglia.beocat.ksu.edu/ http://ganglia.beocat.ksu.edu/].
* A comparatively small [[Hadoop]] cluster
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* A small [[wikipedia:Openstack|Openstack]] cloud-computing infrastructure
* We provide a short description of Beocat for the uses of a proposal or teaching here: [[ProposalDescription|Beocat Info]]


== How Do I Use Beocat? ==
== How Do I Use Beocat? ==
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.
First, you need to get an account by visiting [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our [[LISTSERV]], where we announce any changes, maintenance periods, or other issues.


Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''beocat#cis-ksu-edu''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.
Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''). If you don't know what those are, please see our [[LinuxBasics]] page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.


As mentioned above, we use SGE for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SGEBasics]] page for an introduction on how to submit your first job. If you are already familiar with SGE, we also have an [[AdvancedSGE]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.
As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our [[SlurmBasics]] page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an [[AdvancedSlurm]] page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the [http://www.oscer.ou.edu/education.php Supercomputing in Plain English (SiPE)] series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.
 
==== Get an account at  [https://account.beocat.ksu.edu/ https://account.beocat.ksu.edu/] ====
==== Read about  [[Installed software]] and languages ====
==== Learn about Slurm at [[SlurmBasics]] and [[AdvancedSlurm]] ====
==== Run Interactive Jobs! [[OpenOnDemand]] ====
==== [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] ====
 
==== Big Data course on Beocat! [[BigDataOnBeocat]] ====
==== Interested in Web-Based computational biology research? Check out [[GalaxyDocs|Galaxy!]] ====
==== Looking to utilize the NRP (Nautilus cluster) namespace? Check out [[Nautilus|Nautilus on Beocat]] ====
 
== Transferring data to Beocat ==
Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:
* [[LinuxBasics]] - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
* [[Globus]] - Instructions on transferring files using [https://www.globus.org/ Globus Connect] using the endpoint ''Beocat filesystem (new)''.
* [[OpenOnDemand]] - We offer GUI based file management through OpenOnDemand
* [[Onedrive Data Transfer|Transfer Data to and from your OneDrive]] - We also offer the ability to transfer data to and from OneDrive
 
== Running Software on Beocat ==
Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running.  Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers.  The first link for OpenMPI also provides general information on loading modules and using <B>sbatch</B> and <B>scancel</B> to submit and cancel jobs.
* Running an [[Installed software#OpenMPI|MPI job]]
* Running an [[Installed software#R|R job]]
* Running a [[Installed software#Python|Python job]]
* Running a [[Installed software#MatLab compiler|Matlab job]]
* Running [[RSICC|RSICC codes]]


== Writing and Installing Software on Beocat ==
== Writing and Installing Software on Beocat ==
Line 23: Line 48:
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.
You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our [[FAQ]]. If you're just getting started our [[Training Videos]] might be useful to you.


If your answer isn't there, you can email us at [mailto:beocat@cis.ksu.edu beocat@cis.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.
If your answer isn't there, you can email us at [mailto:beocat@cs.ksu.edu beocat@cs.ksu.edu]. ''Please'' send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the <tt>hostname</tt> command.
 
We are also available on IRC on the [https://libera.chat/guides/connect Libera chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''nathanrwells''' in your message, and it should grab our attention. [https://web.libera.chat/#beocat Available from a web browser here.]
 
For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.


We are also available on IRC on the [http://freenode.net/using_the_network.shtml freenode chat servers] in the channel #beocat. This is useful ''especially'' if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention '''m0zes''' and/or '''kylehutson''' in your message, and it should grab our attention. Available from a web browser [[Special:WebChat|here.]]
<pre style="font-weight: bold;">
Again, when you email us at beocat@cs.ksu.edu please give us the job ID number, the path and script name for the job, and a full description of the problem. It may also be useful to include the output to 'module list'.
</pre>
 
== Twitter ==
We now have [https://twitter.com/KSUBeocat Twitter]. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.


== How do I get priority access ==
== How do I get priority access ==
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat.
We're glad you asked! Contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.
 
== External Computing Resources ==
 
We have access to supercomputing resources at other sites in the country through
the ACCESS program.
We have a large allocation of core-hours that can be used for testing and running
software, plus each user can apply for their own allocation if needed.
These resources can allow users to run jobs if they are not able to get enough
access on Beocat, but they are especially useful for when we don't have the needed
resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit
GPUs, or Matlab licenses.  Click [[ACCESS|here]] to see what resources
we have access to and to get access to some directions on how to use them.
Then contact [mailto:dan@ksu.edu Dr. Dan Andresen] to find out how to access our remote resources.
 
We also have free unlimited access to the Open Science Grid.
This is a high-throughput computing environment designed to efficiently
run lots of small jobs by spreading them across supercomputing systems in the
U.S. and Europe to use spare compute cycles donated to this project.  Beocat is
one of those systems that runs outside OSG jobs when our users are not fully
utilizing all our compute nodes.  For more information on how to get an OSG
account and take advantage of this resource, click [[OSG|here]].
For help in getting access to OSG, email [mailto:daveturner@ksu.edu Dr. Dave Turner].


== Policies ==
== Policies ==
Line 37: Line 93:


== Upcoming Events ==
== Upcoming Events ==
{{#widget:Google Calendar
{{#widget:Google Calendar  
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com
|id=hek6gpeu4bg40tdb2eqdrlfiuo@group.calendar.google.com  
|color=711616
|color=711616  
|view=AGENDA
|view=AGENDA  
}}
}}

Latest revision as of 12:29, 7 November 2024

What is Beocat?

Beocat is the High-Performance Computing (HPC) cluster at Kansas State University. It is run by the Institute for Computational Research in Engineering and Science, which is a function of the Computer Science department. Beocat is available to any educational researcher in the state of Kansas (and his or her collaborators) without cost. Priority access is given to those researchers who have contributed resources.

Beocat is actually comprised of several different cluster computing systems

How Do I Use Beocat?

First, you need to get an account by visiting https://account.beocat.ksu.edu/ and filling out the form. In most cases approval for the account will be granted in less than one business day, and sometimes much sooner. When your account has been approved, you will be added to our LISTSERV, where we announce any changes, maintenance periods, or other issues.

Once you have an account, you can access Beocat via SSH and can transfer files in or out via SCP or SFTP (or Globus Connect using the endpoint Beocat filesystem (new)). If you don't know what those are, please see our LinuxBasics page. If you are familiar with these, connect your client to headnode.beocat.ksu.edu and use your K-State eID credentials to login.

As mentioned above, we use Slurm for job submission and scheduling. If you've never worked with a batch-queueing system before, submitting a job is different than running on a standalone Linux machine. Please see our SlurmBasics page for an introduction on how to submit your first job. If you are already familiar with Slurm, we also have an AdvancedSlurm page where we can adjust the fine-tuning. If you're new to HPC, we highly recommend the Supercomputing in Plain English (SiPE) series by OU. In particular, the older course's streaming videos are an excellent resource, even if you do not complete the exercises.

Get an account at https://account.beocat.ksu.edu/

Read about Installed software and languages

Learn about Slurm at SlurmBasics and AdvancedSlurm

Run Interactive Jobs! OpenOnDemand

Transfer Data to and from your OneDrive

Big Data course on Beocat! BigDataOnBeocat

Interested in Web-Based computational biology research? Check out Galaxy!

Looking to utilize the NRP (Nautilus cluster) namespace? Check out Nautilus on Beocat

Transferring data to Beocat

Transferring data to Beocat can be done through a variety of ways, we offer documentation on a few of them:

  • LinuxBasics - Under the 'Transferring files (SCP or SFTP)' section, we have information regarding SCP and SFTP implementation.
  • Globus - Instructions on transferring files using Globus Connect using the endpoint Beocat filesystem (new).
  • OpenOnDemand - We offer GUI based file management through OpenOnDemand
  • Transfer Data to and from your OneDrive - We also offer the ability to transfer data to and from OneDrive

Running Software on Beocat

Running software on Beocat involves submitting a small job script to the scheduler which will use the information in that job script to allocate the resources your job needs then start the code running. Click on the links below to see examples of how to run applications written in some common languages used on high-performance computers. The first link for OpenMPI also provides general information on loading modules and using sbatch and scancel to submit and cancel jobs.

Writing and Installing Software on Beocat

  • If you are writing software for Beocat and it is in an installed scripting language like R, Perl, or Python, please look at our Installed software page to see what we have available and any usage guidelines we have posted there.
  • If you need to write compiled code such as Fortran, C, or C++, we offer both GNU and Intel compilers. See our FAQ for more details.
  • In either case, we suggest you head to our Tips and Tricks page for helpful hints.
  • If you wish to install software in your home directory, we have a video showing how to do this.

How do I get help?

You're in our support Wiki now, and that's a great place to start! We highly suggest that before you send us email, you visit our FAQ. If you're just getting started our Training Videos might be useful to you.

If your answer isn't there, you can email us at beocat@cs.ksu.edu. Please send all email to this address and not to any of our staff directly. This will ensure your support request gets entered into our tracker, and will get your questions answered as quickly as possible. Please keep the subject line as descriptive as possible and include any pertinent details to your problem (i.e. job ids, commands run, working directory, program versions,.. etc). If the problem is occurring on a headnode, please be sure to include the name of the headnode. This can be found by running the hostname command.

We are also available on IRC on the Libera chat servers in the channel #beocat. This is useful especially if you have a quick question, as you'd be surprised the times when at least one of us is around. If you do have a question be sure to mention nathanrwells in your message, and it should grab our attention. Available from a web browser here.

For interactive assistance, we offer a weekly open support session as mentioned in our calendar down below. Alternatively, we can often schedule a time to meet with you individually. You just need to send us an e-mail and provide us with the details we asked for above.

Again, when you email us at beocat@cs.ksu.edu please give us the job ID number, the path and script name for the job, and a full description of the problem.  It may also be useful to include the output to 'module list'.

Twitter

We now have Twitter. Follow us to find out the latest from Beocat, or tweet to us to find answers to quick questions. This won't replace the mailing list for major announcements, but will be used for more minor notices.

How do I get priority access

We're glad you asked! Contact Dr. Dan Andresen to find out how contributions to Beocat will prioritize your access to Beocat. In general, users contribute nodes to Beocat (aka the "Condo" model), to which their research group has priority access, in addition to elevated general priority for the rest of Beocat. If jobs from other researchers are occupying the node, Slurm will automatically halt and reschedule those jobs immediately to allow contributor access. Unused CPU time on the node is available for other Beocat users.

External Computing Resources

We have access to supercomputing resources at other sites in the country through the ACCESS program. We have a large allocation of core-hours that can be used for testing and running software, plus each user can apply for their own allocation if needed. These resources can allow users to run jobs if they are not able to get enough access on Beocat, but they are especially useful for when we don't have the needed resources on Beocat like access to 4 TB nodes on Bridges2, or more 64-bit GPUs, or Matlab licenses. Click here to see what resources we have access to and to get access to some directions on how to use them. Then contact Dr. Dan Andresen to find out how to access our remote resources.

We also have free unlimited access to the Open Science Grid. This is a high-throughput computing environment designed to efficiently run lots of small jobs by spreading them across supercomputing systems in the U.S. and Europe to use spare compute cycles donated to this project. Beocat is one of those systems that runs outside OSG jobs when our users are not fully utilizing all our compute nodes. For more information on how to get an OSG account and take advantage of this resource, click here. For help in getting access to OSG, email Dr. Dave Turner.

Policies

You can find our policies here

Credits and Accolades

See the published credits and other accolades received by Beocat here

Upcoming Events