SlurmBasics - Revision history

Nathanrwells: /* The Rocky/Slurm nodes */

2025-06-26T13:48:17Z

The Rocky/Slurm nodes

← Older revision		Revision as of 08:48, 26 June 2025
Line 21:		Line 21:
	icr-helios> <B>module load foss</B>		icr-helios> <B>module load foss</B>

	Modules provide an easy way to set up the compilers and libraries you may need to compile your code. Beyond that there are many different ways to compile codes so you'll just need to follow the directions. If you need help you can always ~~email us at <B>beocathelp@ksu.edu</B> or~~ contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket].		Modules provide an easy way to set up the compilers and libraries you may need to compile your code. Beyond that there are many different ways to compile codes so you'll just need to follow the directions. If you need help you can always contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket].

	=== Submitting jobs to Slurm ===		=== Submitting jobs to Slurm ===

Nathanrwells at 20:19, 23 June 2025

2025-06-23T20:19:13Z

← Older revision		Revision as of 15:19, 23 June 2025
Line 21:		Line 21:
	icr-helios> <B>module load foss</B>		icr-helios> <B>module load foss</B>

	Modules provide an easy way to set up the compilers and libraries you may need to compile your code. Beyond that there are many different ways to compile codes so you'll just need to follow the directions. If you need help you can always email us at <B>~~beocat~~@~~cs.~~ksu.edu</B>.		Modules provide an easy way to set up the compilers and libraries you may need to compile your code. Beyond that there are many different ways to compile codes so you'll just need to follow the directions. If you need help you can always email us at <B>beocathelp@ksu.edu</B> or contact Beocat staff through a [https://support.ksu.edu/TDClient/30/Portal/Requests/ServiceDet?ID=44 TDX Ticket].

	=== Submitting jobs to Slurm ===		=== Submitting jobs to Slurm ===

Daveturner: /* Producing memory and CPU utilization tables and graphs */

2024-07-30T05:54:00Z

Producing memory and CPU utilization tables and graphs

← Older revision		Revision as of 00:54, 30 July 2024
Line 277:		Line 277:
	If you want to read more, continue on to our [[AdvancedSlurm]] page.		If you want to read more, continue on to our [[AdvancedSlurm]] page.

	=== kstat is now available to download and install on other clusters.		=== kstat is now available to download and install on other clusters ===

	https://gitlab.beocat.ksu.edu/~~AdminPublic~~/kstat~~.git~~		https://gitlab.beocat.ksu.edu/Admin-Public/kstat

	This software has been installed and used on several clusters for many years.		This software has been installed and used on several clusters for many years.

Daveturner: /* Producing memory and CPU utilization tables and graphs */

2024-07-30T05:52:42Z

Producing memory and CPU utilization tables and graphs

← Older revision		Revision as of 00:52, 30 July 2024
Line 276:		Line 276:

	If you want to read more, continue on to our [[AdvancedSlurm]] page.		If you want to read more, continue on to our [[AdvancedSlurm]] page.

			=== kstat is now available to download and install on other clusters.

			https://gitlab.beocat.ksu.edu/AdminPublic/kstat.git

			This software has been installed and used on several clusters for many years.
			It should be considered Beta software and may take some slight modifications
			to install on some clusters. Please contact the author if you want to give
			it a try (daveturner@ksu.edu).

Daveturner: /* Using Modules */

2024-05-23T18:23:21Z

Using Modules

← Older revision		Revision as of 13:23, 23 May 2024
Line 8:		Line 8:

	eos> <B>module avail</B>		eos> <B>module avail</B>
	eos> <B>module load ~~VASP~~</B>		eos> <B>module load GROMACS</B>
	eos> <B>module list</B>		eos> <B>module list</B>

	When a module gets loaded, all the necessary libraries are also loaded and the paths to the libraries and executables are automatically set up. Loading ~~Vasp~~ for example also loads the OpenMPI library needed to run it and adds the path to the MPI commands and ~~Vasp~~ executables. To see how the path is set up, try executing <B><I>which ~~vasp_std~~</I></B>. The module system allows you to easily switch between different version of applications, libraries, or languages as well.		When a module gets loaded, all the necessary libraries are also loaded and the paths to the libraries and executables are automatically set up. Loading GROMACS for example also loads the OpenMPI library needed to run it and adds the path to the MPI commands and Grimaces executables. To see how the path is set up, try executing <B><I>which gmx</I></B>. The module system allows you to easily switch between different version of applications, libraries, or languages as well.

	If you are using a custom code or one that is not installed in a module, you'll need to recompile it yourself. This process is easier under CentOS as some of the work just involves loading the necessary set of modules. The first step is to decide whether to use the Intel compiler toolchain or the GNU toolchain, each of which includes the compilers and other math libraries. The module commands for each are below, and you can load these automatically when you log in by adding one of these module load statements to your .bashrc file. See <B>/homes/daveturner/.bashrc</B> as an example, where I put the module load statements .		If you are using a custom code or one that is not installed in a module, you'll need to recompile it yourself. This process is easier under CentOS as some of the work just involves loading the necessary set of modules. The first step is to decide whether to use the Intel compiler toolchain or the GNU toolchain, each of which includes the compilers and other math libraries. The module commands for each are below, and you can load these automatically when you log in by adding one of these module load statements to your .bashrc file. See <B>/homes/daveturner/.bashrc</B> as an example, where I put the module load statements .

Daveturner: /* Submitting your first job */

2024-05-23T18:14:29Z

Submitting your first job

← Older revision		Revision as of 13:14, 23 May 2024
Line 53:		Line 53:
	* <code>--mem-per-cpu=</code> tells how much memory I need. In my example, I'm using our system minimum of 512 MB, which is more than enough. Note that your memory request is '''per core''', which doesn't make much difference for this example, but will as you submit more complex jobs.		* <code>--mem-per-cpu=</code> tells how much memory I need. In my example, I'm using our system minimum of 512 MB, which is more than enough. Note that your memory request is '''per core''', which doesn't make much difference for this example, but will as you submit more complex jobs.
	* <code>--time=</code> tells how much runtime I need. This can be in the form of "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". This is a very short job, so 1 minute should be plenty. This can't be changed after the job is started please make sure you have requested a sufficient amount of time.		* <code>--time=</code> tells how much runtime I need. This can be in the form of "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". This is a very short job, so 1 minute should be plenty. This can't be changed after the job is started please make sure you have requested a sufficient amount of time.
	* <code>--cpus-per-task=1</code> tells Slurm that I need only a single core per task. The [[AdvancedSlurm]] page has much more on the "cpus-per-task" switch.
	* <code>--ntasks=1</code> tells Slurm that I only need to run 1 task. The [[AdvancedSlurm]] page has much more on the "ntasks" switch.
	* <code>--nodes=1</code> tells Slurm that this must be run on one machine. The [[AdvancedSlurm]] page has much more on the "nodes" switch.		* <code>--nodes=1</code> tells Slurm that this must be run on one machine. The [[AdvancedSlurm]] page has much more on the "nodes" switch.
	* <code>~~--nodes=4~~ --ntasks-per-node=16 --constraint=~~elves~~</code> ~~requests 4 nodes with 16 cores on each and~~ to only ~~use~~ the ~~Elves~~.		* <code>--ntasks-per-node=16 </code> Request 16 cores on each node.
			* <code>--constraint=moles</code> Request to only run on the Mole class of compute nodes.

	% '''ls'''		% '''ls'''

Daveturner: /* The Rocky/Slurm nodes */

2024-05-23T18:06:44Z

The Rocky/Slurm nodes

← Older revision		Revision as of 13:06, 23 May 2024
Line 1:		Line 1:
	== The ~~CentOS~~/Slurm nodes ==		== The Rocky/Slurm nodes ==

	We have converted Beocat from ~~Gentoo~~ Linux to ~~CentOS~~ Linux on ~~December 26th~~ of ~~2017~~. Any applications or libraries from the old system must be recompiled. We also converted Beocat to use the Slurm scheduler instead of SGE. You will therefore also need to convert all your old qsub scripts over to sbatch scripts. We have developed tools to make this process as easy as possible. If you're needing to convert old SGE scripts to SLURM, please checkout [[kstat.convert]]		We have converted Beocat from CentOS Linux to Rocky Linux on April 1st of 2024. Any applications or libraries from the old system must be recompiled.

	=== Using Modules ===		=== Using Modules ===
Line 16:		Line 16:

	To load the Intel compiler tool chain including the Intel Math Kernel Library (and OpenMPI):		To load the Intel compiler tool chain including the Intel Math Kernel Library (and OpenMPI):
	~~eos~~> <B>module load iomkl</B>		icr-helios> <B>module load iomkl</B>

	To load the GNU compiler tool chain including OpenMPI, OpenBLAS, FFTW, and ScalaPack load foss (free open source software):		To load the GNU compiler tool chain including OpenMPI, OpenBLAS, FFTW, and ScalaPack load foss (free open source software):
	~~eos~~> <B>module load foss</B>		icr-helios> <B>module load foss</B>

	Modules provide an easy way to set up the compilers and libraries you may need to compile your code. Beyond that there are many different ways to compile codes so you'll just need to follow the directions. If you need help you can always email us at <B>beocat@cs.ksu.edu</B>.		Modules provide an easy way to set up the compilers and libraries you may need to compile your code. Beyond that there are many different ways to compile codes so you'll just need to follow the directions. If you need help you can always email us at <B>beocat@cs.ksu.edu</B>.
Line 25:		Line 25:
	=== Submitting jobs to Slurm ===		=== Submitting jobs to Slurm ===

	~~Once your qsub script has been converted to an sbatch script and you have an application compiled for CentOS, you~~ can submit ~~the~~ job using the <B>sbatch</B> command.		You can submit your job script using the <B>sbatch</B> command.

	~~eos~~> <B>sbatch sbatch_script.sh</B>		icr-helios> <B>sbatch sbatch_script.sh</B>
	~~eos~~> <B>kstat --me</B>		icr-helios> <B>kstat --me</B>

	This will submit the script and show you a list of your jobs that are running and the jobs you have in the queue. By default the output for each job will go into a <B>slurm-###.out</B> file where ### is the job ID number. If you need to kill a job, you can use the <B>scancel</B> command with the job ID number.		This will submit the script and show you a list of your jobs that are running and the jobs you have in the queue. By default the output for each job will go into a <B>slurm-###.out</B> file where ### is the job ID number. If you need to kill a job, you can use the <B>scancel</B> command with the job ID number.

Daveturner: /* Monitoring Your Job */

2024-02-29T05:51:19Z

Monitoring Your Job

← Older revision		Revision as of 00:51, 29 February 2024
Line 76:		Line 76:

	Eos> kstat --help		Eos> kstat --help

	USAGE: kstat [-q] [-c] [-g] [-l] [-u user] [-p NaMD] [-j 1234567] [--part partition]		USAGE: kstat [-q] [-c] [-g] [-l] [-u user] [-p NaMD] [-j 1234567] [--part partition]
	kstat alone dumps all info except for the core summaries		kstat alone dumps all info except for the core summaries
	choose -q -c for only specific info on queued or core summaries.		choose -q -c for only specific info on queued or core summaries.
	then specify any searchables for the user, program name, or job id		then specify any searchables for the user, program name, or job id

	kstat info on running and queued jobs		kstat info on running and queued jobs
	kstat -h list host info only, no jobs		kstat -h list host info only, no jobs
Line 269:		Line 269:
	sandung 72 cores % 1.2 used 56 cores queued		sandung 72 cores % 1.2 used 56 cores queued
	zhiguang 80 cores % 1.3 used 688 cores queued		zhiguang 80 cores % 1.3 used 688 cores queued

			=== Producing memory and CPU utilization tables and graphs ===

	kstat can now produce tables or graphs for the memory or CPU utilization		kstat can now produce tables or graphs for the memory or CPU utilization

Daveturner at 05:49, 29 February 2024

2024-02-29T05:49:12Z

← Older revision		Revision as of 00:49, 29 February 2024
Line 74:		Line 74:

	The <B>kstat</B> perl script has been developed at K-State to provide you with all the available information about your jobs on Beocat. <B>kstat --help</B> will give you a full description of how to use it.		The <B>kstat</B> perl script has been developed at K-State to provide you with all the available information about your jobs on Beocat. <B>kstat --help</B> will give you a full description of how to use it.
	~~The Slurm version of kstat is very similar to the SGE version, with the exception that the actual memory usage of each job is not always available so the~~
	~~memory requested is reported, and the memory usage on each node is not always accurate since Slurm includes disk cache. We are continuing to look~~
	~~for better ways to get the memory usage for each job, but at the moment you may need to use [http://ganglia.beocat.ksu.edu/ Ganglia] and look at the~~
	~~memory graph for the node you are running on to get an accurate idea of the memory being used by your application.~~

	Eos> kstat --help		Eos> kstat --help

	USAGE: kstat [-q] [-c] [-g] [-l] [-u user] [-p NaMD] [-j 1234567] [--part partition]		USAGE: kstat [-q] [-c] [-g] [-l] [-u user] [-p NaMD] [-j 1234567] [--part partition]
	kstat alone dumps all info except for the core summaries		kstat alone dumps all info except for the core summaries
	choose -q -c for only specific info on queued or core summaries.		choose -q -c for only specific info on queued or core summaries.
	then specify any searchables for the user, program name, or job id		then specify any searchables for the user, program name, or job id

	kstat info on running and queued jobs		kstat info on running and queued jobs
	kstat -q info on the queued jobs only		kstat -h list host info only, no jobs
	kstat -c core usage for each user		kstat -q info on the queued jobs only
	kstat -g ~~gpu~~ nodes ~~only~~		kstat -c core usage for each user
	kstat -l -h long list - ~~prints full~~ node ~~list~~		kstat -d # show jobs run in the last # days
	kstat -u daveturner job info for one user only		Memory per node - used/allocated/requested
	kstat --me job info for my jobs only		Red is close to or over requested amount
	kstat -j 1234567 info on a given job id		Yellow is under utilized for large jobs
	kstat --nocolor do not use any color		kstat -g Only show GPU nodes
			kstat -o Turner Only show info for a given owner
	--------------------------------------------------------------------------		kstat -o CS_HPC Same but sub _ for spaces
	Multi-node jobs are highlighted in Magenta		kstat -l long list - node features and performance
	~~The switch~~ and ~~nodes/~~switch ~~are on the right~~		Node hardware and node CPU usage
	highlighted in Yellow when nodes are spread across multiple switches		job nodelist and switchlist
	~~Shared jobs are highlighted in Cyan~~		job current and max memory
	~~Memory requested is reported along with the total used when available~~		job CPU utilizations
	~~Total RSS / Total VMSize / Total requested~~		kstat -u daveturner job info for one user only
	~~Runtime~~ is colorized ~~with~~ yellow then red for jobs nearing their time limit		kstat --me job info for my jobs only
	~~Time in the queue~~ is colorized yellow then red for jobs waiting ~~long~~ times		kstat -j 1234567 info on a given job id
	--------------------------------------------------------------------------		kstat --osg show OSG background jobs also
			kstat --nocolor do not use any color
			kstat --name display full names instead of eIDs

			---------------- Graphs and Tables ---------------------------------------
			Specify graph/table, CPU or GPU or host, usage or memory, and optional time
			kstat --graph-cpu-memory # gnuplot CPU memory for job #
			kstat --table-gpu-usage-5min # GPU usage table every 5 min for job #
			kstat --table-cpu-60min # CPU usage, memory, swap table every 60 min for job #
			kstat --table-node [nodename] cores, load, CPU usage, memory table for a node

			--------------------------------------------------------------------------
			Multi-node jobs are highlighted in Magenta
			kstat -l also provides a node list and switch list
			highlighted in Yellow when nodes are spread across multiple switches
			Run time is colorized yellow then red for jobs nearing their time limit
			Queue time is colorized yellow then red for jobs waiting longer times
			--------------------------------------------------------------------------

	kstat can be used to give you a summary of your jobs that are running and in the queue:		kstat can be used to give you a summary of your jobs that are running and in the queue:
Line 157:		Line 170:
	=== Detailed information about a single job ===		=== Detailed information about a single job ===

	kstat can provide ~~a get~~ a great deal of information on a particular job including a very rough estimate of when it will run. This time is a worst case scenario as this will		kstat can provide a great deal of information on a particular job including a very rough estimate of when it will run. This time is a worst case scenario as this will
	be adapted as other jobs finish early. This is a good way to check for job submission problems before contacting us. kstat colorizes the more important		be adapted as other jobs finish early. This is a good way to check for job submission problems before contacting us. kstat colorizes the more important
	information to make it easier to identify.		information to make it easier to identify.
Line 257:		Line 270:
	zhiguang 80 cores % 1.3 used 688 cores queued		zhiguang 80 cores % 1.3 used 688 cores queued

			kstat can now produce tables or graphs for the memory or CPU utilization
			for a job. In order to view graphs you must set up X11 forwarding on your
			ssh connection by using the -X parameter.

	If you want to read more, continue on to our [[AdvancedSlurm]] page.		If you want to read more, continue on to our [[AdvancedSlurm]] page.

Mozes: /* The CentOS/Slurm nodes */

2020-08-31T18:41:09Z

The CentOS/Slurm nodes

← Older revision		Revision as of 13:41, 31 August 2020
Line 1:		Line 1:
	== The CentOS/Slurm nodes ==		== The CentOS/Slurm nodes ==

	We have converted Beocat from Gentoo Linux to CentOS Linux on December 26th of 2017. Any applications or libraries from the old system must be recompiled. We also converted Beocat to use the Slurm scheduler instead of SGE. You will therefore also need to convert all your old qsub scripts over to sbatch scripts. We have developed tools to make this process as easy as possible.		We have converted Beocat from Gentoo Linux to CentOS Linux on December 26th of 2017. Any applications or libraries from the old system must be recompiled. We also converted Beocat to use the Slurm scheduler instead of SGE. You will therefore also need to convert all your old qsub scripts over to sbatch scripts. We have developed tools to make this process as easy as possible. If you're needing to convert old SGE scripts to SLURM, please checkout [[kstat.convert]]

	=== Using Modules ===		=== Using Modules ===