OLD DEPRECATED AdvancedSGE - Revision history

Mozes: Removed content as SGE is gone.

2018-09-20T04:11:52Z

Removed content as SGE is gone.

Mozes: Mozes moved page AdvancedSGE to OLD DEPRECATED AdvancedSGE: old and deprecated

2018-09-20T04:05:19Z

Mozes moved page AdvancedSGE to OLD DEPRECATED AdvancedSGE: old and deprecated

← Older revision	Revision as of 23:05, 19 September 2018
(No difference)

Mozes: /* File Access */

2017-12-04T05:54:21Z

File Access

← Older revision		Revision as of 00:54, 4 December 2017
Line 640:		Line 640:
	per subdirectory applies. There is no limit to the disk space you can use in your bulk directory,		per subdirectory applies. There is no limit to the disk space you can use in your bulk directory,
	but the files there will not be backed up. They are still redundantly stored so you don't need to		but the files there will not be backed up. They are still redundantly stored so you don't need to
	worry about losing data to hardware failures, just don't delete something by accident.		worry about losing data to hardware failures, just don't delete something by accident. Unused files are automatically removed after two years.
	If you need to back up large files in the bulk directory, talk to Dan Andresen (dan@ksu.edu) about		If you need to back up large files in the bulk directory, talk to Dan Andresen (dan@ksu.edu) about
	purchasing some hard disks for archival storage.		purchasing some hard disks for archival storage.

Daveturner: /* File Access */

2017-07-19T22:27:26Z

File Access

← Older revision		Revision as of 17:27, 19 July 2017
Line 680:		Line 680:

	<I>Copy the 'out' directory back to the current working directory after the run</I><BR>		<I>Copy the 'out' directory back to the current working directory after the run</I><BR>
	cp $TMPDIR/out .		cp -rp $TMPDIR/out .

	<B>RAM disk</B>		<B>RAM disk</B>

Daveturner: /* File Access */

2017-07-19T22:24:07Z

File Access

← Older revision		Revision as of 17:24, 19 July 2017
Line 618:		Line 618:
	== File Access ==		== File Access ==
	Beocat has a variety of options for storing and accessing your files.		Beocat has a variety of options for storing and accessing your files.
	Every user has a home directory for general use which is limited in size, has decent file access performance ~~in many cases~~,		Every user has a home directory for general use which is limited in size, has decent file access performance,
	and will soon be backed up. Larger files should be stored in the /bulk subdirectories which have the same decent performance		and will soon be backed up nightly. Larger files should be stored in the /bulk subdirectories which have the same decent performance
	but are not backed up. The /scratch file system will soon be implemented on a Lustre file system that will provide very fast		but are not backed up. The /scratch file system will soon be implemented on a Lustre file system that will provide very fast
	temporary file access. When fast IO is critical to the application performance, access to the local disk on each node or to a		temporary file access. When fast IO is critical to the application performance, access to the local disk on each node or to a
	~~ram~~ disk are options.		RAM disk are the best options.

	<B>Home directory</B>		<B>Home directory</B>
Line 628:		Line 628:
	Every user has a /homes/<I>username</I> directory that they drop into when they log into Beocat.		Every user has a /homes/<I>username</I> directory that they drop into when they log into Beocat.
	The home directory is for general use and provides decent performance for most file IO.		The home directory is for general use and provides decent performance for most file IO.
	~~Each user must keep their files below a~~ 1 TB ~~limit~~, so larger files should be kept ~~on their~~ /bulk		Disk space in each home directory is limited to 1 TB, so larger files should be kept in the /bulk
	directory, and there is a limit of 100,000 files in each subdirectory in your account.		directory, and there is a limit of 100,000 files in each subdirectory in your account.
	This file system is fully redundant, so 3 specific disks would need to fail before any data was lost.		This file system is fully redundant, so 3 specific hard disks would need to fail before any data was lost.
	All files will soon be backed up nightly to a separate file server in Nichols Hall, so if you do accidentally		All files will soon be backed up nightly to a separate file server in Nichols Hall, so if you do accidentally
	delete something it can be recovered.		delete something it can be recovered.
Line 637:		Line 637:

	Each user also has a /bulk/<I>username</I> directory where large files should be stored.		Each user also has a /bulk/<I>username</I> directory where large files should be stored.
	File access is the same speed as for the home ~~directory~~, and the same limit of 100,000 files		File access is the same speed as for the home directories, and the same limit of 100,000 files
	per ~~directory~~ applies. There is no limit to the ~~amount of files~~ you can ~~store~~ in your bulk directory,		per subdirectory applies. There is no limit to the disk space you can use in your bulk directory,
	but the files there will not be backed up. They are still redundantly stored so you don't need to		but the files there will not be backed up. They are still redundantly stored so you don't need to
	worry about losing data to hardware failures, just don't delete something by accident.		worry about losing data to hardware failures, just don't delete something by accident.
	If you need to back up large files in the bulk directory, talk to Dan Andresen (dan@ksu.edu) about		If you need to back up large files in the bulk directory, talk to Dan Andresen (dan@ksu.edu) about
	purchasing some disks for archival storage.		purchasing some hard disks for archival storage.

	<B>Scratch file system</B>		<B>Scratch file system</B>
Line 648:		Line 648:
	The /scratch file system will soon be using the Lustre software which is much faster than the		The /scratch file system will soon be using the Lustre software which is much faster than the
	speed of the file access on /homes or /bulk. In order to use scratch, you first need to make a		speed of the file access on /homes or /bulk. In order to use scratch, you first need to make a
	directory for yourself. ~~/scratch~~ offers greater speed, no limit to the size of files nor the number		directory for yourself. Scratch offers greater speed, no limit to the size of files nor the number
	of files in each ~~directory~~. It is meant as temporary space for prepositioning files and accessing them		of files in each subdirectory. It is meant as temporary space for prepositioning files and accessing them
	during runs. Once runs are completed, any files that need to be kept should be moved to your home		during runs. Once runs are completed, any files that need to be kept should be moved to your home
	or bulk ~~directory~~ since files on the scratch file system get purged after 30 days. Lustre is faster than		or bulk directories since files on the scratch file system get purged after 30 days. Lustre is faster than
	the home and bulk file systems in part because it does not redundantly store files by striping them		the home and bulk file systems in part because it does not redundantly store files by striping them
	across multiple disks, so if a hard disk fails data will be lost. When we get scratch set up to use Lustre		across multiple disks, so if a hard disk fails data will be lost. When we get scratch set up to use Lustre
Line 670:		Line 670:
	have around 600 GB of file space accessible on the local disk.		have around 600 GB of file space accessible on the local disk.

	cp <I>input_files</I> $TMPDIR		<I>Copy input files to the tmp directory if needed</I><BR>
			cp <I>input_files</I> $TMPDIR<BR>

			<I>Make an 'out' directory to pass to the app if needed</I><BR>
	mkdir $TMPDIR/out		mkdir $TMPDIR/out

			<I>Example of running an app and passing the tmp directory in/out</I><BR>
	app -input_directory $TMPDIR -output_directory $TMPDIR/out		app -input_directory $TMPDIR -output_directory $TMPDIR/out

			<I>Copy the 'out' directory back to the current working directory after the run</I><BR>
	cp $TMPDIR/out .		cp $TMPDIR/out .

Line 681:		Line 686:
	If you need ultrafast access to files, you can use a RAM disk which is a file system set up in the		If you need ultrafast access to files, you can use a RAM disk which is a file system set up in the
	memory of the compute node you are running on. The RAM disk is limited to a maximum of half		memory of the compute node you are running on. The RAM disk is limited to a maximum of half
	of the physical memory on ~~the~~ node, and you should account for this usage when you request		of the physical memory on that node, and you should account for this usage when you request
	memory for your job. You'll need to set up a directory on the RAM disk manually at the start of your		memory for your job. You'll need to set up a directory on the RAM disk manually at the start of your
	script, you may need to copy files over before your script runs the application, then you'll need to		script, you may need to copy files over before your script runs the application, then you'll need to
	copy data off the RAM disk ~~at the end of the queue~~ and you absolutely must remove all traces		copy data off the RAM disk and you absolutely must remove all traces
	of the RAM disk at the end of your job or it will remain in memory and affect future jobs. If your		of the RAM disk at the end of your job or it will remain in memory and affect future jobs. If your
	job crashes during the run, you'll need to email beocat@cs.ksu.edu an let us know which node		job crashes during the run, you'll need to email beocat@cs.ksu.edu to let us know which node
	needs the RAM disk cleaned up. Below is an example of how to use the RAM disk.		needs the RAM disk cleaned up. Below is an example of how to use the RAM disk.

Line 693:		Line 698:
	cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID		cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID

	<I> Run the application, ~~possible~~ giving it the path to the RAM disk to use for output files</I><BR>		<I> Run the application, possibly giving it the path to the RAM disk to use for output files</I><BR>
	app -output_directory /dev/shm/<I>username</I>-$JOB_ID		app -output_directory /dev/shm/<I>username</I>-$JOB_ID

	<I> Copy files from the RAM disk and clean it up</I><BR>		<I> Copy files from the RAM disk to the current working directory and clean it up</I><BR>
	cp /dev/shm/<I>username</I>-$JOB_ID/* .<BR>		cp /dev/shm/<I>username</I>-$JOB_ID/* .<BR>
	rm -fr /dev/shm/<I>username</I>-$JOB_ID		rm -fr /dev/shm/<I>username</I>-$JOB_ID

	~~Please let us know if you need any help setting up a RAM disk.~~

	<B>When you leave KSU</B>		<B>When you leave KSU</B>

	If you are done with your account and leaving KSU, please clean up your directory, move any files		If you are done with your account and leaving KSU, please clean up your directory, move any files
	to your supervisor that need to be kept after you leave, and notify us so that we can disable your		to your supervisor's account that need to be kept after you leave, and notify us so that we can disable your
	account. The easiest way to move your files to your supervisor's account is for them to set up		account. The easiest way to move your files to your supervisor's account is for them to set up
	a ~~directory~~ for you with the appropriate write permissions. The example below shows moving		a subdirectory for you with the appropriate write permissions. The example below shows moving
	just a user's data ~~directory~~ to their supervisor.		just a user's 'data' subdirectory to their supervisor. The 'nohup' command is used so that the move will
			continue even if the window you are doing the move from gets disconnected.

	<I> Supervisor:</I><BR>		<I> Supervisor:</I><BR>
Line 715:		Line 719:

	<I> Student:</I><BR>		<I> Student:</I><BR>
	nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/username</I>. &		nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/username</I> &

	~~The 'nohup' command is used so that the move will continue even if the window you are~~
	~~doing the move from gets disconnected.~~

	== Array Jobs ==		== Array Jobs ==

Daveturner: /* File Access */

2017-07-19T06:56:28Z

File Access

← Older revision		Revision as of 01:56, 19 July 2017
Line 690:		Line 690:

	<I> Make the RAM disk and copy input files over if necessary</I><BR>		<I> Make the RAM disk and copy input files over if necessary</I><BR>
	mkdir /dev/shm/<I>username</I>-$JOB_ID		mkdir /dev/shm/<I>username</I>-$JOB_ID<BR>
	cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID		cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID

Line 697:		Line 697:

	<I> Copy files from the RAM disk and clean it up</I><BR>		<I> Copy files from the RAM disk and clean it up</I><BR>
	cp /dev/shm/<I>username</I>-$JOB_ID/* .		cp /dev/shm/<I>username</I>-$JOB_ID/* .<BR>
	rm -fr /dev/shm/<I>username</I>-$JOB_ID		rm -fr /dev/shm/<I>username</I>-$JOB_ID

Line 711:		Line 711:

	<I> Supervisor:</I><BR>		<I> Supervisor:</I><BR>
	mkdir /bulk/<I>supervisorsname</I>/<I>username</I>		mkdir /bulk/<I>supervisorsname</I>/<I>username</I><BR>
	chmod ugo+w /bulk/<I>supervisorsname/username</I>		chmod ugo+w /bulk/<I>supervisorsname/username</I>

Daveturner: /* File Access */

2017-07-19T06:55:26Z

File Access

← Older revision		Revision as of 01:55, 19 July 2017
Line 711:		Line 711:

	<I> Supervisor:</I><BR>		<I> Supervisor:</I><BR>
	mkdir /bulk/<I>supervisorsname</I>/<I>~~studentsname~~</I>		mkdir /bulk/<I>supervisorsname</I>/<I>username</I>
	chmod ugo+w /bulk/<I>supervisorsname/~~studentsname~~</I>		chmod ugo+w /bulk/<I>supervisorsname/username</I>

	<I> Student:</I><BR>		<I> Student:</I><BR>
	nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/~~studentsname~~<I>/. &		nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/username</I>. &

	The 'nohup' command is used so that the move will continue even if the window you are		The 'nohup' command is used so that the move will continue even if the window you are

Daveturner: /* File Access */

2017-07-19T06:53:42Z

File Access

Daveturner: /* File Access */

2017-07-19T06:52:30Z

File Access

Daveturner: /* File Access */

2017-07-19T06:50:59Z

File Access

← Older revision		Revision as of 01:53, 19 July 2017
Line 689:		Line 689:
	needs the RAM disk cleaned up. Below is an example of how to use the RAM disk.		needs the RAM disk cleaned up. Below is an example of how to use the RAM disk.

	<I> Make the RAM disk and copy input files over if necessary</I>		<I> Make the RAM disk and copy input files over if necessary</I><BR>
	mkdir /dev/shm/<I>username</I>-$JOB_ID		mkdir /dev/shm/<I>username</I>-$JOB_ID
	cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID		cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID

	<I> Run the application, possible giving it the path to the RAM disk to use for output files</I>		<I> Run the application, possible giving it the path to the RAM disk to use for output files</I><BR>
	app -output_directory /dev/shm/<I>username</I>-$JOB_ID		app -output_directory /dev/shm/<I>username</I>-$JOB_ID

	<I> Copy files from the RAM disk and clean it up</I>		<I> Copy files from the RAM disk and clean it up</I><BR>
	cp /dev/shm/<I>username</I>-$JOB_ID/* .		cp /dev/shm/<I>username</I>-$JOB_ID/* .
	rm -fr /dev/shm/<I>username</I>-$JOB_ID		rm -fr /dev/shm/<I>username</I>-$JOB_ID
Line 710:		Line 710:
	just a user's data directory to their supervisor.		just a user's data directory to their supervisor.

	<I> Supervisor:</I>		<I> Supervisor:</I><BR>
	mkdir /bulk/<I>supervisorsname</I>/<I>studentsname</I>		mkdir /bulk/<I>supervisorsname</I>/<I>studentsname</I>
	chmod ugo+w /bulk/<I>supervisorsname/studentsname</I>		chmod ugo+w /bulk/<I>supervisorsname/studentsname</I>

	<I> Student:</I>		<I> Student:</I><BR>
	nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/studentsname<I>/. &		nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/studentsname<I>/. &

← Older revision		Revision as of 01:52, 19 July 2017
Line 689:		Line 689:
	needs the RAM disk cleaned up. Below is an example of how to use the RAM disk.		needs the RAM disk cleaned up. Below is an example of how to use the RAM disk.

	== Make the RAM disk and copy input files over if necessary		<I> Make the RAM disk and copy input files over if necessary</I>
	mkdir /dev/shm/<I>username</I>-$JOB_ID		mkdir /dev/shm/<I>username</I>-$JOB_ID
	cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID		cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID

	== Run the application, possible giving it the path to the RAM disk to use for output files		<I> Run the application, possible giving it the path to the RAM disk to use for output files</I>
	app -output_directory /dev/shm/<I>username</I>-$JOB_ID		app -output_directory /dev/shm/<I>username</I>-$JOB_ID

	== Copy files from the RAM disk and clean it up		<I> Copy files from the RAM disk and clean it up</I>
	cp /dev/shm/<I>username</I>-$JOB_ID/* .		cp /dev/shm/<I>username</I>-$JOB_ID/* .
	rm -fr /dev/shm/<I>username</I>-$JOB_ID		rm -fr /dev/shm/<I>username</I>-$JOB_ID
Line 710:		Line 710:
	just a user's data directory to their supervisor.		just a user's data directory to their supervisor.

	== Supervisor:		<I> Supervisor:</I>
	mkdir /bulk/<I>supervisorsname</I>/<I>studentsname</I>		mkdir /bulk/<I>supervisorsname</I>/<I>studentsname</I>
	chmod ugo+w /bulk/<I>supervisorsname/studentsname</I>		chmod ugo+w /bulk/<I>supervisorsname/studentsname</I>

	== Student		<I> Student:</I>
	nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/studentsname<I>/. &		nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/studentsname<I>/. &

← Older revision		Revision as of 01:50, 19 July 2017
Line 689:		Line 689:
	needs the RAM disk cleaned up. Below is an example of how to use the RAM disk.		needs the RAM disk cleaned up. Below is an example of how to use the RAM disk.

	## Make the RAM disk and copy input files over if necessary		== Make the RAM disk and copy input files over if necessary
	mkdir /dev/shm/<I>username</I>-$JOB_ID		mkdir /dev/shm/<I>username</I>-$JOB_ID
	cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID		cp <I>any_input_files</I> /dev/shm/<I>username</I>-$JOB_ID

	## Run the application, possible giving it the path to the RAM disk to use for output files		== Run the application, possible giving it the path to the RAM disk to use for output files
	app -output_directory /dev/shm/<I>username</I>-$JOB_ID		app -output_directory /dev/shm/<I>username</I>-$JOB_ID

	## Copy files from the RAM disk and clean it up		== Copy files from the RAM disk and clean it up
	cp /dev/shm/<I>username</I>-$JOB_ID/* .		cp /dev/shm/<I>username</I>-$JOB_ID/* .
	rm -fr /dev/shm/<I>username</I>-$JOB_ID		rm -fr /dev/shm/<I>username</I>-$JOB_ID
Line 710:		Line 710:
	just a user's data directory to their supervisor.		just a user's data directory to their supervisor.

	## Supervisor:		== Supervisor:
	mkdir /bulk/<I>supervisorsname</I>/<I>studentsname</I>		mkdir /bulk/<I>supervisorsname</I>/<I>studentsname</I>
	chmod ugo+w /bulk/<I>supervisorsname/studentsname</I>		chmod ugo+w /bulk/<I>supervisorsname/studentsname</I>

	## Student		== Student
	nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/studentsname<I>/. &		nohup mv /homes/<I>username</I>/data /bulk/<I>supervisorsname/studentsname<I>/. &