Tips and Tricks

Beocat has a number of tools to make your work easier, some which you may not know about. This is a simple list of these programs and some basic usage scenarios.

Submitting your job to run the fastest

Size your jobs to use the fastest nodes

Specify the proper number of cores

Beocat (nor any other computer or cluster) can make your job run on more than one core at a time if your program isn't designed to take advantage of this. Many people think "I can run this on 40 cores and it will run 40 times faster". This isn't true.

While we have many programs that are designed to take advantage of multiple cores, do not assume this is the case

Optimize your jobs for speed, not for number of cores

It seems that many people pick an arbitrary large number of cores for their jobs. 20 seems to be a common one. However, some of our fastest nodes have 16 cores. It's quite likely if your job will fit on an Elf (16 cores, 8 GB/RAM/core (64 GB RAM total)), it will run faster with 16 cores than by specifying more cores and having it run on slower nodes.

Don't request resources you don't need

The most common culprit here is people specifying they need infiniband when the job is run on a single node. This limits the scheduling such that a perfectly good node for your job may be idle while your job is still waiting.

Programs that make using Beocat easier

nmon

The name is short for "Nigel's Monitor", it's a program written by Nigel Griffiths from IBM.

nmon analyser

A tool for producing graphs and spreadsheets from output generated by nmon.

htop

A prettier, easier to use top. Shows CPU and memory usage in an easy-to-digest format.

screen

A sort of terminal multiplexer, allows you to run many terminal programs at once without mixing them up. Also allows you to disconnect and reconnect sessions. There is a good explanation of how to use screen at http://www.mattcutts.com/blog/a-quick-tutorial-on-screen/.

Ganglia

The web-based load monitoring tool for the cluster. http://ganglia.beocat.ksu.edu . From there, you can see how busy Beocat is.

dstat

A very detailed performance analyzer.

Increasing file write performance

Credit for this goes to http://moo.nac.uci.edu/~hjm/bduc/BDUC_USER_HOWTO.html#writeperfongl

Use gzip

If you have written your own code or are using an app that writes zillions of tiny chunks of data to STDOUT, and you are storing the results on Beocat, you should consider passing the output thru gzip to consolidate the writes into a continuous stream. If you don’t do this, each write will be considered a separate IO event and the write performance will suffer.

If, however, the STDOUT is passed thru gzip, the wallclock runtime decreases even below the usual runtime and you end up with an output file that it already compressed to about 1/5 the usual size.

The here’s how to do it:

someapp --opt1 --opt2 --input=/path/to/input_file | gzip > /path/to/output_file

Use named pipes

Named pipes are special files that don't actually write to the filesystem, and can be used to communicate between processess. Since these pipes are in memory rather than directly to disk, they can be used to buffer writes:

# Create the named pipe
mkfifo /path/to/MyNamedPipe

# Write some data to it
MyProgram --infile=/path/to/InputData1 --outfile=/path/to/MyNamedPipe &
MyOtherProgram < /path/to/InputData2 > /path/to/MyNamedPipe

# Extract the output
cat < /path/to/MyNamedPipe > $HOME/MyOutput
## OR, we could compress the output
gzip < /path/to/MyNamedPipe > $HOME/MyOutput.gz

# Delete the named pipe like you would a file
rm /path/to/MyNamedPipe

One cautionary word. Unlike normal files, named pipes cannot be used between machines, but can be used among processes running on the same machine. So, if you're running an MPI job that will be running completely on one node, you could setup a named pipe and do all your writes to that pipe, and then flush it at the end, but if you're running a multi-node MPI job and your named pipe is on a shared filesystem (like $HOME), each process will need to flush its named pipe to a regular file before the job quits.

Use one big file instead of many small ones

This may seem to be a non-issue, but it's a performance problem we've seen on Beocat many times. I love the term coined by UCI at the link above. They call making many small files "Zillions Of Tiny files (ZOTfiles)". Using files like this is an inefficient use of our shared resources. A tiny file by itself is no more inefficient than a huge one. If you have only 100bytes to store, store it in single file. However, the problems start compounding when there are many of them. Because of the way data is stored on disk, 10 MB stored in ZOTfiles of 100bytes each can easily take up NOT 10MB, but more than 400MB - 40 times more space. Worse, data stored in this manner makes many operations very slow - instead of looking up 1 directory entry, the OS has to look up 100,000. This means 100,000 times more disk head movement, with a concommittent decrease in performance and disk lifetime. We have had Beocat users with several million files of less than 1kB each. Just creating a directory listing with ls would take nearly 1/2 hour. Not only is that inefficient for you, but it also degrades the performance of everybody using that filesystem and degrades our backups as well.

Please use large files instead of ZOTfiles any chance you can!

As a defense against too much abuse of tiny files, there is a limit of 100,000 entries in any directory in our shared filesystem space.