From Beocat

MAINTENANCE -- MAJOR CHANGES STARTING DECEMBER 26th. See here

Revision as of 15:08, 9 February 2017 by Mozes (talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Running jobs on Beocat is normally a simple procedure. Either your application is using MPI and will automatically connect and use all the nodes it has access to, or it will only run within a single machine.

There is a very small subset of jobs that must handle inter-node communication themselves.

Below is a demo script that can startup applications on 3 nodes it has access to. We'll go through its functionality.

 1 #!/bin/bash
 2 # allocate 3 nodes
 3 #$ -pe mpi-16 48
 4 
 5 # run on the 16 core elves, so we get the whole things
 6 #$ -q *@@16c-elves
 7 # 4 GB of memory per core and an hour of runtime
 8 #$ -l mem=4G,h_rt=1:00:00
 9 
10 MASTER_HOST=$(cat ${PE_HOSTFILE} | sed -n '1p' | awk '{print $1}')
11 DEVICE_HOST=$(cat ${PE_HOSTFILE} | sed -n '2p' | awk '{print $1}')
12 CLIENT_HOST=$(cat ${PE_HOSTFILE} | sed -n '3p' | awk '{print $1}')
13 
14 /path/to/master/startup.sh &
15 
16 qrsh -nostdin -inherit ${DEVICE_HOST} "/path/to/device/startup.sh" &
17 qrsh -nostdin -inherit ${CLIENT_HOST} "/path/to/client/startup.sh" &
18 
19 while [[ $(pgrep -c startup.sh) -gt 0 ]]; do
20   sleep 60
21 done

Okay, we start with some fairly standard boiler plate. Tell the scheduler it is bash, and that we need 48 cores in 16-core chunks. This gives us 3 machines. Since we tell it to only run on the 16c-elves, we'll get whole machines for our job.

Next we determine which machines will be which and which machines we have access to. The PE_HOSTFILE variable holds the path to a file that automatically contains that information, as it is created by the scheduler. We just have to parse it out. An example PE_HOSTFILE looks like this:

dwarf31.beocat.ksu.edu 1 killable.q@dwarf31.beocat.ksu.edu UNDEFINED
dwarf21.beocat.ksu.edu 1 killable.q@dwarf21.beocat.ksu.edu UNDEFINED
dwarf29.beocat.ksu.edu 1 killable.q@dwarf29.beocat.ksu.edu UNDEFINED

So, to get our master host, we cat the hostfile, use sed to tell it to give us the first line, and then awk to grab the first column, as it is a hostname. We do the same for the device host and the client host.

From there, we run the master startup script, note the ampersand at the end. This tells the shell to run the command in the background.

Then we connect via qrsh to the device host, calling the devices startup script, then the same for the client host.

Finally we tell our script to loop, waiting for the process startup.sh to end. We'll check every 60 seconds to avoid a busy wait loop.