From Beocat

MAINTENANCE -- MAJOR CHANGES STARTING DECEMBER 26th. See here

Jump to: navigation, search
(Manual multinode job setup)
 
(Efficiency - because I felt like it)
Line 24: Line 24:
 
qrsh -nostdin -inherit ${CLIENT_HOST} "/path/to/client/startup.sh" &
 
qrsh -nostdin -inherit ${CLIENT_HOST} "/path/to/client/startup.sh" &
  
while [[ $(pgrep startup.sh | wc -l) -gt 0 ]]; do
+
while [[ $(pgrep -c startup.sh) -gt 0 ]]; do
 
   sleep 60
 
   sleep 60
 
done
 
done

Revision as of 14:42, 9 February 2017

Running jobs on Beocat is normally a simple procedure. Either your application is using MPI and will automatically connect and use all the nodes it has access to, or it will only run within a single machine.

There is a very small subset of jobs that must handle inter-node communication themselves.

Below is a demo script that can startup applications on 3 nodes it has access to. We'll go through its functionality.

 1 #!/bin/bash
 2 # allocate 3 nodes
 3 #$ -pe mpi-16 48
 4 
 5 # run on the 16 core elves, so we get the whole things
 6 #$ -q *@@16c-elves
 7 # 4 GB of memory per core and an hour of runtime
 8 #$ -l mem=4G,h_rt=1:00:00
 9 
10 MASTER_HOST=$(cat ${PE_HOSTFILE} | sed -n '1p' | awk '{print $1}')
11 DEVICE_HOST=$(cat ${PE_HOSTFILE} | sed -n '2p' | awk '{print $1}')
12 CLIENT_HOST=$(cat ${PE_HOSTFILE} | sed -n '3p' | awk '{print $1}')
13 
14 /path/to/master/startup.sh &
15 
16 qrsh -nostdin -inherit ${DEVICE_HOST} "/path/to/device/startup.sh" &
17 qrsh -nostdin -inherit ${CLIENT_HOST} "/path/to/client/startup.sh" &
18 
19 while [[ $(pgrep -c startup.sh) -gt 0 ]]; do
20   sleep 60
21 done

Okay, we start with some fairly standard boiler plate. Tell the scheduler it is bash, and that we need 48 cores in 16-core chunks. This gives us 3 machines. Since we tell it to only run on the 16c-elves, we'll get whole machines for our job.

Next we determine which machines will be which and which machines we have access to. The PE_HOSTFILE variable holds the path to a file that contains that information. We just have to parse it out. An example PE_HOSTFILE looks like this:

dwarf31.beocat.ksu.edu 1 killable.q@dwarf31.beocat.ksu.edu UNDEFINED
dwarf21.beocat.ksu.edu 1 killable.q@dwarf21.beocat.ksu.edu UNDEFINED
dwarf29.beocat.ksu.edu 1 killable.q@dwarf29.beocat.ksu.edu UNDEFINED

So, to get our master host, we cat the hostfile, use sed to tell it to give us the first line, and then awk to grab the first column, as it is a hostname. We do the same for the device host and the client host.

From there, we run the master startup script, note the ampersand at the end. This tells the shell to run the command in the background.

Then we connect via qrsh to the device host, calling the devices startup script, then the same for the client host.

Finally we tell our script to loop, waiting for the process startup.sh to end. We'll check every 60 seconds to avoid a busy wait loop.