Submitting Condor Jobs to UNICORE

This technical note explains how to submit Condor jobs to a UNICORE VSite. It is similar in character to our Condor-GT4 description in that it elaborates Condor's official documentation (which happens to be almost non-existing for UNICORE). However, unlike Condor-GT4, we have not tested the Condor-UNICORE interface - we only got a basic test running - so we cannot recommend it or provide additional details.

Create a keystore

Unlike Globus Toolkit, UNICORE doesn't use proxy certificates for authentication. Instead, the client (= Condor) needs access to your Grid user certificate, which has to be stored in a format known as JKS keystore. The same keystore file has to also contain trusted CA certificates. If you only have the PEM-encoded user certificate and private key, like with Globus, creating the keystore is a two-step process

Step 1: Convert usercert.pem and userkey.pem to usercert.p12

openssl pkcs12 -export -in usercert.pem -inkey userkey.pem -out usercert.p12

Step 2: Create the keystore usercert.jks

Download UNICORE's GPE application client (GUI, requires Java).

When you first start the client, it will create an empty keystore file and ask you for a new password to protect it.

After that, open Settings/Keystore Editor. Select File/New, which will allow you to create a new keystore file in a location that you specify (e.g. usercert.jks). Then open Actions/Import Keystore... and point to the usercert.p12 created in the previous step.

Next, open Actions/Import Certificate... and in the following dialog enter the directory where your CA certificates are stored. Select all CA certificates to be imported (in D-Grid, the certificates are stored in cryptically named files with .0 extension, e.g. dd4b34ea.0).

Finally, use File/Save to create the keystore.

Step 3: Verify the usercert.jks contents

You can check the contents of usercert.jks using the program keytool from your Java installation. Here is what the output should look like:

jploski@srvgrid01:~/.globus> keytool -list -keystore usercert.jks
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

jan ploski's dfn-verein id, Oct 9, 2008, PrivateKeyEntry, 
Certificate fingerprint (MD5): BA:C5:99:AF:2F:3D:49:4B:DC:AF:48:5D:EA:17:36:D8
dfn-verein pca grid - g0, Oct 9, 2008, trustedCertEntry,
Certificate fingerprint (MD5): 41:39:4A:58:2E:F0:45:B2:29:28:F1:72:AB:F7:05:08

Create a job command file

Here is an example job command file for submitting a simple job to a UNICORE VSite:

executable               = /bin/bash
arguments                = hello.sh
transfer_input_files     = hello.sh,hello_in.txt
transfer_output_files    = hello_out.txt
transfer_executable      = false
whentotransferoutput     = ON_EXIT
notification             = NEVER
universe                 = grid
grid_resource            = unicore zam177.zam.kfa-juelich.de:4009 FZK-DGIREF-NJS
output                   = hello.out
error                    = hello.err
log                      = hello.log
keystore_file            = /home/jploski/.globus/usercert.jks
keystore_alias           = jan ploski's dfn-verein id
keystore_passphrase_file = /home/jploski/.globus/passphrase 
queue

The UNICORE-related parameters are:

  • keystore_file - path to your previously created usercert.jks
  • keystore_alias - alias of the PrivateKeyEntry in the keystore (see keytool output above)
  • keystore_passphrase_file - path to a text file which contains, as a single line, the plaintext password you chose for protecting the keystore
  • grid_resource - consists of the keyword "unicore", followed by the UNICORE gateway hostname and port (the port number must be provided), followed by a VSite name known to the gateway; the gateway is the host which Condor talks to in order to dispatch the job; the VSite is where the actual job is executed

For D-Grid, the gateway address(es) can be looked up here: http://www.fz-juelich.de/jsc/unicore/unicoreSites_D-Grid.xml

Note that as of the time of writing D-Grid installations contain UNICORE 5. We have not tested Condor job submission to UNICORE 6.

The hello.sh script used above contains:

#!/bin/bash

echo Hello, world
cat hello_in.txt
date
uname -a
cp hello_in.txt hello_out.txt

And the hello_in.txt file contains:

This the content of a text file transferred to the execution site (hello_in.txt).

Fix ugahp.jar in Condor

We had problems with the out-of-the-box UNICORE adapter shipped with Condor 7.0.1 in condor/lib/ugahp.jar. Specifically, the submitted job script was rejected by the PBS adapter at the VSite because of a missing email address. We fixed this problem by patching and recompiling the source code for ugahp.jar.

  • If interested, you can download the original ugahp source code used by Condor
  • However, instead of the original you should use the ugahp.jar file from our patched version. (The archive's content was slightly reorganized compared to the original, to make it directly importable as a Java project in Eclipse.)
  • Overwrite the condor/lib/ugahp.jar in your Condor installation with our ugahp.jar.

Submit your job

condor_submit hello.cmd

The detailed protocol of execution can be seen in GridmanagerLog.<your username>.

In our test, we got the following output files:

jploski@srvgrid01:~/condor-unicore> cat hello.err 
id: cannot find name for group ID 64003

This script was created and executed by Unicore
UNICORE - start of user output on stderr


UNICORE - end of user output on stderr
UNICORE EXIT STATUS 0 +

jploski@srvgrid01:~/condor-unicore> cat hello.out 
Hello, world
This the content of a text file transferred to the execution site (hello_in.txt).
Do Okt  9 12:53:58 CEST 2008
Linux wn10 2.6.9-55.0.12.ELsmp #1 SMP Thu Nov 1 09:20:43 CDT 2007 x86_64 x86_64 x86_64 GNU/Linux

jploski@srvgrid01:~/condor-unicore> cat hello_out.txt 
This the content of a text file transferred to the execution site (hello_in.txt).

You can also download the GridmanagerLog file created by running this test.