Subsections


GROWL SCRIPTS

References [#!globus!#,#!globus2!#,#!GROWL!#,#!AHM2007:GrowlScripts!#,#!AHM2005:Growl!#,#!NCeSS2005!#].

This chapter provides an introduction and documentation for the GROWL Scripts [#!AHM2007:GrowlScripts!#]. This suite of scripts was developed during and following the JISC-funded GROWL VRE-1 project [#!GROWL!#] and has been found useful for installing Grid middleware and using remote resources. The GROWL Scripts continue to be maintained and developed by John Kewley at Daresbury.

Some basic documentation and examples are available on-line atx http://www.growl.org.uk/Scripts/growl-scripts.shtml. This chapter is based on this information. Some tutorial examples are provided here http://tardis.dl.ac.uk/Condor/sandbox/growl-scripts.shtml.

Introduction

GROWL Scripts are a set of wrapper scripts designed to facilitate takeup of Grid computing by computational scientists who are already reasonably familiar with parallel and distributed computing on clusters. They comprise wrappers to the standard globus and openssl command-line interfaces and attempt to address the three main barriers faced by newcomers to the Grid: (i) setting up client-side middleware; (ii) handling certificats; and (iii) submitting jobs to the Grid negociating firewalls.

Attributes

Version: 1.6.0
Public calls:
Public modules: Growl/scripts
Other software required: Globus or VDT,GSI-SSH and OpenSSL. These can downloaded from the Internet and be installed from a makefile provided.
Remote services used: MyProxy server, Globus Grid resources
Date: 2006-9
Origin: J. Kewley, CCLRC Daresbury Laboratory
Language: Unix shell
Conditions on external use: Standard, see separate chapter

How to use this Package

The GROWL Scripts can be downloaded from the GROWL Web site at http://www.grids.ac.uk/GROWL/GrowlScripts.tar.gz. Un-tarring this file will create a directory called Growl with sub-directories NOTES, examples, packages, scripts.

Client-side Globus middleware can now be built as follows. This uses wget to pull the latest version from the Internet. If you use a Web cache, you should amend lines in Growl/Makefile.conf to the approriate adddress and port. For instance, for those at Daresbury Laboratory:

[frame=single]
WEB_CACHE = wwwcache.dl.ac.uk
WEB_CACHE_PORT = 8080

You should then be able to build GROWL Scripts by running:

[frame=single]
$ cd Growl
$ make VDT

For this to work cleanly, your operating system must match one of the supported ones for the release of VDT that GROWL Scripts is trying to build (see VDT_VERS in Makefile.conf). Check the VDT Web site for a list of supported platforms for that version. The most common build problems are listed in the README.txt contained in the Growl directory. If there is no suitable VDT version do make gt instead which will use the Globus source distribution (not that this is much slower as it requires extensive compilation). The Globus or VDT distribution will be build in the Growl/packages directory and Grid CA certificates will be installed in Growl/packages/globus/share/certificaes.

Before using the GROWL Scripts, it is necessary to initialise the environment and creat a proxy certificate. This can be done as follows by typing source /Growl/setup.sh.

To procede, you will first need to obtain a personal Grid certificate from the UK e-Science Certification Authority http://www.grid-support.ac.uk/content/view/221/171/. Following their procedure will result in a certificate in your browser. The following instructions are for Firefox on Linux, but using Internet Explorer and Windows is also explained here http://tardis.dl.ac.uk/Condor/sandbox/growl-certs.shtml.

  1. On the Menu Toolbar, select Edit and then Preferences;
  2. Select Advanced, the Encryption tab and then select View Certificates;
  3. Ensure the Your Certificates tab is selected;
  4. Select your current e-Science certificate;
  5. Select Backup;
  6. Choose location to backup to and name the file usercred.p12;
  7. Enter the master password for your Firefox Software Security Device when prompted;
  8. Set a good quality password and confirm it, this will be needed later.

This will result in a file called usercred.p12 which needs to be on your Grid client machine. From there, it can be converted into the correct form and location as follows.

[frame=single]
$ Growl/scripts/mk-cert ~/usercred.p12

Passphrase that protects /home/rja/usercred.p12:
Passphrase to protect /home/rja/.globus/userkey.pem [same]:
MAC verified OK

This Growl script mk-cert has performed the following operations for you:

Note that your original P12 format file also needs protecting. It is probably best if this file is also copied into the /.globus directory and given the same permissions as userkey.pem. The results of this can be seen in /.globus:

[frame=single]
$ ls -la ~/.globus

total 16
drwx------    2 rja    dlarcg       4096 Nov  3 10:50 .
drwxr-xr-x   43 rja    dlarcg       4096 Nov  2 17:13 ..
-r--r--r--    1 rja    dlarcg       1931 Nov  3 10:50 usercert.pem
-r--------    1 rja    dlarcg       1751 Nov  3 10:50 userkey.pem
-r--------    1 rja    dlarcg       5196 Nov  3 10:50 usercred.p12

The certificate can now be used to create proxies which will enable you to use Grid resources.

[frame=single]
$ ~/Growl/scripts/growl-login

Password to protect MyProxy credential:
Your identity: /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan
Enter GRID pass phrase for this identity:
Creating proxy ................................................ Done
Your proxy is valid until: Wed Feb 18 22:57:43 2009

This is made easier if /Growl/scripts is placed on the executable path.

grid-login can also be used to create a credential in the MyProxy repository.

[frame=single]
$ ~/Growl/scripts/growl-login -m

Enter GRID pass phrase for this id:
Password to protect MyProxy credential:
Your identity: /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan
Creating proxy ........................................................... Done
Proxy Verify OK
Your proxy is valid until: Wed Feb 25 10:59:55 2009
Your identity: /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan/CN=1067934402
Creating proxy ........................................... Done
Proxy Verify OK
Your proxy is valid until: Wed Feb 18 22:59:56 2009
A proxy valid for 168 hours (7.0 days) for user rja now exists on myproxy.grid-support.ac.uk.

You can find out what credentials exist after this as follows:

[frame=single]
$ ~/Growl/scripts/growl-info

Certificate
-----------
subject= /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan
notBefore=Mar  4 16:39:57 2008 GMT
notAfter=Apr  3 16:39:57 2009 GMT
SHA1 Fingerprint=9A:89:08:2C:F0:94:30:DB:7D:C0:D3:8E:FB:4E:0A:7D:88:8B:4B:F9
[ /home/rja/.globus/usercred.p12 is also present ]

MyProxy Credential (on myproxy.grid-support.ac.uk)
------------------
username: rja
owner: /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan
  timeleft: 167:57:36  (7.0 days)

Local Proxy Credential
----------------------
subject  : /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan/CN=1067934402/CN=1510373797
issuer   : /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan/CN=1067934402
identity : /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan
type     : Proxy draft (pre-RFC) compliant impersonation proxy
strength : 1024 bits
path     : /tmp/x509up_u10039
timeleft : 11:57:37

Note: there are currently issues with this on 64-bit operating systems.

Specification of GROWL Scripts

Command Description Usage
check-cert checks consistency and validity of credential files (.p12 and .pem)  
growl-cancel cancel a Grid job growl-cancel [jobid]
growl-cat cat a file on a Grid machine growl-cat machine file
growl-clean clean up after a Grid job growl-clean [jobid]
growl-cp allows staging or retrieving of files from a Grid resource and ``3rd party transfers'' growl-cp [-D] from to
growl-create create a remote file with given permissions growl-create machine file permissions
growl-dn prints out your DN, useful if applying for authorisation to use Grid resources growl-dn [-D] [-u username]
growl-get-jobmanager identify available job manager on Grid machine growl-get-jobmanager grid-machine
growl-get-output get output from job growl-get-output [-e] [jobId]
growl-get-queues identify available queues on a Grid machine growl-get-queues grid-machine
growl-getlog get job's log growl-getlog [jobId]
growl-info get info about my proxy credentials growl-info [-D] [-u username] [certfile]
growl-log   growl-log [jobId]
growl-login create proxy credentials so that the Grid can be used growl-login [-D] [-f] [-m] [-v vo] [-u username]
growl-logout remove proxy credentials growl-logout [-D] [-m] [-u username]
growl-ls list files in directory on Grid machine growl-ls machine [ls-options] [files]
growl-mkdir make a directory on a Grid machine growl-mkdir machine dir
growl-mv move files on a Grid machine growl-mv machine file file_or_directory
growl-poll poll a remote job growl-poll [jobId]
growl-pwd print your working directory on a Grid resource growl-pwd [-D] machine
growl-queue lists jobs queued on Grid machine growl-queue machine
growl-rm removes a file/ directory on the Grid resource growl-rm [-s] machine file_or_directory
growl-sh run a remote command growl-sh machine [-D] [command]
growl-status get status of a remote job growl-status [jobId]
growl-submit submit a job to a remote job manager growl-submit [-c] [-d dir] [-i file] [-e file] [-o file] [-f rslfile] [-h hosts] [-v env] [-x rsl] [-p project] [-q queue] [-r] [-n procs] machine[/jobmanager] command args*]
growl-timeout    
growl-version prints version of Growl and dependencies  
growl-which locates given executable in your PATH on the Grid machine growl-which [-d dir to cd to first machine [command]
mk-cert converts a .p12 format credential into .pem format  

check-cert

growl-cancel

growl-cat

growl-clean

growl-cp

growl-create

growl-dn

growl-get-jobmanager

growl-get-output

growl-get-queues

growl-getlog

growl-info

NAME growl-info - prints information about your grid certificates and proxy credentials
SYNOPSIS growl-info [-mv?] [-u username] [certificate]
DESCRIPTION growl-info is a wrapper for grid-cert-info, myproxy-info and grid-proxy-info. It prints identity and validity information about your certificate and proxy credential. If X509_USER_CERT is set, it will use that certificate rather than the default  /.globus/usercert.pem. In the absence of /.globus/usercert.pem it will use /.globus/usercred.p12, but will need to request a password.

It can also optionally provide the same information for a credential lodged on a MyProxy server, but only if a valid proxy credential is present.

If there is no proxy credential in the standard place, it will look in X509_USER_PROXY.

If a certificate is given as an argument, it will use that instead and print out appropriate information.

OPTIONS -? Displays command usage text and exits.

-m Display information on myproxy credentials if present

-u username Use the provided username instead of your login name to locate your credential on the MyProxy server

-v Provide trace/debug information certificate

Use this certificate instead of looking in the standard places.

EXIT STATUS 0 on success, 0 on error
FILES  /.globus/usercert.pem

 /.globus/usercred.p12

/tmp/x509up_uNNNNN

ENVIRONMENT GROWL_HOME Location of the GROWL Scripts installation

MYPROXY_SERVER Name of the MyProxy server on which your certificate is lodged. Note that this is set by default in $GROWL_HOME/.setup.sh and/or $GROWL_HOME/.setup.csh

X509_USER_CERT If this is set, it is used instead of /.globus/usercert.pem to locate the user's certificate

growl-log

growl-login

NAME growl-login - Generates proxy credentials for a user
SYNOPSIS growl-login [-fmv?] [-u username]
DESCRIPTION growl-login is a wrapper for grid-proxy-init, myproxy-init and myproxy-logon. Using the user's certificate and private key, a proxy credential is generated. The passphrase which protects the private key must be provided.

The user's certificate is located by checking in the following locations:

1. X509_USER_KEY This environment variable should be used in conjunction with X509_USER_CERT.

2. /.globus/userkey.pem This should be used in conjunction with /.globus/usercert.pem.

3. /.globus/usercred.p12

If it cannot find a certificate, it will look one for a valid credential on the default MyProxy server and if it finds one, prompts you for its password.

If a valid proxy credential already exists, then growl-login has nothing to do and just exists. [The possibility to over-ride this with the -f option is currently being implemented.]

The -m option forces a check on a MyProxy server for a valid credential. If one is not found, it will lodge one there, prompting for the password to protect it with (not that this is in addition to the passphrase used to protect the certificate which will also need to be provided). If there is a MyProxy credential available, a local proxy credential will be generated from that (in this case the MyProxy password will have to be provided).

OPTIONS -m See above

-u username Use the provided username instead of your login name to locate your credential on the MyProxy server. It also implies -m.

-v Provide trace/debug information -f Not yet implemented

EXIT STATUS 0 on success, 0 on error
FILES  /.globus/usercert.pem

 /.globus/usercred.p12

/tmp/x509up_uNNNNN

ENVIRONMENT GROWL_HOME Location of the GROWL Scripts installation

MYPROXY_SERVER Name of the MyProxy server on which your certificate is lodged. Note that this is set by default in $GROWL_HOME/.setup.sh and/or $GROWL_HOME/.setup.csh

X509_USER_CERT If this is set, it is used instead of /.globus/usercert.pem to locate the user's certificate

growl-logout

NAME growl-logout - Removes user's proxy credentials
SYNOPSIS growl-logout [-mv?] [-u username]
DESCRIPTION growl-logout is a wrapper for grid-proxy-destroy and myproxy-destroy. It will remove your proxy credential and (optionally) a credential stored on your MyProxy server.

After running growl-logout you will have to run growl-login before you can access Grid resources again.

OPTIONS -? Displays command usage text and exits.

-m Display information on myproxy credentials if present

-u username Use the provided username instead of your login name to locate your credential on the MyProxy server. It also implies -m.

-v Provide trace/debug information

EXIT STATUS 0 on success, 0 on error
FILES  /.globus/usercert.pem

 /.globus/usercred.p12

/ tmp/x509up_uNNNNN

ENVIRONMENT GROWL_HOME Location of the GROWL Scripts installation

MYPROXY_SERVER Name of the MyProxy server on which your certificate is lodged. Note that this is set by default in $GROWL_HOME/.setup.sh and/or $GROWL_HOME/.setup.csh

X509_USER_CERT If this is set, it is used instead of /.globus/usercert.pem to locate the user's certificate

growl-ls

growl-mkdir

growl-mv

growl-poll

growl-pwd

NAME growl-pwd - print your working directory on a Grid resource
SYNOPSIS growl-pwd gridhost
DESCRIPTION growl-pwd uses growl-sh (1) to log on to the provided grid resource, runs pwd (1) there, and then returns the results on standard output.
NOTES Although growl-pwd is a simple script, it can be used as an initial ``smoke test'' since successful completion implies the following:

There is no firewall between your client and the grid resource

You are authorised to use the Grid resource

Your client trusts the CA certificate that signed the host certificate of the Grid resource

The Grid resource trusts the CA certificate that signed your certificate

OPTIONS -? Displays command usage text and exits.
EXIT STATUS 0 on success, 0 on error
ENVIRONMENT GROWL_HOME Location of the GROWL Scripts installation

growl-queue

growl-rm

NAME growl-rm - Remove a file/directory on the Grid resource
SYNOPSIS growl-rm grid-resource filename
DESCRIPTION The file is removed. For simplicity, the force (-f) and recursive (-r) options are assumed: there is therefore no growl-rmdir.
OPTIONS none
EXIT STATUS  

growl-sh

growl-status

growl-submit

growl-timeout

growl-version

growl-which

NAME growl-which - locates given executable in your PATH on the Grid machine
SYNOPSIS growl-which gridhost executable
DESCRIPTION growl-which uses growl-sh (1) to log on to the provided grid resource, runs which (1) for the provided executable there, and then returns the results on standard output.
OPTIONS -? Displays command usage text and exits.
EXIT STATUS 0 on success, 0 on error
ENVIRONMENT GROWL_HOME Location of the GROWL Scripts installation

mk-cert

See example above.

Template

NAME  
SYNOPSIS  
DESCRIPTION  
OPTIONS  
EXIT STATUS  

Rob Allan 2009-11-10