Next: Bibliography Up: G-R-TOOLKIT: STFC LIBRARY OF Previous: CONDITIONS OF USE Contents
References [#!globus!#,#!globus2!#,#!GROWL!#,#!AHM2007:GrowlScripts!#,#!AHM2005:Growl!#,#!NCeSS2005!#].
This chapter provides an introduction and documentation for the GROWL Scripts [#!AHM2007:GrowlScripts!#]. This suite of scripts was developed during and following the JISC-funded GROWL VRE-1 project [#!GROWL!#] and has been found useful for installing Grid middleware and using remote resources. The GROWL Scripts continue to be maintained and developed by John Kewley at Daresbury.
Some basic documentation and examples are available on-line atx http://www.growl.org.uk/Scripts/growl-scripts.shtml. This chapter is based on this information. Some tutorial examples are provided here http://tardis.dl.ac.uk/Condor/sandbox/growl-scripts.shtml.
GROWL Scripts are a set of wrapper scripts designed to facilitate takeup of Grid computing by computational scientists who are already reasonably familiar with parallel and distributed computing on clusters. They comprise wrappers to the standard globus and openssl command-line interfaces and attempt to address the three main barriers faced by newcomers to the Grid: (i) setting up client-side middleware; (ii) handling certificats; and (iii) submitting jobs to the Grid negociating firewalls.
Version: 1.6.0
Public calls:
Public modules: Growl/scripts
Other software required: Globus or VDT,GSI-SSH and
OpenSSL. These can downloaded from the Internet and be installed from
a makefile provided.
Remote services used: MyProxy server, Globus Grid resources
Date: 2006-9
Origin: J. Kewley, CCLRC Daresbury Laboratory
Language: Unix shell
Conditions on external use: Standard, see separate chapter
The GROWL Scripts can be downloaded from the GROWL Web site at http://www.grids.ac.uk/GROWL/GrowlScripts.tar.gz. Un-tarring this file will create a directory called Growl with sub-directories NOTES, examples, packages, scripts.
Client-side Globus middleware can now be built as follows. This uses wget to pull the latest version from the Internet. If you use a Web cache, you should amend lines in Growl/Makefile.conf to the approriate adddress and port. For instance, for those at Daresbury Laboratory:
[frame=single] WEB_CACHE = wwwcache.dl.ac.uk WEB_CACHE_PORT = 8080
You should then be able to build GROWL Scripts by running:
[frame=single] $ cd Growl $ make VDT
For this to work cleanly, your operating system must match one of the supported ones for the release of VDT that GROWL Scripts is trying to build (see VDT_VERS in Makefile.conf). Check the VDT Web site for a list of supported platforms for that version. The most common build problems are listed in the README.txt contained in the Growl directory. If there is no suitable VDT version do make gt instead which will use the Globus source distribution (not that this is much slower as it requires extensive compilation). The Globus or VDT distribution will be build in the Growl/packages directory and Grid CA certificates will be installed in Growl/packages/globus/share/certificaes.
Before using the GROWL Scripts, it is necessary to initialise the environment and creat a proxy certificate. This can be done as follows by typing source /Growl/setup.sh.
To procede, you will first need to obtain a personal Grid certificate from the UK e-Science Certification Authority http://www.grid-support.ac.uk/content/view/221/171/. Following their procedure will result in a certificate in your browser. The following instructions are for Firefox on Linux, but using Internet Explorer and Windows is also explained here http://tardis.dl.ac.uk/Condor/sandbox/growl-certs.shtml.
This will result in a file called usercred.p12 which needs to be on your Grid client machine. From there, it can be converted into the correct form and location as follows.
[frame=single] $ Growl/scripts/mk-cert ~/usercred.p12 Passphrase that protects /home/rja/usercred.p12: Passphrase to protect /home/rja/.globus/userkey.pem [same]: MAC verified OK
This Growl script mk-cert has performed the following operations for you:
Note that your original P12 format file also needs protecting. It is probably best if this file is also copied into the /.globus directory and given the same permissions as userkey.pem. The results of this can be seen in /.globus:
[frame=single] $ ls -la ~/.globus total 16 drwx------ 2 rja dlarcg 4096 Nov 3 10:50 . drwxr-xr-x 43 rja dlarcg 4096 Nov 2 17:13 .. -r--r--r-- 1 rja dlarcg 1931 Nov 3 10:50 usercert.pem -r-------- 1 rja dlarcg 1751 Nov 3 10:50 userkey.pem -r-------- 1 rja dlarcg 5196 Nov 3 10:50 usercred.p12
The certificate can now be used to create proxies which will enable you to use Grid resources.
[frame=single] $ ~/Growl/scripts/growl-login Password to protect MyProxy credential: Your identity: /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan Enter GRID pass phrase for this identity: Creating proxy ................................................ Done Your proxy is valid until: Wed Feb 18 22:57:43 2009
This is made easier if /Growl/scripts is placed on the executable path.
grid-login can also be used to create a credential in the MyProxy repository.
[frame=single] $ ~/Growl/scripts/growl-login -m Enter GRID pass phrase for this id: Password to protect MyProxy credential: Your identity: /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan Creating proxy ........................................................... Done Proxy Verify OK Your proxy is valid until: Wed Feb 25 10:59:55 2009 Your identity: /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan/CN=1067934402 Creating proxy ........................................... Done Proxy Verify OK Your proxy is valid until: Wed Feb 18 22:59:56 2009 A proxy valid for 168 hours (7.0 days) for user rja now exists on myproxy.grid-support.ac.uk.
You can find out what credentials exist after this as follows:
[frame=single] $ ~/Growl/scripts/growl-info Certificate ----------- subject= /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan notBefore=Mar 4 16:39:57 2008 GMT notAfter=Apr 3 16:39:57 2009 GMT SHA1 Fingerprint=9A:89:08:2C:F0:94:30:DB:7D:C0:D3:8E:FB:4E:0A:7D:88:8B:4B:F9 [ /home/rja/.globus/usercred.p12 is also present ] MyProxy Credential (on myproxy.grid-support.ac.uk) ------------------ username: rja owner: /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan timeleft: 167:57:36 (7.0 days) Local Proxy Credential ---------------------- subject : /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan/CN=1067934402/CN=1510373797 issuer : /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan/CN=1067934402 identity : /C=UK/O=eScience/OU=CLRC/L=DL/CN=rob allan type : Proxy draft (pre-RFC) compliant impersonation proxy strength : 1024 bits path : /tmp/x509up_u10039 timeleft : 11:57:37
Note: there are currently issues with this on 64-bit operating systems.
| Command | Description | Usage |
| check-cert | checks consistency and validity of credential files (.p12 and .pem) | |
| growl-cancel | cancel a Grid job | growl-cancel [jobid] |
| growl-cat | cat a file on a Grid machine | growl-cat machine file |
| growl-clean | clean up after a Grid job | growl-clean [jobid] |
| growl-cp | allows staging or retrieving of files from a Grid resource and ``3rd party transfers'' | growl-cp [-D] from to |
| growl-create | create a remote file with given permissions | growl-create machine file permissions |
| growl-dn | prints out your DN, useful if applying for authorisation to use Grid resources | growl-dn [-D] [-u username] |
| growl-get-jobmanager | identify available job manager on Grid machine | growl-get-jobmanager grid-machine |
| growl-get-output | get output from job | growl-get-output [-e] [jobId] |
| growl-get-queues | identify available queues on a Grid machine | growl-get-queues grid-machine |
| growl-getlog | get job's log | growl-getlog [jobId] |
| growl-info | get info about my proxy credentials | growl-info [-D] [-u username] [certfile] |
| growl-log | growl-log [jobId] | |
| growl-login | create proxy credentials so that the Grid can be used | growl-login [-D] [-f] [-m] [-v vo] [-u username] |
| growl-logout | remove proxy credentials | growl-logout [-D] [-m] [-u username] |
| growl-ls | list files in directory on Grid machine | growl-ls machine [ls-options] [files] |
| growl-mkdir | make a directory on a Grid machine | growl-mkdir machine dir |
| growl-mv | move files on a Grid machine | growl-mv machine file file_or_directory |
| growl-poll | poll a remote job | growl-poll [jobId] |
| growl-pwd | print your working directory on a Grid resource | growl-pwd [-D] machine |
| growl-queue | lists jobs queued on Grid machine | growl-queue machine |
| growl-rm | removes a file/ directory on the Grid resource | growl-rm [-s] machine file_or_directory |
| growl-sh | run a remote command | growl-sh machine [-D] [command] |
| growl-status | get status of a remote job | growl-status [jobId] |
| growl-submit | submit a job to a remote job manager | growl-submit [-c] [-d dir] [-i file] [-e file] [-o file] [-f rslfile] [-h hosts] [-v env] [-x rsl] [-p project] [-q queue] [-r] [-n procs] machine[/jobmanager] command args*] |
| growl-timeout | ||
| growl-version | prints version of Growl and dependencies | |
| growl-which | locates given executable in your PATH on the Grid machine | growl-which [-d dir to cd to first machine [command] |
| mk-cert | converts a .p12 format credential into .pem format |
| NAME | growl-info - prints information about your grid certificates and proxy credentials |
| SYNOPSIS | growl-info [-mv?] [-u username] [certificate] |
| DESCRIPTION | growl-info is a wrapper for grid-cert-info, myproxy-info
and grid-proxy-info. It prints identity and validity information about
your certificate and proxy credential. If X509_USER_CERT is set, it
will use that certificate rather than the default
/.globus/usercert.pem. In the absence of /.globus/usercert.pem it
will use /.globus/usercred.p12, but will need to request a password.
It can also optionally provide the same information for a credential lodged on a MyProxy server, but only if a valid proxy credential is present. If there is no proxy credential in the standard place, it will look in X509_USER_PROXY. If a certificate is given as an argument, it will use that instead and print out appropriate information. |
| OPTIONS | -?
Displays command usage text and exits.
-m Display information on myproxy credentials if present -u username Use the provided username instead of your login name to locate your credential on the MyProxy server -v Provide trace/debug information certificate Use this certificate instead of looking in the standard places. |
| EXIT STATUS | 0 on success, 0 on error |
| FILES | /.globus/usercert.pem
/.globus/usercred.p12 /tmp/x509up_uNNNNN |
| ENVIRONMENT | GROWL_HOME
Location of the GROWL Scripts installation
MYPROXY_SERVER Name of the MyProxy server on which your certificate is lodged. Note that this is set by default in $GROWL_HOME/.setup.sh and/or $GROWL_HOME/.setup.csh X509_USER_CERT If this is set, it is used instead of /.globus/usercert.pem to locate the user's certificate |
| NAME | growl-login - Generates proxy credentials for a user |
| SYNOPSIS | growl-login [-fmv?] [-u username] |
| DESCRIPTION | growl-login is a wrapper for grid-proxy-init,
myproxy-init and myproxy-logon. Using the user's certificate and
private key, a proxy credential is generated. The passphrase which
protects the private key must be provided.
The user's certificate is located by checking in the following locations: 1. X509_USER_KEY This environment variable should be used in conjunction with X509_USER_CERT. 2. /.globus/userkey.pem This should be used in conjunction with /.globus/usercert.pem. 3. /.globus/usercred.p12 If it cannot find a certificate, it will look one for a valid credential on the default MyProxy server and if it finds one, prompts you for its password. If a valid proxy credential already exists, then growl-login has nothing to do and just exists. [The possibility to over-ride this with the -f option is currently being implemented.] The -m option forces a check on a MyProxy server for a valid credential. If one is not found, it will lodge one there, prompting for the password to protect it with (not that this is in addition to the passphrase used to protect the certificate which will also need to be provided). If there is a MyProxy credential available, a local proxy credential will be generated from that (in this case the MyProxy password will have to be provided). |
| OPTIONS | -m See above
-u username Use the provided username instead of your login name to locate your credential on the MyProxy server. It also implies -m. -v Provide trace/debug information -f Not yet implemented |
| EXIT STATUS | 0 on success, 0 on error |
| FILES | /.globus/usercert.pem
/.globus/usercred.p12 /tmp/x509up_uNNNNN |
| ENVIRONMENT | GROWL_HOME
Location of the GROWL Scripts installation
MYPROXY_SERVER Name of the MyProxy server on which your certificate is lodged. Note that this is set by default in $GROWL_HOME/.setup.sh and/or $GROWL_HOME/.setup.csh X509_USER_CERT If this is set, it is used instead of /.globus/usercert.pem to locate the user's certificate |
| NAME | growl-logout - Removes user's proxy credentials |
| SYNOPSIS | growl-logout [-mv?] [-u username] |
| DESCRIPTION | growl-logout is a wrapper for grid-proxy-destroy and
myproxy-destroy. It will remove your proxy credential and (optionally)
a credential stored on your MyProxy server.
After running growl-logout you will have to run growl-login before you can access Grid resources again. |
| OPTIONS | -?
Displays command usage text and exits.
-m Display information on myproxy credentials if present -u username Use the provided username instead of your login name to locate your credential on the MyProxy server. It also implies -m. -v Provide trace/debug information |
| EXIT STATUS | 0 on success, 0 on error |
| FILES | /.globus/usercert.pem
/.globus/usercred.p12 / tmp/x509up_uNNNNN |
| ENVIRONMENT | GROWL_HOME
Location of the GROWL Scripts installation
MYPROXY_SERVER Name of the MyProxy server on which your certificate is lodged. Note that this is set by default in $GROWL_HOME/.setup.sh and/or $GROWL_HOME/.setup.csh X509_USER_CERT If this is set, it is used instead of /.globus/usercert.pem to locate the user's certificate |
| NAME | growl-pwd - print your working directory on a Grid resource |
| SYNOPSIS | growl-pwd gridhost |
| DESCRIPTION | growl-pwd uses growl-sh (1) to log on to the provided grid resource, runs pwd (1) there, and then returns the results on standard output. |
| NOTES | Although growl-pwd is a simple script, it can be used as an initial
``smoke test'' since successful completion implies the following:
There is no firewall between your client and the grid resource You are authorised to use the Grid resource Your client trusts the CA certificate that signed the host certificate of the Grid resource The Grid resource trusts the CA certificate that signed your certificate |
| OPTIONS | -? Displays command usage text and exits. |
| EXIT STATUS | 0 on success, 0 on error |
| ENVIRONMENT | GROWL_HOME Location of the GROWL Scripts installation |
| NAME | growl-rm - Remove a file/directory on the Grid resource |
| SYNOPSIS | growl-rm grid-resource filename |
| DESCRIPTION | The file is removed. For simplicity, the force (-f) and recursive (-r) options are assumed: there is therefore no growl-rmdir. |
| OPTIONS | none |
| EXIT STATUS |
| NAME | growl-which - locates given executable in your PATH on the Grid machine |
| SYNOPSIS | growl-which gridhost executable |
| DESCRIPTION | growl-which uses growl-sh (1) to log on to the provided grid resource, runs which (1) for the provided executable there, and then returns the results on standard output. |
| OPTIONS | -? Displays command usage text and exits. |
| EXIT STATUS | 0 on success, 0 on error |
| ENVIRONMENT | GROWL_HOME Location of the GROWL Scripts installation |
See example above.
| NAME | |
| SYNOPSIS | |
| DESCRIPTION | |
| OPTIONS | |
| EXIT STATUS |
Rob Allan 2009-11-10