This manual documents the G-R-Toolkit which is a set of client scripts and C library routines providing access to Grid resources for computation and data management. G-R-Toolkit is based on the GROWL and RMCS packages.
The G-R-Toolkit combines the best software developed at SFTC during its e-Science Programme from 2001-2007. It allows users of many applications in computational research to manage their high-performance computing and data and information management tasks directly from their desktop systems. Components of G-R-T, some of which are available separately include:
|GROWL Scripts - facilitate management of digital certificates and access to datasets on remote Grid resources.||SRB Client and RCommands - desktop tools to manage stored datasets and metadata.|
|RMCS - uses Condor DAGMan to create and enact workflows to integrate data management and remote computation.||R, Perl and Python framework - scripting interfaces suitable for many research domains from bio-informatics to chemistry.|
|AgentX - a sophisticated semantic toolset using domain specific ontologies to link applications with ASCII, XML and DB formats.||G-R-T C library - Web service clients appropriate for application programming.|
G-R-T uses Grid middleware to perform its tasks on behalf of the user. Well known technology is re-deployed on a dedicated intermediate server, including Web Services, Condor, Globus, SRB and MyProxy.
G-R-T will work alongside and extend existing toolkits. It has a ``plug-in'' capability allowing Grid client functionality to be imported into applications such as Matlab, Stata, Materials Studio and others. G-R-T is written in a modular style using Web services to achieve a Service Oriented Architecture, a widely adopted pattern in software engineering. This enables its client side to be re-factored or extended to suit most research requirements.
Components of the G-R-Toolkit were developed by Rob Allan, Adam Braimah, Phil Couch, Dan Grose, John Kewley and Rik Tyer in STFC's Grid Technology Group and their partners, see http://www.grids.ac.uk/twiki/bin/view/GridTech/WebHome.
G-R-T has been deployed and tested on the North West Grid, which is described at http://www.nw-grid.ac.uk [#!AHM2007:NWGRIDMiddleware!#]. For more information about access and the commercial services available contact John Bancroft on 01925 603148 or visit the DaComS Web site http://www.dacoms.ac.uk.
RMCS and RCommands have been deployed by Jonathan Churchill on the National Grid Servic, see http://wiki.ngs.ac.uk/index.php?title=Category:Community_Software.
GROWL is the Grid Resources On Workstation Library [#!GROWL!#,#!growl1!#,#!growl2!#,#!AHM2005:Growl!#].
GrowlScripts [#!AHM2007:GrowlScripts!#] is a set of useful command-line scripts which were developed by John Kewley during and after the GROWL VRE project.
RMCS is the Remote My Condor Submit developed in the NERC-funded e-Minerals project [#!mcs!#,#!mcs-user!#,#!e-minerals!#,#!bruin!#,#!calleja!#,#!rmcs-arch!#].
AgentX [#!agentx!#] was developed by Phil Couch in the e-CCP project.
RCommands [#!rcommands!#,#!metadata!#] were developed by Rik Tyer to enhance RMCS.
MultiR [#!multir!#] and SabreR [#!NCeSS2005!#,#!NCeSS2006!#] were developed by Dan Grose based on GROWL but written in the R language [#!R!#,#!R-intro!#,#!R-lang!#,#!R-admin!#,#!R-exts!#,#!GROWL-r!#]. They have been applied to longitudinal statistical analysis [#!sage!#], bio-informatics [#!poster:microarray!#] and geography.
The GROWL Scripts are named growl-name where ``name'' is the name of the function which has a corresponding section appearing in this documentation in the GrowlScripts chapter.
The RMCS scripts are named rmcs_name in a similar way.
In this release, ordering in argument lists is not completely standardised.
We shall assume that the library is to be installed at $(GROWL).
The Library distribution will contain a subset of the following directories. Usually the full source and all machine-specific files will not be included.
/G-R-Toolkit/Doc - various related documents;
/G-R-Toolkit/Doc/Manual - LaTeX user manual and complete specifications;
/G-R-Toolkit/Growl - source and documentation for the Growl Scripts;
/G-R-Toolkit/scripts - soft link to Growl/scripts;
/G-R-Toolkit/base - base Web services code for G-R-T library;
/G-R-Toolkit/NAME - program directory for routines from chapter ``NAME'', e.g. Authentication, Birdbath, Files, Filesystems, GlobJobs, Jobs, Monty, R, RCommands, RMCS, Resources, Rgem, SGEJobs, SRB;
/G-R-Toolkit/lib - user-accessible library archive files (possibly containing module interfaces, e.g. on Cray systems) pre-compiled;
/G-R-Toolkit/include - user-accessible library module interfaces if not contained in library archive files;
files, may not all be present in a particular distribution;
/G-R-Toolkit/Test - examples and test codes;
/G-R-Toolkit/packages - source for required packages; /G-R-Toolkit/man - man pages from packages;
/G-R-Toolkit/share - shared items from packages;
Description of Modules.
|Authentication||C library routines for authentication via MyProxy|
|Birdbath||C library routines for Condor Birdbath broker (experimental)|
|Files||C library routines for managing files on G-R-T server|
|Filesystem||C library routines for managing remote files, uses GROWL scripts|
|GlobJobs||C library routines for managing remote jobs, uses GROWL scripts|
|Growl||source for GROWL scripts|
|Jobs||C library routines for managing remote jobs via GLobus C interface|
|Monty||Perl code for task farming via RMCS|
|R||R wrappers for G-R-T C library routines|
|RCommands||commands for metadata management|
|RMCS||Perl client commands for remote MCS service|
|Resources||C library routines querying Globus resources|
|Rgem||C Web service client for managing output of task farm runs|
|SGEJobs||C library routines for managing remote jobs directly on a machine with Sun Grid Engine, uses GROWL scripts|
|SRB||C library routines for SRB client Scommands|
|Test||C test routines and examples|
Each ``NAME'' subroutine directory, e.g. FFT or CGS contains sub-directories as follows:
src - source code for these library routines;
bin - binaries and module interfaces for these routines;
lib - partial library built from this source directory only;
include - include files (modules) from this source directory only;
doc - specifications chapter for these routines;
test - test suite for these library routines;
examples - examples of using these library routines;
others - various test routines which will not form part of the distribution.
Not all these will be present in the library distribution, but in addition a master Makefile and README will be present.
The GROWL library directory contains a master makefile to build the distribution and test suite. It copies the machine-dependent definitions from arch to the required directories src and test and invokes make for each set of subroutines required.
Each source and test directory contains its own makefile which will
appear for instance as
/G-R-Toolkit/NAME/src/makefile. The makefiles may include a machine-specific portion INCLUDE.make which contains definitions of architecture-dependent compilation flags, libraries and options and shows what modules and binaries will be copied to /G-R-Toolkit/include, /G-R-Toolkit/lib and /G-R-Toolkit/bin which will form the basis of the distribution.
The makefile contains a list of dependencies for building the object code of a particular set of subroutines. This will ensure that modules are compiled in the correct order. Note that the routines in directory base must be built first. This is ensured by the master makefile /G-R-Toolkit/Makefile.
SOAP::Lite Perl Web services
gSOAP, C Web services
A number of technical reports produced by Daresbury Laboratory contain
material relevant to Grid and Web services development. They
may provide background information of interest to users of this
Library. These reports are available from Web URL
Rob Allan 2009-11-10