next up previous contents
Next: Score Justification Up: Technology Evaluation Report and Previous: Introduction   Contents

Method

We have chosen to adopt a Multiattribute Utility Evaluation approach [87] to evaluating the alternative contenders for a VRE. This approach is nevertheless straightforward and simple to apply. The core of the procedure is to identify the most relevant values or criteria that are appropriate to the functioning of a VRE. Measurements are then made to determine the degree to which the criteria are attained. By doing so systematically, and by making numerical judgements wherever possible, we can compare the VRE contenders on a more objective basis than is usually the case. We have identified the following 10 broad criteria, some of which are further subdivided.

These criteria are as follows:

Criterion A:
Can the framework be applied to all disciplines?

Criterion B:
Is the VRE framework useful for users from both the e-science (a) and e-learning (b) domains?

Criterion C:
Criterion C is divided up into two sub criteria:

a) It is generally accepted that software modules that are used by many projects end up being robust and well understood due to the amount of exposure they receive. It thus makes good sense to make use of publicly available libraries when building a software product as opposed to writing the same algorithms over and over again. This criterion is thus intended to reflect the degree of use of open source libraries by the VRE.

b) Conformance to ratified standards is another feature that is generally seen as being important. Standards conformance fosters ease of interoperability and ease of extension via plug-able components. JSR 168 and WSRP have been identified as the main two standards that, when adhered to, will allow Java components and Web service based components to be added to a VRE. What interface standardization also facilitates is reuse. There are many tools currently in circulation, written in Java, which could be re-factored to allow them to be plugged into a standards-based interface like JSR 168. There are also many tools written in other languages. As long as these tools can be re-factored to talk the WSRP protocol, then they could also be re-used in a WSRP compliant container.

Criterion D:
Make UK services and resources available in familiar environments e.g. typically via a Web browser;

Criterion E:
Any open source project worth its salt is supported by a decent online community. Discussion forums, chat rooms and mailing lists have all proven to be incredibly useful tools for spreading know how about a software product. The following two criteria are an attempt to measure this kind of support.

a) This measures the degree of support there is for developers who wish to start writing tools for, or even extending, the VRE framework.

b) This measures the degree of support there is for users and administrators who wish to install, configure and use the VRE framework.

Criterion F:
Offer choice in presentation or delivery for (a) services and (b) tools;

Criterion G:
How steep is the learning curve required to use the VRE framework?

Criterion H:
This criterion measures the amount of functionality you get ``out of the box'' from a particular framework. In other words, if an institution installed the basic version of this VRE framework and just left it at that, would it be any use?

Criterion I:
Presence and extent of a future funding stream, for (a) $<$12 , (b) $>$12 and $<$24 and (c) $>$24 months

Have all the criteria been listed? There are others we could include like the track record of the developer team, but this is taken into account when we allocate a score under sub criterion F(a). We felt that other, perhaps domain specific criteria, would be less important in an overall evaluation and are thus likely to have a lower impact and not affect the overall rankings. This has been partially tested with a sensitivity analysis (see below).

We then ranked the 9 criteria in order of importance and allocated a score out of 10, this was then standardized to sum to 1. If criteria had a lot of overlap with another criterion they would be given similar rankings. For simplicity, and to start with, we have given them all equal weights (0.1).

The same process was then applied to the sub criteria. This gives us the weights $w_{Xj}$ that can be used to aggregate the score of each component of criterion $X$ to produce a total score for criterion $X$. So for example, for criterion $B$ which is subdivided into 4 components, the total score for criterion $B$ is given by $B = \sum_j
w_{Xj} B_j $. The total score over all the criteria is then given by $U =
\sum_X r_X X$, where $r_X$ is the weight applied to criterion $X$. The weightings we used are shown in parentheses in the following figure.

Figure 6: Value tree for comparing open source, platform independent VREs
\includegraphics[width=6.5in]{VALUETREE.eps}

The scores (out of 100) for each component were then obtained using our judgment.


next up previous contents
Next: Score Justification Up: Technology Evaluation Report and Previous: Introduction   Contents
Rob Allan 2005-05-09