\begin{article}
\title{Implementation, challenges and future directions of integrating services from the GIS and decision science domains}
\subtitle{A case of Distributed Spatial Multi-Criteria Evaluation}

\author{Luc Boerboom \\
Faculty of Geo-Information Science and Earth Observation (ITC) University of Twente, 
P.O. Box 217, 7500 AE  Enschede, The Netherlands, boerboom@itc.nl  
\\
Özgün OskayAlan\\
Faculty of Geo-Information Science and Earth Observation (ITC) University of Twente, alan@itc.nl  
}

\maketitle

\section{Abstract}
We are implementing an open source project for spatial decision making called Distributed Spatial Multi-Criteria Evaluation (DSMCE) under an EU project for inter-regional development on forestry and climate change adaptation (ForeStClim). 

In this paper, we first describe what DSMCE is and what it does. We have designed an extensible architecture for integrating services of two domains, respectively the GIS and Decision Sciences domain. Thereby we delegate domain expertise to available implementations. We use the Service Oriented Architecture (SOA) paradigm to build our DSMCE service and application. Integration is implemented by the use of open specifications and protocols coming from these domains. DSMCE is not only extensible in terms of the external services it uses, it also is extensible as an application because it is developed with OSGi technology which that brings advanced modularity. 

Second we share observations about implementation challenges we have addressed. These challenges are related to the design of integration of the two domains, the ability of specifications to address real implementation problems, and the reliability and quality of available open source tools. These lead us to conclusions about the solutions we had to implement.

Third and finally we give an overview of future directions. Some of these topics relate to the spatial domain, e.g. the use of Web Processing Services (WPS) for pre and post processing around decision analysis, others to the decision sciences domain, e.g. the integration of other non-spatial data sources and services, or collaborative decision making. 
 
\section{Introduction}
If spatial multi-criteria evaluation (SMCE) (Herwijnen, 1999, Sharifi and Retsios, 2004)  methodologies, implementations, and expertise could evolve on the web, both decision making and the decision aiding methodologies could develop in new directions. SMCE in desktop applications has been used in analytical academic or consulting studies such as transport (Sharifi and Boerboom, 2006, Keshkamat et al., 2009), environmental management (Zucca et al., 2008) in poverty assessment (ODPM, 2004, Baud et al., 2009). It is also used in participatory decision processes either normatively motivated because it should make decisions more democratic, or rationally motivated because it should make decisions more informed, or instrumentally because it should make decision responsibility for possible failures shared (Stirling, 2006). The number of publications has experienced a strong increase (Malczewski, 2006).

But on the web it could aid and change individual and collaborative decision making. It could create need and opportunity for expanding the range of methods for the analysis of spatial preference of groups of people, for collaborative analysis of conflict and consensus, and for learning about decision making and decision making processes. It could add value to and between spatial data infrastructures, and perform integrated assessment between organizational mandates. And it could get infrastructural properties (Boerboom, 2010). 

So far there has only been one implementation of SMCE that is server-based with the lightness of a browser client, which is ParticipatoryGIS.com (Boroushaki and Malcewski, 2010), but unlike our implementation it has not been implemented as a generic tool but for a specific project nor does make use of OGC web services standards and implementations.

We present the first prototype implementation of the open source distributed spatial multi-criteria evaluation (DSMCE) web application. It is distributed, not only in the concept of distributed computing,  because it can collect data from distributed data sources, i.e. Web Feature Services (WFS), or, if data cannot be exchanged because of data policies, data value, bandwidth, and other reasons, it can be distributed to these data sources and collect only the intermediary outputs. Also, decision makers are geographically distributed or in time. And development of it can be distributed, given the open source nature, provided good programming practices and systems are maintained. Distributed SMCE is an open source web application development project hosted on the source forge Kenai.\footnote{\url{http://kenai.com/projects/distributed-DSMCE}}

Implementing DSMCE with spatial OGC standards was not as straight forward as the intended by OGC (Percivall, 2010). We did not find a systematic review of issues around implementations and use of OGC standards in literature, although (He et al., 2009) address one of the issues using multi-version WFS’. It is beyond the scope of this paper to do so. But we consider it useful for future development of standards and implementations to explore the challenges we have faced and the solutions we have developed, which could become part of future more systematic studies. 

So the outline of this paper is as follows. We first briefly describe the particular use case for which this web application is being developed and its generalization. Second we describe the web application. Then we discuss implementation challenges with open source modules. Finally, we describe future directions.

\section{A specific use case and its generalization}
The specific use case is the following. Most European forest organizations have included climate change in their strategy documents. Now these strategic intentions need to be translated to adaptation plans in the different regions. This occurs as part of the regular forest management adaptation planning processes. Therefore spatial evaluation of vulnerability and adaptive options needs to be considered in these processes. In the ForeStClim project on “Transnational Forestry Management Strategies in Response to Regional Climate Change Impacts“, within which the distributed spatial multi-criteria evaluation web application is developed, we intend to compare several regions. But rather than applying a uniform evaluation approach we recognize the regional variability.

So the problem is that evaluation of climate change vulnerabilities and suitability of adaptive option is distributed. There are different forest management organizations in the different regions in North-western Europe. Each organization works in its specific forest policy environment and has specific policy objectives.  And they all have different data, technical data environments, and data policies. To summarize, decisions of these foresee management organizations are idiosyncratic in only partly shared policy and market environments, and there is neither value in making databases and datasets interoperable nor in semantically harmonizing data. 

Finally, the project hopes that in the exchange of different evaluation approaches the different regions will gain ideas to improve their own understanding of vulnerability and adaptive options. And that these can be communicated to European policy bodies as the Ministerial Conference on the Protection of Forests in Europe

Generalizations from this specific use case are the following. As far as decisions are concerned, DSMCE can be used for decisions where data can be the same for different decision makers, but decision makers can partially or fully use different data and data sources on internet or intranet. Also it is meant to communicate preference structure on Internet for decision makers to learn from each others’ approaches to evaluation and integrated assessment. Often data cannot be shared or made interoperable because it is just not worth the effort since use of data is infrequent and/or idiosyncratic, and does not pay off the effort of making databases interoperable. Also data can be too valuable to be shared or outdated data should not be used. DSMCE supports decisions with organizational databases and with data infrastructures (SDIs) but also at the fringes of SDIs.

\section{DSMCE}
First we describe what DSMCE does. Then we describe its architecture. Finally we list functional and technical innovations. DSMCE consists of two parts, the web application (frontend service) and the backend service. 

The web application (Figure \ref{fig.boerboom_fig1}) opens in any web browser. It initially is a single screen with four panels. Spatial multi-criteria evaluation takes place in the central panel. Here objectives and criteria can be structured, standardized with maximum standardization to a 0-1 scale, prioritized with the expected value ranking method, and aggregated (Nijkamp, 1990, Sharifi and Retsios, 2004). The Data panel (left) provides access to web feature services.  From the service the user receives some technical information and a list of layers offered. After selecting a layer, its attributes will be listed as thumbnail maps and their attribute names. Multiple thumbnails can be popped up (pop up window) to an image and some key statistics of mean, standard deviation, minimum and maximum value in order to obtain a feeling for the data. The thumbnails can be dragged and dropped into the maps field. The spatial view panel (to the right) is a viewer that uses OpenLayers, and shows the transparent output map of well (green) and poorly (red) performing areas on top of base layer. Here it is OpenStreetMap. Clicking on one of the polygons, attribute names and values for that polygon will be displayed in the Spatial Info panel (lower right corner). 

\begin{figure}{
{\includegraphics[width=\linewidth]
{boerboom/figure1}}}
\par \caption{Screenshot of the Distributed Spatial Multi-Criteria Evaluation web application}\label{fig.boerboom_fig1}
\end{figure}

As depicted in Figure \ref{fig.boerboom_fig2} the architecture consists of three main components: Spatial Services, Decision Support Services and finally the D-SMCE service. Spatial Services consist of data services such as WFS and data processing services such as WPS. In the decision support service component we have the Decision Deck as a multi-criteria aiding service. D-SMCE itself is a client of all those external services and plays a role of a mediator service to bridge two domains (Spatial Domain and Decision Support domain) to perform required tasks of a Spatial Multi Criteria Analysis. It is implemented on the Java platform and uses modular design following OSGi specifications. 

If we look inside the D-SMCE component there we have two services as well. We have the backend service. In this backend service we implement data access and business layers. Here we have modules of several client implementations to access external services for data retrieval and data processing. Also we have some other modules for required calculations that external services cannot offer. Then there is the frontend service. This service consists of the presentation layer together with some utility modules that are needed by the web application (e.g. user profile management, access control). The presentation layer is implemented with the Vaadin framework.

Our idea is to clearly separate these two services, backend and frontend, by running them as two independent services. If a necessity arises, we may like to add a new component to this architecture such as statistics service by implementing a required client in the backend service, as it is depicted in the bottom of the figure.

\begin{figure}{
{\includegraphics[width=\linewidth]
{boerboom/figure2}}}
\par \caption{Architecture of Distributed Spatial Multi-Criteria Evaluation}\label{fig.boerboom_fig2}
\end{figure}

The core of this new web-application is functionally and technologically innovative. Functional innovative aspects are:
\begin{itemize}
\item SMCE on the web. Spatial data is currently shown and sometimes downloadable on internet. A map can be looked at one at a time. With DSCME multiple maps can be viewed, interpreted, and aggregated to perform spatial evaluation.
\item Integration with MCDA web services. (fig 2).We will apply non-spatial multi-criteria decision analysis web services standards (XMCDA) (Decision Deck, 2011) in the spatial domain. Weighted summation MCE is the first method but once embedded others can be added. 
\item Spatial data from different locations on internet can be brought to the web-application
\item Distributed calculation. If data providers do not want data to be moved, parts of the backend of the DSCME can be served from different locations where parts of a criteria tree can be analyzed in different web locations and results shared to a “central” location. 
\end{itemize}

Technological innovative aspects are: 
\begin{itemize}
\item Extensibility of services. DSMCE is a service oriented platform that mediates the interaction between spatial and non-spatial services available on the web. This feature is unique since the majority of decision support software are either desktop or isolated web applications. This feature opens opportunities for extension of the system (such as adding statistical capabilities, etc.)
\item Open standards and communities. DSMCE uses open standards developed by communities. It uses OGC Standards (Web Feature Service, Web Processing service, and Web Mapping Service) to process spatial data and Decision Deck standards for the non-spatial multi-criteria decision analysis (MCDA) methods. The underlying philosophy is to delegate domain expertise to other implementations that are represented on the web (i.e. as web services) and build a reliable, extensible infrastructure for the mediation of all these delegated services. Therefore comparing to other software systems in the field of spatial decision making, Distributed SMCE can be positioned as a framework rather than a tailor-made application. 
\item Use of OSGi technology. (fig. 2) OSGi technology is a specification to create modular applications in the Java platform. The choice of OSGi technology for DSMCE has been a key decision. It is technically and administratively motivated. Technically, the extensibility that is gained by the use of Web Services cannot be utilized without a modular web application that mediates interaction between different kinds of web services. Possibly more important is the administrative motivation of distributed partial analysis within different organizational boundaries. Such scheme requires both modularity and convenient tools for distribution of logical components. Distributed SMCE can follow this scheme by using OSGi.
\item Contribution of own services. Distributed SMCE is not only mediating web services, but also performing intermediate computations which are not available on the web or too specific to be standardized under OGC, Decision Deck or other standards. For instance it extracts relevant information from one service, e.g. meta-data and certain descriptive statistics from maps like the maximum and minimum values, and uses this information for other services such XMCDA services. Although the computation might be specific still it may have demand as a module or as service on the web. Also for these kind of use cases, OSGi technology and the current architecture of DSMCE gives enough room and flexibility. 
\item Server side rapid development with Ajax/Vaadin. The client, i.e. User Interface (fig. 2), is based on the Vaadin Server Side Ajax UI Framework. Like all other Ajax frameworks Vaadin provides rich user experience. Vaadin has some advantages compared to other frameworks in terms of Rapid Application Development. This is essential for an open source project where 3rd party developers of Distributed SMCE would like to extend Distributed SMCE and so its web client. The choice of Vaadin has some important implications. First, since Vaadin is a server side Ajax framework, it has a fairly ‘thin’ User Interface layer that runs on web browsers of the end users. Second, browser compatibility issues are handled by Vaadin. So the developer does not have to worry whether the developed code is working with different browsers. Third, the big majority of operations, communication and security is handled in the server on a Java platform. This feature of Vaadin gives us opportunity to develop a good degree of modularity in combination with the modularity enforced by OSGi that applies to the Java platforms. Finally, since most of the current web applications are heavily based on Javascript, it is hard to modularize them by using specifications such as OSGi, or frameworks developed for imperative programming language platforms such as Java. We have chosen Vaadin because it minimizes the use of Javascript.
\end{itemize}

\section{Discussion}
With the prototype we aim to explore possible limitations and future challenges. We for instance need to experiment with performance. Some design and implementation issues we have already encountered and addressed. 
\subsection{Design Issues}
Two major design choices were made. The first was to make the web application highly modular and distributable. The second was substituting WPS for Decision Deck web services. 

\subsubsection{Data exchange restrictions}
Two challenges in the design were to handle the size of geographic data layers and the assumption that data policies require data to reside inside organizational boundaries. Both challenges violated our initial design where we wanted to rely fully on external services. But now the application had to become distributable as well in cases data services could not directly be provided. So these challenges required that the application at least partially be local (insider the organizational boundary) and created the need to design a distributed application using distributed services. 

As a solution to these challenges we consider the use of meta-data and delegating the analysis (e.g. calculation of descriptive statistics by WFS) to the data services wherever possible. And for further analysis on the dataset within organization boundaries we deploy a utility service. This service can perform analysis and then can transmit partial and intermediary results to the main system.

Using OSGi technology gives this possibility of designing a highly modular application. Another advantage are its “Remote Services” (OSGi Alliance, 2011) for distributing modules over the web. We aim to use “Remote Services” of OSGi to be able to distribute our necessary modules across organizational boundaries to give us opportunity to retrieve only meta-data and partial and intermediary results to finalize the analysis. 

\subsubsection{Finding proper service for decision aiding}
At the start of the project we considered to implement the multi-criteria decision aiding components as Web Processing Services (WPS). However, we encountered several challenge about WPS in relation to our project. One challenge was already formulated by (Friis-Christensen et al., 2007) as the absence of “separation of geometry and attribute data: Geometry information, though not required for a large number of processing operations (like classification and attribute normalisation) is dragged along as information ballast slowing down the performance of applications. Examples for specifications looking into this issue are the related OGC discussion papers on the Geolinking Service (OGC, 2004b) and the Geolinked Data Access Service (OGC, 2004a).” Indeed in our application the majority of use cases required only attribute data of features to be processed and analysed and the geometric information served just the mapping. These attributes are, partially in pre-processed form, input to XMCDA web services. So we abandoned our initial idea to implement all analysis as WPS. 

Another reason to abandon WPS was the lack of concepts to provide semantics of the multi-criteria decision aiding domain. This problem of a lack of concepts to provide semantics was already observed by (Foerster and Stoter, 2006). This issue together with the fact that WPS provides very generic interface, would have required us to spend considerable effort to implement decision aiding algorithms for spatial data in WPS. 

Therefore we looked at an alternative solution where we could separate analysis of attribute data from analysis of and operation on geometry. And we could also find web services that provide decision semantics. We found a solution in the decision sciences domain where open standards for multi-criteria decision aiding (MCDA) web services have recently been developed in the Decision Deck Project. Now we only consider WPS for truly geometric pre and post processing operations such as overlay analysis or calculation of spatial metrics as criteria.

Finally, we abandoned WPS for decision aiding algorithms because of the possibility to process large volumes of data because many criteria maps may be involved and more advanced decision aiding methods are more complex and resource consuming too. As discussed by (Michaelis and Ames, 2009) in such situations it can be more efficient to perform the processing locally. 
\subsection{Implementation Issues }
We have faced three implementation issues. First, we have had to address inconsistencies between schema and schema instances. Second we have had to address lack of information about the data. Here we do not mean so much meta-data, but descriptive data. And third we missed open source GIS toolkits with proper documentation. 

\subsubsection{Inconsistencies between schema and schema instances}
We noticed that retrieval of data (or meta-data) becomes fragile because of difference between WFS schema and WFS schema instances or because of missing schemas while retrieving complex types. Since the data retrieval in DSMCE is made from remote WFS servers, which we do not have control of, human errors or bugs in OGC Service implementations can cause bad user experience.

As a solution, we implemented parsers that use a domain model which is a collection of Java objects based on OGC Web Service Common and Web Feature Service specifications. We implemented with Apache Commons Digester (Apache Commons Digester, 2011) library. Although this approach requires implementation of java objects following the domain model, it gives nice flexibility and more tolerance for errors. For retrieval of the data we preferred to use Geo JSON format since it is a lighter format comparing to GML and accessing attribute data is easier.

\subsubsection{Data ambiguity}
Another main problem is the lack of support for units of attributes and descriptions of attributes. In GML3, schemas for units are defined (Cox et al., 2002), however this information is not being used by the available online Web Feature Services yet. But decision makers will need to know such attribute information  and the currently supports name and type elements are not sufficient for a decision maker. Moreover, most of the time, the name field is cryptic and not explanatory either. 

We have not addressed this issue of data ambiguity in the current prototype. For those cases where WFS are used that the user has no control over, we will be providing annotation tools to the decision maker (end user) units and data descriptions assuming the user has other means to obtain units and data descriptions. For those cases where users of WFS are in the same organization as the WFS supplier, custom solutions could be made to create unambiguous data interpretation. 

\subsubsection{Lack of WFS support for descriptive statistics about the data}
The general user of WFS, but certainly the user of WFS via DSCME, needs descriptive statistics such as the maximum, minimum, mean, and standard deviation. For instance of the standard deviation of a certain attribute is small, it has little discriminatory value between the alternatives. Particularly in spatial data sets where the number of alternatives (points, lines, polygons, cells) can be very large, descriptive statistics are very important. So although a decision maker may find the criterion that uses the attribute important, if it is not discriminating, it might even be discarded altogether. But also for properly styled WMS visualization, not only in DSMCE, maximum and minimum values are important. 

We expected more support from OGC data services (WFS, OGC). Descriptive statistics are optional services in section 13.3.2 of WFS 1.1.0 specification (Vretanos, 2005): "The schema of the Filter Capabilities Section is defined in the Filter Encoding Implementation Specification. This is an optional section. If it exists, then the WFS should support the operations advertised therein. If the Filter Capabilities Section is not defined, then the client should assume that the server only supports the minimum default set of filter operators as defined in the Filter Encoding Implementation Specification."

Since these descriptive statistics are crucial for decision making, we implemented a simple statistics facility by fetching the whole feature set and using Apache Commons Math library to compute descriptive statistics. We considered checking if a filter is available from a WFS in the capabilities response, but this burdens our system with complexity of checking and error handling. Although we have a working solution, it breaks with our initial idea of using the meta-data and capabilities of external services prior to the core analysis.

\subsubsection{Lack of lightweight open source GIS toolkits with proper documentation}
As described under section design issues, the majority of operations in DSMCE only use attribute information, not geometric information. However in the available open source GIS toolkits, data structure designs are naturally affected by the traditional structure of GIS data where features are a composition of geometry and attributes. We also noticed many inter-dependencies between libraries and lack of documentation about dependencies. So it becomes really hard to use toolkits for our lighter needs and we did not find a lightweight GIS toolkit which is efficient for attribute data and helpful for simple mapping. These difficulties and poor documentation motivated us to implement lightweight OGC service clients for WFS and WMS. For that purpose we used Apache Commons HTTP (Apache Commons HttpClient, 2011) library and to be able to create POST requests with XML encoding we used WAX library for JAVA (Volkmann, 2011).

\section{Future directions }
Since we have finished only a prototype so far, a lot can still be done: 
\begin{itemize}
\item First, we want to modularize our systems with OSGi and run it in an OSGi container. 
\item Second we need to create state persistency, user profiles and workspace.
\item Third, we want to proceed with the integration of Decision Deck to provide a good amount of multi-criteria decision aiding algorithms. 
\item Fourth, we want to add preprocessing WPS so that users can create a suitable criterion map from a geometry of another map or of geometries of different maps. 
\item Fifth, we have not addressed the issue of discovering data but evaluation and use of OGC Catalogue Service is in our agenda. If DSCME becomes an application that runs within organizations it will need to be customized to use the organizational catalogue. 
\item Sixth, because of the challenges in data formats and available tools that support Web Coverage Service (WCS) we started our project with vector support. However we know of very good experience of usefulness of raster-based SMCE with the SMCE module we developed earlier in the desktop ILWIS GIS (52North, 2011) and would like to include a raster version in the agenda. 
\item Seventh, we want to develop the potential for collaborative decision making and explore new decision aiding algorithms.
\item And finally, we know chaining services and managing it by the use of workflow managers is very interesting. To be able to satisfy different and complex scenarios in decision making we would like to develop a workflow mechanism for our application. We believe prior to that we need a good degree of modularity in our application.
\end{itemize}

\section{Conclusions}
We have presented a prototype of distributed spatial multi-criteria evaluation web application, which integrates OGC and Decision Deck web services, thereby delegating functionality to the respective expertise domains. It is distributed, not only in the concept of distributed computing,  because it can collect data from distributed data sources, i.e. Web Feature Services (WFS), or, if data cannot be exchanged because of data policies, data value, bandwidth, and other reasons, it can be distributed to these data sources and collect only the intermediary outputs. Also, decision makers are geographically distributed or in time. And development of it can be distributed, given the open source nature, provided good programming practices and systems are maintained.

We have described its workings and architecture. But importantly we have explained several design and implementation solutions which we had to follow because of partially functioning implementations of OGC standards. We are offering anecdotal evidence of shortcomings of these standards but also of open source software. It would be worthwhile to do a more systematic analysis but that is beyond the scope of this paper. 

\section{Acknowledgements}
The ForeStClim project has received European Regional Development Funding through INTERREG IVB NEW. 

\section{References} 
\begin{enumerate}
\item 52NORTH. 2011. ILWIS Community [Online]. 52North. Available:  http://52north.org/ communities/ilwis [Accessed July 2011].
\item APACHE COMMONS DIGESTER. 2011. Commons Digester 3 website [Online]. Apache Commons TM. Available:  http://commons.apache.org/digester/ [Accessed July 2011].
\item APACHE COMMONS HTTPCLIENT. 2011. Apache Commons HttpClient website [Online]. Apache Commons TM. Available:  http://projects.apache.org/projects/ commons\_httpclient.html [Accessed July 2011].
\item BAUD, I., PFEFFER, K., SRIDHARAN, N. \& NAINAN, N. 2009. Matching deprivation mapping to urban governance in three Indian mega-cities. Habitat International, 33, 365–377.
\item BOERBOOM, L. G. J. 2010. Integrating Spatial Planning and Decision Support System Infrastructure and Spatial Data Infrastructure. International Journal of Spatial Data Infrastructures Research (IJSDIR), (accepted with revision:  http://ijsdir.jrc.ec.europa.eu/index.php/ijsdir/ article/viewFile/223/269).
\item BOROUSHAKI, S. \& MALCEWSKI, J. 2010. ParcitipatoryGIS.com: A WebGIS-based collaborative multicriteria decision analysis. Journal of the Urban and Regional Information Systems Association, 22 23-32.
\item COX, S., DAISEY, P., LAKE, R., PORTELE, C. \& WHITESIDE, ARLISS 2002. OpenGIS Geography Markup Language (GML) Encoding Specification. Report no. OGC 02-023r4. Open Geospatial Consortium, Inc.
\item DECISION DECK. 2011. Welcome to the Decision Deck Project website [Online]. Decision Deck Consortium, Ecole Centrale Paris. Available:  http://www.decision-deck.org/contact.html [Accessed July 2011].
\item FOERSTER, T. \& STOTER, J. 2006. Establishing an OGC Web Processing Service for generalization processes. Workshop of the ICA Commission on Map Generalisation and Multiple Representation. Portland, United-states: International Cartographic Association, Commission on Map Generalisation and Multiple Representation.
\item FRIIS-CHRISTENSEN, A., LUTZ, M., OSTLÄNDER, N. \& LARS, B. 2007. Designing Service Architectures for Distributed Geoprocessing: Challenges and Future Directions. Transactions in GIS, 11, 799–818.
\item HE, J., CHEN, N., WANG, W. \& CHEN, Z. 2009. A Dynamic Schema Matching Approach for Multi-Version Web Feature Service Retrieve.
\item HERWIJNEN, M. V. 1999. Spatial Decision Support for Environmental Management, Amsterdam, Free University Amsterdam.
\item KESHKAMAT, S. S., LOOIJEN, J. M. \& ZUIDGEEST, M. H. P. 2009. The formulation and evaluation of transport route planning alternatives: a spatial decision support system for the Via Baltica project, Poland. Journal of Transport Geography, 17, 54-64.
\item MALCZEWSKI, J. 2006. GIS-based multicriteria decision analysis: a survey of the literature. International Journal of Geographical Information Science, 20, 703-726.
\item MICHAELIS, C. D. \& AMES, D. P. 2009. Evaluation and Implementation of the OGC Web Processing Service for Use in Client-Side GIS. Geoinformatica, 13, 109-120.
\item NIJKAMP, P. 1990. Multicriteria Analysis: A Decision Support System for Sustainable Environmental Management. In: ARCHIBUGI, F., NIJKAMP, P. (ed.) Economy and Ecology: Toward Sustainable Development. Kluwer Academic.
\item ODPM 2004. The English Indices of Deprivation 2004 (revised). London: Neighbourhood Renewal Unit, Office of the Deputy Prime Minister.
\item OGC 2004a. Geolinked Data Access Service v0.9.1. Wayland, MA: Open Geospatial Consortium.
\item OGC 2004b. Geolinking Service v0.9.1. Wayland, MA: Open Geospatial Consortium.
\item OSGI ALLIANCE 2011. OSGi Service Platform Core Specification, Release 4, Version 4.3. OSGi Alliance.
\item PERCIVALL, G. 2010. The application of open standards to enhance the interoperability of geoscience information. International Journal of Digital Earth, 3, 14-30.
\item SHARIFI, M. A. \& BOERBOOM, L. G. J. 2006. Spatial multiple criteria decision analysis in integrated planning for public transport and land use development study in Klang valley, Malaysia. In: Proceedings of Vol. XXXVI, part 2. Technical Commission II, ISPRS Vienna 2006 symposium, 12-16 July 2006, Austria / ed. by W. Kainz and A. Pucher. Vienna : ISPRS, 2006. pp. 125-130 This paper has been published earlier: In: Proceedings of the ISPRS midterm conference, Commission VI, WG VI/4, Theory and concepts of spatio-temporal data chandelling and information, Vienna, Austria, 10-14 June, 2006. 7 p.
\item SHARIFI, M. A. \& RETSIOS, V. 2004. Site selection for waste disposal through spatial multiple criteria decision analysis. In: Journal of telecommunications and information technology, (2004)3, 11 p.
\item STIRLING, A. 2006. Analysis, participation and power: justification and closure in participatory multi-criteria analysis. Land Use Policy, 23, 95-107.
\item VOLKMANN, M. 2011. Writing API for XML (WAX) [Online]. Object Computing, Inc. (OCI). Available:  http://java.ociweb.com/mark/programming/ WAX.html [Accessed July 2011].
\item VRETANOS, P. A. 2005. Web Feature Service Implementation Specification, Version: 1.1.0. Report No. OGC 04-094. Open Geospatial Consortium Inc.
\item ZUCCA, A., SHARIFI, A. M. \& FABBRI, A. G. 2008. Application of spatial multi-criteria analysis to site selection for a local park: A case study in the Bergamo Province, Italy. Journal of Environmental Management, 88, 752-769.
\end{enumerate}
\end{article}
