Wednesday, 30 May 2007

Federated Geo-synchronization

Standards and tools for reliable data synchronization in Spatial Data Infrastructure and field based data collection.


This article describes the issues and technical solutions associated with Federated Geo-synchronization.

Lisasoft aims to contribute to these solutions as part of the OGC’s Open Web Services Testbed 5.2.

To strengthen and refine our requirements, we are looking for Agencies which would benefit from solutions identified here. Please leave a comment, or contact me if you are interested.

Technical Problem Statement

As spatial databases become distributed and collaboratively maintained, traditional database transaction models ineffectively handle modern scenarios.

Figure 1 Synchronising databases in a Spatial Data Infrastructure

Users require current data from remote agencies. Data may be stored on a slow or unreliable server or behind an unreliable internet connection.

Updates may come from remote field workers, trusted external organizations, or general internet users. Identities must be confirmed, updates validated and applied, or rolled back to a previous version.

Technical Design

Figure 2 Caching WFS-T in field, local and remote networks

This project will:

  • Make a Spatial Data Infrastructure (SDI) fast and robust by caching remote WFSs locally.

  • Provide WFS Synchronization to allow real time data updates between agencies.

  • Ensure interoperability between agencies and applications by proposing required extensions to Open Standards.

  • Ensure wide adoption by providing all components free as Open Source Software.

  • Provide desktop and mobile, field based data collection tools.

Version WFS-T

A Transactional Web Feature Service (WFS-T) provides an OGC standards compliant web interface for downloading and updating vector features over the internet. To date the WFS-T standard doesn’t address version history.

Versioned WFS-T enables users to roll back to previous versions, track update history, check differences between updates. A versioned WFS is required to support a cached WFS.

Version WFS-T Development

Geoserver developers have developed a Versioned WFS-T by extending the WFS-T specification to include standard version attributes. As at May 2007, the Geotools version code is complete, but still in alpha state. It requires configuration web pages to ease operator use, packaging into a release and real world testing.

The extensions to the WFS-T specification need to go through the OGC standards process.


Security involves: authentication (to verify who a user is) and authorization (to specify what a user can view or update).

Security Development

Geoserver has prototype authorization and authentication code. Access is provided to the level of WFS. Granular access to a layer or a specific feature is not supported. The code still requires refinement, a user interface and integration with the baseline.

Clients like Udig, Mapbuilder and OpenLayers require security logic. Some of this will be addressed during the Canadian Geographic Data Infrastructure Interoperability Pilot, due to complete October 2007.

Cached WFS-T

A Cached WFS mirrors a remote WFS locally. A Cached WFS-T also caches WFS-T updates when disconnected from the remote WFS.

A Cached WFS-T is used when:

  • The remote WFS uptime is not guaranteed.

  • The remote WFS connection is unreliable or unable to handle traffic required.

Cached WFS-T depends upon the Version WFS-T protocol.

Cached WFS-T Development

An alpha version of Cached WFS (read only) is implemented by Geoserver. A friendly user interface is required to bring this to COTS quality.

Minimal development is required to implement Cached WFS-T (with writes) which has a basic conflict management interface.

Business rules for managing updates and conflicts from disconnected clients will be addressed in a second development phase.

Desktop Mapper

Figure 3 JGrass desktop mapping application

Desktop Mapping offers powerful data manipulation and analysis. There are a number of clients available both proprietary and open source with varying levels of functionality and standards compliance.

Desktop Mapper Development

A prime candidate for an Open Source Desktop Mapper is UDig and JGrass which are combining forces to provide:

  • User friendly mapping interface.

  • Extensive map analysis tools from Grass.

  • Access to numerous mapping format and data sources from Geotools.

  • Extensible architecture from Eclipse

Development is required to include:

  • Embedded cached WFS-T from Geoserver (developed but requires integration and testing)

  • Embedded data store using H2 (in development)

  • Access to images in a compressed format like ECW or JPG2000.

More work is required to add:

  • Business logic, views and reports to manage collaborative editing and information from a versioned WFS-T.

Mobile Mapper

Field operators need to create or update geographic data while in the field.

A typical use case involves:

  • Download geographic data while in the office

  • Disconnect from the network

  • Modify, create and delete features and datasets. Interface with a GPS to collect feature information.

  • Synchronize changes with local or remote data-stores via Standards compliant WFS-T protocol.

Mobile Mapper - Tablet Development

Figure 4 Ultra Mobile PC with slide down keyboard

A Tablet or Ruggedized PC provides the same operating environment as a desktop PC. So the Desktop Mapper will port directly to the Tablet.

Integration with a GPS is the only extra development required for the Mobile Tablet.

Mobile Mapper - PDA Development

Figure 5 Mapping on a PDA

PDAs are often used for field work because they are cheaper and smaller than laptops. Along with smaller size they are also less powerful and have less storage capacity.

Cut down versions of Windows (Windows CE) and Java (J2ME) run on most PDAs.

Investigation is required to determine effort required to port the Desktop Mapper to the PDA and whether alternative development would be more effective.

Browser Editor

Figure 6 Mapbuilder, a browser map editor/viewer

Browser editors efficiently enable data collection from the public or remote workers.

Browser clients can also publish public map data.

Browser Editor Development

Openlayers and Mapbuilder are working together to produce Open Source, Open Standards Browser Based mapping client. WFS-T editing is supported but needs to include business logic associated with user authentication and access rights.

Open Standards

Multiple agencies tend to run multiple technical solutions. This is fine so long as they interoperate through Open Standards.

The Versioned WFS-T protocol will be presented to the OGC to be formalized as an Open Standard.

Open Source

Free tools reduce entry costs to a Spatial Data Infrastructure which will maximize participation.

Open Source software already provides the majority of the functionality required by this project which means tools can be built for minimal cost.


Essential Deliverables

Essential Deliverables are required to meet immediate customer needs. These phases are low risk as the functionality already exists in tested or prototype code.

Phase 1: Mirror remote WFS locally

  • Cached WFS (view only). Builds upon Geoserver/PostGIS.

  • Standard UDig for WFS viewing

Phase 2: Update remote WFS-T from remote or disconnected client

  • Cached WFS-T (read/write). Builds upon Geoserver/PostGIS. Add simple update business rules.

  • Standard UDig for WFS-T editing

Phase 3: Security - Role based editing and views

  • Security added to Geoserver

  • Role based options available in UDig

Optional Deliverables

Optional Deliverables are nice to have and involve further development with associated risk.

Phase 4: Universal Client for easy install

  • UDig with embedded database (H2) and Cached WFS (Geoserver)

Phase 5: Universal Client on PDA

  • Port universal client to PDA


OWS 5.1

RFQ (5.1) Issued

May 11, 2007

OWS 5.2

Revised RFQ issued

July 9, 2007

Questions Due & Bidders’ Conference

July 16, 2007 (TBR)

Clarifications Posted

July 23, 2007 (TBR)

RFQ Responses Due

August 3, 2007

Kickoff Meeting

week of September 10, 2007

Interim Milestone

week of November 12, 2007

Demonstration Milestone

week of January 7, 2008

Final Delivery

February 18 – February 22, 2008

Commercialize product, provide support, consulting and customized solutions.

March 2008 – 2009.

Sunday, 20 May 2007

Geospatial Industry moves Web Publishing to Open Source

James Free writes his observations of ESRI MVP types moving away from ESRI servers (ArcGIS Server, ArcIMS, ArcSDE) to Open Source for web publishing. The MVPs report Open Source is faster and offers better value.
This lines up with anecdotal reports I'm hearing from developers. For instance, developers on the West Australia's Landgate project compared optimized Oracle Spatial + ESRI SDE with unoptimized PostGIS and found PostGIS ten times faster for the queries they were running.

Saturday, 19 May 2007

Open Standards & Open Source at SSC Conference, Hobart, Australia.

There was significant buzz about Open Standards and Open Source around the Terrapages/Lisasoft and OSGeo stands at the Australian Spatial Sciences Institute conference 14-18 May 2007 in Hobart, Australia.
Open Standards and testbeds
There was a lot of interest from a number of Government Departments in using Open Standards. Some departments like Landgate in Western Australia have already made major inroads into a Spatial Data Infrastructure using Open Standards and a lot of Open Source.
Opportunities to tap into international testbeds exist for Australian Industry and Government Departments. Investment in international testbeds allows best practices from Overseas to be deployed locally, and for local industry to break into international markets.
Federated Geo-synchronisation
Federated Geo-synchronisation involves synchronising a remote WFS locally so that the local WFS users don't require a stable internet connection.
Many people noted their desire to have this functionality for current needs. This is an unfunded thread in the upcoming OGC Testbed (OWS5) but I think there is enough desire to initiate Australian funding for this functionality.
Metadata collection
A few people, including Rob Atkinson from Socialchange Online discussed automating much of the metadata entry for Web Services, which can then be fed into Catalogs.
Rob talked about the need to create default Metadata schemas for different industries which services can subscribe to. There is room for a Research Project here, and backing implementation.
A number of agencies discussed their needs to comply with ANZLIC metadata standards. Again, there are opportunities for agencies to work together on this problem to create a shared solution.
Canadian Geographic Data Infrastructure Interoperability Pilot
The CGDI IP will provide widespread, cross agency dissemination of data which can be updated locally. This same use case is similar to the Federated Geosynchronisation required by the Australian Spatial Data Infrastructure and we should be looking to deploy results of this project locally.
Lisasoft is negotiating to build an integrated, browser based client for this application.
Open Web Services Testbed 5
OWS Testbed 5 is this year's major testbed sponsored by the OGC. There are significant opportunities for Australian Government Departments and Australian Industry to participate in these testbeds.
Open Source
Geoff Zeiss from Autodesk talked about the Open Source business. Unfortunately I wasn't able to make his talk, but heard a lot about it from enlightened people walking past our booth.
A high government official noted that it was important for the Australian government to promote Open Standards for interoperability and to a lesser extent, to support Open Source to promote adoption of technologies being developed.
We announced our Commercial Support for Open Source Geospatial Software. This attracted a lot of positive response. One techie noted that he would be recommending his department buy support after he left so that his applications would keep running. Another talked about having problems running Open Source applications after their key techie left. We hope Commercial Open Source Support will provide the stepping stone required to migrate to Open Source.
Open Source Education
We had a fruitful conversation with Anthony O Flaherty from South Australia TAFE. Anthony noted that some of his students were using Open Source and was happy to encourage it. We pointed out that we'd be happy to provide industry relevance to his training and would be looking at his graduates in future when hiring. In particular, we pointed Anthony at Ian Turton's Open Web Mapping Course.

Sunday, 13 May 2007

The Economic Motivation of Open Source Software: Stakeholder Perspectives

Dirk Riehle provides an excellent explanation of business behind Open and Closed Source Software:

LISASoft/TerraPages apply these Open Source business principles to Geospatial Software. LISASoft provide value using Open Source Geospatial Software.

Using free Open Source allows Systems Integrators to increase services or reduce price.

The price of dominant, closed source software reflects market demand rather than development cost. As open source projects mature, their cost undercuts closed source competitors. Dominant projects are forced to use lock in tactics or to open source as well.
Many Open Source Geospatial Products offer equivalent functionality and quality to existing dominant Closed Source products. Initial costly development is complete and Open Source offers better value for money.