State of GeoServer 2012

And now for the Latinoware 2012 presentation people actually came to see - the State of GeoServer 2012. Once again the content is CC by Attribution and build on earlier talks.

State of GeoServer 2012 from Jody Garnett

The talk raised a lot of questions, both directly after the talk and in the breaks between presentations. The questions all came down to catalog service web support - and what it means for GeoServer 2.3.

I also got to play the careful balancing act between representing GeoServer (as PSC) and representing the Open Source Geospatial Foundation (ie product neutral). There is still a lot of confusion around how to evaluate and select open source projects for an successful engagement.

The most appropriate course of action is to engage with the local community - a major strength of the OSGeo Chapter setup.

Once again the important status update is the release of GeoServer 2.2. This release was a long time coming and improves major headline features, and important changes under the hood.

Recent activities covered by the presentation:

• Time Boxed Release once again this a deep change that will effect the developer community, our customers and how end users of the application work with GeoServer.
• Catalog Service a very simple catalog service, used to publish the GeoServer contents out via CSW. This should allow for easy harvesting by full featured products such as GeoNetwork. The initial service is working, but I expect more funding will be required based on the enthusiasm shown online and at events like Latnioware.
• Sensor Observation Service currently under construction
• OSGeo Incubation

From Latinoware 2012

WPS Personality - Chris Tweedie

One of the pleasures of taking in the Spatial@Gov conference was a chance to catch up with Chris Tweedie. I first met Chris when he was an employee for Landgate over in Western Australia and he served as quite an advocate for open standards and open source. Chris has been picked up by ERDAS and is thus emphasising the open standards half of that equation.

Chris Tweedy

Chris was kind enough to demonstrate the ERDAS support for web processing service. As always I am impressed with the ease of use provided by integrated solutions.

The ERDAS browser client was impressive and easy to use; although Chris does not normally demo this product we were quickly able to figure out what the screens were asking of us and collect "Elevation Change Defection" results back for display.

When we got down to technical details ERDAS had reached the limits of the WPS specification. The WPS DescribeProcess data structure does not supply quite enough information for their user interface needs (example field validation).  I hope the future versions of WPS will be more helpful in this regard.

That said they were not limited to their WPS server; and had performed testing with either 52N or deegree (sorry I cannot remember which as they did not have one in their booth to test against).

I also recognised an old friend in the NDVI vegitation model, a classic we had slated for last years WPS shootout.

Aside: Thanks to LISAsoft for sending me to Spatial@Gov it was a fascinating look at the Australian Geospatial scene.

WPS Personality Markus Schneider

One contributors to the WPS-Shootout that was not able to attend in person was Markus Schneider from the deegree project.

I have a small association with deegree acting as their mentor during OSGeo incubation. Indeed deegree is the only project taking part that sports the OSGeo stamp of approval.

In this case indicating the project has their open source license story in order and follow an open development process.

The recently released deegree 3 supports WPS giving them an opportunity to participate in this years shootout. I was able to catch up with Markus Schneider and ask him the same questions covered in the panel discussion.

Introduction

Markus Schneider is part of the core deegree team; working at Occam Labs / lat/lon. He has been working as an open source geospatial developer for the last 10 years.

Q: Tell us about the deegree community?

Markus: Our mission is to provide an advanced and thorough implementation of OGC web services (WMS, WFS, CSW, WPS, WCS, SOS) and a geospatial base library that covers the relevant OGC standards (GML, SLD, SE, FE, ...). It's for those who seek a framework that doesn't stop at simple features and handles the complex stuff (e.g. complex application schemas) as well.

Q: What interested you in Web Processing Service?

Markus: Well... as often with OGC specs, I am "impressed" with the abundance of options that the specification team managed to put into it ;-) The most cool thing is probably that you can process the streamed output from other OGC services (e.g. WFS) and even pipe it through several WPS processes.

Q: Example of a deegree Success Story?

Markus: For instance, there's a comprehensive coordinate transformation service powered by deegree WPS out there (at the Central Basic Geodata Service for Germany).

Q: Shootout Results?

no comment!

Q: Anything else?

Marks: What we're really proud of is the almost 100% coverage of the standard and all options, as well as the scalability -- due to it's streaming architecture, there shouldn't be any general limits on the size of processed inputs/outputs.

Q: Future plans for deegree WPS?

Markus: deegree 3.1 is scheduled for October 2011 (currently we're at 3.1-pre-13). 3.1 will bring WFS 2.0 and other stuff, and the WPS got a new way of deploying processes without that need for restarting the whole webapp.

WPS Shootout

Thanks for attending the WPS Shootout. The slides are available below:

WPS Shootout

View more presentations from jgarnett.

You should be able to click through and download the presentation if needed (perhaps your browser is modest and does not approve of flash).

Tips!

• The slides have been updated to reflect the questions asked during the presentation
• The tests results are directly available from the slides by clicking on the *conformance* links

If you would like to skip the talking bit and go straight to the good bits I have dragged out the following slides.

PyWPS

You can review the request/response for PyWPS:

• PyWPS test script results

In this case the results show an small mistake which we expect will be fixed shortly.

deegree

You can review the request/response for deegree:

• deegree test script results

52N

You can review the request/response for 52N.

• XMLSpy shows 100% success
• waiting on the test script fix

Zoo-Project

You can review the request/response for Zoo-Project:

• Zoo-Project test script results

GeoServer

There are no Request / Response samples for GeoServer.

Constallation

There are no test results as the constellation project (as it under development).

Happy Trails

We will be packaging up these results as a "white paper" for OSGeo; thank you for your support and encouragement in 2011.

Until then "Happy Trails!"

WPS Personality Simone Giannecchini

Simone Giannecchini was a surprise last minuet addition as Andrea could not join us. Simone brought a nice business perspective to the wps-shootout and was very focused on the steps needed to take GeoServer WPS to production quality.

The GeoServer project is well known for its WMS, WFS and WCS implementation. What is little known is that a WPS "extension" has been available on and off since 2008. Originally started by Refractions in 2008; work was maintained by the community (Andrea and Jody) with GeoSolutions really taking over the reins in 2011.

GeoServer was not able to take part in the wps-shootout due to time constraint (no time to set up a server for others to test against). As such I would like to thank a couple wps-shoout members who were able to pitch in so that GeoServer could be represented:

• Gérald was kind enough to test GeoServer out 2 hours before the presentation!
• Jachym was kind enough to test GeoServer against several WPS clients

Finally thanks to Simone and GeoSolutions for representing the GeoServer community in person and on stage.

WPS Personality Gérald Fenoy

I would love to tell you more about my co-presenter / organiser for the wps-shootout - Mr. Gérald Fenoy. Only trouble is I cannot find him. Why is this you ask? Well I think it comes down to Zoo-Project being very social.

The Zoo-Project is easily the most widely dispersed in terms of development teams located around the world. As a consequence of this they have done an amazing amount of promotion of the Web Processing Service standard and really raised the profile of WPS internationally. Their project is also very inviting allowing process developers to work with their choice of languages; with the result being served up from an engine written in C++.

What I can do instead is describe some of the hard work he has done for the wps-shoot out. When I manage to upload the slides from the wps-shootout you can get a sense of the effort Gérald has gone to by clicking on the "links" in the test results as shown below.

The website pages list all the checks performed; and give you a chance to review the request/response and any validation problems encountered.

Thank you Gérald for making this years wps-shootout a success.

WPS Personality Bastian Schäffer

Another WPS personality for the shootout this year is Bastian Schäffer. I actually had a bit of fun catching up with Bastian - and learned a bit about him and 52N as a company.

Bastian fell into this geospatial processing work in the course of his masters thesis at University of Muenster. He joined the 52N community in 2007 as they struggled with WPS 0.4 specification.

As an organisation 52N operates in collaboration with the university with a mandiate to push concepts and ideas into production. This launched Bastian into work with the OGC and he had many ideas on how the WPS specification could be improved in the future. I the wps-shoot experience can help highlight ways to improve interoperability.

I would like to thank Bastian for his participation / motivation / and help in making this years wps-shootout such a success.

WPS Personality Vincent Heurteaux

Vincent Heurteaux is part of the team responsible for the constellation project - which apparently is going to support WPS when it comes out later this year. Very exciting to see another player enter the ring.

Vincent is the one in the middle trapped between two Martins; something that only happens at FOSS4G.

To meet Vincent or any of the other WPS personalities come by the wps-shootout Thu 4:00pm.

WPS Personalty Jachym Cepicky

This afternoon I managed to hunt down Jachym Cepicky best known as the author of PyWPS.

I had a great time talking to him; and plotting the wps shootout. A couple really good ideas came out of the conversation.

• He was able to look into the existing OpenLayers code for parsing a WPS Capabilities document for me
• I made a bit of a discovery - he has been working on the WPS client side of the fence with a Javascript client.
• Some interesting history into the PyWPS project
• Some strong words about raster support in WPS being where the real heavy lifting is to be found

I was overjoyed to find another developer who (although separated by a Java vs Javascript divide) could talk about he experience of working as a general WPS client trying to avoid any server specific assumptions.

If you are interested in learning more, and some of the suggestions we have for working on your WPS client code; stop by the WPS Bird of a Feather Session Wednesday at 5pm. You can sign up on the wiki link if you would like to take part and add to the list of ideas.

Catching up on Web Processing Service

Make no mistake web processing service is once again a hot topic at this years foss4g with 8 different workshops/presentations/tutorials for you to catch up on the current state of the art.

One thing that makes this so exciting is that their are many different project teams; and they are are playing very hard right now. The projects are rapidly improving and which one is the best fit for your organisation is a real matter for consideration.

With that in mind I would like to offer a small supplement to the program:

• Thursday 4:00 WPS-Shootout has the following listing: Presenters: Jody Garnett, OSGeo; Gérald Fenoy, GeoLabs SARL

While Gérald and myself have helped set things up the truth is much more exciting than just listening to the two of us speak. We actually have rounded up everyone!

• 52N - Bastian Schäffer
• deegree (not represented ... but I will try and have some details from email)
• GeoServer - Simone Giannecchini
• PyWPS - Jachym Cepicky
• ZOO-project - Gérald Fenoy
• Constellation - Vincent Heurteaux

The format is a panel discussion with time devoted to Q&A (so you can start thinking of a nice awkward question to ask now). There is however a western theme (since this is the west and it is a shootout) so try for a good western accent.

Start your WPS Services

Now that I have made contact with the different groups a couple things are clear:

• Everyone is very enthusiastic about having additional clients to talk to (this is the traditional chicken and egg problem with standing up either a client or a service - you need a friend in order to have a conversation - or a chicken I guess)
• Demo services are available for anything not under active development (this is great news for me)
• While deegree 3 WPS is still technically under development they are producing a downloadable war making their service easy to test
• GeoServer documentation needs some work (sad news for me but I can fix it)
• PyWPS was recommended

Today's task is to connect to each service and make sure I can parse the capabilities document; and if things go well the describe process documents.

52 North
52 North have been very supportive with both a stable service to test against, and in standing up a service from their development branch. Their development branch makes use of GeoTools 2.6 and I am keen to hear how their transition went. Recently we have made some usability improvements for GeoTools 2.7 which will make those updating older applications even easier.

• http://giv-wps.uni-muenster.de:8080/wps/WebProcessingService

Thanks to Bastian for setting up the a server using the development branch.

ZooWPS
The ZooWPS mailing list got back to me today and quickly pointed me to both examples and a sample WPS service I can test against.  The examples confused me a bit (as the skip straight to the execute requests and rush over the whole capabilities and describe process steps).

• http://shilpa.media.osaka-cu.ac.jp/zoo/?Service=WPS&Request=GetCapabilities&Version=1.0.0&Language=en-CA

Thanks to Nicolas for the sample server to test against.

While the examples are confusing (and show a danger of just using links for data) they have a simple great picture explaining how their WPS functions.

deegree
Deegree makes a number of demonstration services available for testing; but they make use of an older 0.4 version of the WPS specification (and my dedication to standards compatibility has a limit). The new deegree 3 is implementing WPS 1.0 and has a war available that fits my needs.

• http://wiki.deegree.org/deegreeWiki/deegree3/WebProcessingService

(can I capitalise deegree if it is used to start a sentence ... or is it like "iPod" and the shape of the word matters more then silly english sentence conventions?)

GeoServer WPS Community Module

While I can quickly produce a war of the community module; my preferred method of testing is to use a very lightweight application server called "Jetty". Indeed use of Jetty is rolled into the maven build system:

cd web/app
mvn jetty:run -Pwps

The only difficulty is that the build tool maven has grown a bit responsible since I last used it and will no longer install plugins such as jetty without me modifying a couple of configuration files first. I am going to sort out what is needed and update the GeoServer docs later today.

PyWPS
A recommendation from yesterday (thanks Tom) which appears to be a contemporary of deegree in terms of years of experience. One thing that really attracts my eye when looking at a new project is:

• recent news (showing that the project is alive)
• documentation (even better if it is called course material)

And guess what is on the PyWPS home page?

2010-05-05 New course material added

New course material added to PyWPS source. See documentation for details.

I will sign up to the email list and try and introduce myself shortly.


uDig (ie client)
The other thing I am working on is user interface ideas to present the idea of external processing. My thinking thus far is to cheat - and represent external results (and if processing is still going use a progress bar as a placeholder).

If I consider it as a list of results it becomes a more interesting and productive user interface concept:

• results can be "tagged" to define ad-hoc grouping according to server, process, processing status
• results that were produced externally (such as to an ftp site) can be listed, and downloaded if needed
• using a wps could be considered "adding" a result to the list and handled using a wizard (although a wizard is not the best for interacting with the map - such as selecting a calculation area)
• I should be able to record the steps that were used to produce each result and "rerun" if needed

Web Process Service Round Up

I have a fun bit of work lined up - updating the web processing service client code in uDig.

It is no secret that I am a huge fan on the idea of Web Processing Service - I am excited about the possibilities in using a WPS as a front to a grid of computers (a strategy 52North seems to be pursuing), the ability to bundle up processes written in a number of languages (something ZooWPS is really going after).

The part I am really keen on does not seem to be tackled yet: I am very interested in chaining processes using standard diagrams such as BPEL - this represents a really nice olive branch between GIS and the business analysts that would love to know what the department is doing). There is some confusion in this area as the diagrams end up looking similar to those provided by BI tools (since GIS is used for decision making) or similar to ETL tools (since chains of processing are required).

Today am making contact with the different web processing service implementations and warning them what I am up to and generally finding out where they live and what is a good contact point for communications.

Thus far:

• 52North - 30 mins to respond to email, seems to be very active and able to link to an example WPS service right out of the gate. This is the established open source WPS solution and I am looking forward to seeing how it handles feature collections and raster processing.
• ZooWPS - no response to email yet, but the IRC channel was well populated (turns out half the members were my LISAsoft co-workers from different offices around Australia). This is the new kid on the block in the WPS space
• GeoServer WPS Community Module - no email since I had already been following that email list. The GeoServer WPS community module has been very quiet in its development but has made recent progress in the two areas I am interested in testing.
• deegree 3 is working on their second generation WPS implementation and is under active development - I may end up building from source in order to have something to test. It is great to see the continued support of WPS here (deegree 2 worked against an earlier version of the specification).

The two areas I am targeting each have their own special risks.

Features should be the bread and butter of GIS processing and we are held back in this area by the generally hap hazard support for GML. I can see nailing everything to the wall using GML and XML Schema - this is really what should be done - (since it is a data interchange format) when shuttling data between services. GML allows us to communicate the range and limits of the data and be able to negotiate differences between data models. I could see using this approach in an ETL context or when doing scientific work.

The expectations of the current crop of implementations are in a slightly different direction: focus on geometry (hey it is spatial!) and have the attributes carried along for the ride. The ZooWPS implementation also supports GeoJason which is very good for this style of ad-hoc collaboration. Even for this ad-hoc style we will need to indicate "which" geometry in a feature needs to be acted on ... so it should be fun seeing what the different implementations have provided.

Raster data is also interesting/scary. There is an answer in place for the obvious question of data size (the WPS specification accounts for this by allowing long running processes making use of FTP sites for staging results). The other question is the same one encountered by web coverage service; what does the data mean? Which bands mean what and how is your DEM height measured etc. I am really not sure if WPS is up to capturing this information; will the file format headers capture this in enough detail; or will each process need to be supplied hints to sort out how to interact with the information.