Improvements to Mesh4x KML adapter ("Mesh4Maps"?)

After feedback from Where 2.0 we updated the Mesh4x KML adapter to Mesh4x to embed all versioning metadata for the items in the file itself.

This allows you to have one KML file, send it via email, copy it on USB drives, and have your team do changes on them anywhere they are. When you get their copy back again, or if they meet each other, they can use the sync utility to make sure changes are merged both ways. They can also use the sync utility to sync to a server via http, by just putting the server URL in the text box.

This week Juan Marcelo aka 'Tondat' worked on some refining touches on the adapter:

• One File: You only need to keep track of your KML file now. The sync utility will add versioning metadata as needed to the file.
• KMZ: We support KMZ which is google earth's native save format (A KMZ file is really a zip file with KML and additional resources like images or icons inside)
• Sync any KML:  You can sync placemarks, folders, styles and stylemaps.
• Folders and item hierarchies: We sync placemarks even as you move them around in the tree, or change the tree itself. Tondat tells me next week we'll separate placemark versioning from the versioning of where it is in the tree, so moving items around does not create conflict for the items.

We are also starting to update the server-side 'cloud storage' component to allow external applications to drive it. This would make it trivial to make a web app that:

• Lets you create 'shared maps'
• Lets you download the KML and you can work on it offline
• Gives you a URL to sync to with the sync utility
• Has an online web page to see the current map
• Maybe the page itself allows editing online? For example via the google maps API.

If you are interested in putting something like this together, please let us know.

Thank you to all translation volunteers for Sahana!

After an amazing work on behalf of the volunteers doing the translation from English to Burmese, we have translations for all Sahana strings, which will make a deployment of Sahana in the country reach a broader audience and be useful beyond the few foreign aid workers allowed in.

FlashCrowdSourcing

We've heard about crowdsourcing (e.g. as supported by Amazon's mechanical turk and other similar infrastructures) but what tools and approaches can help with flash-crowdsourcing (a flash-flood of crowds)? Volunteer management in disaster situations is a big part of relief tasks (there's even a Sahana module dedicated to that), and I wonder if we can take a bit more of advice from the 'real' world volunteer management into the 'virtual volunteer' world.

Our challenges were the need for fast coordination and high-bandwidth communication. We jumped onto Skype pretty often the first nights. It took time and false starts to figure out and describe the job well enough. I was ecstatic when the first translated line came in, and another one volunteers started explaining the task to other volunteers, and the speed picked up pretty fast.

We used Google spreadsheets to co-edit the master list of all translation batches and get a live chat amongst all volunteers, without having to agree on any IM technology. That feature by itself was great as it gave a chance for volunteers to see each others' online status and ask questions of each other. Without it, we would have been caught in the middle brokering every conversation!

The last 20% took a big chunk of the time. As we neared the end, minor issues on collisions and questions compounded with the 24-hour cycle of getting questions asked and answered from different sides of the planet.

Next steps for Sahana

Next steps that various volunteers will carry on in parallel:

1. Cleanup of work - fixing and editing the odd line and translation here and there
2. Re-encoding files- different Burmese fonts use different encodings. We chose to have faster translations at the expense of more tech work at the tail-end; which means we have to take the strings as entered by each translator in their preferred font and make it into something common. Coban Tun and other Burmese-speaking brave souls (e.g. from Burglish)are tackling the choice of fonts, encodings, and conversion tools.
3. Merging files - once converted the files will be able to be merged into the format required to Sahana
4. Test
5. Check-in, deploy to the Virtual Machines, which are being made available for download

If you think you can help with any of the above please let us know! And congratulations to all the translators again!

Where 2.0

Just returned from Where 2.0, an amazing conference about "everything geo".

Robert and I went there to present some of InSTEDD's work and learn as much as we could from the geo-demigods in the event, as well as to connect with folks we had crossed paths in other forums (such as Erik Hersman - aka WhiteAfrican -pic below, of http://ushahidi.org fame).

Many folks we wanted to meet were over there and had some great dialogues about how InSTEDD approaches projects. My takeaways: communicate what we are doing more often, and have clear channels to participate in design. We are taking the feedback and this week we re-opened the InSTEDD online forums on our website and a collaborative design group for GeoChat. One of the challenges with having such a great team is that things happen fast  for example, Mesh4x over 2 weeks, the KML Sync work in 5 days) and we need to consider this when working with a community of folks whose insight would make the stuff better and who could imagine new uses for the technologies.

In that spirit we started an open discussion about the direction for our technologies. The GeoChat work attracted a lot of attention. Jonathan Thompson was particularly engaged giving scenarios about position updates via Thuraya satellite phones and email. Not only he embarked with us on some interactive design there-and-then but also started contacting his buddies in the far field to ask them for feedback on the scenario. Awesome! For some reason the code in the Google Code project has fallen behind the tree we check-in to, we'll be fixing it next week.

One of the demos was about our KML Mesh4x adapter (a preview of the type of things you can do with Mesh4x). We got some good feedback on getting the versioning info embedded into the KML file, making it equally functional but more elegant. Play with it! See how you can sync one or more local maps with each other or with a cloud-based service, for example, setting up the N-way topology below (I was saving this sweet pic for another blog post but what the heck)

The morning of our talk Mikel Maron and Jesse Robbins gave a new variation their "Disaster Tech" presentation (eTech presentation from them on similar topic), and did a great job of describing the tensions that exist when high-tech stuff meets high-risk environments.

We had the honor to have our batch of talks presented by Tim O'Reilly himself. I was humbled by the reaction of the crowd to the Sahana localization to Burmese for Myanmar Nargis relief crowdsourcing effort. With the smarts, expertise, and experience in the audience, better and more efficient approaches can be invented. Here are our slides, without the associated bobbing heads:

Last but not least, this event was particularly well-put together for speakers. The O'Reilly staff was super friendly and the backstage folks very professional. I've spoken in countless events of different magnitudes, with audiences of dozens to thousands, and was positively impressed.

It's impossible to record every interaction and some things are worth blogging by themselves but hopefully this gives you an idea of what's been going on the last days.

Sahana installation poised for Myanmar disaster support

Upon the request from Lanka software, we have successfully brought up a virtualized Sahana instance.

You can see it here: https://sahana.instedd.org

(Note you may need to ignore a certificate warning to see this until we deploy a new certificate for this server)

Learn more about Sahana at: http://sahana.lk

Juan was instrumental in getting the Debian virtualized image running well on our Red Hat host OS, thank you! The compressed Sahana VM is about 300mb, which will allow a quick re-deployment of a hardened configuration in Myanmar as necessary. I think this VM would be a good asset to keep around, allowing anyone running Windows or Linux to bring up a running Sahana server with little to no effort.

Volunteer-based Sahana localization effort

There are over 30 volunteers across 4 continents working on localizing Sahana to Burmese. We ran into multiple issues, most stemming from the lack of Unicode standardization of Burmese. We are using Google Spreadsheets to coordinate the work (the embedded live chat is an amazing feature for live coordination) and folks are using mostly MS Word to do the translations, which we accumulate on a Google Groups page. Many thanks to all involved, I'm afraid to start mentioning folks by name because I'll mess things up or miss key individuals. Jesse Robbins blogged about this in O'Reilly Radar, and Bill Behrman from Stanford has worked his rolodex though which helped us get additional volunteers. Many folks at the NetHope summit had the chance to refer folks as well. Thanks!

Google Groups for localization

Translation is hard - especially for the fonts and encodings to work together. See the awesome burglish site to see what I mean... the translated docs end up having strings like tcef;u¾rsm; which is really encoded Wwin_burmese, which would look like this with Padauk  

Burglish project

One of the main issues with the localization is that it isn't just about translating strings-  there is also a need to accept input in the right format. This isn't trivial with all combinations of fonts and input methods people use, and especially not trivial on a web page that has to work in multiple browsers!

There is also ongoing work on InSTEDD's GeoChat system with usual suspects and new volunteers, preparing for a potential use in Myanmar, which is topic of another blog post entirely.

Build maps collaboratively with new Mesh4x KML adapter

A handful of months ago I met Kersten Jauer, UN Information Officer for the Central African Republic (CAR). CAR is a large country in Central Africa, surrounded by Sudan, Chad, Cameroon, Congo, and DRC; 67% of its population lives with under $1 a day and is scoured by constant internal rebellions and gender-based violence.

Kersten spends a lot of the time in the field in CAR, and put together an amazing map of the whole country to support logistics and NGO programs. Roads, provinces, bridges, fuel pumps, it all got captured by hand in Google Earth and saved as KML files. By the time I got it, Kersten's KML had grown to be 11 MB, an amazing amount of information patiently collected and edited, and periodically shared online with all those working to improve the region.

Contrast the map above showing the CAR KML with the map on the right showing the same region as seen by default in Google Earth. What got my attention was a little note in the KML:

If you would like to comment on this file or have suggestions please email to MapsAndGoogleEarth+car@hcpt.jot.com

To add a placemark just email it with a short description to the same address or kersten.jauer@undp.org

Please also check out the maps section on http://hcpt.jot.com

The map was built collaboratively, but imagine the workload Kersten must have had getting little snips, integrating them on the larger map, and then letting folks know of updates. And how would the map be maintained whenever Kersten was attending to some emergency?

Mesh4x KML Adapter

We started building a simple instance of a KML adapter for Mesh4x this week. This adapter would allow a team of people edit a KML file and then 'synchronize' it with all the others. For example, I could add a pushpin saying a bridge is down, and you could be editing another pushpin or moving it around to represent that a logistics truck has moved. When we synchronize, the truck moves around in my KML and the broken bridge appears in yours.

This could be synchronized peer-to-peer (a KML on your disk to a KML on a USB drive or someone else's box) as well as via a 'cloud' web service. Note this is changing the data inside the KML, it is not just 'file sharing'. The adapter knows about KML and keeps track of versions of fine-grained elements (pushpins, placemarks, polygons) inside the same file. It is an example of how a data mesh could be used to synchronize fine-grained data between applications.

We chose KML for this adapter as it is a standard ("OGC KML") that is widely used and supported by Google Earth (of course), Microsoft Virtual Earth, as well as nice tools that work offline and can be used in the field such as GeoPDF.

We have a sample UI (shown here) to let you play around with the basics. The effort is still on the libraries and we don't have a neat UI to let you choose endpoints or resolve conflicts, but all will come in due time. Other restrictions include having to put your placemarks in a "Shared Items" folder in your KML, and styles don't get replicated. We foresee no problems working out these constraints over the coming weeks.

To try it out, make sure you have Java installed and:

1. Get the sample application from http://code.google.com/p/mesh4x/downloads/list
2. Double click on mesh4j-KML-DemoApp.jar
3. Point to a KML or open the sample ones
4. Edit the location of Sample Pushpin 1 in File 1
5. Add a new pushpin in File 2
6. Press synchronize, and after both files should have the updated Sample pushpin 1 AND the new pushpin!

Another advantage of a data mesh is that endpoints can be heterogeneous, as long as you do the appropriate mapping. Eventually you will be able to sync a spreadsheet with columns such as Title/Description/Lat/Long into KML pushpins and back quite easily.

We hope to be showing this at Where 2.0. A lot of the team has been focusing on supporting the Myanmar disaster relief, so progress this week has been a bit random, but we still want your feedback!

Learn more about Kersten's Work in CAR at www.hdptcar.net, or get Kersten's latest epic KML.

See the Mesh4x project at http://mesh4x.org

Mesh4x adds generic database support

First of all - a very heartfelt support to the Myanmar population in this times of crisis. Many friends are either already there or on their way to help as part of UNDAC teams. It's a tough situation in a tough context, and all my hopes reach out to the communities there so they can recover soon. Unfortunately, it won't go back to "normal" for a long time, if ever. I was in Peru last week and the August '07 earthquake still defines how people live in Pisco. The press and much of the aid has left and the town is still...leveled. Throw in a major disaster in a non-resilient environment, with a bunch of foreign aid with varied commitments to the region, and the long term outcomes are very hard to predict.

This week we made significant updates in mesh4x. One of them is a Hibernate adapter, which allows you to plug into the mesh almost any relational database available in the market

Hibernate Adapter

In our first scenario, let's say you have, or you are quickly hacking together, an application to help enter, analyze and report information. You have a database schema, and you'd like to integrate it with an excel database that field folks are using for data entry. You need to make sure updates and deletes somehow make it out to the spreadsheets, and that folks' updates make it back in. Furthermore, you'd like folks in the field to synchronize spreadsheets with each other directly - thus making it a classic mesh scenario. With the Hibernate adapter, our goal is to allow you to mesh-enable your database by just mapping your entity fields to your database fields.

Hibernate, as most developers know, is an Object-Relational Mapper library for Java. With this adapter you can now integrate into a data mesh any database engine that Hibernate supports, which is an impressive list. By supporting Hibernate as an adapter we allow every user to customize the mapping of the mesh data to their database schema using familiar tools, and get support for a lot of databases. There is still some work to do - for example, as of today the adapter still requires the database schema to revolve around the fact that the rows are being synchronized in a mesh. We expect in the upcoming weeks to remove this restriction and use two separate 'repositories', one for the synchronization information (which you shouldn't care about) and another one for your data.

This will allow you to point to almost any existing database schema and mesh it up without messing it up. (Apologies, couldn't resist).

You can always reach the project through http://mesh4x.org.

Here you can see a list of adapters and suggest your own.