Deployment with a safety net

We now write Drush Make files for even our smallest one-off builds. This discipline saves us time and makes us more efficient as a team by knowing what code actually powers a site. It also helps us transition projects to our partners or to the shops that will be supporting the sites we build, helping us more quickly cut the long tail of a build. To give a sense of just how much this can help internal processes, I'll explain how I was robbed of three days this week by jumping onto a project we worked on before we formalized using Drush Make in our build process.

This week we started sprinting on a custom Open Atrium build for Teachers Without Borders, an organization with a somewhat obvious multinational reach that wanted us to extend Open Atrium in a few interesting ways. They needed Open Atrium to be an identity hub, be localized into multiple languages, and have more social features like following. We'd already done some work for Teachers Without Borders deploying a set of Managing News sites, including one localized into Chinese, and an Open Atrium hub site with the OpenId Simple Sign On stack.

When we originally deployed these sites, we didn't have a good way of tracking what had been added or changed from the standard Open Atrium and Managing News platforms. Each of these platforms had a number of extra modules deployed, mostly to power the authentication infrastructure, and a patch or two to some modules. As someone who didn't work at all on either of those projects, it made getting into these two sites, as well as fixing bugs on them, a very confusing experience.

Copyright, social media, movies, and soccer this week in Washington, DC

After several days of off the chart temperatures, it's feeling a little more reasonable outside with temperatures only in the 90s. Take advantage of the break in the heat to get out of the house and check out a technology event. While there's not a whole lot on the calendar this week, there are still some interesting events. Have a great week!

Monday, July 26

Dusk

Screen on the Green: With much cooler temperatures and a forecast for clear skies, it's shaping up to be a great night to catch a movie on the Mall. Tonight 12 Angry Men will be playing as part of the Screen on the Green festival, starting at dusk.

Tuesday, July 27

6:00 - 8:00 pm

CopyNight: If you're interested in copyright issues around technology, you'll want to check out this meetup. This month there will be a discussion on the implications of the recent ruling finding LimeWire liable for inducing copyright infringement.

Closed beta testing for offline mapping

Today we start closed beta testing of Maps on a Stick v.2, an application for making interactive maps and visualizations offline. This version is thoroughly rethought to make the application more flexible and efficient. For example,

• No more complex deployment process. Users can easily copy single-file SQLite tilesets to Maps on a Stick for viewing.
• Adding online KML is as simple as inputting the address of the KML file and clicking 'Add.'
• Uploading KML consists of clicking 'Upload KML' and uploading it.

With this release, the simplicity and flexibility of Maps on a Stick have increased tremendously, as a result of improvements to the underlying infrastructure.

The first version of Maps on a Stick revealed a few important principles of portable, cross-platform applications:

• Distributing tilesets as massive collections of files is disadvantageous due to the inefficiency of filesystems, especially those that serve as the lowest common denominator for USB sticks, like FAT32.
• Distributing pure HTML and Javascript revealed the serious and hard limitations of that approach. Cross-domain KML was not supported, and even loading files from the local disk is not supported in Google Chrome because of security restrictions.

Drupal Peru is hosting another Camp this Saturday, July 24

Tomorrow I am headed to DrupalCamp Perú Invierno where roughly fifty Drupalers will get together to talk about topics ranging from installing Drupal to writing your first module to using Drush for simplified site building.

I will present on how to use Open Atrium effectively. Open Atrium is a flexible platform that does a lot out of the box and can also be customized and extended to meet specific needs. This means that Open Atrium can be used well for different use cases. Here at Development Seed we use it internally as a project management tool for our team and for our clients. Meanwhile, the Open Atrium Community site is a public-facing site used to manage public support, development, and documentation around Open Atrium. I'll discuss the strategies we use in both cases to effectively harness the power of Open Atrium, with out of the box Open Atrium. There are quite a few Peruvian developers and shops using Open Atrium, so I am particularly looking forward to this talk.

There will also be a code sprint room open where developers will be testing and rolling core patches for Drupal 7, as part of the Drupal Peru community's current support of the larger Drupal project.

HaitiAidMap.org opens up operational data

HaitiAidMap.org launched this past week with a large data update. The site maps hundreds of projects in Haiti being run by dozens of NGOs operating on the ground. This visual linking of project data to organizations, clusters, and communes is key for InterAction – the NGO coordinating body running the site and collecting and cleaning all the data – as they work to improve coordination and resource allocation while transparently showing where external organizations are operating in the country.

The site focuses on showing project density across the whole country, allowing users to quickly filter by "cluster" (the issue area a program focuses on), and changing out base layer maps to show terrain maps and earthquake magnitude maps for added context. You can jump directly to a cluster, commune, or organization page from the front page.

InterAction manages their data in a series of CSV files (most of the data comes in via email in a combination of Excel & Word files, and the team at InterAction then cleans all the data and puts it into CSV with some help from FortiusOne's GeoIQ tool). The site, built on Drupal, then uses Feeds to take this cleaned data and pull it into the site to populate pages, allowing InterAction to rapidly communicate their data with all of their partners. Here is a review of the subpages that are automatically powered from the CSV import.

Filter by Commune: From the front page, you can quickly jump to see all projects in a commune. Here is a look at the projects happening in Port au Prince.

Federal agencies can now easily start using free custom designed maps

MapBox is now an approved software-as-a-service provider on Apps.gov. All federal agencies can freely use MapBox map tiles and only need to go to Apps.gov and enroll to get started. (Direct links aren't supported, look under "Social Media Apps" >> "Display of Multimedia, Data, Maps").

This all started back in May when we learned that the Department of Education team behind Data.Ed.Gov was interested in using MapBox map tiles on their new website, (a Drupal build by the folks at NGP Software). The Department of Education team were building pages to help the public browse grant application data and wanted to make sure they had a nice cleanly designed base layer like MapBox's World Light - USA tileset to show off their data.

CSO meeting in Abuja, July 19th & 20th

A host of civil society organizations gathered today in Abuja, Nigeria to share strategies and discuss tools for better monitoring the upcoming presidential elections (currently scheduled or January 2011). Against the backdrop of heavy irregularities in the 2007 election, there is a good opportunity during this election cycle for both domestic and international election monitors to better use technology to help communicate the data they are collecting and speed up data collection time.

Today I presented on the latest mapping work we are doing with NDI's team in Afghanistan, explaining how mapping and visualizing results can help tell better stories and make information more concrete. We also discussed how these tools should be used not just for elections, but during then entire run up to the elections, from mapping voter registration to redistricting.

Back from State of the Map and looking forward to advance the Humanitarian OSM Team

I'm back in the office after a great few days in Girona, Spain for State of the Map, the annual OpenStreetMap (OSM) conference. One of the highlights of the trip was that it brought together the Humanitarian OSM Team and gave us the opportunity to talk about what we've learned from three successful deployments to Haiti and plan our next activities.

OpenStreetMap in use by search and rescue teams after the earthquake in Haiti. Photo taken by a member of Fairfax County's Urban Search and Rescue Team.

One of our current focuses is to pull together resources and training materials that have already been created as part of different projects, and to finalize the Humanitarian Data Model that will provide standard terms for data collection and help OSM data transfer seamlessly into the workflows of responding organizations. A lot of work has already been done in these areas by MapKibera in Kenya, OpenMapsCaucasus in Georgia, and the three Humanitarian OSM Team deployments to Haiti, and we are in a good place to start consolidating versions, polishing them, and getting them translated in the various languages of the places where they are needed.

Bring together open data leaders in government, media, international development, and technology

Today's brown bag lunch panel on Making Data Work for You features data experts from the U.S. Department of Education, the World Bank, the New York Times, the New America Foundation, and Eric from our team. These leaders from different sectors are going to share their experience opening up data and making interesting and useful tools and visualizations with it. Here's the run down on the speakers and what they'll be presenting on:

• Jason Hoekstra and Alula Asfaw will present on the the Department of Education's brand new Data.Ed.gov.

• Derek Willis will talk about three open data initiatives from the New York Times - the Times Developer Network, its API resource Open, and its blog on all of this The Scoop.

• Jeff McCoy and Neil Fantom will talk about Data.WorldBank.org, the huge open data initiative recently launched by the World Bank that gives access to more than 2,000 indicators from World Bank data sources.

• Eric will talk about our work to open up data on and map ongoing recovery projects in Haiti with InterAction in HaitiAidMap.org and chime in on the technical side of Data.WorldBank.org with the World Bank.

The talk is part of Media Future Now's monthly brown bag lunch series. It will get going at noon and last until 1:30 pm, and it will be held at 1133 19th St., 9th floor. Come on by if you're in Washington, DC!

Drupal, php, and open data this week in Washington, DC

With the World Cup over for another four years, we'll all have more free time to do other things - like check out some of the fun technology events happening around town. And even though it's the middle of July, there are plenty of events happening. Below are some of the events that we're participating in or hoping to attend this week. Hope to see you around!

Monday, July 12

7:00 - 9:00

Drupal Meetup: This monthly event is a great opportunity to learn more about Drupal - one of the fastest growing content management systems out there - and meet local Drupal developers in a casual setting. This month there will be three short talks on security in Drupal, the tool Kit that helps streamline site building, and smallcore, the movement to make Drupal out of the box smaller and more dynamic.

Tuesday, July 13

Noon - 1:30 pm

Brown Bag: Making Data Work For You: The Media Future Now folks are hosting this brown bag discussion with folks from the U.S. Department of Education, the World Bank, and the New York Times who will be talking about how they've opened up their data, how they've benefited from this, and what challenges they've faced. Eric will be there to talk about the strategy behind opening up data and to talk specifically about the work we've done with two of these organizations.

The July Drupal Meetup in Washington, DC is tonight (7/12) at 7:00 pm

The July Washington, DC Drupal meetup is happening tonight at 7:00 pm at Stetson's Bar and Grill. It should be a fun event, with three interesting lightning talks on the schedule:

• Jeff Eaton will talk about smallcore, the movement in Drupal to make Drupal core smaller so it's a stronger, more flexible web framework.

• Ben Jeavons will talk about the Drupal Security Report, released in April, what was found, and what it means.

• Alex Barth will talk about Kit, a compilation of tools and best practices that makes site building in Drupal easier and faster.

• UPDATE: Laia Grino will present HaitiAidMap.org, a project built on Drupal that opens up and maps 950 projects by 44 different organizations operating on the ground in Haiti to help with recovery efforts - six months after the devastating earthquake. Laia will talk about the background of the project, how it's helping collaboration on the ground, and the challenges that were encountered along the way.

More details on the meetup are up in the Washington DC Drupalers group. Hope to see you tonight!

Voting for DrupalCon Copenhagen sessions closes Sunday, July 11

We have proposed to lead or be a part of several sessions at DrupalCon Copenhagen on our development processes using Drush Make, Features, and other tools and our site management with Aegir, making and building business models around Drupal distributions, and aggregation and the real time web and localization. If you're headed to Copenhagen and interested in seeing these topics on the schedule, go vote for them!

• For Every Site, a .Make File: Building Drupal sites has gotten faster with tools like Drush and .make files. Jeff will demonstrate how we use these tools - along with Features and Kit - in our development process and walk through examples of how the process has worked when implemented on real client work.

• One Drupal to Rule Them All!: Aegir - the hosting system that allows you to easily manage thousands of Drupal sites on as many servers as you need - development is moving fast. Led by Adrian, this session will will walk through how to install Aegir, how to use it to clone, upgrade, and manage sites, and how it improves the staging workflow in Drupal.

• Drupal Distributions: The Dos and Don'ts: Drupal distributions like Open Atrium, Managing News, Open Publish, and Tattler are becoming increasingly popular and common. Jeff and Irakli Nadareishvili will talk about the challenges of building and managing a distribution and how tools like Installation Profiles, CTools, Features, Context, Drush, and others are crucial to it.

• Go Real-Time with Pubsubhubbub and Feeds: In this session, Alex will talk about how to bring realtime web feed subscriptions into your Drupal site using Pubsubhubbub. He'll explain how Pubsubhubbub works, how the Feeds module can be used to subscribe to PuSH feeds, and how the PuSH Hub module can push content from Drupal to other web applications in real time.

• Backstage with Drupal Localization: Jose will join Gábor Hojtsy in this session to show how localization works in Drupal, how you can make your themes and modules localizable, and how you can utilize APIs to help with it all. Jose will also talk about how localization works on Translate.OpenAtrium.com - the community based effort to translate Open Atrium which is now up to 38 languages.

• The State of Drupal as a Web Application and Product Platform: Looking at the other side of Jeff and Irakli's session on Drupal Distributions, this session will look at the business of building a distribution on Drupal.

• Managing a Drupal Consulting Firm: In this panel Eric will join folks from NodeOne and Four Kitchens to talk about running a Drupal shop, explaining how we do things here at Development Seed and why and running through some approaches that have - and absolutely have not - worked for us.

Voting on sessions for DrupalCon Copenhagen closes this Sunday. With well over 200 sessions proposed for the three day conference - happening August 24 to 26 - voting will play a huge part in narrowing down the program. You can vote at DrupalCon.org.

2010 International OpenStreetMap Conference in Girona, Spain

I'm on my way to the 4th Annual State of the Map International OpenStreetMap Conference in Girona, Spain. It will be great to catch up with friends and colleagues on the Humanitarian OpenStreetMap Team who have deployed to Haiti twice since my trip there in March with Nicolas, and to discuss next steps and ways to continue supporting the Haitian OSM community.

If you are interested in learning more about OpenStreetMaps's potential for humanitarian operations, Nicolas Chavent and I will be sharing specifics about our recent experiences using OSM in Haiti on Sunday at 3:30 pm. We´ll discuss the role of OSM in the response to the January 12th earthquake, the work we conducted with civil society groups in Cite Soleil around participatory mapping in slums, and partnering with the Haitian government to help reinforce the national mapping infrastructure.

I will also be speaking on the "Cocktails on the Titanic" panel on Friday at 6:00 pm discussing new business models involving OpenStreetMap and open data and listening to community members present their ideas in lightning talks.

If you can't make the conference this year, I will be tweeting from both @rsoden and @MapBox.

Summary of the architecture changes that will be in the next release of Aegir

The next Aegir release will enable managing sites across multiple servers. This means that you no longer need to have Aegir on the same box as your site, and you can even migrate your sites between servers. Big sites can now also be spread across multiple web servers, giving your hosting infrastructure the ability to grow with your sites.

Below are the details of what's come from our June sprint where we worked closely with Chapter Three's team. While all code will not be officially packaged until mid July in Aegir 0.4 Alpha 9, you can check out the latest code from our git repository at git.aegirproject.org.

And then there were servers

The heart of this sprint was introducing the concept of "servers" and "services" into Aegir. We have rewritten both the front end and the backend of Aegir with a new OOP API to properly make use of the entities. So out with the "db_server" and "web_server" node types, and in with the new "server" node type.

Each server can now have one or more service associated with it, such as a HTTP service or a DB service. What's more is that there can be multiple implementations of a service type. So, for example, you can now have Apache and Nginx support, and not only can you have different implementations of a service, you can also create entirely new service types, such as a DNS service.

On a conceptual level this means that we can cleanly distinguish situations where everything is on the same server, as you will only have a single server node with http_apache and db_mysql services associated to it. If you then add a remote database server, you will have a second server node with just the db_mysql service associated to it. Just being able to cleanly model these relationships is already a significant improvement to our codebase.

This new architectural change has been taken into the backend too, allowing us to much more easily extend the functionality of Aegir. When you create a platform or a site, all it cares about is that it has it has a service it can connect to - it doesn't care about the internals of how it's all put together. This means that for the first time it's completely feasible to use provision without needing to have it manage your virtual hosts in apache.

It also means that we can implement some fairly complex situations we never could accomplish before. The prime example of this is the 'web cluster' service. If you have multiple web servers, you can create a new server to represent a cluster of web servers. When you create a new site, you can simply select the new cluster to host your site on.

PHP, Nerd Nite, and TEDxPennQuarter this week in Washington, DC

We hope everyone had a very happy 4th of July yesterday filled with good times with friends and family, delicious barbecues, and of course fireworks. With much of the city celebrating the holiday with a short work week, it's no surprise that the pickings for technology events this week are slim. Below are a few events that caught our eye, and there's a complete - though still small - list of all technology events happening in Washington, DC over at DC Tech Events. Have a great week!

Tuesday, July 6

6:00 - 8:00 pm

DCPHP Beverage Subgroup: Interested in talking about php code in a casual environment and chatting with other programmers over a few beers? Then check out this monthly meetup.

An early preview of the latest developers for the World Bank

At today's Peru Drupal Meetup, I will present on open data outreach strategies, specifically showing a sneak preview of our integration of web widgets into the next round of work on the World Bank's Data website (which should be live to the greater public in mid July).

Web widgets will help take both the World Bank's site and its data further. I will show how the widgets are powered by the World Bank's API and how to get started building your own integration with the World Bank Data API. Here is the first public look at what a basic map widget looks likes, in this case of internet users (per 100 people) from the site. As you can see, you can zoom in on the custom tiles (down to Peru) and interact with the data.

There will also be other presentations, as well as a coders' room which has been quite active during other meetups. Last month, for example, several people from the Drupal Peru group got together to work on and review core patches to help with the Drupal 7 release.

Mapping the global public health marketplace

Results for Development just launched the Center for Health Market Innovations (CHMI), a project that opens up data on and maps public health projects happening around the world. This portal represents an important open data advance for the global public health community. The site currently profiles hundreds of private sector health programs and allows practitioners and researchers to filter, map, export, and even add to the data to produce better analytical research in the global public health sector. It also has a "pipeline" feature that encourages colleagues in the field to add data on projects that aren't currently profiled, in one of the project's efforts to expand its coverage.

CHMI has an ambitious goal: to improve the capacity of global public health professionals to analyze the success of various approaches to delivering health care to the poor.

Copyrights and democracy, nonprofits, and mobile phones this week in Washington, DC

It's promising to be another hot week here in Washington, DC as we move into July. In between preparing for the weekend Independence Day festivities, take some time to check out a technology event to have a couple cold beers, learn something new, and meet some new people. Below are the events happening this week that caught our eye, and you can check out a full listing of events at DC Tech Events. Have a great week!

Monday, June 28

6:30 - 8:30 pm

How Copyright Threatens Democracy: Where do freedom of speech and the democratic principles of the internet intersect with current copyright laws, the business goals of commercial media behemoths, and democracy as we know it? Cory Doctorow will discuss this and the conflicts that arise from these dueling paths, as well as what we should all be doing about it, at this event hosted by the New America Foundation.

A display layer on top of the World Bank's API

The Data.WorldBank.org project we built for the World Bank makes 339 human development data indicators open and available to the public (with 700+ more coming soon), but the site itself is really just a display layer on top of the World Bank's API. In this post I'll talk about our team's key take aways from building Data.WorldBank.org, as well as the technology we used. A site like this is a bit different than most - it's much more common for a site to display the same data it manages and not data from somewhere else - so we had to do things a bit differently.

API performance

When we started to build Data.WorldBank.org, the World Bank already had a public API thanks to the great work of Phase2 Technology and especially Irakli that exposed a subset of the data they needed on the site. However, there were a number of concerns and limitations surrounding it. It wasn't known how much of a load the API could support, and in its current form it had a limited number of users, and though some users like Google were very large, beyond them the API had very little constant usage.

There were a set of changes that we knew would need to happen to the API for this project. The API application was already in the midst of being made multilingual. For this site we needed everything in the API available in four languages - English, Arabic, French, and Spanish. This required a substantial, though predictable, level of change to the API application and across the board changes to the public API methods. This is normal, particularly with this being the first time the World Bank was building off of its API. Also, with the amount of data being added to the API, no one knew what other changes would need to be made to keep the API application fast. Lastly, we didn't want to disrupt the current user base with these potential issues and API changes.

Nearly all of the content and data on Data.WorldBank.org come from this API. While we had the option of accessing the API's database directly, we decided early on that we needed to be consumers of the API, even if it meant running into issues and running a fork of the code base. In the end we had to do just that. In order to get the level performance we needed for the site (and with the number of indicators increasing by 300% for the April launch and the huge planned increase this summer), we had to change some of the API's internals - some of these changes were reflected in the API's methods and others meant that IDs had to be changed.

These changes prevented us from being able to offer this version publicly right away. However these changes will be slowly introduced into the public API. The process of building the site directly against the API provided valuable feedback of what needs to be changed in the public API. So while we settled on using a private copy of the API during the build, we think the process will help make public API stronger.

Open data can improve efficiencies, collaboration, and decision making for nonprofits

Tomorrow Eric will lead an Ask the Expert session on open data and what it means for the nonprofit sector with NTEN, the Nonprofit Technology Network. Open data can drastically improve efficiencies within organizations and fuel collaboration between organizations working for common causes, so it is particularly relevant to nonprofits.

Eric will talk specifically about some of our work in this area and its impact, such as project mapping in Haiti to improve collaboration between organizations working on the ground, vote count and other data visualizations for Afghanistan's presidential election to combat fraud, and opening up the World Bank's extensive data set with Data.WorldBank.org. And for the majority of the hour long session, Eric will answer questions from participants on open data and data visualizations, successful approaches to opening up and using data, and issues relating to participants' projects an concerns.

The session will take place at 2:00 pm EST (11:00 am PST) on Wednesday, June 23. More details and signup information are available on NTEN's website.