OSSMichigan.org - Not quite a planet but a state.

Last week I blogged about the new SourceForge.net and one of the first questions I got was when are we going to “lift the covers” and show off our new tech.

There’s definitely more to come in terms of releases and code, but I thought it’d be worthwhile to start with a quick run through of the tech stack and a bit of a description of what we’re doing.

Our first rule for libraries and tools on the new forge, was that we needed to use open source everywhere. Partly this is just because having the freedom to look at the code and modify it where we need fixes, makes it’s the easiest and best way to develop software. Partly it’s because we’re an open source code hosting platform, and we want to use what we promote. But perhaps most importantly, it means that we’re not prevented from sharing our work with others, or from inviting others to work with us in the future.

At the same time we had a company wide decision to standardize on the technology stack that we’d used in the “consume” project last year. So, we’re using:

• Python,
• TurboGears,
• MongoDB,
• and AMQP (RabbitMQ).

The combination of these means that we have:

• a huge number of libraries available to us,
• a web framework that we can turn into a plugin framework for projects and the tools they want,
• a schema-free database that lets us easily version documents to keep history on wiki pages, tickets, and other “artifacts” within the new forge
• a scalable system for handling asynchronous tasks, and propagating update notifications

The choice to use Python has been particularly valuable, since there are (literally) dozens of libraries that we were able to use to help us with everything from encrypted cookie sessions, and mongodb drivers, to markdown text processing, and syntax highlighting.

We’re still in the early days and have a lot more to do, but the goal is an open extensible, system that supports open source projects, and ultimately encourages more people do download and use a wider variety of open source applications.

@robconnery I’m really glad that you are excited. I think anytime someone is healthily and safely passionate about something, it can only be a good thing.

Rob has a great post where he lists 4 cases where he likes Ruby and compares to the same thing in C#. Case 1 Expressiveness: Rob likes the unless statement and the post expression if statement. Case 2: Rob likes Gems. Case 3: Rob likes simple things. Case 4: Rob likes sending messages, open classes and method missing.

Python and Perl already did all this, so why Ruby?

Case 1 and 3 were true in python when i started writing it in 1996 and case 1, 2 and 3 were true in perl when i started writing it in 2000. I’m sure case4 is true in both python and perl too, but I never went that deep into either of them. Much like in Ruby, you don’t have to go that deep to get things done.

I am of the opinion that if you have never seen a dynamic typed language before, or maybe a dynamic typed language other that BASIC or VB before, that Ruby has all of the appeal which you tout. However, there are some of us who write C# because we actually like it, we write desktop applications, and find it to be the best static and strong typed language around. We came to C# and were super impressed because the weak typing of C wasn’t there. The rough edges of C++ wasn’t there and for nearly all applications there is no performance difference and sometimes the GC and managed environment actually gives a boost in performance over some of the bad C++ we were writing before.

So should someone who has never written in Perl, Python, Pike and PHP go try out Ruby… absolutely… get the exposure.

Alternatively, if you have done some Perl or Python and now you are a C# guy. Ruby might not seem so impressive. In fact, it looks more like the same thing with a new coat. I can’t tell what the hype is about. There isn’t much new and different.

All that said, after years of Perl, learning C# was a challenge, especially since I was using it to solve many of the same problems for which I had been using Perl. WHY? was I doing it that way? Well I wanted Windows Forms UI front ends on my Perl versions of programs there were ultimately just sed/awk/grep and some ldapsearch/ldapadd/ldapmodify commands. Not commands really, but calls to libraries.

There is a good reason that the “simple things” aren’t AS simple.

What I learned was that there is a damned good reason that Case 3 “The Simple Things” were a little more complex in C#. The separation of stream and textreader abstract types in C# make huge sense once you realize that doing the same thing in perl or python (or ruby) can be a bit of a hassle. The organization of decorator streams in the .NET BCL just makes sense. Want to compress? Decorate with the stream compressing class. Want to encrypted? Decorate with the stream encrypting class. Want to do both? In either order? Decorate appropriately.

I do share Rob’s opinion. It is a little prettier in Ruby. I’ve already gone on record as saying that “var” in C# should be optional. In VB6 the Let statement was optional. In VB.NET the Let statement is no supported. IMO C#’s var isn’t much different than VB.NET’s Let and Dim. Sure would be nice if it were optional.

I’ve also requested static imports so that we could do things like just call the open method instead of saying File.Open. When you are in a nice tiny singly responsible file, it just makes sense.

These things don’t change my ability to write code.

On .NET’s lack of a CPAN, Cheeseshop, Gem equivalent: YES! YES! YES WE NEED IT NOW!

I can’t say anything other than .net needs CNAN (comprehensive .net archive network) or maybe CCAN (comprehensive CIL archive network). I can’t decide which name I like better.

As for metaprogramming, I think that Python, Perl and Ruby’s ability for runtime metaprogramming will continue to be far beyond anything you see in the C# world. That is not say that metaprogramming is not possible in C#. Its just very different. Its typically compile time metaprogramming. Thanks to the addition of T4 in VS2008 and 2010, metaprogramming in C# is readily available and powerful.

I could go on and insert above about how I learned to love the .NET RegularExpression API after having it blow my mind in comparison to perl’s. Or about how poorly documented the System.DirectoryServices API is, but that once I got it I loved it so much more than Net::LDAP. Or about the extreme pain in building CPAN modules on a Sun Sparc and how installing Mono and using Visual Studio and C# was actually easier than making Perl work properly.

But rather than elaborate on those things, I’ll end by saying, yes, Ruby is awesome, if you have never seen any of the things which make it awesome before.

A little more than a year ago, a small group of volunteers coordinated to host the first CrisisCamp in Washington, DC. At the time, we just wanted to pull together first responders, technologists, government, NGO, and interested citizens to discuss crisis mitigation, response, and humanitarian relief efforts. The two-day event was a complete success in connecting these communities in dialogue and projects that led to field deployed projects. In the last meeting of CrisisCampDC we discussed the potential future of these camps – and on a whim I registered crisiscommons.org, installed MediaWiki and Mikel provided a logo.

For the next 9 months, side projects occured and interesting conversations continued, but without a single coherent focal point. What happened in early January completely changed how we thought about volunteer crisis response. In the hours and days following the Haitian earthquake thousands of volunteers around the world began brainstorming and contributing to projects that would hopefully have a positive benefit to the response and affected communities.

CrisisCampHaiti

By Thursday we had decided to host a CrisisCampHaiti in Washington DC and very quickly similar groups decided to hold events in 4 other cities. The CrisisCamps provided a focused venue for developers, volunteers and organizations to coalesce and collaborate on developing needed solutions and information that would assist on the ground efforts.

OpenStreetMap had already been identified as a key resource in the response – starting first with the use of unclassified 1990’s paper maps, and then increasingly with the availability of high-resolution and up-to-date commercial satellite imagery. This provided for a very simple task for general volunteers with a computer and internet connection to begin tracing road networks and infrastructure. Videos like iMapHaiti.com got new volunteers up to speed and mapping within 10 minutes.

The technical expertise brought to bear was powerful. Mobile phone apps such as Tradui for translating between Kreyol and English; We Have We Need, a place where relief organizations can quickly post their most urgent needs and have them matched by generous donors during a time of crisis, and more.

Developers conceptualized and created green field applications, others worked on adapting existing tools to new uses or connecting them together – such as an Ushahidi to OpenStreetMap bridge that would allow for people on the ground to send mobile messages that could update the actual map data.

The outpouring of effort was amazing. In essence, the realization was that people wanted to contribute. And instead of sending $5 via a text message they wanted to donate their even more valuable time and expertise to provide true value and support.

Crisis Continuity

These efforts have been widely discussed, and the power of thousands of connected, capable, and caring technical and helpful people immediately pointed at a problem is compelling. However, what is not immediately apparent is that these efforts, tools, and communities are not completely ad-hoc and spontaneous. They have evolved through joint experiences, social networks, technical exchanges, and personal needs. The tools were developed around an initial kernel of a problem, and then modified, evolved, cajoled, and carried from one event and use to the next. Jesse & Mikel have espoused this concept before.

It is this continuity through many experiences and efforts that forges the applications and organizations. Following the initial surge, a core component of the community continues to talk about lessons learned, how to expand the tool, integrating with other workflows. An interim solution in one event slowly becomes more integrated as part of a response with new features, languages, and capabilities along each step.

This is primarily possible through openness: open-source, open-data, open-collaboration. Open Source software means that any solution developed can be reapplied and improved upon as new requirements and capabilities are needed. Open data guarantees that there is a free flow of information before, and during an event that can reach to any and all responders and volunteers as appropriate. Unforeseen needs can be met by modification and analysis of the data. And finally Open Collaboration means that people freely exchange needs, solutions, and ideas that ensure best options are available.

The continuity is further expressed in the tools and data remaining in the affected areas for citizens and government to utilize. There is less of a vacuum remaining after organizations withdraw as local groups can take ownership of the tools as well as stay connected with the community to build capacity.

CrisisCommons

What has been missing is a community that provides support and coordination of these various efforts. New projects will start and be deployed. But how do NGO’s and response communities identify which tools are available, reliable, and meet their operational requirements? How do they work with the volunteer communities to identify needs, provide ideas and specifications and adopt these tools as they are developed, tested, and supported?

A goal of CrisisCommons is to provide this role. Through international communities as well as local and regional organizations and camps that understand relevant risks and responses to provide for pertinent and continued support.

Organizations such as the World Bank, MapAction and others clearly have identified the potential of working with organizations such as CrisisCommons that can be an interface to the moving surges of volunteers, companies, and tools that they can leverage in reconstruction efforts.

There is a change in how the public is engaging and supporting in crisis response. They are able to augment capabilities and provide surge support. But it is necessary to recognize that the capability to respond and engage quickly and effectively occurs through continuous evolution. In preparation, prevention, and mitigation of disasters we can apply our tools and knowledge. In reconstruction we can modify and integrate the viable solutions into sustainable operations.

CrisisCongress

There are still a number of questions that have yet to be answered about this type of model. This week the first international CrisisCongress is convening with individuals from around the world to discuss the models of volunteer crisis response and technology. Through our discussions, shared experiences and problem solving we will have a clearer vision for how to continue the successes we have had and grow the capability for people to respond and help in moments and places of crisis, whether across the globe or in their own community.

JoDee is right.
JoDee is always right.
Craig is lazy.
Craig can’t manage to move several small items in a bin to find some of the things that he was looking for.
Craig lost a bet, and had to post this on his website.
JoDee rules.
JoDee is standing over my shoulder making me write this.
That is all.

So far 2010 has been incredible, and hectic. There have been numerous great projects, collaborations, and work that have prevented me from taking the time to blog. As I’ve noted in the past, Twitter defuses just enough of idea sharing that I don’t readily go to write articles. However, while these micro-messages relieve the immediate pressure of a concept they lack the general feeling of satisfaction that a more expressive and coherent article provides.

In quick summary of what I’ve been up to – as a means of providing a sort of excuse, preview, and immediate alleviating of the overwhelming feeling that “I haven’t posted in a while, so it’s difficult to start again”:

Besides a two week trip to India with Corrie, I gave a plenary lecture at the Library of Congress on Neogeography and digital preservation of geospatial data that will soon be online, spoke at the UK Socio-Cultural workshop on the use of community and citizen generated geospatial data in crisis response and development work. There is also some hopefully soon news on new countries opening up data.

At FortiusOne we’ve been fortunate to work with many great partners this Spring and Summer in providing open collaborative platforms that we’ll soon be able to share with everyone. In addition, we’ve been heads down building out a host of new features to GeoCommons that will really open the GeoWeb and provide more than just visualization. We also participated in the OGC testbed that experimented with the sharing and annotation of authoritative and crowd-sourced data, much of the lessons and capabilities that are already exemplar in GeoCommons, but we’ll be adding more features to enhance the interoperability.

DC continues to be an interesting place to live – and I’ve definitely had more exposure to government than I ever expected. The area is surprising in the innovation and connectedness that is definitely worth sharing.

While we’re about to launch a number of new capabilities, I’m also able to come up for a bit more air. My aim over the next few months is to dramatically increase my posts. Consider this one as a way to poke through the shroud that will enable more regular posting.

So, I’ve been working on sf.net in various ways for about a year now.
http://sourceforge.net/p/. It’s written in Python using modern open source tools, from RabbitMQ, and MongoDB, to Git and Mercurial. And we are committed to making this the most open forge possible. We’re committed, to open processes, open code, and perhaps most importantly open data.

The first thing we did was create some new pages for downloads. Recently we releases a new service designed just for open source project leaders who want to use sf.net as a directory and downloads service.

But, we’re also aware that one of the most important services we provide is project hosting. For the last several months a small group of us have been trying to bring sourceforge.net’s tools into 2010. And now we’re releasing an early preview of those new developer/community tools:

We have a long way still to go, but every long journey begins with a single step, and today’s step is allowing you to try the new forge, to create new projects at:

https://sourceforge.net/register

Where you can go to get a new project, with our new tracker, wiki, git, svn, and other tools. Projects can have subprojects, and links to other tools hosted off site, along with the many features that sf.net brings (free web hosting, hosted apps, etc).

But, why do all this?

In 1999 SourceForge was cool.

It provided all the tools that an open source project needed to get going, from cvs hosting, to bug tracking, and e-mail list support.

They pioneered free free software project hosting, and helped to transform the software development culture from one which barely new about free software or open source, to one where nearly everybody I know uses open license software. Oh sure, some of them might not know it, but they have it on their phones, in their TVs, their wireless routers — not to mention all the websites they use everyday that run on open source.

But, time passed.

More alternatives came out, more projects (including my own) started self hosting, and the landscape of open source software development changed. SourceForge.net took a long time coming out with support for new tools like svn, and then git.

Still, SourceForge has a special place in my heart. Partly it’s nostalgia, I suppose, but I still think:

• the core mission is still right
• and there is still a real need

We (Open Source developers) still need tools like git, mercurial, and svn hosting. We still need bug trackers and mailing lists. And in a meeting of other open source project leaders last fall, nearly every single one of them identified the time wasted integrating and administering these tools as one of their most important frustrations.

Not enough…

But, for many sourceforge.net and other free project hosting services were just not good enough, they weren’t scriptable, the weren’t extensible, their data wasn’t portable, and so they felt like they had to take on that cost.

And I fundamentally believe that open source projects live an die by communication, and that sourceforge.net can do something new by integrating the various kinds of “conversations” that happen around the project. We can integrate mailing lists and forums, we can integrate SCM and ticket trackers, etc.

New and improved

So, a couple of us have been quietly working on something new. The new forge is designed around a few core ideas:

• that data should be portable (every project gets their own database, which they can take with them if they want),
• that the open source community ought to be able to extend and enhance the tools they need,
• that integrating and cross linking the various kinds of conversations that open source projects need to have ought to be easier.

So, what we’re announcing today is more of a commitment to getting there on all these things, and a commitment to the “release early, release often” project management strategy.

So, expect us to take your feedback and make things better. Expect us to release lots of small fixes, and expect a few places where things are broken/incomplete because we value feedback more than polish at this point.