mehack

Synthesis’s network went down last Friday. Not down in the sense of our network connection. Not down in the sense of WiFi. I mean really down. Our main server decided that it was time to call it quit, and left the building. Really out. Like bye bye. Time to rebuild.

Some advice for people who find themselves in a similar predicament:

• Ditch Windows Server if you can. Synthesis used to rely on it for Exchange (to support the Outlook and Windows Mobile junkies in the office), but when the processor melts down on that computer, you can’t just take the hard drives and plug it into another machine;
• minimize the number of your machines in your closet and run Ubuntu JeOS — if you do need to run Windows, at least you get a lot more flexibility;
• Zimbra, zimbra, zimbra! We’re still testing it, but so far we absolutely love it. The web client rocks, it has Windows Mobile push support, and I’m now finally using iCal and Mail.app instead of Outlook under Parallels; and
• for backups, Jungledisk is the way to go. We mount S3 from inside our Linux instances and do daily rsyncs of critical data for backup. And, once week, we automatically suspend each instance, hot copy it, and rsync that for backup.

Better, stronger, faster.

One of the items that we’re constantly looking at perfecting at Synthesis is our project process — since we work on such a wide variety of projects, its sometimes hard to converge on a single process that all of us can know, embody, learn, and execute on; its really a question of, “write once, run anywhere”. The one thing that I’ve gravitated towards is (and thanks to Glyph for putting it so succinctly): run it like an open source project.

Our client interaction lifecycle has people coming in and out of it at every stage — we have practitioners interacting heavily with clients during “project inception”, engineers are going rampant during “development and build out”, and our clients have internal teams that we’re working with during “the transition”. People need to get up to speed really quickly because their expertise may be needed for a brief moment, or because we’re handing off ownership of a project. And knowledge transfer is just a plain, hard, problem.

Just like any good open source project, we’ve structured all our projects to have top level and sub-level README files — with this, I, a developer, can go from SVN checkout to compiling code in a short amount of time. That, armed with the URL of a well-maintained and up to date Trac instance means that I, again as a developer, know “what needs to be done”. Jabber chat rooms (we were never IRC fans) and e-mail lists for each project means that a “community” can be found relatively quickly. And code reviews mean that the changes committed to the tree are correct, with the great side effect of forcing us to get other junior developers up to speed.

Having projects run like this makes our life that much easier. We just pretend that we’re running a good open source project hosted on Sourceforge and then we’re forced to contend with ebbs and flows. Our process, we like to think, supports developers popping in and out of a projects, and allows us to answer to those who just demand a snapshot view of “where things stand”. And, when we don’t have to worry about “how” things are getting done, then they can really just get done.