Welcome to Pegasus Mail & Mercury Sign in | Join | Help
Everything you never wanted to know about releasing software

With Mercury/32 v4.5 now into the end stages of pre-release confirmation (the release candidate archive went out to the beta testers last night), it's safe to say that the light is visible at the end of what has been a very, very long tunnel.

Now, I would never attempt to defend the fact that there has been a three year gap between releases - a delay like that is damaging, and it should never have happened. Suffice it to say that I have learned a number of difficult lessons about managing large, complex projects as a result of this process, and that it should thus be possible to avoid having it ever happen again. That said, though, I suspect that very few people out there really have any idea what it's like to make a release of a piece of major software, especially when that software is commonly used as critical infrastructure... So, here are some purely personal insights into why I find getting up to and through a release so hard.

1: Testing, testing, testing   Those of you who use it may have some appreciation of just how feature-rich Mercury is: the program has settings for almost anything imaginable, and under the hood, there is a very rich plugin interface that allows clever people like Lukas Gebauer to add the things I either can't or haven't. The more feature-rich a program becomes, the harder it is to test, because there are so many more variables and so many more paths through the code. Add to this the fact that Mercury is heavily threaded (a technical term meaning that it can be doing many things at the same time) and that thread synchronization issues are among the hardest problems to reproduce, diagnose, and fix, and the testing process becomes a literal minefield. Testing Mercury is far, far harder than testing Pegasus Mail - orders of magnitude harder. Every time a feature is added to Mercury, testing begins again almost afresh, because every addition can have side-effects - unanticipated problems where the addition disturbs something already in existence - that have to be tracked down.

2: The dreariness of maintenance   There was an interesting thread in the off-topic forum here recently where a community member asked why programs had bugs. On the surface, his comment was perfectly fair, but in truth, any program larger than a few lines is going to have problems that creep in: the larger the program, the more possibilities there are for bugs, and complexity increases those possibilities even further. Software is never finished - it's always growing and changing, which means that the bugs are growing and changing too. Allied to this is the perpetual process of rewriting code - taking things that used to be "good enough" but now are not, and making them "good enough" again. At some stage in the life of all large software projects, you reach a stage where you spend more time tracking down bugs and rewriting old code than you do adding new features. This maintenance can't be avoided and mustn't be rushed (or else you'll typically just make things worse), but unfortunately, programmers are almost always relatively normal human beings, and like change and variety in their lives: having to spend most of your time reinventing or reworking old things can be pretty dreary, and on a cold Winter's morning, it's often the last thing you'll want to do. Maintenance, therefore, can be a slow process, if only for human reasons.

3: User expectations
   It's a common truism in the industry now that user expectations get higher with each passing day: users expect programs to run flawlessly, to be easy to use, to cost nothing, and to have feature sets that match their every need. With higher expectation has also come lower tolerance, whether tolerance for errors, tolerance for ease of use, or even tolerance for appearance. The problem with higher expectations is that they are, obviously, harder to meet - but worse, they are usually exponentially harder to meet: making something "prettier" is actually a very difficult and time-consuming process, and even though it produces little actual tangible benefit, users expect it, even in programs like Mercury. If it's not there, lower tolerance means that people get more aggressive and more turned off. The difficulty and expense of producing programs that meet user expectations these days is reaching the point where only large corporations with huge resources will ever be able to afford the process. User expectations are basically killing the small-to-medium sized player.

4: Resources, resources, resources   I suspect most non-software-developers really have no conception of what it costs to produce a piece of software - "A PC on a table in your bedroom - what more do you need?" (that's an actual quote from someone who shall remain nameless, by the way). There are compilers (Visual Studio costs over $2000 here now), documentation tools (FrameMaker costs $1700), file servers for each supported environment, test servers, testbed systems for various versions of Windows, high-speed Internet links (unfortunately very expensive in New Zealand), domain names, certificate fees, support subscriptions and upgrades for software and development tools, books and reference materials, third-party software licenses (we pay over US$2000 per annum for the very small number of third-party tools we license), sundry development tools (such as Igor Arsenin's excellent TaskInfo product), electricity, banking and business costs, accounting costs... The list goes on. The cost of doing business gets higher every year, and peoples' willingness to pay in support seems to dwindle (and please, don't think of this as me criticizing my user community - it's a general statement about the world the Internet has become). Reduced resources mean making do with less, which in turn extends the time to release.

5: The dread of feedback   This one's a strange one, because it's a complete flip-flop from what I used to experience. In the earliest days of Pegasus Mail and Mercury, releases were exciting, because they were my chance to show the world all the cool new things I'd done. Because the world was more of a community back then, I got lots of supportive mail each time I did a release, which was also exciting and uplifting. These days, in the first few days after a release, the mail I tend to get is different: sure, there are still people who write to say thank you without strings or qualifications, but more normally the messages start with "Version xx is great, but...". It's really hard to describe how soul-destroying it is to release something you've worked really hard to produce, only to start getting wishlists and gripes almost immediately. When I say "gripes", I'm not saying that people are necessarily rude - it's usually just people expressing their annoyance that their pet feature or fix didn't make it into the release. And then there is the out-and-out rude mail; sadly, I always get a couple of those: these are the people who tell me that I should "just stop wasting my time and get a job at McDonalds (if they'll have me)". (And yes, that one's an actual quote as well). There's only a few, and they're always from people who either have a serious grudge of some kind or are simply ill-bred or psychologically troubled, but they have an impact out of all proportion to their importance. Just one of those messages can depress me so badly that I'm incapable of being a functioning human being for the remainder of the day. After seventeen years of doing this, you would think that I might have developed a thicker skin about this kind of thing, but I haven't: it really hurts, and over time, the anticipation of the mail I know I'm going to have to handle has made the process of release more and more daunting, more and more difficult.

Now, this probably all sounds like me complaining, but I'm not really... All I'm doing is summarizing a reality that became inevitable once the Internet reached the level of being a commodity. I should probably toughen up and lighten up, especially since I still actually enjoy the development process for the most part. If only the business of doing a release wasn't so stressful... But wishful thinking doesn't change the world.


-- David --

PS: If there are other software developers out there who read this, I'd love to hear your comments and experiences on the release process.

Posted: Wednesday, June 06, 2007 1:56 PM by David Harris


Lars said:

Hi David,

I am an electronic engineer but I do over 90 percent software development. I program the main-firmware for test/measure units. This units are multicontroller systems, with LCD and different industrial interfaces, like RS-232, Profibus, CAN, digital I/O and some more. The maintance of the firmware and feature wishes from our customers and sales engineer makes this project a never ending story.

I work on this main project since 2002. I have made EXACTLY the same expierence like you. I agree 100 percent to every point on your list, I could not say it clearer as you did...especially not in english language, my native language is german ;-).

Testing, testing, testing...yeah, that's the most important work before a new firmware version will be released to the customers. We have two stages of testing: The so called alpha-testphase and beta-testphase. The results of every stage have to be documentated (ISO 9001). If the alpha-test failed, the test ends and the developer has to fix the problem. This has never happened till now, because the developer who has worked on the firmware does the alpha test by himself. Before the developer write down the officially result of the alpha-test, he tests his changes to the firmware many times before he builds the release.

The beta-test is done by another person, who has only knowledge about using the test device. The beta tester has to use a beta-test document, which describes every test step in depth. For example, if he tests an input box for signed values, he has to test every key, the navigation within the input box and the value ranges. At the moment the complete beta test needs three to four weeks. The functionality of the changes will test first, of course. For a complex software, no test can give 100 percent safety that it works under all circumstances.

Documenting the software process spent a lot of time, and I know no software developer who likes to commenting his work. But it is very important! For example, the function description is needed to upgrade the beta test document.

We have rules how the developers have to write code. This speeds up the familiarisation phase for source code, which is done by someone other. For documenting source code, we use the very helpful doxygen program and other tools.

To come to an end, I have to say: THANK YOU for this excellent post. And thank you for your excellent software products.

Greetings from germany,


# June 8, 2007 11:16 AM
Anonymous comments are disabled