With Mercury/32 v4.5 now into the end stages of pre-release confirmation (the release candidate archive went out to the beta testers last night), it's safe to say that the light is visible at the end of what has been a very, very long tunnel.
Now, I would never attempt to defend the fact that there has been a three year gap between releases - a delay like that is damaging, and it should never have happened. Suffice it to say that I have learned a number of difficult lessons about managing large, complex projects as a result of this process, and that it should thus be possible to avoid having it ever happen again. That said, though, I suspect that very few people out there really have any idea what it's like to make a release of a piece of major software, especially when that software is commonly used as critical infrastructure... So, here are some purely personal insights into why I find getting up to and through a release so hard.
1: Testing, testing, testing Those of you who use it may have some appreciation of just how feature-rich Mercury is: the program has settings for almost anything imaginable, and under the hood, there is a very rich plugin interface that allows clever people like Lukas Gebauer to add the things I either can't or haven't. The more feature-rich a program becomes, the harder it is to test, because there are so many more variables and so many more paths through the code. Add to this the fact that Mercury is heavily threaded (a technical term meaning that it can be doing many things at the same time) and that thread synchronization issues are among the hardest problems to reproduce, diagnose, and fix, and the testing process becomes a literal minefield. Testing Mercury is far, far harder than testing Pegasus Mail - orders of magnitude harder. Every time a feature is added to Mercury, testing begins again almost afresh, because every addition can have side-effects - unanticipated problems where the addition disturbs something already in existence - that have to be tracked down.
2: The dreariness of maintenance There was an interesting thread in the off-topic forum here recently where a community member asked why programs had bugs. On the surface, his comment was perfectly fair, but in truth, any program larger than a few lines is going to have problems that creep in: the larger the program, the more possibilities there are for bugs, and complexity increases those possibilities even further. Software is never finished - it's always growing and changing, which means that the bugs are growing and changing too. Allied to this is the perpetual process of rewriting code - taking things that used to be "good enough" but now are not, and making them "good enough" again. At some stage in the life of all large software projects, you reach a
stage where you spend more time tracking down bugs and rewriting old code than you
do adding new features. This maintenance can't be avoided and mustn't be rushed (or else you'll typically just make things worse), but unfortunately, programmers are almost always relatively normal human beings, and like change and variety in their lives: having to spend most of your time reinventing or reworking old things can be pretty dreary, and on a cold Winter's morning, it's often the last thing you'll want to do. Maintenance, therefore, can be a slow process, if only for human reasons.
3: User expectations It's a common truism in the industry now that user expectations get higher with each passing day: users expect programs to run flawlessly, to be easy to use, to cost nothing, and to have feature sets that match their every need. With higher expectation has also come lower tolerance, whether tolerance for errors, tolerance for ease of use, or even tolerance for appearance. The problem with higher expectations is that they are, obviously, harder to meet - but worse, they are usually exponentially harder to meet: making something "prettier" is actually a very difficult and time-consuming process, and even though it produces little actual tangible benefit, users expect it, even in programs like Mercury. If it's not there, lower tolerance means that people get more aggressive and more turned off. The difficulty and expense of producing programs that meet user expectations these days is reaching the point where only large corporations with huge resources will ever be able to afford the process. User expectations are basically killing the small-to-medium sized player.
4: Resources, resources, resources I suspect most non-software-developers really have no conception of what it costs to produce a piece of software - "A PC on a table in your bedroom - what more do you need?" (that's an actual quote from someone who shall remain nameless, by the way). There are compilers (Visual Studio costs over $2000 here now), documentation tools (FrameMaker costs $1700), file servers for each supported environment, test servers, testbed systems for various versions of Windows, high-speed Internet links (unfortunately very expensive in New Zealand), domain names, certificate fees, support subscriptions and upgrades for software and development tools, books and reference materials, third-party software licenses (we pay over US$2000 per annum for the very small number of third-party tools we license), sundry development tools (such as Igor Arsenin's excellent TaskInfo product), electricity, banking and business costs, accounting costs... The list goes on. The cost of doing business gets higher every year, and peoples' willingness to pay in support seems to dwindle (and please, don't think of this as me criticizing my user community - it's a general statement about the world the Internet has become). Reduced resources mean making do with less, which in turn extends the time to release.
5: The dread of feedback This one's a strange one, because it's a complete flip-flop from what I used to experience. In the earliest days of Pegasus Mail and Mercury, releases were exciting, because they were my chance to show the world all the cool new things I'd done. Because the world was more of a community back then, I got lots of supportive mail each time I did a release, which was also exciting and uplifting. These days, in the first few days after a release, the mail I tend to get is different: sure, there are still people who write to say thank you without strings or qualifications, but more normally the messages start with "Version xx is great, but...". It's really hard to describe how soul-destroying it is to release something you've worked really hard to produce, only to start getting wishlists and gripes almost immediately. When I say "gripes", I'm not saying that people are necessarily rude - it's usually just people expressing their annoyance that their pet feature or fix didn't make it into the release. And then there is the out-and-out rude mail; sadly, I always get a couple of those: these are the people who tell me that I should "just stop wasting my time and get a job at McDonalds (if they'll have me)". (And yes, that one's an actual quote as well). There's only a few, and they're always from people who either have a serious grudge of some kind or are simply ill-bred or psychologically troubled, but they have an impact out of all proportion to their importance. Just one of those messages can depress me so badly that I'm incapable of being a functioning human being for the remainder of the day. After seventeen years of doing this, you would think that I might have developed a thicker skin about this kind of thing, but I haven't: it really hurts, and over time, the anticipation of the mail I know I'm going to have to handle has made the process of release more and more daunting, more and more difficult.
Now, this probably all sounds like me complaining, but I'm not really... All I'm doing is summarizing a reality that became inevitable once the Internet reached the level of being a commodity. I should probably toughen up and lighten up, especially since I still actually enjoy the development process for the most part. If only the business of doing a release wasn't so stressful... But wishful thinking doesn't change the world.
Cheers!
-- David --
PS: If there are other software developers out there who read this, I'd love to hear your comments and experiences on the release process.
<p>With Mercury/32 v4.5 now into the end stages of pre-release confirmation (the release candidate archive went out to the beta testers last night), it's safe to say that the light is visible at the end of what has been a very, very long tunnel.
Now, I would never attempt to defend the fact that there has been a three year gap between releases - a delay like that is damaging, and it should never have happened. Suffice it to say that I have learned a number of difficult lessons about managing large, complex projects as a result of this process, and that it should thus be possible to avoid having it ever happen again. That said, though, I suspect that very few people out there really have any idea what it's like to make a release of a piece of major software, especially when that software is commonly used as critical infrastructure... So, here are some purely personal insights into why I find getting up to and through a release so hard.
<font color="#0000ff"><i>1: Testing, testing, testing</i></font>&nbsp;&nbsp; Those of you who use it may have some appreciation of just how feature-rich Mercury is: the program has settings for almost anything imaginable, and under the hood, there is a very rich plugin interface that allows clever people like Lukas Gebauer to add the things I either can't or haven't. The more feature-rich a program becomes, the harder it is to test, because there are so many more variables and so many more paths through the code. Add to this the fact that Mercury is heavily threaded (a technical term meaning that it can be doing many things at the same time) and that thread synchronization issues are among the hardest problems to reproduce, diagnose, and fix, and the testing process becomes a literal minefield. Testing Mercury is far, far harder than testing Pegasus Mail - orders of magnitude harder. Every time a feature is added to Mercury, testing begins again almost afresh, because every addition can have side-effects - unanticipated problems where the addition disturbs something already in existence - that have to be tracked down.
</p><p><font color="#0000ff"><i>2: The dreariness of maintenance&nbsp;&nbsp; </i></font>There was an interesting thread in the off-topic forum here recently where a community member asked why programs had bugs. On the surface, his comment was perfectly fair, but in truth, any program larger than a few lines is going to have problems that creep in: the larger the program, the more possibilities there are for bugs, and complexity increases those possibilities even further. Software is never finished - it's always growing and changing, which means that the bugs are growing and changing too. Allied to this is the perpetual process of rewriting code - taking things that used to be "good enough" but now are not, and making them "good enough" again. At some stage in the life of all large software projects, you reach a
stage where you spend more time tracking down bugs and rewriting old code than you
do adding new features. This maintenance can't be avoided and mustn't be rushed (or else you'll typically just make things worse), but unfortunately, programmers are almost always relatively normal human beings, and like change and variety in their lives: having to spend most of your time reinventing or reworking old things can be pretty dreary, and on a cold Winter's morning, it's often the last thing you'll want to do. Maintenance, therefore, can be a slow process, if only for human reasons.
<font color="#0000ff"><i>
3: User expectations</i></font>&nbsp;&nbsp; It's a common truism in the industry now that user expectations get higher with each passing day: users expect programs to run flawlessly, to be easy to use, to cost nothing, and to have feature sets that match their every need. With higher expectation has also come lower tolerance, whether tolerance for errors, tolerance for ease of use, or even tolerance for appearance. The problem with higher expectations is that they are, obviously, harder to meet - but worse, they are usually <i>exponentially</i> harder to meet: making something "prettier" is actually a very difficult and time-consuming process, and even though it produces little actual tangible benefit, users expect it, even in programs like Mercury. If it's not there, lower tolerance means that people get more aggressive and more turned off. The difficulty and expense of producing programs that meet user expectations these days is reaching the point where only large corporations with huge resources will ever be able to afford the process. User expectations are basically killing the small-to-medium sized player.
<font color="#0000ff"><i>4: Resources, resources, resources</i></font>&nbsp;&nbsp; I suspect most non-software-developers really have no conception of what it costs to produce a piece of software - "A PC on a table in your bedroom - what more do you need?" (that's an actual quote from someone who shall remain nameless, by the way). There are compilers (Visual Studio costs over $2000 here now), documentation tools (FrameMaker costs $1700), file servers for each supported environment, test servers, testbed systems for various versions of Windows, high-speed Internet links (unfortunately very expensive in New Zealand), domain names, certificate fees, support subscriptions and upgrades for software and development tools, books and reference materials, third-party software licenses (we pay over US$2000 per annum for the very small number of third-party tools we license), sundry development tools (such as Igor Arsenin's excellent <i>TaskInfo </i>product), electricity, banking and business costs, accounting costs... The list goes on. The cost of doing business gets higher every year, and peoples' willingness to pay in support seems to dwindle (and please, don't think of this as me criticizing my user community - it's a general statement about the world the Internet has become). Reduced resources mean making do with less, which in turn extends the time to release.
<font color="#0000ff"><i>5: The dread of feedback</i></font>&nbsp;&nbsp; This one's a strange one, because it's a complete flip-flop from what I used to experience. In the earliest days of Pegasus Mail and Mercury, releases were exciting, because they were my chance to show the world all the cool new things I'd done. Because the world was more of a community back then, I got lots of supportive mail each time I did a release, which was also exciting and uplifting. These days, in the first few days after a release, the mail I tend to get is different: sure, there are still people who write to say thank you without strings or qualifications, but more normally the messages start with "Version xx is great, but...". It's really hard to describe how soul-destroying it is to release something you've worked really hard to produce, only to start getting wishlists and gripes almost immediately. When I say "gripes", I'm not saying that people are necessarily rude - it's usually just people expressing their annoyance that their pet feature or fix didn't make it into the release. And then there is the out-and-out rude mail; sadly, I always get a couple of those: these are the people who tell me that I should "just stop wasting my time and get a job at McDonalds (if they'll have me)". (And yes, that one's an actual quote as well). There's only a few, and they're always from people who either have a serious grudge of some kind or are simply ill-bred or psychologically troubled, but they have an impact out of all proportion to their importance. Just one of those messages can depress me so badly that I'm incapable of being a functioning human being for the remainder of the day. After seventeen years of doing this, you would think that I might have developed a thicker skin about this kind of thing, but I haven't: it really hurts, and over time, the anticipation of the mail I know I'm going to have to handle has made the process of release more and more daunting, more and more difficult.
Now, this probably all sounds like me complaining, but I'm not really... All I'm doing is summarizing a reality that became inevitable once the Internet reached the level of being a commodity. I should probably toughen up and lighten up, especially since I still actually enjoy the development process for the most part. If only the business of doing a release wasn't so stressful... But wishful thinking doesn't change the world.
Cheers!
-- David --
PS: If there are other software developers out there who read this, I'd love to hear your comments and experiences on the release process.
</p>
