I've been doing this (writing software) for most of my 46 years, and professionally for well over 20 now: one notion that imprints itself more and more forcibly upon me with each passing year is how much more complicated everything seems to get.
I suppose it is in the nature of the human animal to elaborate: caves become tents, which become mud huts, then grow into drafty stone castles, and eventually into towering skyscrapers... Or, bones become sharpened sticks, which become arrows and spears, then muskets, rifles, machine guns until now the US military is testing the "vomit cannon" (a weapon that induces mass nausea by sound waves). With each innovation, every incremental step along the path from humble hamlet to sparkling skyline, things get more complex... One seldom-understood effect of this is that knowledge becomes more and more specialized: any chimp can wield a bone as a weapon, but how many single people would know how to build a vomit cannon from scratch? As tasks get more complex, they progressively surpass the ability of any individual to encompass them in their entirety. The way humans deal with this problem is by specializing - by splitting the task into smaller, more manageable pieces then splicing them together at the end.
The problem with dealing with complexity in this way is that it also introduces complication. Fred Brooks famously stated that "Adding manpower to a late software project makes it later", which can be taken as a general comment on the complication that arises when you increase the number of people involved in any task (not just software tasks). Different people approach problems in different ways, have different priorities, and differing sets of skills: considering this, it always seems miraculous to me that *any* large software project ever yields results at all, let alone working well.
Pegasus Mail is now around 460,000 lines of C code, and Mercury around 200,000. By commercial standards, these are middle-sized projects, and well beyond the optimum size for a single developer. With size comes complexity and complication - nothing in these programs is easy any more - indeed, it makes me weep when I compare the productivity that I could achieve when I was writing the early DOS versions of Pegasus Mail back in 1990 with the almost glacial progress I can manage now. Features that I could have added in a day back then would take a month now, and the depth of knowledge and background required to do them is vastly greater. I think I'm a better programmer now than I was in 1990, but it takes me ten times longer to achieve anything, and it's hard to describe how frustrating that is.
I must admit that when I think of Windows Vista with its tens of millions of lines of code, I can scarcely credit that it works: and the more I consider it, the more a kind of visceral, almost feral mistrust arises in me - how can I trust something so complex that no single human being is capable of fathoming it in its entirety? Would I trust my life to it? My reputation? My identity? Honestly, it's beginning to scare the hell out of me that we're becoming so dependent on this type of technology.
Perhaps I'm being silly, wistful and naive, but wouldn't it be nice if things were simpler? "Simple" is the lost virtue of this age, and I can't help but worry that we're going to miss it far more than we realize.
-- David --