Just a short update for now - I realize I have been more than usually quiet and remote recently, and for that, I apologize - I have had a significant amount on my plate.
Over the course of this year, I have completely rewritten my TCP/IP code (the part of the program that actually connects to the Internet), to make it more modular, more reliable, and more maintainable. A huge part of this process has involved moving away from the third-party package I previously used to handle SSL/TLS secure connections and instead using the industry-standard OpenSSL suite. With this code complete and tested, I am now well into reworking the various Mercury protocol modules to use it. This has a number of ramifications: firstly, SSL support should be much more reliable, and should work correctly with a much wider range of external clients and servers. Secondly, it will be much easier to keep the SSL components of Mercury up-to-date, and only a single point of change will be necessary at my end to update all the modules, compared with the way it is now, where I have to update each module separately. Thirdly, you will be able to use "real" certificates for SSL, and Mercury will be able to generate the necessary signing requests you can give to certification authorities (CAs). A useful side-effect of the process is also that TCP-based operations in Mercury will be somewhat faster than they have previously been.
Other developments in Mercury are proceeding, most specifically the separation of interface from server process, and the reworking of the help system to use my own help engine: these processes, however, are taking longer than I would like, and it seemed best to bring out a version with the new SSL code as soon as possible. As a result, as soon as the Mercury protocol modules are all converted over to the
new TCP/IP libraries, I'll be giving them to my test team for a short
round of testing, the aim being to release an interim cut of Mercury
(tentatively to be called v4.8), either late in December or early in
January. Other developments will follow in due course over the subsequent months. Licensees need not be worried about this update - all existing licenses will continue to work with it without change.
Again, my apologies for the lack of regular updates: I keep saying I'll get better at it, yet I never seem to be... <slightly embarrassed grin>
All my very best to you all.
-- David --
With the release of Mercury v4.73, it became clear that it was time to start work on major developments within Mercury that are long overdue. The most obvious and immediately necessary of these is the complete separation of the user interface from the working processes at the core of the program. Under Windows 7, Windows Server 2008 and, we presume, upcoming versions of Windows, service processes can no longer have user interfaces that interact with the desktop, which makes configuring Mercury in these environments a bit fiddly. What is needed is a user interface that can run anywhere, connecting to the working processes to configure them: this would allow you to configure the copy of Mercury running on the server in your machine room from the comfort of your own desktop PC, or from home if there was an after-hours problem. Allied with this is the need to manage mailing lists remotely as well - I am building the new remote facilities in such a way that there will be a simple application allowing list moderators to manage just the lists for which they are responsible from anywhere on the Internet. For unlicensed copies of Mercury, the program will continue to run as an application with the full user interface built-in - and indeed, if you have a license, you can run the program as an application with its full UI *and* still configure it remotely. This ability to have the UI or not as you please is quite powerful and took quite a bit of effort.
After quite a bit of development over the last few months, Mercury is now in a state where the separation of UI and working core is complete. It is now possible to run Mercury as a service with no user interface, and configure it using the full user interface running as a separate process on the same machine: from this, it is a fairly short step to allow the configuration to be fully remote as well. After that, it's just making the consoles for the various processes available remotely and the job will be effectively done. One of the things that makes the process more time-consuming is the need to ensure that very high levels of security apply to the process, so everything is getting bullet-proofed and double-checked to that end.
In other areas, Mercury, like Pegasus Mail, will be migrating to use the industry-standard OpenSSL library for its secure communications, which should improve interoperability and will bring the new ability to use "real" SSL certificates (i.e, ones issued by CAs such as Verisign). As I remarked in a recent Pegasus Mail blog posting, the development of the new common message store for both programs is proceeding well and should finally come to fruition this year. A side-effect of the new message store will be that it will be possible for me to rework large parts of the MercuryI IMAP server that are currently limited by the old message store. The aim is to produce faster, more reliable IMAP service with support for a number of the more important modern IMAP extensions, such as SORT and IDLE.
Well that's it for now - time to get back to working on any of the thousands of things I seem to have piled up on my desktop. All my very best to you all!
-- David --
In response to considerable user demand, I have begun work on a significant change to the way Mercury manages domain names. In all versions of Mercury up to and including the current ones, it has been possible to have multiple domain names serviced by a single Mercury instance, but all such domains shared a single user database (so, if you had a user called "john", then that user would appear in all the domains serviced by Mercury). Over time, it has become desirable to allow different domains to have their own users and mailing lists, separate from other domains on the server. There has also been growing demand for specialized types of domain, such as "MX Domains", where a Mercury instance can act as a secondary mail server for a domain that is not otherwise local (i.e, has no local users on the Mercury system). Finally, with a move to truly separate domains, it makes sense to allow domains to have their own specific filters, content and spam controls, whitelists and other mail processing controls, something not possible in the current implementation.
Providing these and other new capabilities will require considerable internal restructuring of the way Mercury works, and as an unavoidable consequence, certain internal Mercury API calls (the special functions made available to developers of third party expansion and plugin modules) are going to be affected. It is my aim to keep the disruption to the interfaces as small as possible, but there is inevitably going to be the possibility that some some existing third-party Mercury Daemons might require some adjustments. If you are a developer of Mercury Daemons, I would like to invite you to join the Mercury beta testing process so you can be kept informed of the changes well before the new code sees public release - please mail me directly, using my David.Harris@pmail.gen.nz
address if you would like to take part.
Just a short note to let you know that Mercury/32 v4.71 is ready for release. We're currently trying to get a problem sorted out with Symantec - their SEP product is incorrectly identifying the help files in the Mercury archive as viral, which results in failed installations. Why is it always Symantec who mess us around, I wonder? This is the third time in three years, yet no other vendor of A/V products has incorrectly tagged us in that time. Oh well...
Once the problem with Symantec is sorted, the release archive will be made publicly available. The big new feature in v4.71 is the ability to run as a native Windows Service, which is a major step forward for usability in server-based environments.
Release announcements will be made here and in the forums as soon as the archive is available. A formal release of Pegasus Mail v4.5 (with some cool new capabilities) should be out very soon as well.
-- David --
Just a short posting to let you know that there should be a minor update release of Mercury/32 v4.6, probably with the version number v4.7, late in February.
The changes will not be gigantic - mostly bug fixes and small incremental improvements, such as allowing IMAP connections via direct-connect SSL.
We have begun a significant redevelopment process on some of the core innards of Mercury - most specifically the folder management layer and the user database subsystem: these changes, along with increased Windows integration, native service support, and increased web management capabilities will be the focus of our work this year (although I do have one "surprise" feature coming up in the next few months that I believe many sites will find extremely useful).
While I hate putting in shameless plugs, times are very hard, and all license purchases for Mercury are gratefully received at the moment. Furthermore, if you know of people you think might benefit from Mercury, please introduce them to it - the more the word gets around, the more likely it is that the program can continue to thrive.
-- David --
Mercury/32 v4.6 is now late in testing, and I thought I'd give you a preview of some the new capabilities it holds.
- Threaded core: The core module (the central part of the program that handles mail delivery and routing) is now somewhat multithreaded, resulting in throughput performance increases of anything up to 350% on busy queues.
- MercuryP (the POP3 server) has been totally overhauled. It is now vastly faster, and a number of problems have been fixed (in particular, an issue arising from a long-standing bug in Windows where message UIDs would change after a daylight savings time change, resulting in mail being downloaded again). It also now includes the short-term blacklisting capabilities found in MercuryI and MercuryS for handling pests and brute-force password crackers.
- MercuryP now supports login-time filtering: simply by adding any of several filter criteria to the username you use to login to MercuryP, you can control the mail it will present to you. This means that if you only want to see unread urgent mail from "bob" on your PDA, you can now do this, just by adding the string "(urgent,new,from=bob)" to your login name.
- Mercury now includes a new commandline mail generator called MSendTo: MSendTo can generate mail in a wide range of formats and writes directly to the Mercury queue. It will be very useful for system administrators and anyone wanting to send mail under programmatic control.
- New consoles: the Mercury protocol modules now have a new console design and interface. I know this doesn't sound like much, and you won't notice a lot of difference, but it's a huge step towards making the Mercury user interface remotable.
- The MercuryB MLSS mailing list subscription management interface has been totally overhauled. It now offers important new features like the ability to change your status for all lists on the server and to retrieve your list password by e-mail, and has numerous improvements and fixes. A mailing list moderator utility is in development but probably won't be available until the next release.
- MercuryI (the IMAP server) has had a number of fixes, and now caches the inbox, resulting in a huge performance increase when opening that folder.
- Lots of small bug fixes.
- Update notifications and advisories: licensed copies of Mercury will be able to receive notifications of new updates and advisories on security and other issues. License uptake on Mercury has been fair since the licenses became available, but it could definitely be better, and I'm hoping this feature might act as a motivator.
It's possible that this feature list may expand prior to release, but right now, release is itself a feature. I hope to get v4.6 out the door soon.
-- David --
With Mercury/32 v4.5 now into the end stages of pre-release confirmation (the release candidate archive went out to the beta testers last night), it's safe to say that the light is visible at the end of what has been a very, very long tunnel.
Now, I would never attempt to defend the fact that there has been a three year gap between releases - a delay like that is damaging, and it should never have happened. Suffice it to say that I have learned a number of difficult lessons about managing large, complex projects as a result of this process, and that it should thus be possible to avoid having it ever happen again. That said, though, I suspect that very few people out there really have any idea what it's like to make a release of a piece of major software, especially when that software is commonly used as critical infrastructure... So, here are some purely personal insights into why I find getting up to and through a release so hard.
1: Testing, testing, testing Those of you who use it may have some appreciation of just how feature-rich Mercury is: the program has settings for almost anything imaginable, and under the hood, there is a very rich plugin interface that allows clever people like Lukas Gebauer to add the things I either can't or haven't. The more feature-rich a program becomes, the harder it is to test, because there are so many more variables and so many more paths through the code. Add to this the fact that Mercury is heavily threaded (a technical term meaning that it can be doing many things at the same time) and that thread synchronization issues are among the hardest problems to reproduce, diagnose, and fix, and the testing process becomes a literal minefield. Testing Mercury is far, far harder than testing Pegasus Mail - orders of magnitude harder. Every time a feature is added to Mercury, testing begins again almost afresh, because every addition can have side-effects - unanticipated problems where the addition disturbs something already in existence - that have to be tracked down.
2: The dreariness of maintenance There was an interesting thread in the off-topic forum here recently where a community member asked why programs had bugs. On the surface, his comment was perfectly fair, but in truth, any program larger than a few lines is going to have problems that creep in: the larger the program, the more possibilities there are for bugs, and complexity increases those possibilities even further. Software is never finished - it's always growing and changing, which means that the bugs are growing and changing too. Allied to this is the perpetual process of rewriting code - taking things that used to be "good enough" but now are not, and making them "good enough" again. At some stage in the life of all large software projects, you reach a
stage where you spend more time tracking down bugs and rewriting old code than you
do adding new features. This maintenance can't be avoided and mustn't be rushed (or else you'll typically just make things worse), but unfortunately, programmers are almost always relatively normal human beings, and like change and variety in their lives: having to spend most of your time reinventing or reworking old things can be pretty dreary, and on a cold Winter's morning, it's often the last thing you'll want to do. Maintenance, therefore, can be a slow process, if only for human reasons.
3: User expectations It's a common truism in the industry now that user expectations get higher with each passing day: users expect programs to run flawlessly, to be easy to use, to cost nothing, and to have feature sets that match their every need. With higher expectation has also come lower tolerance, whether tolerance for errors, tolerance for ease of use, or even tolerance for appearance. The problem with higher expectations is that they are, obviously, harder to meet - but worse, they are usually exponentially harder to meet: making something "prettier" is actually a very difficult and time-consuming process, and even though it produces little actual tangible benefit, users expect it, even in programs like Mercury. If it's not there, lower tolerance means that people get more aggressive and more turned off. The difficulty and expense of producing programs that meet user expectations these days is reaching the point where only large corporations with huge resources will ever be able to afford the process. User expectations are basically killing the small-to-medium sized player.
4: Resources, resources, resources I suspect most non-software-developers really have no conception of what it costs to produce a piece of software - "A PC on a table in your bedroom - what more do you need?" (that's an actual quote from someone who shall remain nameless, by the way). There are compilers (Visual Studio costs over $2000 here now), documentation tools (FrameMaker costs $1700), file servers for each supported environment, test servers, testbed systems for various versions of Windows, high-speed Internet links (unfortunately very expensive in New Zealand), domain names, certificate fees, support subscriptions and upgrades for software and development tools, books and reference materials, third-party software licenses (we pay over US$2000 per annum for the very small number of third-party tools we license), sundry development tools (such as Igor Arsenin's excellent TaskInfo product), electricity, banking and business costs, accounting costs... The list goes on. The cost of doing business gets higher every year, and peoples' willingness to pay in support seems to dwindle (and please, don't think of this as me criticizing my user community - it's a general statement about the world the Internet has become). Reduced resources mean making do with less, which in turn extends the time to release.
5: The dread of feedback This one's a strange one, because it's a complete flip-flop from what I used to experience. In the earliest days of Pegasus Mail and Mercury, releases were exciting, because they were my chance to show the world all the cool new things I'd done. Because the world was more of a community back then, I got lots of supportive mail each time I did a release, which was also exciting and uplifting. These days, in the first few days after a release, the mail I tend to get is different: sure, there are still people who write to say thank you without strings or qualifications, but more normally the messages start with "Version xx is great, but...". It's really hard to describe how soul-destroying it is to release something you've worked really hard to produce, only to start getting wishlists and gripes almost immediately. When I say "gripes", I'm not saying that people are necessarily rude - it's usually just people expressing their annoyance that their pet feature or fix didn't make it into the release. And then there is the out-and-out rude mail; sadly, I always get a couple of those: these are the people who tell me that I should "just stop wasting my time and get a job at McDonalds (if they'll have me)". (And yes, that one's an actual quote as well). There's only a few, and they're always from people who either have a serious grudge of some kind or are simply ill-bred or psychologically troubled, but they have an impact out of all proportion to their importance. Just one of those messages can depress me so badly that I'm incapable of being a functioning human being for the remainder of the day. After seventeen years of doing this, you would think that I might have developed a thicker skin about this kind of thing, but I haven't: it really hurts, and over time, the anticipation of the mail I know I'm going to have to handle has made the process of release more and more daunting, more and more difficult.
Now, this probably all sounds like me complaining, but I'm not really... All I'm doing is summarizing a reality that became inevitable once the Internet reached the level of being a commodity. I should probably toughen up and lighten up, especially since I still actually enjoy the development process for the most part. If only the business of doing a release wasn't so stressful... But wishful thinking doesn't change the world.
-- David --
PS: If there are other software developers out there who read this, I'd love to hear your comments and experiences on the release process.
Think of the Czech Republic and depending on your frame of mind, images of fantastic architecture, beautiful women, or a long and rich cultural heritage might flicker through your thoughts. For me, though, when I think of the Czech Republic, I think of Lukas Gebauer (and I apologize now for not getting the accents in his name right).
Lukas is a software developer, and a very clever one indeed. He wrote the SpamHalter Bayesian spam filter for Pegasus Mail, and has written numerous plugins for Mercury, three of which (SpamHalter, ClamWall and GreyWall) are now standard components in the Mercury distribution. What probably amazes me more than anything else about Lukas is that he has written these very high-quality programs with almost no assistance from me - not that I wouldn't have been glad to provide it, but he seemingly hasn't needed it! I doubt I could have written code to use my own interfaces with so little assistance, so his work is all the more special because of that.
His latest plugin, GreyWall, is possibly the most startling plugin you'll ever encounter for Mercury. It uses a new event interface I developed for Mercury/32 v4.5, and simply turns away any connections from unknown server/sender combinations with a temporary (400-series) error. For real mail, all this means is that the first message from a particular server may take a few extra minutes to be delivered (all subsequent mail goes through with no delays). For spam zombies, however, the effect is quite dramatic. Current generation spam zombies are not set up to handle retry conditions - if they get one, they typically just skip the address and move on: this means that enabling GreyWall will result in a huge amount of your incoming spam being "turned away at the door", costing you no bandwidth, time or processing power. Experience in the test team has shown that enabling GreyWall can reduce spam levels by up to 90% even with no other filtering in place, and with a near-zero false positive rate!
As I noted above, GreyWall is now a standard part of Mercury, installable directly from the Mercury installer. When you decide to install GreyWall, or SpamHalter, or ClamWall, think of the Czech Republic and of Lukas, and consider going to his site to offer a little financial support to him: he's a very rare breed, and we all benefit from his work.
-- David --
May 6 2007
I've recently had to take a small diversion from getting Mercury/32 v4.5 ready for release to prepare a patch for v4.01. The patch works around a vulnerability in the IMAP server where sending IMAP "literals" formatted in a particular way would crash the running copy of Mercury/32. While I was at it, I corrected a couple of fairly significant memory leaks in the IMAP server - these had been discovered during the implementation of the audited memory allocation manager in v4.5 (see my last blog post for more on this).
Having to prepare patches like this raises very mixed feelings in me: on the one hand, I accept that vulnerabilities are inevitable in software that exposes itself to the world, and feel that I should be grateful that the vulnerability was discovered before (as far as we know) any particular damage resulted from it. On the other hand, though, some of the people who seem to spend so much time discovering these vulnerabilities also strike me as being sad, dangerous individuals with fairly significant social problems. The fact that I had to learn about this vulnerability very indirectly (i.e, the person who discovered it made no effort to report it to me) annoys and saddens me. OK, get your jollies by trying to break other peoples' work - that's fine, and it's even quite constructive and helpful from a certain perspective; but if you succeed, for Heaven's sake tell the author so he can fix it! Any kudos you might get for being such a clever bunny and discovering an obscure problem is immediately eviscerated by not acting responsibly with that discovery.
Anyway, to whomever discovered this weakness - my sincerest thanks: one day, perhaps I'll be able to meet you and shake you warmly by the throat.
-- David --
Mercury/32 v4.5 is all about stability and robustness, and it was with this in mind that I began a major overhaul of the way memory is handled in Mercury about two weeks ago. It was a toss-up whether it was more important to get a release out the door or to make these important changes, but in the end, I decided that I've spent so long trying to get v4.5 right that it was really essential to do this work.
The changes insert a new layer between Mercury and the system, responsible for tracking the way memory is allocated, used and deallocated. For those of you who aren't programmers, the computer's RAM is a finite resource that you have to ask to use, and when you've finished using it, you have to give it back to the system. If you use a piece of memory, but don't give it back, then you end up with what's commonly known as a "memory leak" - the program uses more and more memory until eventually the system runs out and crashes. This all sounds straightforward, but as a famous man once said, "nothing is ever that easy"... There are many paths through a program, and it's perilously easy to lose track of a chunk of memory you've allocated, resulting in it never being freed. The problem is so fundamental that I don't think it's unreasonable to say that just about every widely-used Windows application will have memory leaks at some level - it's just that for many programs, particularly those that run for short periods then terminate, it's not an issue (Windows always recovers all memory allocated by a program when it terminates).
In a program like Mercury, which can run for weeks at a time, though, memory leaks can get to be a serious issue - hence the need to audit memory use closely. This is what the new code I have now implemented does - it allows me to see at a glance any memory that is not being released when the program exits. The new audited memory allocator has already proven worth the release delay it has caused, though, because it has allowed me to track down a number of memory leaks in the MercuryI IMAP server - leaks that I would never have had a chance of finding otherwise.
In some ways, the new allocator is a crystallization of what has happened right throughout the whole process of developing Mercury/32 v4.5 - it's crucially important, has had an enormous impact on the reliability and robustness of the program, and, alas, is totally invisible to the end user. As a programmer, it can be frustrating to know just how much effort has gone into something "under the hood", as it were, yet also to know that the end user will have no idea or understanding of that effort... *sigh*. Oh well, I guess that's why programmers get all the money, gorgeous chicks and fast cars they do... <wry grin>
-- David --
This is a short post to indicate where we stand with Mercury/32 at March 26 2007.
The next release of Mercury/32 will be v4.5 - the skip in version numbers indicates that a huge amount has gone on internally, but that it doesn't look that different. V4.5 has been almost entirely about reliability, scalability, performance and robustness: on the surface, not much has changed, but "under the hood" there have been thousand of changes and improvements. That said, there are some new features there as well:
- Disclaimers: a new filtering rule action makes it easy to insert text into any message. While intended mainly to allow sites to inject disclaimers into the mail sent by their users, it can also be used for many other purposes.
- Public folder delivery: Mercury can now deliver directly to Pegasus Mail public folders, making the facility vastly more useful
- Delivery status notifications: Mercury can now generate customizable progress notifications for messages that are delayed or being retried
- Mail Queue overhauled: The Mercury main mail queue has been totally overhauled and should now be both vastly more reliable and anything up to 100 times faster than before
- Spamhalter and Clamwall included: Lukas Gebauer's terrific Spamhalter Bayesian filter and Clamwall Antivirus integration modules are now a standard part of the program
- Many reliability improvements, especially to the mailing list editor, the MercuryI IMAP server and to error reporting in most modules
Starting with v4.5, Mercury will become a licensed product. We have spent a long time agonizing over this move, and exactly what it means: what we have come up with is essentially an honour system. As shipped, the program will have no restrictions or limitations, but will have an enlarged status bar indicating that it is unlicensed, and will inject a "Sent by an unlicensed copy of Mercury" header in all outgoing mail. Licenses will be free for individuals, families and charitable organizations. Businesses, government bodies and academic institutions will be obliged to purchase licenses based on the number of users they have. Once a license is installed, the enlarged status bar will return to normal and the outgoing mail header will be removed. These license terms represent a compromise between my personal desire to make Mercury totally free, and the reality of needing funds to live and continue development; the idea is to provide sufficient motivation for sites who are likely to pay for a license to do so, without placing overly onerous restrictions on those who cannot justify doing so.
Mercury/32 v4.5 is essentially ready for release now: I just have to finalize the licensing issues, and it will then be made available almost immediately.
-- David --
This blog is intended as a kind of diary on the development of Mercury/32. As I work on the program, I will periodically make
postings here describing what's being done, and what plans are being
Only I and my formal beta test team can post or reply in this blog - if
you are a registered member and see an issue on which you would like to
comment, please start a new thread in the appropriate forum.
-- David --