Just a short update on the status of WinPMail v4.5.
I had hoped to have the program released by now, but we've run into a late problem that I feel we have to resolve before a release can take place.
It is currently my intention to release a v4.5 public beta to this community as soon as the code is next in a stable state - hopefully before the end of this week. Once the public beta has had a chance to circulate a little and we're sure there are no remaining serious issues standing in the way of a full release, v4.5 will be made available on an unrestricted public basis.
Cheers!
-- David --
I've had a number of people ask me if I could put together a blog posting describing what I'm doing at the moment, and what plans there are for Pegasus Mail and Mercury in the short-to-medium term.
Well, the first thing I'll tell you is that my current priority is to get Pegasus Mail v4.5 out the door. On the face of it, it won't look like much has happened, and I suspect many people will be rather disappointed by v4.5, but in reality, more than 50% of the entire codebase of the program has been modified in some way during the shift from Borland C++ to Visual C++; it's been a huge, largely unrewarding job that simply had to be done. In the process, though, we've caught and fixed literally hundreds of small and medium-sized bugs that have been present in the program for periods ranging from recent history to more than a decade.
A huge amount of effort this year has gone into releasing Mercury/32 v4.6 - once again, there was a lot of code modernization going on that doesn't yield obvious visible benefits, but which simply had to be done.
But not everything is porting code and fixing bugs. I've also been spending a lot of time familiarizing myself with new technologies that will have significant bearing on both Mercury and Pegasus Mail:
- For Mercury, I've been spending a lot of time learning about CSS and XHTML, because the future of Mercury is clearly heavily web-oriented.
- I've been spending a lot of time learning about SQL and particularly SQLite, because it's clear that both Pegasus Mail and Mercury are going to need robust database facilities in future.
- I've developed my own object programming interface for both programs: this is a key technology that will have a huge bearing on future development by making it easier to extend the programs' capabilities without side-effects.
- I'm well into developing a complete replacement for the fundamental message store used by both Pegasus Mail and Mercury. This new component, called MailStore, is easily the single most important development in either product in more than a decade, and should be in a working state towards the end of this year.
- I've spent a lot of time this year working on documentation - the bane of all programmers... The Mercury/32 Daemon Developer Kit in particular was a huge amount of work that was vastly overdue, and I do hope to start seeing new Mercury Daemons appearing before much longer.
- Finally, I've spent a lot of time developing tools - either because I was forced to do so (as in the case of the new help system, which I was forced to do by Microsoft), or because I believe they'll be important in upcoming releases of either program.
I realize I'm not being quite as specific as I'm sure many of you would like: I'm not saying that "X feature will be working by August", or that "Y feature is now operational": I promise that as I get to a position where I can make reasonable statements about new developments I will do so, but in truth, the last two years have been a gigantic, painful, tedious process of modernization and tooling up for new capabilities, rather than actually working on those capabilities per se.
I also realize that I've been somewhat of a stranger to the forums this year: I apologize for that, and will say honestly that my absence has largely been a result of tech-support burnout... Doing technical support is a very soul-destroying process, because you're typically seeing people at their worst and most stressed, and nobody really likes spending all of their time dealing with the bad aspects of their work. As a result, spending too long doing technical support is a recipe for depression and loss of motivation, and I've been doing technical support at some level for nearly two decades now. It's my aim to become a more regular participant in the forums, although I will be consciously and actively trying to resist the self-destructive urge to solve every problem I see posted. I'd like to get back into the forums so I can enjoy interacting with the people who use my work without always burning myself out trying to fix all their problems.
So, there you have it - a slightly longer, and maybe slightly more candid posting than I intended, but I hope it serves to give you a little insight into where I am at the moment.
Cheers to you all!
-- David --
I don't think there's any other profession quite like writing software - short of warfare, I can't think of any other area of human endeavour where the potential for casual disaster is so high.
For the last eighteen months, I have been involved in rewriting and modernizing a huge chunk of the 460,000 lines of code that makes up Pegasus Mail. I've written about this (I just can't bring myself to use the verb "blogged", I'm afraid) before, but I thought it might be mildly diverting for some of you to see just exactly how bizarre and off-the-wall this process can often get.
In the course of porting Pegasus Mail from the now ancient Borland compiler I have used for many years to Visual C++, there are various code changes that have been forced on me. One of them is quite basic - the code I use to get listings of the files in a directory. On the face of it, this was straightforward: I evaluated the problem and thought that it might take me a couple of hours to fix. You can see where this is going, I assume... It eventually took me almost five weeks of repeated debugging and testing to get this one simple function working. When I had finally solved it, I sent a message to my beta test team describing what had been involved: here is the text... (note that the reference to "VC12" is to a beta build of the program which is only available to the test team. At the time of writing, we're up to VC13, and are getting ready for a formal release of Pegasus Mail v4.5).
The problem with files not showing up was quite a lot more complicated, and is part of the ongoing saga of the Windows FindFirstFile and FindNextFile functions. To understand this problem (and it's worth summarizing it, I think), you need to have a little history.For the longest time, I have used the Borland compiler family to write my code. I never used much Borland-specific code, but some that I *did* use extensively was a pair of functions called findfirst and findnext, which were used to enumerate the files in a directory. They worked reliably for a long time, but it eventually became clear that they had some problems. Furthermore, there was no direct or portable equivalent of these functions in Visual C++. So, I decided to write my own functions to replace them, making my code much more portable.
Now, Windows has a group of functions called FindFirstFile, FindNextFile and FindClose, which on the surface appear to duplicate the operation of findfirst/findnext... On the surface. In reality, these function calls are quirky and buggy as hell. As an example, they treat wildcard characters quite differently depending on the underlying file system: so, the pattern "*.?" will return one set of files if the underlying file system is FAT, another if it's NTFS, and a third, different set if the file system is NetWare. What's more, different combinations of Windows and file system will return different sets of file attributes (so, a plain search on FAT WILL return files marked readonly, but the same search on NetWare WON'T return files marked readonly unless you specifically request them).
Finally, Windows defines an extensive set of file attributes - things like read-only, hidden, system, compressed and so forth... The problem is that they have kept defining attributes over the years, simply adding new ones whenever they want. As a result, it can be quite tricky to work out from its attributes whether a file is an "ordinary" file or not. To understand what I mean by this, consider that you typically WON'T be interested in files that are marked "Offline", because they either cannot be opened, or will take an inordinately long time to be opened because they have to be recovered from backing store. Similarly, files marked as a "Device" should probably never be accessed other than on explicit requirement.
The reason various of you have found that WinPMail is not seeing certain files on your system (typically files without the archive bit set) is because Windows is returning different sets of attributes for these files depending on the OS and file system... So, on NetWare systems, the archive bit is set or not set (no problem), but on NTFS, if the archive bit is not set, another bit called FILE_ATTRIBUTE_NORMAL *IS* set (a situation not true if the file is on a NetWare volume). All of this is interfering with the test I was using to work out whether a file is "ordinary" or not.
I have now completely abandoned the "ordinary file" test I was using and have changed it to an inverse test - I now check to see if a file is "unusual" (i.e, has certain specific attributes that I'm not interested in by default) and omit it if it is. This is a considerably easier and more deterministic test and using it bypasses all the bizarre vagaries of OS/file system combinations.
VC12, which I'll release tonight, implements the new filescanning code, and I'm 99.9% sure that I've finally got it sorted out. But it's symptomatic of how difficult things are getting that such a basic, fundamental piece of code has been so bloody awkward. I've wasted dozens of hours diagnosing and solving these problems - problems that simply shouldn't have existed in the first place. Hmph.
Sorry if this is tedious or over-technical - I just thought it might be interesting for you to see a little of the kind of thing that goes on "behind the scenes".
More on the v4.5 release soon.
Cheers!
-- David --
[Rant mode on]
Another week where I've been very quiet... This time, though, I find I'm actually quite angry at the reason. For the last three-odd weeks, I have been writing a help system and porting the Pegasus Mail help to it. Why have I been doing this? Because Microsoft broke its contract with developers.
Ever since it was first released, Pegasus Mail has used a Windows help system called WinHelp: WinHelp was a standard part of the operating system and worked quite well. Over time, for reasons I've never really quite understood, Microsoft drifted away from the WinHelp model towards a new help system called HTMLHelp, which uses Internet Explorer to display help pages. I personally never thought HTMLHelp offered any real advantages; furthermore, changing to HTMLHelp would have taken considerable time and effort, and
worse, would have required all my translators to go through the same
awful conversion process, something I wouldn't wish on anyone. I and my translation teams have made a very extensive investment of time and effort in WinHelp, and based on the feedback I regularly get from people, nobody was noticeably unhappy with it (one of the more regular pieces of "nice" feedback I get from people is how comprehensive the help system is).
Enter Windows Vista. Quite late in the pre-release process for Vista, Microsoft suddenly announced that they didn't like WinHelp any more, they didn't think anyone should be using it, and they wouldn't be including it in Vista. What's more, they specifically forbade developers dependent on WinHelp from distributing it with their applications. Microsoft's position was apparently that everyone should be using HTMLHelp, regardless of the effort, expense or practicality of doing so. I also have to say that Microsoft offered exactly zero assistance in moving between formats; there was an existing conversion tool, but it was useless.
That's not the end of the story, though. At some point in the endless patching process for Internet Explorer, Microsoft made a change that effectively crippled HTMLHelp: if an attempt was made to display a help file stored on a shared volume (for instance, a file server), the help file would simply fail to work with an incomprehensible error message ("Action cancelled"). While there are some possible fixes for this problem, they all involve an overall reduction of security and are fairly technical (registry editing). What's more, because the problem would occur in our help file, people would come to *us* for technical support on the issue, even though it's actually a bug in a Windows system component.
So here we are: under Vista, the only help system available to us is one which will fail to work in one of the most common installation scenarios for Pegasus Mail. The one that works fine (WinHelp) can only be downloaded from Microsoft by the end user - we can't supply and install it ourselves. From a developer's perspective, this is a nightmare. In the end, the ONLY solution available to me was to write my own help compiler and display subsystem, so I can continue to use the sources that I and my translators have so laboriously produced. This has not been an easy process, but I think the results are good: my testers got the first working cut of this code last night, and so far the feedback has been quite positive.
But I can see you're asking why I claim that Microsoft has broken a contract... ? When programmers develop for an environment like Windows, they use programming interfaces called "APIs", provided by the environment's developer - in this case, Microsoft. The API is a contract between Microsoft and the developers who use it, a contract that says "Provided you follow the rules we lay out in the documentation, we'll guarantee that your program will work as you expect, and that it will continue to work as you expect in future". In this case, Microsoft have reneged on the deal. Instead of simply leaving developers like me in the lurch, they could have done any of a number of things: they could have allowed us to distribute WinHelp with our applications; they could have provided proper transition tools; they could have ensured that the system they WERE providing actually worked properly... But they did none of these things. They didn't even give much in the way of warning (six months is far from adequate for something as important and substantial as a help system).
So, I've written a complete help system. I think it's probably quite a lot better than what we had, and it's at least as good as HTMLHelp; what's more, it will work correctly even where HTMLHelp does not. But I should not have had to waste a month of my life doing this when there are so many more interesting and important things to do. I really wouldn't have thought Microsoft could have sunk much lower in my estimation, but it's one of their talents that they always seem to be able to do so. Grrr.... 
[ Rant mode off ]
-- David --
Regulars here may have noticed that I've been a bit quiet for the last week or so... I've been putting in one last heave of effort to complete the internal interface overhaul that I've been working on for a while (it has felt like forever at times).
It's been quite a task: something like 25,000 lines of code have had to be more or less completely rewritten from scratch, and a considerable amount of debugging has been required to get everything just right... So what benefits do you, the end user, get from all this work? A fancy new user interface? A whole lot of new and powerful features? Pet peeves finally corrected? Well, although it pains me to say it, the visible benefit you will see from all this effort is precisely nothing. 
It's a bit like getting the engine in your car overhauled. It costs a lot of money, takes time, and is quite inconvenient, yet when it's done you don't really feel like you've gained anything - the car just goes the way it did before. The only real satisfaction is knowing that you'll probably get another 100,000km before you have to have it all done again. Well, a large software project is not much different, really: over time, things get tacked on and hacked on until eventually the whole thing looks like it's being held together by the machine-code equivalent of duct tape. You finally reach a point as a software author where the whole shebang becomes so hard to maintain that it's like a seized-up engine: only a major overhaul will get things running smoothly again. So it has been with Pegasus Mail - seventeen years of development has taken its toll, even though I consider myself a careful and tidy programmer: the best part of a year ago, it consisted of 460,000 lines of code written for a compiler that hadn't been supported in over five years, with as many untidy ends as a packet of homemade spaghetti: now, though, the program is hosted in a current compiler environment, and has internal interfaces that will allow it to grow in the ways it will need to produce version 5 in a maintainable manner.
So, doing this overhaul was about as much fun as a root canal from a sadistic dentist, but it was essential, and now we can move on. The next goal is to release a v4.42 build as soon as possible (lots and lots of small problems have been fixed, and my testers assure me that the new builds are much faster than the old ones), then to move onto v5, for which I have so many cool new ideas that I can barely wait to begin realizing them.
Cheers!
-- David --
I'm perpetually mystified by the human propensity for self-delusion: we seem to be able to talk ourselves into the most ridiculous things, whether through hopelessly naive optimism, stupidity or selective memory. A wiser head than mine (George Santayana, if I recall correctly) once summed it up quite well, saying "Those who do not learn from the mistakes of history are doomed to repeat them".
From the preceding paragraph, you should be able to glean the idea that I'm not best pleased with myself at the moment. I've got myself into a quagmire, and like all quagmires, it's wider and stickier than I ever expected. I'm talking about the changes to the internal object interface used by Pegasus Mail.
When I originally took this on, it was a logical, worthwhile futureproofing modification that was only going to take a week or two to finish: that was nearly two months ago. Now, don't get me wrong - the "worthwhile", "logical" and "futureproofing" parts are all still completely valid, but seventeen years of doing this should have taught me that it was
never going to take only a week or two - especially when the addressbooks became involved.
Pegasus Mail's legacy addressbooks are a source of unending grief for me: they're some of the oldest data structures in the program, dating from the DOS version. Back in the DOS days, programmers had to be really, really careful about how they used memory, because there wasn't much of it available. As a result, the addressbook code is incredibly contorted in places to allow the smallest possible memory use, even though in the days of gigabytes of RAM, the notion is now totally irrelevant. Unfortunately, slanting the entire development of a piece of code towards one priority like this is quite pernicious, because it debilitates almost every other part of the code. The result is that the legacy addressbook code in Pegasus Mail is fragile, awkward, convoluted and unbearably tedious in places, and I really don't like working on it very much.
Now, Pegasus Mail v5 will have a new addressbook format which will, I believe, be an absolute market leader - much more an information manager than an addressbook, its capabilities will make it enormously useful to almost everyone, unlike the antiquated and minimalist excuse for an addressbook we have at the moment. The design for the new format is done, and quite a lot of the code is even written: the problem is that I can't abandon the existing format - I'd get lynched - so I have to make sure that the two formats co-exist, if only to provide an easy migration path for existing users. This means that I have had to overhaul the old addressbook code once more so it can use the new interface I've designed.
The update on the old code is now mostly done, fortunately - although it's been quite a trial. There's still a bit to do though, so it's back to the salt mines for me. *sigh*.
Cheers!
-- David --
Just a short initial note to describe where we stand at 26 March 2007...
For the last few months, I have been porting the Pegasus Mail codebase (300+ source files, 450,000+ lines of C code) from the ancient Borland compiler I have used for many years to Microsoft Visual C++. The process has proven to be quite a lot more involved than I expected, and I and the test team have spent a lot of time tracking down issues resulting from the port. The primary benefit of this move is improved debugging, and we have already tracked down numerous small to medium reliability problems as a result of doing so. Moving to Visual C++ also gives me a future development path for the Windows environment, so it was really an essential task.
In the course of the port, I also decided to make some major changes to one of the internal interfaces Pegasus Mail uses. This process has been even more complex, but provides the basis for a very significant new plugin programming API that will be available in future.
The overall process of porting, checking, and modifying the interfaces is, however, now mostly done, and I hope to get back to significant coding of new features soon. I wouldn't care to estimate when we might have another release of the program, but probably not before the end of May at the earliest at this stage. The exact feature set that will appear in the next version is also still not firmly decided - as we get tighter on it, though, I'll provide information here.
Cheers!
-- David --
This blog is intended as a kind of diary on the development of Pegasus Mail. As I work on the program, I will periodically make postings here describing what's being done, and what plans are being made.
Only I and my formal beta test team can post or reply in this blog - if you are a registered member and see an issue on which you would like to comment, please start a new thread in the appropriate forum.
Cheers!
-- David --