Pegasus Mail & Mercury

Repetitive work in SQL

Peter Strömblad — Thu, 04 Sep 2008 18:14:00 GMT

Just wanted to share the two SQL scripts I produced last week. They are stored procedures that create standard looking CreateUpdate and Get procedures for any given table in SQL-Server. I've published the scripts at sqlservercentral.com and they are free to use and tweak.

Stored procedure mkProc_CreateUpdate

Stored procedure mkProc_Get

Here is a brief description for both of the procedures

mkProc_CreateUpdate v1.0 (c) Petena AB 1993-2008, http://www.petena.se/, http://www.praktit.se/

Creates stored procedure of given table name for the purpose of adding or updating a record
Primary keys are to be set to null to trigger the insert, and the key is returned.
Signals null/not null and default values.

Input Params:
@_tableName - the table you want to script
@prefix - add string here if you want your stored procedure name prefixed, like 'sp_'
@AddParams - add additional parameters to the stored procedure call
@DoCreate - true/false : when true the stored procedure is created
@Debug - true/false : true only works when @DoCreate is false, presents all internal tables

Example: Exec mkProc_CreateUpdate 'Orders', 'crm_', '@userID int,' false, true

Pegasus Mail v4.5 public beta available

David Harris — Wed, 03 Sep 2008 11:31:00 GMT

A public beta of Pegasus Mail v4.5 is now available for anyone who wishes to do so to try out. The beta is very complete and stable, but is provided without formal technical support - you should almost certainly apply due diligence testing to it before using it in a production environment.

You can download the public beta from http://download-us.pmail.com/w32-450-pb1.exe, or a full German version of the beta from http://download-us.pmail.com/w32-451-de-pb1.exe (and yes, we realize they appear to have different version numbers).

Peter Stromblad will post an announcement telling you about new beta-related forums that will be made available in the community in the near future - using these forums, you can discuss the beta and provide feedback if you wish.

Please read the "What's new" notes in the beta installer or help file for information on the v4.5 release.

Cheers to all!

-- David --

C# Null, DBNull and Entities

Peter Strömblad — Mon, 25 Aug 2008 08:59:00 GMT

I've been working like mad to get my 3-tier construction to work. As I'm not all that great in C# yet, I delightedly discovered that C# supports nullable types. Now finally I can match my database constructions with a proper entity in code, and as well build the business layer so that it deals with the values as it should.

Some of you know that a relational database is perfect when it comes to relating a drop down with a key value, then fetching the meaning of the key value from the related table. But if the drop down hasn't been selected, you should not store an arbitrary value - you store NULL = unknown.

Now I came across a tool with source from Rick Strahls weblog, called DataRowGenerator. I've managed to alter this so that it now suits my needs by creating one entity file for each datatable. Below is a sample of the code that the program now generates automatically for all my 54 tables.

Here are some reference links that you may find useful.

http://blogs.msdn.com/thottams/archive/2008/06/30/dbnull-and-nullable-types.aspx
http://sab39.netreach.com/Blog/Blog/12/vobId__172/pm__18/
http://www.codeproject.com/KB/architecture/three_tier_architecture.aspx
http://west-wind.com/weblog/posts/147.aspx

/* ArtikelGrp.cs (c) Petena AB 1994-2008, http://www.petena.se, http://www.praktit.se


DEVELOPMENT HISTORY: ******************************************************************************
v0.01
  20080825 PiS -> Start

BUGS & NOTES: *********** ( !=Warning, +=Additions to be made, -=Comment ) ************************
  - 
 */

using System;
using System.Collections;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;

namespace PraktIT.CRM.Data {

	[Serializable]
	public partial class ArtikelGrp {
				
    #region Private Members
		  private Int32? _ID = null;   //  
		  private Guid? _UID = null;   //  
		  private Int32? _ArtikelGrp_ID_Parent = null;   //  
		  private Int32? _Nr = null;   //  
		  private String _Kod = null;   //  
		  private String _Namn = null;   //  
		  private String _Titel = null;   //  
		  private Boolean? _Aktiv = null;   //  
		  private Guid? _Firma_UID_Owner = null;   //  

    #endregion

    #region Public Properties
        public Int32? ID
        {   get { return _ID; }
            set { _ID = value; }  }

        public Guid? UID
        {   get { return _UID; }
            set { _UID = value; }  }

        public Int32? ArtikelGrp_ID_Parent
        {   get { return _ArtikelGrp_ID_Parent; }
            set { _ArtikelGrp_ID_Parent = value; }   }

        public Int32? Nr
        {   get { return _Nr; }
            set { _Nr = value; }  }

        public String Kod
        {   get { return _Kod; }
            set { _Kod = value; }  }

        public String Namn
        {   get { return _Namn; }
            set { _Namn = value; }    }

        public String Titel
        {   get { return _Titel; }
            set { _Titel = value; }   }

        public Boolean? Aktiv
        {   get { return _Aktiv; }
            set { _Aktiv = value; }  }

        public Guid? Firma_UID_Owner
        {   get { return _Firma_UID_Owner; }
            set { _Firma_UID_Owner = value; }  }


    #endregion

    #region Populate
      public static ArtikelGrp Populate(IDataReader dr) 
      {
        ArtikelGrp _ArtikelGrp = new ArtikelGrp();
        Populate(dr, ref _ArtikelGrp);
        return _ArtikelGrp;
      }

      public static void Populate(IDataReader dr, ref ArtikelGrp _ArtikelGrp)
      {
        _ArtikelGrp.ID = dr["ID"] as Int32?;
        _ArtikelGrp.UID = dr["UID"] as Guid?;
        _ArtikelGrp.ArtikelGrp_ID_Parent = dr["ArtikelGrp_ID_Parent"] as Int32?;
        _ArtikelGrp.Nr = dr["Nr"] as Int32?;
        _ArtikelGrp.Kod = dr["Kod"] as String;
        _ArtikelGrp.Namn = dr["Namn"] as String;
        _ArtikelGrp.Titel = dr["Titel"] as String;
        _ArtikelGrp.Aktiv = dr["Aktiv"] as Boolean?;
        _ArtikelGrp.Firma_UID_Owner = dr["Firma_UID_Owner"] as Guid?;
        }
    #endregion

	}

	public class ArtikelGrpDataProvider : DataProvider {

  #region Get using Select
    public ArtikelGrp GetArtikelGrp(Int32 ArtikelGrp_ID)
    {
      SqlConnection conn = new SqlConnection(ConnectionString);
      string _SqlCmdTxt = "Select * FROM ArtikelGrp WHERE ID = @pk";
      SqlCommand cmd = new SqlCommand(_SqlCmdTxt, conn);
      cmd.Parameters.AddWithValue("@pk", ArtikelGrp_ID);
      conn.Open();
      SqlDataReader dr = cmd.ExecuteReader(CommandBehavior.CloseConnection);
      if (!dr.Read())
      {
        dr.Close();
        conn.Close();
        throw new Exception("ArtikelGrp not found!\r\nID = " + ArtikelGrp_ID);
      }
      else
      {
        ArtikelGrp _ArtikelGrp = ArtikelGrp.Populate(dr);
        dr.Close();
        conn.Close();
        return _ArtikelGrp;
      }
    }
  #endregion

  #region Get using Stored Procedure
    public ArtikelGrp GetArtikelGrp(Int32 ArtikelGrp_ID, Guid Person_UID)
    {
      SqlConnection conn = new SqlConnection(ConnectionString);
      SqlCommand cmd = new SqlCommand("sp_ArtikelGrp_Get", conn);
      cmd.CommandType = CommandType.StoredProcedure;
      cmd.Parameters.AddWithValue("@ID", ArtikelGrp_ID);
      cmd.Parameters.AddWithValue("@Person_UID", Person_UID);
      conn.Open();
      SqlDataReader dr = cmd.ExecuteReader(CommandBehavior.CloseConnection);
      if (!dr.Read())
      {
        dr.Close();
        conn.Close();
        throw new Exception("ArtikelGrp not found!\r\nID = " + ArtikelGrp_ID + "\r\nPerson_UID = " + Person_UID);
      }
      else
      {
        ArtikelGrp _ArtikelGrp = ArtikelGrp.Populate(dr);
        dr.Close();
        conn.Close();
        return _ArtikelGrp;
      }
    }
  #endregion
	}

}

Codename: Madmin

Peter Strömblad — Tue, 19 Aug 2008 13:48:00 GMT

The time has come to start a long awaited project!

Madmin is my codename, until someone has a better name for it.

Madmin is short of Mailserver Administrator.

All the way back from 2000 we've administered our Mercury installations through a web interface, with a SQL-database and a separate program called MercurySynch.

Madmin's objective is to replace MercurySynch.

As such, it will be:
1. a webservice
2. utilizing IIS
3. directly manipulate one or more Mercury/32 Mailservers.

Our old MercurySynch polls the database at 20 minute intervals, but a webservice with direct access will do its work directly when called upon.

A webservice has the benefit that it is a stand alone application, and can just as Rolfs HTTPServer start, stop, pause, reload any instance via Windows Messaging. Designing a webservice also has the benefit that it is fairly easy to create administrational modules for that can be integrated into larger packages, or created as a stand-alone Mail Administrator web-site.

How do we begin?

Well, the only way to start is to iron out the relational data model. I have a few requisites:
1. Multiple Mercury installations has to be supported
2. Multiple domains
3. Local users, tied to organizational entity
4. Domains, tied to organizational entity
5. Alias handling
6. Relay
7. Autoresponders

Anything missed?

If you feel you want to be part of this, or if you have expectations or valuable input - now is the time to put that forth in the notes to this post.

Tool and reporting.

As I go along, I'll post my thoughts here - and I'll also update this space regularly as I make progress. When there is something to test I'll share it in the downloads section here at the community. I know we will later get into design issues, as well as PDA support for a proper and sleek Mail Server Administration web site. The tool I use is Visual Studio 2008 and the language will be C#.

You know Windows Update is flawed - don't you?

Peter Strömblad — Sun, 17 Aug 2008 08:13:00 GMT

Automatic updates of software are good - aren't they? Well I'd answer yes in normal cases, but anything that is automatic can behave in ways the creators never imagined. So it is with fire alarms, burglar alarms, and of course with computing. This patch tuesday from Microsoft contains quite a number of fixes. Some that are very important to servers, regarding name resolution. However smart we are, automatic updates are not enabled on our servers - we consider them too dangerous - since in the past some driver updates have caused machines never to start again.

And so it happens - again. And now, with virtual servers running on top of the base server - when updates fail, you render three-four five or even eight machines stranded.

What happened then, why doesn't the server start as it should? - well it does, but just barely. I've never seen windows so inoperable that not even a task monitor is accessible, not even in safe mode. A small clue exists on sys\windows in the file WindowsUpdate.log - that is accessible over the network. It states

2008-08-17 10:19:44:472 4328 1594 AUClnt WARNING: Shell_NotifyIcon failed (dwMessage=0x0, uFlags=0x3, hr=0x80070002)
2008-08-17 10:19:44:472 4328 1594 CltUI FATAL: Failed to show client UI, directive=5, hr=80070002

Aha, something in the windows shell isn't starting as it should. And with the debate of removing Media Player from windows, since explorer is so tightly integrated, led me to investigate - what changes were made to explorer this time. Within KB953838-IE7.log I found it. It says:

6.937: DoInstallation:AnalyzeDiskUsage failed

Aha - so the sys volume ran out of disk space during the install - and let's not bother with the why or how at the moment. The reason I state that Windows Update is flawed, is that it requires a re-boot for some operations. This Patch Tuesday's pack of files required a re-start, and between the update and the restart, that continues the installation - disk space ran out. Sure, yes it probably did - but when updating really cruical files, shouldn't this be checked before? In fact it does, but the limit of free space is too tight - and there you have the result - a server heavily depended upon - needs to be re-installed.

Well, then, why wasn't 16 GB of system space enough? Because of all darned copies that Windows generates. In the Windows directory there are hidden KB directories. Besides these you have another directory called ServicePackFiles and another directory with files. Only the Windows directory contains more than 8 GB of files today, on a server that was installed december 2006. That is insane. To the downfall, we support many systems, older than some of my boys. When they we're installed with RAID controllers and high speed SCSI drives disk space was quite expensive and a total 32GB server system was considered more than enough. Some of the directories are safe to delete, some are not - and I haven't found any official information from Microsoft on how to conserve system volume space. If anyone has, please let me know.

So for the time being, take this advice if you run cruical machines:
Make sure you have plenty of free space on you system volume before running Windows Update aka Microsoft Update.

....

and now on to the task of backing up all drives (just about a terabyte of data.... - sigh)

arrrrgh web apps

Peter Strömblad — Tue, 12 Aug 2008 21:29:00 GMT

Writing apps for the webb is a challenging task. What takes a normal hour when creating "normal" apps takes four times when working a three tier webb-app. There are simply so many pits you can fall in.

In any case I thought I'd share some very interesting read-up.

First of all, creating forms: Read up on Rick Strahl's excellent piece at msdn: http://msdn.microsoft.com/sv-se/magazine/cc163505(en-us).aspx

Then dwelve down into his weblog and you'll find an absolute excellent tool on how to create the windows classes directly from your database: http://www.west-wind.com/WebLog/posts/147.aspx.

So, after that we're on to Ajax. The resource for that is naturally http://asp.net.

There's a lot more. One needs to familiarize one self with css (zen-garden is excellent), sql calls, stored procedures, java-script, cookies, performance issues and not to forget loose binding. A web-app is always in a "new" state - there is very little you can do to compensate for the round-trips to the server side.

In the future I hope to share code when I get to hacking a Mercury web-service.

cheers for now / Peter

WinPMail v4.5 public beta imminent

David Harris — Mon, 11 Aug 2008 00:42:00 GMT

Just a short update on the status of WinPMail v4.5.

I had hoped to have the program released by now, but we've run into a late problem that I feel we have to resolve before a release can take place.

It is currently my intention to release a v4.5 public beta to this community as soon as the code is next in a stable state - hopefully before the end of this week. Once the public beta has had a chance to circulate a little and we're sure there are no remaining serious issues standing in the way of a full release, v4.5 will be made available on an unrestricted public basis.

Cheers!

-- David --

Every day something new...

David Harris — Mon, 21 Jul 2008 03:15:00 GMT

I've had a number of people ask me if I could put together a blog posting describing what I'm doing at the moment, and what plans there are for Pegasus Mail and Mercury in the short-to-medium term.

Well, the first thing I'll tell you is that my current priority is to get Pegasus Mail v4.5 out the door. On the face of it, it won't look like much has happened, and I suspect many people will be rather disappointed by v4.5, but in reality, more than 50% of the entire codebase of the program has been modified in some way during the shift from Borland C++ to Visual C++; it's been a huge, largely unrewarding job that simply had to be done. In the process, though, we've caught and fixed literally hundreds of small and medium-sized bugs that have been present in the program for periods ranging from recent history to more than a decade.

A huge amount of effort this year has gone into releasing Mercury/32 v4.6 - once again, there was a lot of code modernization going on that doesn't yield obvious visible benefits, but which simply had to be done.

But not everything is porting code and fixing bugs. I've also been spending a lot of time familiarizing myself with new technologies that will have significant bearing on both Mercury and Pegasus Mail:

For Mercury, I've been spending a lot of time learning about CSS and XHTML, because the future of Mercury is clearly heavily web-oriented.
I've been spending a lot of time learning about SQL and particularly SQLite, because it's clear that both Pegasus Mail and Mercury are going to need robust database facilities in future.
I've developed my own object programming interface for both programs: this is a key technology that will have a huge bearing on future development by making it easier to extend the programs' capabilities without side-effects.
I'm well into developing a complete replacement for the fundamental message store used by both Pegasus Mail and Mercury. This new component, called MailStore, is easily the single most important development in either product in more than a decade, and should be in a working state towards the end of this year.
I've spent a lot of time this year working on documentation - the bane of all programmers... The Mercury/32 Daemon Developer Kit in particular was a huge amount of work that was vastly overdue, and I do hope to start seeing new Mercury Daemons appearing before much longer.
Finally, I've spent a lot of time developing tools - either because I was forced to do so (as in the case of the new help system, which I was forced to do by Microsoft), or because I believe they'll be important in upcoming releases of either program.

I realize I'm not being quite as specific as I'm sure many of you would like: I'm not saying that "X feature will be working by August", or that "Y feature is now operational": I promise that as I get to a position where I can make reasonable statements about new developments I will do so, but in truth, the last two years have been a gigantic, painful, tedious process of modernization and tooling up for new capabilities, rather than actually working on those capabilities per se.

I also realize that I've been somewhat of a stranger to the forums this year: I apologize for that, and will say honestly that my absence has largely been a result of tech-support burnout... Doing technical support is a very soul-destroying process, because you're typically seeing people at their worst and most stressed, and nobody really likes spending all of their time dealing with the bad aspects of their work. As a result, spending too long doing technical support is a recipe for depression and loss of motivation, and I've been doing technical support at some level for nearly two decades now. It's my aim to become a more regular participant in the forums, although I will be consciously and actively trying to resist the self-destructive urge to solve every problem I see posted. I'd like to get back into the forums so I can enjoy interacting with the people who use my work without always burning myself out trying to fix all their problems.

So, there you have it - a slightly longer, and maybe slightly more candid posting than I intended, but I hope it serves to give you a little insight into where I am at the moment.

Cheers to you all!

-- David --

No corny jokes about "New Vistas opening up", please

David Harris — Sun, 08 Jun 2008 15:29:00 GMT

I've resisted doing anything very much with Windows Vista until now, but eventually reality catches up with fantasy, so I finally decided it was time to get going. So, I've now got Windows Vista UItimate running here on a laptop (a Dell XPS1530), and over the next few weeks will be taking a bit of time to familiarize myself with its ins and outs.

First impressions usually don't really mean too much - the most I'd usually say about them is that they're a useful indicator of the things you *liked* in previous versions of the OS. In this case, there are a few first impressions that come to mind.

Vista seems to want to connect to the Net for *everything*. My Dell came with a TV tuner card, and Windows even wants to connect to the Net before I can use that. Now, I accept that the Net is pretty much omnipresent now, but even so, there are still going to be plenty of times when you want to use a laptop disconnected.

UAC... Well, I guess that about says it all. I've used this machine for probably three hours so far, and I'm already SO frustrated with the unending UAC prompts that I'm going to turn them off. This is a case of Microsoft getting an important feature so totally wrong that I don't even know why they bothered.

The Windows "sidebar"... I've thought and thought, and I just can't see what use it has. None of the gadgets that comes with the system really seems to have any point to me at all, and I struggle to imagine anything that I would want there. It looks to me like a feature looking for a justification... But of course, other people's mileage may vary.

The arbitrary changes to the "Start" menu - in particular the ugly new "programs" menu with its odd scrollbar and strange positioning - these seem like changes that will only serve to frustrate and annoy users used to the way XP worked. Of course, many times you think this, then a few months later (after you've become used to the new approach) you wonder how you ever lived without it... But I don't think this will be one of those cases.

The arbitrary changes to where everything is really annoy me - control panels are changed, things have been moved, simple tasks have become enormously difficult (for example, I actually had to open the machine up by removing two torx screws to find the WLAN card's MAC address so I could tell my WLAN about it - something that was a simple right-click under XP).

I don't know if I'm ever going to get used to the new Window close/minimize/maximize controls - they just look as though someone has positioned them carelessly during window layout, the way they are (for those few of you who might not have encountered Vista yet, these controls are now half the height of the caption bar and anchored to the top edge of the bar - they look really out of place to me).

The look of the system is quite slick, but doesn't really seem complete to me somehow... I can't quite put my finger on it yet, but I think possibly it seems a little "over-stylized" to me - it seems kind of "busy". I'm sure I'll get used to that, though.

As an aside, the XPS1530 is a pretty cool piece of kit - slim and light, attractive to look at, yet still with everything you could ever want on board. I'd have no problems recommending it, regardless of what I end up thinking of Vista.

As I use the system more and develop clearer feelings about the issues, I may write more here on the subject.

Cheers!

-- David --

Time off

Peter Strömblad — Sun, 06 Apr 2008 07:57:00 GMT

Imagine doing whatever you love most, 24 hours a day - how long would you be able to stay awake? My body and mind defenitely needed a good rest. Summer 2007 was the WORST summer I can remember. The first day my Wife (who has an ordinary job as a mech. constructor) got off for vacation was nice, but the following five weeks - the rain was pouring down. So I worked.... - This winter wasn't any better. The WARMEST winter ever in Sweden overall. Meaning, instead of snow we got .. - you guessed it - RAIN.

The day we left was finally a day of snow. Winter has normally passed by march 17th, but not this year. So on slippery summer tires, the car drove slowly at 0300h towards Copenhagen Airport. We we're going to Tenerife!

Of course the flight was delayed! What do you do with three small children (2,5,9 & ½all), and a wife who is terrified of flying, in an airport, when they are over-tired from having gotten up at 2.30 am - when your flight is delayed until lunch?
Yep: You let them sleep on the floor !!! - at the gate!

Well, once we got there it took a good week for us all to unwind, and another to recharge. One can say a lot about going on a charter trip, with Coco the clown entertaining the kids at 2000h - it is relaxing - especially since the TV is dubbed in a completely incomprehensible language (Spanish). What you do is fall asleep with the kids, in the middle of a bed time story, a long long time ago, in a galaxy far faaaaar aaaahhhh.....

At the firm we're not too many people, and my responsibility lies with the servers. Compared to the early 90s properly configured servers aren't likely to crash from hardware issues. It certainly pays off to get fault tolerant memory, redundant disks and redundant networking. Above all, is to stray off from wanting to put too much into one single server. Therefore on our E-Mail servers we tend to put nothing else, than just Windows, Windows Firewall, and an Antiviral program (currently NOD32). We also limit the rights of the auto-login account, so that it can't do any harm to the rest. Besides this we have a separate inbound email server that does all the anti-spam and anti-viral treatment. By far the Mercury/32 we run today is the steadiest software I know.

However getting a Windows Mobile phone to work IMAP over GPRS wasn't all that easy, I can see that as a pro on the road a lot. It will be convenient to work the Mercury/32 server with a simple web interface. Currently our add-on system lets me configure nearly all aspects, but I can't do the more server based tasks - like adding or altering a filter for both local delivery and forward to a customer in China. Nor can I work IMAP if something goes wrong. So in a Mobile world a lot remains to be done.

The creative side finally started to work again and I don't know if you guys think it is a good idea to have a specialized IMAP forum - sinze I believe this will be the key most important issue as the world goes more and more mobile. I had a number of problems, mostly due to that Microsoft Mobile Outlook 6 doesn't behave well. When f.ex. "standard" folders aren't there, the folders are not created - and to know this - when you're thousands of miles away - and only get a "Synchronization error" - there isn't much to do about it.

Finally if you have ideas on how we should develop the community, please let me know what you think. Also, if you have the chance to get some free time - take time off - it is truly vitalizing!

Cheers / Peter S.
ps - Thanks dad for celebrating your birthday by a pool side, sorry I beat you in Golf though... - ds

Mercury/32 v4.6 in late testing

David Harris — Tue, 19 Feb 2008 23:20:00 GMT

Mercury/32 v4.6 is now late in testing, and I thought I'd give you a preview of some the new capabilities it holds.

Threaded core: The core module (the central part of the program that handles mail delivery and routing) is now somewhat multithreaded, resulting in throughput performance increases of anything up to 350% on busy queues.
MercuryP (the POP3 server) has been totally overhauled. It is now vastly faster, and a number of problems have been fixed (in particular, an issue arising from a long-standing bug in Windows where message UIDs would change after a daylight savings time change, resulting in mail being downloaded again). It also now includes the short-term blacklisting capabilities found in MercuryI and MercuryS for handling pests and brute-force password crackers.
MercuryP now supports login-time filtering: simply by adding any of several filter criteria to the username you use to login to MercuryP, you can control the mail it will present to you. This means that if you only want to see unread urgent mail from "bob" on your PDA, you can now do this, just by adding the string "(urgent,new,from=bob)" to your login name.
Mercury now includes a new commandline mail generator called MSendTo: MSendTo can generate mail in a wide range of formats and writes directly to the Mercury queue. It will be very useful for system administrators and anyone wanting to send mail under programmatic control.
New consoles: the Mercury protocol modules now have a new console design and interface. I know this doesn't sound like much, and you won't notice a lot of difference, but it's a huge step towards making the Mercury user interface remotable.
The MercuryB MLSS mailing list subscription management interface has been totally overhauled. It now offers important new features like the ability to change your status for all lists on the server and to retrieve your list password by e-mail, and has numerous improvements and fixes. A mailing list moderator utility is in development but probably won't be available until the next release.
MercuryI (the IMAP server) has had a number of fixes, and now caches the inbox, resulting in a huge performance increase when opening that folder.
Lots of small bug fixes.
Update notifications and advisories: licensed copies of Mercury will be able to receive notifications of new updates and advisories on security and other issues. License uptake on Mercury has been fair since the licenses became available, but it could definitely be better, and I'm hoping this feature might act as a motivator.

It's possible that this feature list may expand prior to release, but right now, release is itself a feature. I hope to get v4.6 out the door soon.

Cheers!

-- David --

Sorry - site was inaccessible last nite

Peter Strömblad — Wed, 23 Jan 2008 19:00:00 GMT

A fault occured last night, occupying the server at a 100% - thus making responses so slow that browsers timed out.

The problem wasn't all easy to locate, but eventually we found that the server logs, and database transaction logs had grown beyond reasonable sizes. Once we managed to shrink the transaction logs and the database, all was back to normal again.

I apologize for any inconvenience this caused.

cheers / Peter S.

Firewalling

Peter Strömblad — Sun, 20 Jan 2008 22:05:00 GMT

No Fire, just a wall - or another brick ...

True Firewall Appliances are fine, complicated and feature rich. Some have a feature called DNS Doctoring, which means that a DNS A request is translated from one IP into another. This is very useful, when you have a private network but with only some public IP-addresses. F.ex. 192.168.0.2 could be translated into 84.20.9.2 as a DNS service. But this translation is done at the cost of also translating the originating caller IP.

Problems can turmoil quickly, since you can have a DMZ (De Militarized Zone) and a VPN-Tunnel. We have both!. If you have a VPN tunnel, you want the traffic to flow from one internal net to the other, without any restrictions or address translations. The problem arises when you combine these two, meaning DNS-Doctoring and a VPN-Tunnel - from two sites. When you do DNS-Doctoring from just one site, then there are no real hurdles. The actual obstacle lies in f.ex. DNS-redundancy.

Today I have spent most of the day trying to get my scenario to work. What it gives is actually a DMZ with private addresses, since you have to configure inbound and outbound rule sets for all the external addresses you want to use in both ends, and all explicit traffic - which is alot in an AD environment. The best solution would have been to separate the DNS A request translation from the actual traffic translation - but I guess that is too simple to ask.

Sorry for the technicality in this post, just wanted to write this off my chest.

Networking is sometimes a real mind job - sigh

"That should only take a few minutes"... [Sardonic Laughter]

David Harris — Mon, 14 Jan 2008 02:27:00 GMT

I don't think there's any other profession quite like writing software - short of warfare, I can't think of any other area of human endeavour where the potential for casual disaster is so high.

For the last eighteen months, I have been involved in rewriting and modernizing a huge chunk of the 460,000 lines of code that makes up Pegasus Mail. I've written about this (I just can't bring myself to use the verb "blogged", I'm afraid) before, but I thought it might be mildly diverting for some of you to see just exactly how bizarre and off-the-wall this process can often get.

In the course of porting Pegasus Mail from the now ancient Borland compiler I have used for many years to Visual C++, there are various code changes that have been forced on me. One of them is quite basic - the code I use to get listings of the files in a directory. On the face of it, this was straightforward: I evaluated the problem and thought that it might take me a couple of hours to fix. You can see where this is going, I assume... It eventually took me almost five weeks of repeated debugging and testing to get this one simple function working. When I had finally solved it, I sent a message to my beta test team describing what had been involved: here is the text... (note that the reference to "VC12" is to a beta build of the program which is only available to the test team. At the time of writing, we're up to VC13, and are getting ready for a formal release of Pegasus Mail v4.5).

The problem with files not showing up was quite a lot more complicated, and is part of the ongoing saga of the Windows FindFirstFile and FindNextFile functions. To understand this problem (and it's worth summarizing it, I think), you need to have a little history.For the longest time, I have used the Borland compiler family to write my code. I never used much Borland-specific code, but some that I *did* use extensively was a pair of functions called findfirst and findnext, which were used to enumerate the files in a directory. They worked reliably for a long time, but it eventually became clear that they had some problems. Furthermore, there was no direct or portable equivalent of these functions in Visual C++. So, I decided to write my own functions to replace them, making my code much more portable.

Now, Windows has a group of functions called FindFirstFile, FindNextFile and FindClose, which on the surface appear to duplicate the operation of findfirst/findnext... On the surface. In reality, these function calls are quirky and buggy as hell. As an example, they treat wildcard characters quite differently depending on the underlying file system: so, the pattern "*.?" will return one set of files if the underlying file system is FAT, another if it's NTFS, and a third, different set if the file system is NetWare. What's more, different combinations of Windows and file system will return different sets of file attributes (so, a plain search on FAT WILL return files marked readonly, but the same search on NetWare WON'T return files marked readonly unless you specifically request them).

Finally, Windows defines an extensive set of file attributes - things like read-only, hidden, system, compressed and so forth... The problem is that they have kept defining attributes over the years, simply adding new ones whenever they want. As a result, it can be quite tricky to work out from its attributes whether a file is an "ordinary" file or not. To understand what I mean by this, consider that you typically WON'T be interested in files that are marked "Offline", because they either cannot be opened, or will take an inordinately long time to be opened because they have to be recovered from backing store. Similarly, files marked as a "Device" should probably never be accessed other than on explicit requirement.

The reason various of you have found that WinPMail is not seeing certain files on your system (typically files without the archive bit set) is because Windows is returning different sets of attributes for these files depending on the OS and file system... So, on NetWare systems, the archive bit is set or not set (no problem), but on NTFS, if the archive bit is not set, another bit called FILE_ATTRIBUTE_NORMAL *IS* set (a situation not true if the file is on a NetWare volume). All of this is interfering with the test I was using to work out whether a file is "ordinary" or not.

I have now completely abandoned the "ordinary file" test I was using and have changed it to an inverse test - I now check to see if a file is "unusual" (i.e, has certain specific attributes that I'm not interested in by default) and omit it if it is. This is a considerably easier and more deterministic test and using it bypasses all the bizarre vagaries of OS/file system combinations.

VC12, which I'll release tonight, implements the new filescanning code, and I'm 99.9% sure that I've finally got it sorted out. But it's symptomatic of how difficult things are getting that such a basic, fundamental piece of code has been so bloody awkward. I've wasted dozens of hours diagnosing and solving these problems - problems that simply shouldn't have existed in the first place. Hmph.

Sorry if this is tedious or over-technical - I just thought it might be interesting for you to see a little of the kind of thing that goes on "behind the scenes".

More on the v4.5 release soon.

Cheers!

-- David --

Ever Onwards.

David Harris — Tue, 01 Jan 2008 10:29:00 GMT

Well. 2008... Amazing.

It's been 2008 for about 23 hours now, and I admit I'm still wondering what happened to 2007 - it seemed to be over almost before it began.

Many years ago, Queen Elizabeth, in one of her Christmas Broadcasts, described the year she had just had as her "Annus Horribilis", a term that has been widely reused, recycled and abused since then. Although I don't much like resorting to cliché, I have to admit that her little latinate touch was just the right term to describe the way I feel about 2007. Put simply, 2007 was, overall, not a great year for me - it started out badly, and got worse. BUT, and this is a big but, towards the end of the year, it also began to get better, and I now allow myself to feel some quiet hope that the road ahead is finally clearer and less treacherous.

2007 was a year of rebuilding for me - in almost every way. For Pegasus Mail and Mercury, the year was about massive reconstruction, followed by a process of consolidation - making right, then making good. Although frustrating, many of the processes I've had to go through with both products this year were necessary and unavoidable, but with them largely complete, the way is open for smoother development in future. Expect to see more releases in 2008 than there have been in the last two years combined, as we finally begin moving towards Pegasus Mail v5, and as Mercury gets more and more robust and refined.

No blog reflecting on 2007 would be complete without mentioning the work of Peter Stromblad in setting up this community: I believe this site has created a wonderful forum for both programs, and its existence has certainly eased the burden of technical support (which was getting very hard to cope with in previous years) on me and my team. I realize that I've been quiet for the last couple of months, after an initial burst of postings - I expect to get back into more regular attendance in the course of the next few weeks.

Finally, I'd like to thank everyone who has believed in Pegasus Mail and Mercury during the hardest year of their long lives, but especially my beta test team, without whom I simply could not function at all.

So then, in these final minutes of January 1st 2008, I'd like to offer you all my warmest wishes for the New Year: let's get this show on the road - I'm looking forward to enjoying the journey. :-)

Cheers!

-- David --

Happy Holidays

Peter Strömblad — Fri, 21 Dec 2007 15:26:00 GMT

Wish you all a nice couple of days off.

Cheers / Peter

If only it were so simple...

David Harris — Wed, 24 Oct 2007 13:44:00 GMT

I've been doing this (writing software) for most of my 46 years, and professionally for well over 20 now: one notion that imprints itself more and more forcibly upon me with each passing year is how much more complicated everything seems to get.

I suppose it is in the nature of the human animal to elaborate: caves become tents, which become mud huts, then grow into drafty stone castles, and eventually into towering skyscrapers... Or, bones become sharpened sticks, which become arrows and spears, then muskets, rifles, machine guns until now the US military is testing the "vomit cannon" (a weapon that induces mass nausea by sound waves). With each innovation, every incremental step along the path from humble hamlet to sparkling skyline, things get more complex... One seldom-understood effect of this is that knowledge becomes more and more specialized: any chimp can wield a bone as a weapon, but how many single people would know how to build a vomit cannon from scratch? As tasks get more complex, they progressively surpass the ability of any individual to encompass them in their entirety. The way humans deal with this problem is by specializing - by splitting the task into smaller, more manageable pieces then splicing them together at the end.

The problem with dealing with complexity in this way is that it also introduces complication. Fred Brooks famously stated that "Adding manpower to a late software project makes it later", which can be taken as a general comment on the complication that arises when you increase the number of people involved in any task (not just software tasks). Different people approach problems in different ways, have different priorities, and differing sets of skills: considering this, it always seems miraculous to me that *any* large software project ever yields results at all, let alone working well.

Pegasus Mail is now around 460,000 lines of C code, and Mercury around 200,000. By commercial standards, these are middle-sized projects, and well beyond the optimum size for a single developer. With size comes complexity and complication - nothing in these programs is easy any more - indeed, it makes me weep when I compare the productivity that I could achieve when I was writing the early DOS versions of Pegasus Mail back in 1990 with the almost glacial progress I can manage now. Features that I could have added in a day back then would take a month now, and the depth of knowledge and background required to do them is vastly greater. I think I'm a better programmer now than I was in 1990, but it takes me ten times longer to achieve anything, and it's hard to describe how frustrating that is.

I must admit that when I think of Windows Vista with its tens of millions of lines of code, I can scarcely credit that it works: and the more I consider it, the more a kind of visceral, almost feral mistrust arises in me - how can I trust something so complex that no single human being is capable of fathoming it in its entirety? Would I trust my life to it? My reputation? My identity? Honestly, it's beginning to scare the hell out of me that we're becoming so dependent on this type of technology.

Perhaps I'm being silly, wistful and naive, but wouldn't it be nice if things were simpler? "Simple" is the lost virtue of this age, and I can't help but worry that we're going to miss it far more than we realize.

Cheers!

-- David --

Here in spirit, if not in posting

David Harris — Mon, 01 Oct 2007 12:08:00 GMT

Just wanted to put up a short posting to explain why I've been almost entirely absent from the community forums for the last couple of weeks.

I'm dealing with a number of issues at the moment, and simply haven't had any time available to me. I'm working hard on finishing the WinPMail to Visual C port, aiming for a release in the very near future, and am completing a v4.53 release of Mercury which should also be out very soon. There's also a relatively low-importance security update for MercuryI which I've had to track down and fix - I should be able to release that in the next couple of days, although I stress it's a fairly low-risk issue, because it requires a successful login to work.

As well as this, we're doing some system modifications here and I have had some personal issues to attend to.

So, I apologize for the silence, but I should be back in the mix before very much longer.

Cheers!

-- David --

Microsoft breaks the contract

David Harris — Wed, 08 Aug 2007 03:45:00 GMT

[Rant mode on]

Another week where I've been very quiet... This time, though, I find I'm actually quite angry at the reason. For the last three-odd weeks, I have been writing a help system and porting the Pegasus Mail help to it. Why have I been doing this? Because Microsoft broke its contract with developers.

Ever since it was first released, Pegasus Mail has used a Windows help system called WinHelp: WinHelp was a standard part of the operating system and worked quite well. Over time, for reasons I've never really quite understood, Microsoft drifted away from the WinHelp model towards a new help system called HTMLHelp, which uses Internet Explorer to display help pages. I personally never thought HTMLHelp offered any real advantages; furthermore, changing to HTMLHelp would have taken considerable time and effort, and worse, would have required all my translators to go through the same awful conversion process, something I wouldn't wish on anyone. I and my translation teams have made a very extensive investment of time and effort in WinHelp, and based on the feedback I regularly get from people, nobody was noticeably unhappy with it (one of the more regular pieces of "nice" feedback I get from people is how comprehensive the help system is).

Enter Windows Vista. Quite late in the pre-release process for Vista, Microsoft suddenly announced that they didn't like WinHelp any more, they didn't think anyone should be using it, and they wouldn't be including it in Vista. What's more, they specifically forbade developers dependent on WinHelp from distributing it with their applications. Microsoft's position was apparently that everyone should be using HTMLHelp, regardless of the effort, expense or practicality of doing so. I also have to say that Microsoft offered exactly zero assistance in moving between formats; there was an existing conversion tool, but it was useless.

That's not the end of the story, though. At some point in the endless patching process for Internet Explorer, Microsoft made a change that effectively crippled HTMLHelp: if an attempt was made to display a help file stored on a shared volume (for instance, a file server), the help file would simply fail to work with an incomprehensible error message ("Action cancelled"). While there are some possible fixes for this problem, they all involve an overall reduction of security and are fairly technical (registry editing). What's more, because the problem would occur in our help file, people would come to *us* for technical support on the issue, even though it's actually a bug in a Windows system component.

So here we are: under Vista, the only help system available to us is one which will fail to work in one of the most common installation scenarios for Pegasus Mail. The one that works fine (WinHelp) can only be downloaded from Microsoft by the end user - we can't supply and install it ourselves. From a developer's perspective, this is a nightmare. In the end, the ONLY solution available to me was to write my own help compiler and display subsystem, so I can continue to use the sources that I and my translators have so laboriously produced. This has not been an easy process, but I think the results are good: my testers got the first working cut of this code last night, and so far the feedback has been quite positive.

But I can see you're asking why I claim that Microsoft has broken a contract... ? When programmers develop for an environment like Windows, they use programming interfaces called "APIs", provided by the environment's developer - in this case, Microsoft. The API is a contract between Microsoft and the developers who use it, a contract that says "Provided you follow the rules we lay out in the documentation, we'll guarantee that your program will work as you expect, and that it will continue to work as you expect in future". In this case, Microsoft have reneged on the deal. Instead of simply leaving developers like me in the lurch, they could have done any of a number of things: they could have allowed us to distribute WinHelp with our applications; they could have provided proper transition tools; they could have ensured that the system they WERE providing actually worked properly... But they did none of these things. They didn't even give much in the way of warning (six months is far from adequate for something as important and substantial as a help system).

So, I've written a complete help system. I think it's probably quite a lot better than what we had, and it's at least as good as HTMLHelp; what's more, it will work correctly even where HTMLHelp does not. But I should not have had to waste a month of my life doing this when there are so many more interesting and important things to do. I really wouldn't have thought Microsoft could have sunk much lower in my estimation, but it's one of their talents that they always seem to be able to do so. Grrr....

[ Rant mode off ]

-- David --

Dollars to dimes

David Harris — Thu, 19 Jul 2007 01:56:00 GMT

Yesterday, the US Dollar hit its lowest point ever against the New Zealand Dollar.

In that one sentence, you can see encapsulated much of the financial pain and difficulty that developers like me have been experiencing for quite some time. Many people don't realize that different countries use different currencies, and that those currencies' values change over time in relation to each other. What this means is that when someone pays me some amount in US Dollars, I actually receive a different amount in New Zealand Dollars (my local currency). The amount I receive depends on a floating, variable rate called an "exchange rate", which is the amount a bank will pay for a foreign currency, and which is affected by all sorts of economic factors.

For the last five years, the US Dollar has been in a kind of freefall against most other currencies, as the US economy flirts around the edge of international collapse. This has meant that US currency has been worth less and less - but most people probably don't realize *how much* less. Well, as a matter of interest for myself, I did a comparison yesterday. On July 18 2002, if someone had paid me US$1000, I would have received NZ$2039.15. Five years later, the same US$1000 would have yielded NZ$1250.93 - that's a difference of $788.22, or well over a third. My New Zealand dollars still buy me pretty much what they bought me five years ago (inflation is pretty low over here), but I'm getting far fewer of them. Standard economic wisdom has it that I should simply increase my prices to cover the currency changes, but if I did that, a US$150 license would now be costing close on US$250, and most customers simply won't wear increases like that. What's more, my so-called "business model" has traditionally made it quite difficult for me to increase prices in any event, because I have always felt it was a matter of honour to accept any price I had previously advertised.

By contrast, if someone had paid me EUR1000 in 2002, I would have received NZ$2024.29, whereas now I would receive NZ$1724.13 - still a drop (reflecting the relative strength of the New Zealand economy), but nothing like as severe a drop as the USD, where more than a third of my income has simply vanished in a puff of banker's logic.

I bet most people have never even thought about things like this - after all, not many people really do much in the way of international transactions... But for people like me, for whom almost every transaction is international, it's quite a serious issue, and there's no sign of relief on the horizon. So, we tighten our belt a little more and eat out a little less... <grin>.

Important disclaimer: This is not intended as any form of solicitation of support from my users, who have been generous and kind beyond my expectations: it's just an exposition of a problem most people probably aren't even aware exists, and is provided purely as an insight.

Cheers!

-- David --

Getting this car back on the road

David Harris — Mon, 16 Jul 2007 03:30:00 GMT

Regulars here may have noticed that I've been a bit quiet for the last week or so... I've been putting in one last heave of effort to complete the internal interface overhaul that I've been working on for a while (it has felt like forever at times).

It's been quite a task: something like 25,000 lines of code have had to be more or less completely rewritten from scratch, and a considerable amount of debugging has been required to get everything just right... So what benefits do you, the end user, get from all this work? A fancy new user interface? A whole lot of new and powerful features? Pet peeves finally corrected? Well, although it pains me to say it, the visible benefit you will see from all this effort is precisely nothing.

It's a bit like getting the engine in your car overhauled. It costs a lot of money, takes time, and is quite inconvenient, yet when it's done you don't really feel like you've gained anything - the car just goes the way it did before. The only real satisfaction is knowing that you'll probably get another 100,000km before you have to have it all done again. Well, a large software project is not much different, really: over time, things get tacked on and hacked on until eventually the whole thing looks like it's being held together by the machine-code equivalent of duct tape. You finally reach a point as a software author where the whole shebang becomes so hard to maintain that it's like a seized-up engine: only a major overhaul will get things running smoothly again. So it has been with Pegasus Mail - seventeen years of development has taken its toll, even though I consider myself a careful and tidy programmer: the best part of a year ago, it consisted of 460,000 lines of code written for a compiler that hadn't been supported in over five years, with as many untidy ends as a packet of homemade spaghetti: now, though, the program is hosted in a current compiler environment, and has internal interfaces that will allow it to grow in the ways it will need to produce version 5 in a maintainable manner.

So, doing this overhaul was about as much fun as a root canal from a sadistic dentist, but it was essential, and now we can move on. The next goal is to release a v4.42 build as soon as possible (lots and lots of small problems have been fixed, and my testers assure me that the new builds are much faster than the old ones), then to move onto v5, for which I have so many cool new ideas that I can barely wait to begin realizing them.

Cheers!

-- David --

Thank you

Peter Strömblad — Wed, 27 Jun 2007 20:23:00 GMT

First of all, thank you all - for making the community work the way it works. I think it works very well, and I am very pleased to see that we have passed 700 registered users. This sunday we had over 200 visitors simultaneously, searching and browsing information. Very seldom I or other moderators have to delete or edit content in violation of the rules. When browsing deleted posts, I see that you who have deleted a post you have made, have indeed written a proper explanation - some even make me laugh.

My aim with the site was to make Pegasus Mail and Mercury much more visible. Now searching some of popular search sites on the internet for keywords of f.ex. "Pegasus Mail" or "Mercury" / "Mercury/32" the community is listed among the top results. I know the SEO can be a lot better, but I'm very pleased to see that many others are referring to the site for answers.

And the site has only been live for 2 months.

Statistics
For those of you who are interested in stats of the site, what it takes to run it here are some figures:

The first week of may, some 83000 pages were served. Every week the pages served is increasing. Last week 150' pages were sent out.
Time of the day difference isn't that big though UTC 0200 - 0500 are somewhat less intense
Sundays the traffic is lower than the other 6 days in a week.
Search engines that show any significance in traffic are Google, Live and Yahoo.
Among the search words are the most obvious ones most frequent, mail, mail server, Pegasus, Pegasus Mail, Mercury, Mercury Mail, SMTP and so on .
Web browser use, in order: Firefox, Netscape compatible, MSIE, Opera, ShopWiki and others
52% of the traffic is coming from Windows clients
Majority of the page size is less than 100K, with a steep access curve down to 10K
An absolute majority is being served pages under half a second and only exceptional page requests are served longer than 2 seconds. (This is due to caching, and most likely lies under the administration pages that are not that frequented than the site in general)

Future
But, now is not the time to rest - What do you think should be improved? Should the site be available in other languages than English? Should there be nationalized forums? Make your opinion heard at the "about this site" forum!

During the summer I will start the transition to CS2007 and multi-language support of the site. I'm not making any promises about when the release is to be made. For the time being I'm coding what I hope will be the license handler for Mercury - it all depends on if David will approve of it once he sees it.

Everything you never wanted to know about releasing software

David Harris — Wed, 06 Jun 2007 00:56:00 GMT

With Mercury/32 v4.5 now into the end stages of pre-release confirmation (the release candidate archive went out to the beta testers last night), it's safe to say that the light is visible at the end of what has been a very, very long tunnel.

Now, I would never attempt to defend the fact that there has been a three year gap between releases - a delay like that is damaging, and it should never have happened. Suffice it to say that I have learned a number of difficult lessons about managing large, complex projects as a result of this process, and that it should thus be possible to avoid having it ever happen again. That said, though, I suspect that very few people out there really have any idea what it's like to make a release of a piece of major software, especially when that software is commonly used as critical infrastructure... So, here are some purely personal insights into why I find getting up to and through a release so hard.

1: Testing, testing, testing Those of you who use it may have some appreciation of just how feature-rich Mercury is: the program has settings for almost anything imaginable, and under the hood, there is a very rich plugin interface that allows clever people like Lukas Gebauer to add the things I either can't or haven't. The more feature-rich a program becomes, the harder it is to test, because there are so many more variables and so many more paths through the code. Add to this the fact that Mercury is heavily threaded (a technical term meaning that it can be doing many things at the same time) and that thread synchronization issues are among the hardest problems to reproduce, diagnose, and fix, and the testing process becomes a literal minefield. Testing Mercury is far, far harder than testing Pegasus Mail - orders of magnitude harder. Every time a feature is added to Mercury, testing begins again almost afresh, because every addition can have side-effects - unanticipated problems where the addition disturbs something already in existence - that have to be tracked down.

2: The dreariness of maintenance   There was an interesting thread in the off-topic forum here recently where a community member asked why programs had bugs. On the surface, his comment was perfectly fair, but in truth, any program larger than a few lines is going to have problems that creep in: the larger the program, the more possibilities there are for bugs, and complexity increases those possibilities even further. Software is never finished - it's always growing and changing, which means that the bugs are growing and changing too. Allied to this is the perpetual process of rewriting code - taking things that used to be "good enough" but now are not, and making them "good enough" again. At some stage in the life of all large software projects, you reach a stage where you spend more time tracking down bugs and rewriting old code than you do adding new features. This maintenance can't be avoided and mustn't be rushed (or else you'll typically just make things worse), but unfortunately, programmers are almost always relatively normal human beings, and like change and variety in their lives: having to spend most of your time reinventing or reworking old things can be pretty dreary, and on a cold Winter's morning, it's often the last thing you'll want to do. Maintenance, therefore, can be a slow process, if only for human reasons.

3: User expectations   It's a common truism in the industry now that user expectations get higher with each passing day: users expect programs to run flawlessly, to be easy to use, to cost nothing, and to have feature sets that match their every need. With higher expectation has also come lower tolerance, whether tolerance for errors, tolerance for ease of use, or even tolerance for appearance. The problem with higher expectations is that they are, obviously, harder to meet - but worse, they are usually exponentially harder to meet: making something "prettier" is actually a very difficult and time-consuming process, and even though it produces little actual tangible benefit, users expect it, even in programs like Mercury. If it's not there, lower tolerance means that people get more aggressive and more turned off. The difficulty and expense of producing programs that meet user expectations these days is reaching the point where only large corporations with huge resources will ever be able to afford the process. User expectations are basically killing the small-to-medium sized player.

4: Resources, resources, resources   I suspect most non-software-developers really have no conception of what it costs to produce a piece of software - "A PC on a table in your bedroom - what more do you need?" (that's an actual quote from someone who shall remain nameless, by the way). There are compilers (Visual Studio costs over $2000 here now), documentation tools (FrameMaker costs $1700), file servers for each supported environment, test servers, testbed systems for various versions of Windows, high-speed Internet links (unfortunately very expensive in New Zealand), domain names, certificate fees, support subscriptions and upgrades for software and development tools, books and reference materials, third-party software licenses (we pay over US$2000 per annum for the very small number of third-party tools we license), sundry development tools (such as Igor Arsenin's excellent TaskInfo product), electricity, banking and business costs, accounting costs... The list goes on. The cost of doing business gets higher every year, and peoples' willingness to pay in support seems to dwindle (and please, don't think of this as me criticizing my user community - it's a general statement about the world the Internet has become). Reduced resources mean making do with less, which in turn extends the time to release.

5: The dread of feedback   This one's a strange one, because it's a complete flip-flop from what I used to experience. In the earliest days of Pegasus Mail and Mercury, releases were exciting, because they were my chance to show the world all the cool new things I'd done. Because the world was more of a community back then, I got lots of supportive mail each time I did a release, which was also exciting and uplifting. These days, in the first few days after a release, the mail I tend to get is different: sure, there are still people who write to say thank you without strings or qualifications, but more normally the messages start with "Version xx is great, but...". It's really hard to describe how soul-destroying it is to release something you've worked really hard to produce, only to start getting wishlists and gripes almost immediately. When I say "gripes", I'm not saying that people are necessarily rude - it's usually just people expressing their annoyance that their pet feature or fix didn't make it into the release. And then there is the out-and-out rude mail; sadly, I always get a couple of those: these are the people who tell me that I should "just stop wasting my time and get a job at McDonalds (if they'll have me)". (And yes, that one's an actual quote as well). There's only a few, and they're always from people who either have a serious grudge of some kind or are simply ill-bred or psychologically troubled, but they have an impact out of all proportion to their importance. Just one of those messages can depress me so badly that I'm incapable of being a functioning human being for the remainder of the day. After seventeen years of doing this, you would think that I might have developed a thicker skin about this kind of thing, but I haven't: it really hurts, and over time, the anticipation of the mail I know I'm going to have to handle has made the process of release more and more daunting, more and more difficult.

Now, this probably all sounds like me complaining, but I'm not really... All I'm doing is summarizing a reality that became inevitable once the Internet reached the level of being a commodity. I should probably toughen up and lighten up, especially since I still actually enjoy the development process for the most part. If only the business of doing a release wasn't so stressful... But wishful thinking doesn't change the world.

Cheers!

-- David --

PS: If there are other software developers out there who read this, I'd love to hear your comments and experiences on the release process.

So the world moves on

David Harris — Wed, 16 May 2007 00:36:00 GMT

Last weekend, I wrote my first ever program in C#. Now, admittedly, in keeping with long-established historical tradition, it was only "Hello world", but even so, it felt like a step into a whole new universe.

I've been writing programs for a very long time now - getting on for thirty years, in one form and capacity or another. I've mostly written in C, although I did a lot in Pascal back in the CP/M days (I can hear the younger members saying "CP what?!?" as I write this), and have even written programs in (ugh) Basic. I'm probably one of only a handful of people left alive who can actually remember how to program in Focal-8, and I've spent years trying to forget how to write in Fortran... But C# is a sign of how much the world has moved around me, and how different everything has become.

For someone used to using the Windows Win32 API and writing procedural code, the C# and .NET way of doing things can be quite mind-boggling, particularly when you throw in some of the new .NET 3 features such as Avalon (Windows Presentation Foundation). There is quite literally so much information density there that I find myself asking again and again how anyone can ever learn enough of this stuff to be able to work with it competently. It's also daunting to think that the way you've been doing things for nearly twenty years is finally reaching the end of its usefulness, and that soon (all too soon!) change is going to be forced upon you whether you like it or not... So the world moves on.

I guess there are two ways you can deal with change - you can either resist it and hope that enough other people resist it to cause it to fail, or you can embrace it early and hope to reap the benefits over time. Both approaches have risks, but historically it's a truism that (in the immortal words of Douglas Adams) "Resistance is useless!" - or, if you take a more sinister viewpoint, "Resistance is futile - you will be assimilated". I for one find the intricacy and depth of the C# / .NET combination strangely seductive - the first time I've been able to say that about a Windows technology since Windows 95 was introduced. Whether this is the path I need to be walking is something I'm thinking about a lot at the moment: I haven't made my mind up for sure one way or another, but I'm definitely approaching it with an open mind, and even some enthusiasm.

Time will tell, I guess.

Cheers!

-- David --

Aaarrghh!The Pain!

David Harris — Fri, 11 May 2007 00:00:00 GMT

I'm perpetually mystified by the human propensity for self-delusion: we seem to be able to talk ourselves into the most ridiculous things, whether through hopelessly naive optimism, stupidity or selective memory. A wiser head than mine (George Santayana, if I recall correctly) once summed it up quite well, saying "Those who do not learn from the mistakes of history are doomed to repeat them".

From the preceding paragraph, you should be able to glean the idea that I'm not best pleased with myself at the moment. I've got myself into a quagmire, and like all quagmires, it's wider and stickier than I ever expected. I'm talking about the changes to the internal object interface used by Pegasus Mail.

When I originally took this on, it was a logical, worthwhile futureproofing modification that was only going to take a week or two to finish: that was nearly two months ago. Now, don't get me wrong - the "worthwhile", "logical" and "futureproofing" parts are all still completely valid, but seventeen years of doing this should have taught me that it was never going to take only a week or two - especially when the addressbooks became involved.

Pegasus Mail's legacy addressbooks are a source of unending grief for me: they're some of the oldest data structures in the program, dating from the DOS version. Back in the DOS days, programmers had to be really, really careful about how they used memory, because there wasn't much of it available. As a result, the addressbook code is incredibly contorted in places to allow the smallest possible memory use, even though in the days of gigabytes of RAM, the notion is now totally irrelevant. Unfortunately, slanting the entire development of a piece of code towards one priority like this is quite pernicious, because it debilitates almost every other part of the code. The result is that the legacy addressbook code in Pegasus Mail is fragile, awkward, convoluted and unbearably tedious in places, and I really don't like working on it very much.

Now, Pegasus Mail v5 will have a new addressbook format which will, I believe, be an absolute market leader - much more an information manager than an addressbook, its capabilities will make it enormously useful to almost everyone, unlike the antiquated and minimalist excuse for an addressbook we have at the moment. The design for the new format is done, and quite a lot of the code is even written: the problem is that I can't abandon the existing format - I'd get lynched - so I have to make sure that the two formats co-exist, if only to provide an easy migration path for existing users. This means that I have had to overhaul the old addressbook code once more so it can use the new interface I've designed.

The update on the old code is now mostly done, fortunately - although it's been quite a trial. There's still a bit to do though, so it's back to the salt mines for me. *sigh*.

Cheers!

-- David --