Progress updates

Progress updates

123
0
closed
David Harris posted Dec 2 at 9:10 am

[ Note: this update cannot currently be posted to the main Pegasus Mail web site because of access problems caused by changes made by Amazon, who host the site. ]


For much of this year, I have been working on a message validation protocol calld DKIM (DomainKeys Identified Mail) in Mercury. Last weekend, I finally sorted out the last glitch and went live with it on Hera, my primary mail exchanger. Since that time it seems to have worked as expected, although I still have a couple of areas I want to double-check to ensure there are no lingering issues (these are mostly to do with mail sent to mailing lists, which are a long-standing issue with technologies like DMARC, SPF and DKIM).


DKIM has an awful lot of moving parts - RSA key generation, hash generation, digital signing, DNS integration, quite complex message parsing and normalization... But I believe it's worth the effort, if only because it significantly decreases the likelihood of your mail being quarantined as spam by the big mail service providers (assuming you have valid SPF and DMARC records to go with it).


As soon as I'm sure there are no residual issues with mailing lists, I'll be making it available to my test team for a short beta period, then I'll bring it out as a new release, aiming for late in December or early in January.


In another significant development this week, I have finally overcome my distrust and reluctance, and have begun using ChatGPT as a search and code generation tool. I have to say, the results are really pretty astonishing. As an example, I asked it to write a routine that would open specified ports in the Windows Firewall, so I can add that to Mercury, and it did it effortlessly with only a couple of small tweaks required. This is something I've wanted for years, but couldn't work out how to do myself.


Anyone who has known me for a long time knows that I'm not very good at using Google (or DuckDuckGo, which I prefer for privacy reasons) to find information - I tend to get a bit lost and eyes-glazed-over in the result lists and often miss things that would be helpful. For this reason if for no other, ChatGPT is going to have a significant impact on the way I work, all for the better, leading to more efficient and quicker development.


As for WinPMail... Well, if I'm honest, it's been taking a bit of a back seat to Mercury recently as I trudged my way through the complexity of doing DKIM, but that changes from here on in. I've identified an issue to do with the way the program attempts to use the Windows Registry when running on Windows 11, and will be rushing out a patch for that as soon as I can; after that, a number of long overdue UI overhauls will finally get their moment in the spotlight, in particular major reworks of the message reader and composer windows, and the Contact Manager (the replacement for the now utterly ancient Pegasus Mail addressbook). I look forward to offering previews of these new features early in the New Year.


Well, that about covers it for now - watch for another instalment in a month's time.


Cheers!


-- David --


53
0

Introduction


This is the first instalment of what is intended to be a regular monthly
progress report on the development of Pegasus Mail and Mercury. Each
month, on the first day of the month, I will provide a brief summary of
what I have been working on in the programs in the preceding period,
posting it both on the main web site, and on the
community server at https://community.pmail.com.


If there are other pieces of software that have been in continuous
existence for nearly thirty-five years, I can't quickly think of one, but
Pegasus Mail sent its first ever message at the end of 1989, and Mercury
had its first release in 1993. During that time I have tried to be as
available to my users as possible, but I won't deny that after
thirty-five years I am quite burned-out when it comes to communication,
and I'm simply not very good at it any more. It's my hope that this
progress report will make up for some of the relative absence you may
have sensed from me in recent years, and for which I cannot offer enough
in the way of apology.


This progress update will be a little longer than the ones I expect to
post in future, because I want to give a slightly broader overview of
what's happening behind the scenes, in the hope that it might explain
why things seem to be taking so long. So, without further waffle then,
let's get into telling you what's been going on.


Background


One of the problems with a big program that is developed over a long
period of time is that parts of it can suffer "update starvation" - they
work well enough that even though they're not ideal, there are always
other things that seem more important to spend time on. As the years
pass, the less-than-ideal parts get more code bolted on to them, so they
get even less ideal, until eventually the program becomes so difficult to
maintain that you have to make a crucial decision: do I spend a lot of
time fixing what I have, or do I start from scratch and write a
completely new program? A number of years ago I was faced with this
decision and decided to overhaul what I already had.


With the benefit of hindsight, I can now see that this may have been the
wrong choice.


A lot of the code in WinPMail v4.8 was written in the 1990s, and some of
it even dates back to the DOS era. Replacing it with modern code that
works better and offers much easier future maintainability has ended up
being a quite monumental task. To give you an example, I have torn out
vast amounts of general purpose core code from WinPMail and produced
modern versions of it, often from scratch, in a library I call BaseLib:
BaseLib consists of some 58 separate files totalling well over 100,000
lines of code. To put that in perspective, Leo Tolstoy's "War and Peace"
has 71,500 lines - and BaseLib is just one component of a larger
ecosystem: there is also OIF (my object library, 24,000 lines), DHUI (my
user interface library, 64,000 lines), FS3 (my multi-purpose I/O library,
5,500 lines) and TCPIP (my network and cryptographic library, 23,000
lines).


Of course, all of this is before I even start talking about changing the
code that was left behind after I moved to the new libraries. Pegasus
Mail is now about 250,000 lines of code, and Mercury is about 125,000
lines, so that's still a lot to bring up to date.


Obviously, there are real benefits to having done all this work - all of
the libraries work identically in both Mercury and Pegasus Mail, so I'm
now having to maintain much less code than was previously the case. The
same is true of one of the most important pieces of code I have ever
written, which I call M3 (MailStore version 3), and which is near
completion: it replaces the entire mailbox management layer in both
programs with one that is identical for both, truly modern and enormously
robust. You'll notice it's called "M3": that's because there was also
"MailStore" and "MailStore v2" preceding it, both of which I ended up
scrapping before they were entirely complete because they did not meet
the standards or specifications I expect of myself; I don't think of
those abandoned versions of MailStore as wasted time, because I learned
a great deal from them, but they certainly represent lost time, and
when you're a solo developer, lost time is time you can never make up.


Had I started the programs from scratch, there is the possibility that I
might have been able to use a more modern language than the one I use, or
that I could have taken advantage of third-party libraries instead of
having to write my own... But I would have had to learn these things, and
it's my experience that the things you write while you're learning a new
language or environment inevitably get scrapped later once you gain
enough familiarity to understand the way you should have done them...
More lost time.


Things might be even more difficult if I weren't helped by some talented
and committed people, like Michael in der Wiesche and Martin Ireland,
who handle the HTML rendering in Pegasus Mail; or Sven Henze, who
tirelessly maintains the German version of Pegasus Mail; or Han van den
Bogaerde, who puts countless hours into supporting people who have
problems in the program; or my test team, who help iron out the bugs
before they can reach you: without people like this backing me up, I
don't believe it would be possible to keep the programs going.


But that's enough background and history though - what are we doing at
the moment?


Current focus


The overwhelming focus this year has been on Mercury, in particular
adding support for DKIM (a standard for verifying message origins) and
ACME (automated installation and renewal of SSL certificates). Mercury's
DKIM support is now finished, apart from a little work in one of the
backing libraries I mentioned above, FS3; a bit of testing and polishing
on it and I expect to have a release with DKIM support before the end of
this year. ACME support is a little more difficult, with the "Certbot"
program I expected to use to implement it ceasing support at the start of
the year. I've found a replacement though - a system called WinACME,
which I believe I can integrate quite easily into Mercury: expect to see
a release with ACME support early in 2025. Other things I'm working on in
Mercury are a completely new domain system which will allow separate user
lists for each domain the program supports, IPv6 support, support for
OpenSSL v3.x, and support for a wider range of IMAP extensions in
MercuryI.


As far as Pegasus Mail goes, it's been a year of refactoring - pulling
out more of that old code and replacing it with newer stuff; not very
satisfying either for me (because it's hard work) or for you (because it
doesn't result in anything that looks very different) - but essential
nonetheless. Sven and I have been putting a lot of work into rearranging
the program's resources (things like dialogs, graphics and menus) to make
them easier to maintain moving forward. I hope to have a full v4.91
release of WinPMail out in December or January.


That's about it for now - watch out for the next progress update on
December 1st.


All my best to you,


-- David --


2
2
1
Actions
Hide topic messages
Enable infinite scrolling
All posts under this topic will be deleted ?
Pending draft ... Click to resume editing
Discard draft