Just tried two messages I've seen this week.

1) Loooong header line (MS Outlook x-cr-hashedpuzzle:)

2) Long lines in an inline text attachment sent by Thunderbird 2.0.13. Format "flowed".

Both worked fine when downloaded from 4.72, not from 4.01b.

Guess I'll have to upgrade. 

This refers to a bug that was reported in detail on the Mercury mailing list several years ago, but still seems to be present in the current version (4.01).  Specifically, if a header line exceeds 253 characters, Mercury splits it by inserting a CR/LF after the 253rd character.  The split is not done correctly, causing two undesirable side effects:

(a) if the header line has more than 253 printable characters, then the characters after the split start immediately at the beginning of the next line, without any preceding whitespace, so the part after the split is regarded as a new (and most likely unrecognisable) header, rather than as a continuation.

(b) If the header line has exactly 253 printable characters, the CR/LF is inserted immediately before the header line's own CR/LF.  This effectively results in the insertion of a blank line after the header, which causes all following headers to be regarded as being part of the email body.

My setup is: email picked up from a POP3 account with MercuryD, and distributed to two user mailboxes from where the users pick up the mail via MercuryP.  Outgoing mail is sent via MercuryC, though I am not aware of any issue with headers on outgoing mail. 

This bug never troubled me before. Unfortunately, I have just changed my DNS host and the new hosting service adds a Received-SPF: header to forwarded emails, which just happens to be generally between 250 and 260 characters long.  This means that around 10% of my incoming mails, when the line is exactly 253 printable characters long, are being corrupted.  This isn't really very acceptable.

I can provide more information if required.  Is there any likelihood of this bug being fixed in the near term, and is there any formal bug tracking system where I should enter it so that it doesn't get forgotten (as seems to have been the case so far)?

Thanks
Tony

As far as I recall, this is the first I have heard of this "bug". Remember that I have not been a regular participant in the public mailing lists for a number of years due to overload, and if this was reported back to me by my test team, I'm afraid I missed it.

Firstly, I assume you are *certain* that the line breaking is not being done by the POP3 server from which MercuryD is retrieving mail? The reason I say this is because if you were reaching a hard line-length limit in MercuryD, then the header line would be truncated, not wrapped in the way you describe - the characters beyond the line break would simply be omitted altogether. MercuryD doesn't have any line folding mechanism in place, because it doesn't need it.

Secondly, I've just been through the MercuryD POP3 and distribution code and have verified that line length limits are never shorter than 1000 characters (the SMTP mandated maximum). In fairness, this is the v4.5 code, and it is possible that the limits may have been extended since v4.01 (extending line length limits has been a general and ongoing process during the v4.5 development).

I definitely don't want to sound like I'm in denial here, but what you're describing does not match any of the possible paths I can see through the MercuryD source code - the code doesn't appear to be able to generate the symptoms you're describing. Now, I'm willing to grant that I may have missed something - there's a fair amount of code involved, and I've tended to look at the most likely parts of it in detail, but in isolation. I focused on the areas that actually read and write headers: I don't think there are any other areas that could be relevant, but I am too cautious to exclude the possibility. With that disclaimer in mind, I have to say that my gut feeling based on what you describe is that it's the POP3 server that's doing the line breaking. If you can check this out and confirm categorically it is not the case (by producing a MercuryD session log showing the header being received in a correct, unfolded state), though, then I'm willing to look into it further.

Cheers!

-- David --

First of all, David, thanks for the quick reply.  I quite understand your scepticism.

Firstly, the main reason why I think that this is a Mercury issue is that it is remarkably similar to bug reports that I found while searching the Mercury list archive for clues.  Specifically, the topics started at http://bama.ua.edu/cgi-bin/wa?A2=ind0106D&L=mercury&P=R971&I=1 and http://bama.ua.edu/cgi-bin/wa?A2=ind0003C&L=mercury&D=0&I=1&P=3173 (dating back to 2000 and 2001) describe line folding problems that are very similar indeed to what I see.  However, I have no other evidence, except that the position of other headers that are inserted by Avast! (my local antivirus) show that the corruption has occurred before it reaches my PC, and hence my antivirus and local mail client (Thunderbird) are not to blame.

Secondly, I notice that you are focusing on MercuryD -- I didn't intend to imply that I thought that the fault specifically lay there.  I'm just saying that my incoming mail path is MercD -> core -> MercP, and the problem (if it is a Mercury problem) could be in any of these three components, but not in any other part of Mercury.  Interestingly, one of the linked topics above refers to the "line folding" problem being in MercuryP.

I will collect logs from both MercD and MercP to try to show the problem - if this isn't conclusive, I can use WireShark to sniff the messages into and out of the PC running Mercury to eliminate sources external to the PC.  Incidentally, I'm also running AVG and SpamPal in front of Mercury, so they could be causes - I'll remove them too.

By the way, the message at http://bama.ua.edu/cgi-bin/wa?A2=ind0108C&L=MERCURY&P=R8566 (also on this subject) has an interesting closing paragraph.  Did you remember? :-)

Regards

Tony

OK, I have some session logs from MercuryD and MercuryP that demonstrate the incorrect insertion of a blank line in the message.  I don't know how well they're going to come out here, but let's try anyway.

This is an excerpt from the MercuryD log, giving the problem header and a couple of context lines on either side (I've obfuscated the email addresses but kept the same number of characters):

15:17:47.808: >> Received: from mail9.freedom2surf.net ([194.106.56.33] helo=mx3.freedom2surf.net)<cr><lf>
15:17:47.808: >>     by avs01.freedom2surf.net with esmtp (Exim 4.43)<cr><lf>
15:17:47.818: >>     id 1HraSK-0006hk-7H<cr><lf>
15:17:47.818: >>     for tonyeva@xxx.com; Fri, 25 May 2007 15:07:48 +0100                       <cr><lf>
15:17:47.818: >> Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.179])<cr><lf>
15:17:47.818: >>     by mx3.freedom2surf.net (Postfix) with ESMTP id 0C10B8B5D6A<cr><lf>
15:17:47.818: >>     for <tonyeva@xxx.com>; Fri, 25 May 2007 15:06:59 +0100 (BST)               <cr><lf>
15:17:47.828: >> Received-SPF: none (mxeu16: 66.218.67.71 is neither permitted nor denied by domain of returns.groups.yahoo.com) client-ip=66.218.67.71; envelope-from=sentto-3661725-2653-1180101995-tony=xxx.xx.uk@returns.groups.yahoo.com; helo=n11e.bullet.scd.yahoo.com;<cr><lf>
15:17:47.828: >> Received: from [66.218.67.71] (helo=n11e.bullet.scd.yahoo.com)<cr><lf>
15:17:47.828: >>     by mx.kundenserver.de (node=mxeu16) with ESMTP (Nemesis),<cr><lf>
15:17:47.828: >>     id 0MKxE4-1HraRB1H4T-0006jC for tony@xxx.xx.uk; Fri, 25 May 2007 16:06:38 +0200<cr><lf>
15:17:47.828: >> Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys              <cr><lf>
 
 This is the same section of the same email as logged by MercuryP:

17:40:01.709: << Received: from mail9.freedom2surf.net ([194.106.56.33] helo=mx3.freedom2surf.net)<cr><lf>
17:40:01.709: <<     by avs01.freedom2surf.net with esmtp (Exim 4.43)<cr><lf>
17:40:01.709: <<     id 1HraSK-0006hk-7H<cr><lf>
17:40:01.709: <<     for tonyeva@xxx.com; Fri, 25 May 2007 15:07:48 +0100                       <cr><lf>
17:40:01.709: << Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.179])<cr><lf>
17:40:01.709: <<     by mx3.freedom2surf.net (Postfix) with ESMTP id 0C10B8B5D6A<cr><lf>
17:40:01.709: <<     for <tonyeva@xxx.com>; Fri, 25 May 2007 15:06:59 +0100 (BST)               <cr><lf>
17:40:01.719: << Received-SPF: none (mxeu16: 66.218.67.71 is neither permitted nor denied by domain of returns.groups.yahoo.com) client-ip=66.218.67.71; envelope-from=sentto-3661725-2653-1180101995-tony=xxx.xx.uk@returns.groups.yahoo.com; helo=n11e.bullet.scd.yahoo.com;<cr><lf>
17:40:01.719: << <cr><lf>
17:40:01.719: << Received: from [66.218.67.71] (helo=n11e.bullet.scd.yahoo.com)<cr><lf>
17:40:01.719: <<     by mx.kundenserver.de (node=mxeu16) with ESMTP (Nemesis),<cr><lf>
17:40:01.719: <<     id 0MKxE4-1HraRB1H4T-0006jC for tony@xxx.xx.uk; Fri, 25 May 2007 16:06:38 +0200<cr><lf>
17:40:01.719: << Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys              <cr><lf>

This is a case where the Received-SPF: header is exactly 253 printable characters.  I have other logs with longer lines, showing the incorrect splitting, but this is the case that causes the most trouble because it prematurely terminates the header.

Let me know if you need any more info.  I'd be very grateful to know when this bug is likely to be fixed.

Regards

Tony
 

It may be something in MercuryP - when I originally looked into this, I focused on MercuryD. I'll take a look this afternoon (although as I remarked elsewhere, it's quite possible this has already been fixed, so I might not find anything).

Cheers!

-- David --

Any luck yet?

I just noticed that this happens with more than just the header lines. I had a couple of emails with long content lines, and they had been wrapped at 253 characters, just like the header.  I don't have any logs but I see no reason to doubt that it's the same bug.  Of course, in the data field, folded lines don't have much of an effect - an occasional broken word, or perhaps a broken link or HTML tag.  Nothing very serious, but I just wanted to mention that this isn't solely confined to header lines.

Regards

Tony
 

[quote user="tonyeva"]

Any luck yet?

Further to this thread - it now appears that the fixes I made (see above) have corrected the problem for v4.5, so I'm going to consider this thread "resolved".

Cheers!

-- David --

Old thread, I know, but I've only recently had the misfortune to spend some time with this problem (yes, still on 4.01 due to upgrade anxiety and license change).

 Would you know if the fixes you applied also pertained to the body section of the email being downloaded from MercP? I see the same line split behaviour for text bodies and any inline attachments of type text.

 Kind regards

hawk 

I don't know of any such problem, but if you have a sample message with that behavior in 4.01 I'd be happy to try it in the current version of Mercury.

/Rolf