Thank you, Thomas.  I think that I have a handle on this now.  As I noted earlier some servers are sending several single messages in Bcc situations and some are sending a single message.  However, I know this now and know how to cope with it.

Gordon

I am puzzled how I received one of my e-mails.  The sender is known to me and has personally confirmed that they sent the e-mail.  The X-Originating-IP address is in the sender's ISP range.  However, there are no To: or X-Apparently-To: headers when the message source is viewed by Thunderbird or by Outlook Express.  Clearly, there was a To: header as there is a RCPT TO: line showing in the Mercury SMTP Server transaction.  Has Mercury removed this?

Can anyone explain to me what is going on here.  The complete header follows, with identifying information removed.

Thank you

Gordon

Return-path: <somebody@sympatico.ca>
Received: from col0-omc1-s1.col0.hotmail.com (65.55.34.11) by Delta-SMTP (Mercury/32 v4.74) .........;
   17 May 2012 07:16:20 -0400
Received: from COL118-W36 ([65.55.34.8]) by col0-omc1-s1.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
  Thu, 17 May 2012 04:16:20 -0700
Message-ID: <COL118-............@phx.gbl>
Content-Type: multipart/alternative;
 boundary="_5eb96bcb-f9ea-4f07-8b7a-22e67cc47695_"
X-Originating-IP: [174.89........]
From: <somebody@sympatico.ca>
Subject: FW: .......
Date: Thu, 17 May 2012 11:16:19 +0000
Importance: Normal
In-Reply-To: <D9746CB0F64D49......@maryc74be6f927>
References: <D9746CB0F64D49......@maryc74be6f927>
MIME-Version: 1.0
Bcc:
X-OriginalArrivalTime: 17 May 2012 11:16:20.0492 (UTC) FILETIME=[7C6EECC0:01CD341E]
X-AC-Weight: [####] (Whitelisted) -9999
X-CC-Diagnostic:
X-PMFLAGS: 570950016 0 1 MNBWC8FP.CNM

The To header isn't required, and Mercury won't add one if there isn't one in the received message. For SMTP messages from other servers delivery is performed according to the SMTP envelope recipient information only. I can't say why there isn't a To header in this message, but either there wasn't any when Mercury received the message, or there was some strange formatting problem with the message headers.

/Rolf 

Interesting.  So once a message like this is in Mercury's mail queue, does it know what account to deliver it to?  In this case, the message ended up in the wrong account, but that might be an issue with the filtering I am doing.

This message came from a customer of a major (in Canada) ISP, so I am surprised that the header isn't more "normal".  It was a forwarded message, so I don't know whether that has anything to do with the issue.

Thank you

Gordon

Yes, Mercury will know where to deliver a message without a To header; the SMTP envelope information is available to Mercury core. Otherwise BCC messages would never be delivered to the right recipient!

Could be that something happened to this message during forwarding that removed the header.

/Rolf 

[quote user="GordonM"]Interesting.  So once a message like this is in Mercury's mail queue, does it know what account to deliver it to?[/quote]

Yes, there are 2 files in the queue for each message - a .qdf file containing just the message body, and a .qcf file containg the envelope information including the recipients.  Mercury usually adds an 'X-Envelope-To:' header before delivering to a mailbox - unless the message is filtered.

[quote]In this case, the message ended up in the wrong account, but that might be an issue with the filtering I am doing.[/quote]

Maybe.  What are you filtering on?

The whole filtering process is rather complex (especially as I have some temporary things in, as I am changing my ISP).  I think that the key thing is that I am filtering on an Expression *username@mydomain.com*.  This should accommodate the stucture of X-Envelope-To:< username@mydomain >.  However, what I have noticed is that the message that started all this issue has no X-Envelope-To: header (I was a Bcc on this message).  I then sent a Bcc only message to my same account from another external e-mail account that I have.  When this arrived and I examined it, there was a correct X-Envelope-To:< username@mydomain > header.  I don't understand this.

Gordon

Mercury's X-Envelope header is not available for filtering as it is only created just before writing the message to the mailbox.

Your second message may not have triggered any filtering and that's why it has the header.

Paul - I am still looking for some consistency here.  The original problem Bcc: message showed no To: or X-Envelope-To:, a Bcc: test message from one of my external accounts showed a To: and a test message from my GMail account showed the To: as "Undisclosed recipients" and a X-Envelope-To: header.  I would have thought that any filtering I am doing would operate the same for the original message and for the GMail Bcc message.  However, I need to investigate some more.

Potentially, this may mean that Bcc messages can't be filtered based on the recipients address.

Thank you

Gordon

[quote user="GordonM"]

Paul - I am still looking for some consistency here.  The original problem Bcc: message showed no To: or X-Envelope-To:, a Bcc: test message from one of my external accounts showed a To: and a test message from my GMail account showed the To: as "Undisclosed recipients" and a X-Envelope-To: header.  I would have thought that any filtering I am doing would operate the same for the original message and for the GMail Bcc message.  However, I need to investigate some more.

Potentially, this may mean that Bcc messages can't be filtered based on the recipients address.[/quote]

Correct - filtering can only be on the body of the message and the envelope address won't always be in the body.

> Hi. I just want to confirm if the bcc's cannot be filtered and only
> the body of the message can? Is it possible to update or include this
> in the future? Thanks.

If you are receiving mail via MercuryS then you can filter on the SMTP e-mail address.  Also I am running a couple of Mercury daemons that add the RCPT-TO address to the headers so it can be filtered on by the mail client.  Drop me a line at the address below and I provide a list of the daemons I use that add this address.

Thomas - Thank you for the daemon.  It's a big step forward.  The only problem that still occurs is when there are two Bcc addresses, e.g. for my wife and me.  In this case, the two RCPT TO: transactions are showing in the MercuryS window, but there is delivery to only one account.  It seems to be the second RCPT TO: address that is successful.  The message for the first one is lost completely.  I am not sure whether the daemon that you gave me is capable of handling this two Bcc situation.

Regardless of the small problem, I am pleased with the result.

Thank you.

GordonM

> Thomas - Thank you for the daemon.  It's a big step forward.  The only
> problem that still occurs is when there are two Bcc addresses, e.g.
> for my wife and me.  In this case, the two RCPT TO: transactions are
> showing in the MercuryS window, but there is delivery to only one
> account.  It seems to be the second RCPT TO: address that is
> successful.  The message for the first one is lost completely.  I am
> not sure whether the daemon that you gave me is capable of handling
> this two Bcc situation.
>
> Regardless of the small problem, I am pleased with the result.

Strange, these two messages received separately via MercuryS should be received as two separate messages, one with a RCPT TO of your e-mail address and one with your wife's address.  Are you sure you are not using some sort of filter to block duplicates?  

This is all rather curious.  I did tests sending Bcc messages from 4 different accounts; Hotmail, my local Freenet, my ISP's webmail, and GMail.  Each message had two Bcc entries with no To: and no Cc.  GMail and Freenet sent the message as two separate messages (in the SMTP transaction), each to one of the individual addressees. The messages appeared in the Inboxes of each of the addresses, as expected.  Hotmail and my ISP sent the message as a single message and the SMTP transaction shows two RCPT TO lines one after the other.  In the Hotmail and ISP cases, the mail was delivered only to the Inbox of the second RCPT TO address.  My next test was to send another message from my ISP but with the order of the addressees reversed.  This resulted in a single message being sent, as before, with two RCPT TO addresses, but with the order reversed.  Again only one Inbox received the message and it was the one listed by the second RCPT TO.

So, I don't think the problem is anything to do with any filtering I may be doing.  It seems to be consistent that when only one message is sent for two recipients, only the second recipient receives the message.  So I am wondering whether this is a problem with Mercury, is it a problem with the daemon or is it something else.

Failing anything else, my next step is to remove the daemon and turn off the global filtering and then see if the single message with two RCPT TO entries is delvered to one or both addressees.  This will tend to point to either Mercury or the daemon as the problem, depending what the result is.  I will be doing this tomorrow as it's getting rather late here now.

Any suggestions appreciated.

GordonM

I think that I am now getting somewhere.  If I remove the daemon and exit global filtering without doing anything and send a message from my ISP (which sends a single message with two RCPT TO lines), the message is delivered to both addressees.  If I put the daemon back, while still exiting global filtering immediately, the result is the same.

My speculation is that at the time the message is coming through global filtering, it is still juse one message with  two addressees.  If so, the message must be split after global filtering to go to the two addressees.

My filtering had moved the message to each addressee in turn, but this means that, with only one message, the message will be removed from the mail queue before the second addresses can be dealt with.  I have now copied rather than moved the message to the two addressees and they are now both receiving the message.

I don't know whether my speculation is correct, but the process is now working for me.

GordonM

> My filtering had moved the message to each addressee in turn, but this

means that, with only one
> message, the message will be removed from the

mail queue before the second addresses
> can be dealt with.  I have now

copied rather than moved the message to the two addressees and
> they are

now both receiving the message.

Since the message was addressed to both users why were you doing any filtering at all?

Thomas - Maybe I am missing something but, I don't understand why I wouldn't want to do do filtering just because I want the message to go to both of the Bcc addressees.  I use filtering for several things including taking action as a result of content control, deleting message from people who are forever forwarding unwanted "funnies" to me, change of address messages, moving messages from certain users into special accounts, selecting messages that I want to be vislble on my phone (I don't want to see commercial stuff), and recognizing and dealing with SPAM in various ways.  I am also in the process of transitioning as a result of moving from using MercuryD to MercuryS, encouraged by my change of ISP.  This means that I am changing some of my filtering rules.

If I want to do any of the above filtering, even If these problem Bcc messages don't necessaily need to be filtered, they still need to be recognized and caused to exit filter processing.  This goes back to the original problem of no To: address, which your daemon solved.

From http://wiki.pmail.com/index.php?title=KB:Mercury32/Mercury32_Message_Processing_Flowchart, it seems to me that my speculation is correct that only a single message goes through global filtering, even if there are serveral addressees.  It is then distributed to local addressees after global filtering.

GordonM

> If I want to do any of the above filtering, even If these problem Bcc
> messages don't necessaily need to be filtered, they still need to be
> recognized and caused to exit filter processing.  This goes back to
> the original problem of no To: address, which your daemon solved.

I have quite a few global filters and never use an exit out of the filtering process. If it does not match a rule the system does nothing at all, just delivers the mail to the RCPT TO addresses.   I literally do not understand a system where an global exit rule is used for all messages, or for any messages.

> it seems to me that my speculation is correct that only a single message

goes through global filtering,
> even if there are several addressees. 

It is then distributed to local addressees after global filtering.

Generally Bcc: means you get a separate message for each Bcc: address, otherwise it's possible for someone to recover all of the other Bcc: addresses.  That said, even when a single message is sent with multiple RCPT TO addresses as long as the processing if not stopped by some sort of filtering rule. everyone gets a separate copy of the message when sent via MercuryS.