Thomas wrote:  "There is nothing that MercuryS, MercuryI and Mercury P can do to identity anything in the customers equipment or mail client."

 The Directive says "identify users' communication equipment or what purports to be their equipment"

 I don't think, from that wording, that there's a requirement to identify the type of equipment, or anything in it. All it requires is to identify the equipment, ie, its FQDN or IP address. I assume the second part " or what purports to be their equipment" is avoid providers being in default when they receive mail from spoofed addresses.

I'm raising this issue on a number of forums as preparation.

In short, there exists a European legislation directive that states that the member states shall adopt the directive 2006/24/EC which states in article 5, for publicly available electronic communication services, that we shall save for at least 6 months, records of email origin, name of sender, ip-numbers, tools used, paths, recepient, date & times and some more - so that message flows are traceable. Message content does not need to be stored.

The directive is readable in bi-lingual at: http://eur-lex.europa.eu/Notice.do?mode=dbl&lng1=en,de&lang=&lng2=cs,da,de,el,en,es,et,fi,fr,hu,it,lt,lv,mt,nl,pl,pt,sk,sl,sv,&val=425159:cs&page=&hwords=null

To sum up the demands of the directive;
We who have publicly available email services need to store nearly all received email headers for at least 6 months.

Regarding our use of Mercury/32, the current logging level doesn't support this legislation. It might well be so that we have to adopt this by mid year 2008.

David, could you please read up on this issue, and state if you think this is doable within Mercury?
Another issue is the byte space needed, and if compression can be invoked, as well as usability in producing reports from saved data headers.

Other comments of course welcome.

[quote user="Peter Strömblad"]

I'm raising this issue on a number of forums as preparation.

In short, there exists a European legislation directive that states that the member states shall adopt the directive 2006/24/EC which states in article 5, for publicly available electronic communication services, that we shall save for at least 6 months, records of email origin, name of sender, ip-numbers, tools used, paths, recepient, date & times and some more - so that message flows are traceable. Message content does not need to be stored.

The directive is readable in bi-lingual at: http://eur-lex.europa.eu/Notice.do?mode=dbl&lng1=en,de&lang=&lng2=cs,da,de,el,en,es,et,fi,fr,hu,it,lt,lv,mt,nl,pl,pt,sk,sl,sv,&val=425159:cs&page=&hwords=null

To sum up the demands of the directive;
We who have publicly available email services need to store nearly all received email headers for at least 6 months.

Regarding our use of Mercury/32, the current logging level doesn't support this legislation. It might well be so that we have to adopt this by mid year 2008.

David, could you please read up on this issue, and state if you think this is doable within Mercury?
Another issue is the byte space needed, and if compression can be invoked, as well as usability in producing reports from saved data headers.

Other comments of course welcome.

[/quote]

If you use an "always" filter in Mercury/32 that copies all mail to an "archive" user then you are going to get a copy of everything passing through the server.  I personally put my always filter after the filter that moves the spam off to a spam account so that I'm not archiving all of the spam.  

I work the archive user on a regular basis to organize the mail to monthly folders.  Some may want to move the monthly mail off to a CDROM by putting all of the mail into a new mail directory so it can be easily read from the CDROM if required.

Not sure if this completely meets the requirements of the proposed law but if this were saved with the System, MercuryS and MercuryE it sure seems to me like it meets the intent of the law.

1. We're not allowed to store the actual contents of a message.
2. All traffic needs to be stored, ie messages routed off-host as well as inbound, as ... probably retrieved, pop3/imap connection. Purpose is to have a clear trace from origin all the way to the end-receiver actually receiving it in the client box.

[quote user="Peter Strömblad"]

1. We're not allowed to store the actual contents of a message.
2. All traffic needs to be stored, ie messages routed off-host as well as inbound, as ... probably retrieved, pop3/imap connection. Purpose is to have a clear trace from origin all the way to the end-receiver actually receiving it in the client box.

[/quote]

I must have miss-read the requirements then.  I did not read that you must not store the body of the message, all I read is you must save the headers.   The "always" filter keeps anything that passes through the system, if it's forwarded then you get a copy of it coming in and a copy going out.  The POP3, MercuryS, MercuryD, MercuryE and system logs also allow you to trace connections.  If you are saying that the EU is going to require all of the ISPs to replace there current SMTP mail systems I certainly do not see this as a requirements at all.  They are saying to me is that you must keep a copy of the message headers of all mail passing through the system.

I read it as applying to ISPs, ie, people running publicly available services, not publicly available servers.

The definitive law, for any individual or corporate entity, is the law enacted by their national government. Individuals are not bound by a Directive, only by the law which implements it - which may be subtly different.

EU/EG/EC Directives state minimum demands. National laws are meant to adopt minimum requirements or baldly go beyond these limits.

Here in Sweden, public services are services offered that any or some entity can subscibe to. An entity can be a person, company, association, school, union etc. Therein do I qualify, as does this site with its emailing capabilities. 

Current Swedish proposal targets january 1st 2009 as adoption. Meaning the bill processing has to be decided at least by mid fall 2008. The Swedish proposal extends some of the minimum requirements, as for example the storage period is set to 1 year and not 6 months.

So to sum up the demands, as for Mercury, the easiest adoption is to save all received and added headers, for inbound as well as outbound traffic regardless of any filtering, spamhalting, content-rbl, antiviral treatment etc., and store this logged content in a chronological format, and delete the log automatically after a set period of months.

[quote user="Peter Strömblad"]

EU/EG/EC Directives state minimum demands. National laws are meant to adopt minimum requirements or baldly go beyond these limits.

Here in Sweden, public services are services offered that any or some entity can subscibe to. An entity can be a person, company, association, school, union etc. Therein do I qualify, as does this site with its emailing capabilities. 

Current Swedish proposal targets january 1st 2009 as adoption. Meaning the bill processing has to be decided at least by mid fall 2008. The Swedish proposal extends some of the minimum requirements, as for example the storage period is set to 1 year and not 6 months.

So to sum up the demands, as for Mercury, the easiest adoption is to save all received and added headers, for inbound as well as outbound traffic regardless of any filtering, spamhalting, content-rbl, antiviral treatment etc., and store this logged content in a chronological format, and delete the log automatically after a set period of months.

[/quote]

And as such a simple always filter filtering the mail off to a "archive" user should do it. You'll need to be saving all the logs as well for the same period if you really need to be displaying what you have rejected as well.  This is what all corporations in the US and most government agencies have been doing fo years.  You save everything that passes through the server just in case someone comes after you.  ;-(  This can be archived off to a CDROM and then destroyed after a year in your case.  The hands off operation will depend a lot on the tools that yuo develop to control this data and heven help yuo if they ever message up and delete everything.  That said, I do not think this is something that should be built into Mercury/32 since this is too much of a risk for David if something goes wrong.

This sounds like ideal territory for a plugin daemon that just dumps headers to a logfile.

I don't quite agree, although I understand the standpoint. Reason is;

a product that in its "packaged" state does not comply with current law (read this as if it already is 2009), will most likely not be adopted by public hosters. It's a bit like saying it's up to you to fulfill the rfc's, the product itself only supports portions.

Secondly, on technology, the daemon would have to be invoked in MercuryC/S/E/D and possibly the core as well - that is, for a closed source/binary, too daunting task for end users to crack.

Just my opinion though: product's target market could well be not to support European ISPs - but I sincerely hope not to be excluded that way.

[quote user="Peter Strömblad"]

I don't quite agree, although I understand the standpoint. Reason is;

a product that in its "packaged" state does not comply with current law (read this as if it already is 2009), will most likely not be adopted by public hosters. It's a bit like saying it's up to you to fulfill the rfc's, the product itself only supports portions.

Secondly, on technology, the daemon would have to be invoked in MercuryC/S/E/D and possibly the core as well - that is, for a closed source/binary, too daunting task for end users to crack.

Just my opinion though: product's target market could well be not to support European ISPs - but I sincerely hope not to be excluded that way.

[/quote]

I somewhat agree that a packaged product is better for the specific market but as you have stated each government in the EU is going to implement this via their own laws.   Unless David spends a whole lot of time on this making it possible for users in each country in the EU to have a turnkey system based on the way their country implements the law it's going to be chaos. 

This sort of data retention has been the law for some time in the US, all message traffic must be saved for at least some period of time.  The logs and the always filters of Mercury/32 (and other SMTP hosts as well) allows the user to record the message traffic and store the logs and data for the required period of time.  Does take some system admin work to manage the data but system admins have been managing backup data for years.

[quote user="Peter Strömblad"]

To sum up the demands of the directive;
We who have publicly available email services need to store nearly all received email headers for at least 6 months.

[/quote]

I believe you're mis-interpreting the directive. It requires retention of data on the source, the destination, and the date and time of communications, I don't see how it requires anything above retention of the existing logs generated by Mercury.

In addition it applies to Service Providers which under UK law has been defined as providing a service to the public. It does not apply to organisations such as ours since we are considered a Private Network for the purposes of other existing legislation and this would also apply to a large number of others (nearly all?) using Mercury, I await to see what will happen in terms of national legislation but I think adding features to Mercury may be putting the cart before the horse.

David.
 

The logs from the various modules are there yes, but they make it impossible to track sender from/To, IP numbers used down to the very maildrop in use. Nearly all this info is recorded along the way within each message. Lastly, we're not allowed in any way to store message content (just like any unauthorized phone tap is illegal). The sum was, and is - that the easisest method is to store the full message headers.

So: 1. I'm a service provider, 2. I rely on Mercury today, 3. I've already had our NSA and the feds demanding logs in connection to 9/11, 4. Swedish law, will most likely be in place by january 2009 - and it goes beyond the directive on a number of counts. 5. I have to comply to our national law (me sense of morale tells me to), 6. I'd rather see this solved within Mercury/32 than a) creating my own daemons, b) not complying to law, c) leaving Mercury ....

Lastly, I imagine universities will be regarded as public service providers regarding student communication since there is no employee/employer contract with the students.

You could save the email headers via a simple policy, but it seems to me that a major task in your case will be the storage, archiving and rotation of the data - is that really a task for Mercury?

[quote user="Peter Strömblad"]

The logs from the various modules are there yes, but they make it impossible to track sender from/To, IP numbers used down to the very maildrop in use. Nearly all this info is recorded along the way within each message. Lastly, we're not allowed in any way to store message content (just like any unauthorized phone tap is illegal). The sum was, and is - that the easisest method is to store the full message headers.

[/quote]

The only thing I can see that's missing is matching up the message id from the core module log with the incoming smtp session in the mercurys log, the core module logs the destination mailbox, size, date, time and the mercurys log logs the source ip address and to and from addresses and date and time. Such a level of logging is not dissimilar from most other mail servers.

[quote user="Peter Strömblad"]

Lastly, I imagine universities will be regarded as public service providers regarding student communication since there is no employee/employer contract with the students.

[/quote]

For the purposes of UK law this isn't correct, from page 2 of the ja.net factsheet on guest access (http://www.ja.net/services/publications/factsheets/073-guest-and-public-network-access.pdf):

[quote]
Organisations considering such partnerships should note that any network that offers Internet access to the public is likely to be classified in law as a public network, whereas JANET and the networks of its customers are generally classified as private. Operating a public network is likely to involve more onerous duties, for example:

• protection of the privacy of users (Regulation of Investigatory Powers Act 2000 and Privacy and Electronic Communications (EC Directive) Regulations 2003)
• provision of information to users (Communications Act 2003) and
• (possibly in the future) retention of data about usage for criminal investigations (Anti-Terrorism, Crime and Security Act 2001 and the European Data Retention Directive 2006/24/EC).

[/quote]

David.
 

[quote user="PaulW"] You could save the email headers via a simple policy, but it seems to me that a major task in your case will be the storage, archiving and rotation of the data - is that really a task for Mercury?[/quote]

storing I think yes, but the rest not neccesarily the mail server task.

[quote user="Sully"]The only thing I can see that's missing is matching up the message id from the core module log with the incoming smtp session in the mercurys log, the core module logs the destination mailbox, size, date, time and the mercurys log logs the source ip address and to and from addresses and date and time. Such a level of logging is not dissimilar from most other mail servers.[/quote]

The directive states:
(a) data necessary to trace and identify the source of a communication:
(b) data necessary to identify the destination of a communication:
(c) data necessary to identify the date, time and duration of a communication:
(d) data necessary to identify the type of communication:
(e) data necessary to identify users' communication equipment or what purports to be their equipment:
(f) - doesn't apply to email.

More modules than just MercuryS are needed to fulfill (a), as I read that not just inbound connections are bound by this. (e) is quite effectively stored within the message headers, nowhere else. But regarding f.ex. MercuryP or MercuryI we have no log nor communication within the rfc I believe at all to "identify users' communication equipment" // ie mailer // like is done within the http request object.

If it's not email we are talking about then it sure does not pertain to Mercury or Pegasus Mail.  There is nothing that MercuryS, MercuryI and Mercury P can do to identity anything in the customers equipment or mail client.  If it's in the headers of the message you can identify this for the sender assuming it's not all faked but there is nothing other than the IP address that can be verified in any mail transaction.

What specific changes are you recommending to be done to Mercury/32 to solve the problem as you see it?

We do not archive and will not archive spam messages beyond a month or two whatever the EU may direct us to do!

 We are of course a private network so perhaps the directive does not apply to us!