Pegasus Mail & Mercury

One of the simplest things you can do to reduce the amount of spam your system has to process is to add a couple of simple MercuryS transaction filtering rules that detect spam zombies.

For the uninitiated, a "spam zombie" is a personal computer that has been compromised by a trojan horse or other malware package, and which can be remotely controlled. These machines, also called "bots" are an enormous problem - they tend to be organized into networks controlled by shadowy criminals (often in the former Soviet Union countries or in China); these networks may consist of thousands or even hundreds of thousands of compromised systems. One very common use for such "botnets" is to make them available for hire as a means of distributing spam. Anecdotal evidence suggests that this practice can be very lucrative, so don't expect it to go away any time soon.

Fortunately, whether through simple incompetence on the part of the writers of these malware packages, or because it's somehow necessary if they're to keep a low profile, these spambots typically do a laughably poor job of implementing the very basic SMTP protocol used for mail. We can take advantage of these weaknesses to detect and eliminate their connections before they even send any data and waste our time.

Here are a few simple tests you can do: I've been using them for a long time as part of my spam-defeating strategy, and they work well for me - they may work well for you too.

1: Check the HELO parameter to see if the connecting host is identifying itself as you.  This may sound idiotic, but an awful lot of spam zombies don't seem to be able to do domain name resolution properly, and instead simply use your domain name or address in the HELO greeting. This one is an absolute guarantee that the connecting client is bogus, and you can use it to form reliable blacklists. So, if your mail server's hostname is mail.example.com, any time you see a connection issue the command HELO mail.example.com or EHLO mail.example.com, you can drop the connection and blacklist the host. The MercuryS transaction filtering rule that does this might look like this:  H, "*mail.example.com*", RS, "554 Format of command not acceptable here". Similarly, the zombie may attempt to pass your own IP address to you in the HELO line: this is another easy and utterly foolproof test: assuming your mail server's IP address is 190.140.4.4, you could use a MercuryS rule like this: H, "*190.140.4.4*", RS, "554 Format of command not acceptable here".

2: Check for a fully-qualified domain name in the HELO command.   The SMTP specification requires all HELO commands to include the fully-qualified domain name (or FQDN) of the client requesting service. An FQDN must by definition contain at least one period character, so you can use a MercuryS transaction filtering rule like this one to reject non-compliant hosts:  H, "*.*" RSN, "554 Format of HELO/EHLO greeting unacceptable." (the "N" in the action group says that the rule only triggers if the string does NOT match the expression supplied). This test is not as clear-cut as the tests in section 1, but any properly-configured mailer should be generating compliant commands, so you're only going to be turning away non-compliant clients. In more than a year of using this rule, I'm not aware of a single legitimate false positive it has generated.

3: Set up a honeypot address then blacklist any attempt to send mail to it.   A "honeypot" address is an address that exists only to attract spam - the address itself is typically not even a valid address. To create a honeypot, find some obscure location in a web page you control and create a hyperlink containing a mailto: link for the honeypot address. A good approach is to find a page that is not frequently accessed, then hyperlink a single period character somewhere on the page. Make sure you include an ALT tag in the HTML telling any real person who actually finds the link not to send mail to it. Once you've done this, set up a transaction filtering rule that checks for any attempt to send mail to that address. Assuming your honeypot address is sugar@example.com, you could create a rule like this: R, "*sugar@example.com*" BS, "554 Spam is not welcome here - go away."  Now, it's fainly possible that a spammer might route a message to your honeypot address through a legitimate host, so you might want to be careful about blacklisting, but it has caused no obvious problems for me. And in case you're wondering how the spammer is going to find your honeypot address when you've gone to such lengths to make it obscure, believe me, you'll be stunned how quickly you start getting hit. Spammers use very active "web spiders" to harvest addresses from web pages, and you'll almost certainly find that your first honeypot hit will occur less than three hours after your honeypot link goes public.

As with all things, your mileage may vary on these tests, but they have proven helpful and worthwhile for me. In time, the spammers will undoubtedly adapt and improve what they do to get around tests like these, but for now, they work quite well. I hope they are useful to you.

Cheers!

-- David --

If I add a rule like this: 

R, "*sugar@example.com*" BS, "554 Spam is not welcome here - go away."

 
Is it possible to see the list of short-term blacklisted sites?
Can offending sites be permenantly blacklisted?

 
No doubt the rule David suggests is  suitable for practical use, but I'm just curious & would like to see how many blacklisted sites would build up over say a 24 hour period.

 Richard

 

 

 

Richard:

If I add a rule like this: 

R, "*sugar@example.com*" BS, "554 Spam is not welcome here - go away."

Is it possible to see the list of short-term blacklisted sites?
Can offending sites be permenantly blacklisted?

No doubt the rule David suggests is  suitable for practical use, but I'm just curious & would like to see how many blacklisted sites would build up over say a 24 hour period.

You can't currently see the list of short-term blacklisted sites (not even in v4.5). To be honest, it never occurred to me that anyone would want to see it.

Offending sites can be permanently blacklisted manually, using access control lists. I originally intended to add the facility to have transaction filters automatically generate permanent blacklists as well as short-term blacklists, but decided in the end that doing that was just too dangerous.

My own production mail server has an extensive range of rules that result in short-term blacklisting (including various RBL hits that lead to short-term blacklisting). At the moment, it's been up since April 18, the current number of entries in the short term blacklist is 165, and the peak value is 1262. The program can easily have 30,000 entries in the STB without any performance hit, so clearly loading is not an issue. I estimate that I reduce the overall bandwidth I would otherwise consume by over 90% by doing this, so it's a very significant contributor to my anti-spam strategy.

Cheers!

-- David --

I am having difficulty with the logic of item #3:

3: Set up a honeypot address then blacklist any attempt to send mail to it.

 Why do I want to set up a honeypot just to reject the mail at the transaction level? To me it seems that I'm just inviting more needless hits on my mail server. Am I not better off not even having this type of honeypot?

 
Curt Cuppels
 

 

David Harris:

2: Check for a fully-qualified domain name in the HELO command.   The SMTP specification requires all HELO commands to include the fully-qualified domain name (or FQDN) of the client requesting service. An FQDN must by definition contain at least one period character, so you can use a MercuryS transaction filtering rule like this one to reject non-compliant hosts:  H, "*.*" RSN, "554 Format of HELO/EHLO greeting unacceptable." (the "N" in the action group says that the rule only triggers if the string does NOT match the expression supplied). This test is not as clear-cut as the tests in section 1, but any properly-configured mailer should be generating compliant commands, so you're only going to be turning away non-compliant clients. In more than a year of using this rule, I'm not aware of a single legitimate false positive it has generated.

 

Will Mercury accept locally send by Pegasus this way? I am sending that way in our setup and all machines announce themself with their local name (i.e PC1 etc.). So the transaction with EHLO looks like:

Connection from 10.0.0.173
EHLO PC1
AUTH CRAM-MD5
etc. 

 

Should I exclude these first with something like:

 H, "PC1", X, "sending machine known"

or will Mercury accept them because they are local anyway? Maybe another syntax to exclude them?

 

TIA,

Joop.

 

Curt Cuppels:

I am having difficulty with the logic of item #3:

3: Set up a honeypot address then blacklist any attempt to send mail to it.

Why do I want to set up a honeypot just to reject the mail at the transaction level? To me it seems that I'm just inviting more needless hits on my mail server. Am I not better off not even having this type of honeypot?

If you have no e-mail addresses that are currently receiving spam, then I agree that adding a honeypot probably isn't a great idea... But not that many people who run a mail server are in this position. Most people have to expose their e-mail addresses in some way that will allow them to be harvested, which means that there's always going to be zombies hitting your server fairly regularly (in my case, I'm getting anywhere from 4,000 - 6,000 connections an hour from zombies) . It's in this type of situation where adding a honeypot really starts showing its worth.

In the typical modern case, the odds are that you're going to get mail from a zombie at some point anyway, whether or not you have a honeypot active: using a honeypot allows you to detect it as such and turn it away altogether without receiving so much as a line of data. Remember that the whole point of the honeypot address is that you're blacklisting any machine that tries to offer it to you in a RCPT, so for 30 minutes after that point, no connection from that machine will even be accepted, let alone processed. If you didn't blacklist when you saw the honeypot come in, then there would be much less point to the exercise.

The other thing is that any time you see the honeypot address, you know the entire mail transaction is bogus. This can be important, because depending on how the spammer's address harvester works, he may have a collection of valid addresses at your site that would not otherwise generate any errors: because of the way mail tends to get grouped, the honeypot address will end up being submitted along with the valid addresses, thereby allowing you to catch and discard unwanted messages that would otherwise either get delivered or require anti-spam processing further down the chain.

This thread wasn't intended to be prescriptive - I was just passing on some ideas that have worked well for me, and honeypots are definitely in that category. Your mileage may definitely vary depending on your circumstances, though.

Cheers!

-- David --

J-Fix:

Will Mercury accept locally send by Pegasus this way? I am sending that way in our setup and all machines announce themself with their local name (i.e PC1 etc.). So the transaction with EHLO looks like:

Connection from 10.0.0.173
EHLO PC1
AUTH CRAM-MD5
etc. 

 

Should I exclude these first with something like:

 H, "PC1", X, "sending machine known"

or will Mercury accept them because they are local anyway? Maybe another syntax to exclude them?

Under "SMTP Server" -> "Connection Control" add an IP range for your local net, edit the properties, and check the box "Connections from this range are exempt from transaction filtering"

I also have rules in my transaction filter to allow specific external machines with dynamic ip's

# Exempt the following HELO's
H, "[heHE][heHE]LO laptop01??*", X, "Laptop"

And since David is in this thread I have a question that has been bugging me for a while

Why does the "Mail From" filter ignore the "exempt" checkbox?

I would like to use this to prevent spammers using my valid users as From: addresses when connecting to my server.

I could have a trans filter rule blacklisting anyone using a Mail From: *@mydomain and then use exceptions for local net and roaming users.

Thanks David. You are right. Mileage may vary. I don't even get 4000-6000 emails a week, much less an hour. :-) I do see your points concerning the honeypot in those situations.  Hopefully I never have to deal with that but the day will possibly come.

I've been using items 1 & 2 effectively here for quite some time and can vouch for them.

Thanks for the clarification. Not having the experience with zombie attacks had me blind to what you were trying to convey.

 

Curt Cuppels
 

dilberts_left_nut:

And since David is in this thread I have a question that has been bugging me for a while

Why does the "Mail From" filter ignore the "exempt" checkbox?

Transaction filtering isn't a single process - it's actually five separate processes controlled by a single configuration file. Using the "X" command for any single phase of that process only exempts the remainder of that phase from processing: so, if you have an X (exempt) action as a HELO rule, it will terminate all remaining HELO processing, but it won't have any effect on the other phases, such as MAIL, RCPT, DATA.

This has been addressed in Mercury/32 v4.5 - I've added a new "S" (for "sanctify") action that terminates all further transaction processing wherever it is encountered... So, if you have a HELO rule that triggers with the new 'S' action, the remaining transaction filtering phases won't even be applied. The 'X' action is still there, and still behaves as it currently does.

Hope this clarifies things for you.

Cheers!

-- David --

PS: It just occurred to me that you might be talking about something different - that the access control lists don't apply to the MAIL command; this has also been fixed in v4.5.

J-Fix:

Will Mercury accept locally send by Pegasus this way? I am sending that way in our setup and all machines announce themself with their local name (i.e PC1 etc.). So the transaction with EHLO looks like:

Connection from 10.0.0.173
EHLO PC1
AUTH CRAM-MD5
etc. 

If you're using WinPMail v4.41, then this shouldn't happen any more - it should now always be using FQDNs in the HELO command. Could you check this out and get back to me?

Cheers!

-- David --

David Harris:

PS: It just occurred to me that you might be talking about something different - that the access control lists don't apply to the MAIL command; this has also been fixed in v4.5.

That's what I was after .... and......."Excellent"
 

David Harris:

J-Fix:

Will Mercury accept locally send by Pegasus this way? I am sending that way in our setup and all machines announce themself with their local name (i.e PC1 etc.). So the transaction with EHLO looks like:

Connection from 10.0.0.173
EHLO PC1
AUTH CRAM-MD5
etc. 

If you're using WinPMail v4.41, then this shouldn't happen any more - it should now always be using FQDNs in the HELO command. Could you check this out and get back to me?

Cheers!

-- David --

 

You're right David. Your answer made me wonder so I checked and created a new user and then it identifies as local IP number in the EHLO greeting and I think I know why. In the past years I changed some flags in PMAIL.INI files manually (got tips on the BAMA list) and maybe that these are the reason on that machines to identify that way?

 

I don't remember exactly what flags (maybe "Transport control word"?) I changed (long time ago) but I rember that it had to do with letting Mercury create a message ID header using the domainname instead of the id_number@local as Pegasus created it back then (looks like it uses the domainname now). Could these changes to the flag numbers be the reason for using the local machinename instead of the assigned local IP?

 

TIA,

Joop.

 

dilberts_left_nut:

Under "SMTP Server" -> "Connection Control" add an IP range for your local net, edit the properties, and check the box "Connections from this range are exempt from transaction filtering"

 

 

Duh.. thanks for the heads up, I completely forgot about that checkmark :-)

 

Thanks,

Joop. 

 

Richard:

Is it possible to see the list of short-term blacklisted sites?

Can offending sites be permenantly blacklisted?

If you check the MercuryS log files, you can see entires where an IP address has been added tot he short term blacklist. For example, I have a DNSBL check titled "Spamhaus Zen - Reject". When a transaction arrives that returns a positive on the Zen list, the following entry is made in the MercuryS log file:

E  20070529  004133  4640efe8  Host (IP.Address) blocked by Spamhaus Zen - Reject - dropped and blocked.

So I can scan the MercuryS log file for "dropped and blocked". If the host later tries to connect again, you will see entries like this:

E  20070529  004610  0   Connection from (IP.Address) refused because of short-term restriction.

If you want to permanently blacklist the address, then you can add their IP address to the Connection Control list.

You can also check your daily Mercury/32 statistics report to see how many are currently blocked, and the maximum achieved.

  Short-term blacklist - current               5 (3)
  Short-term blacklist - peak                  28 (28)
 

Item 3 "honeypot" addy is an absolute in fighting spam.

With every website I setup, I always include a "Gotcha@domainname.com" addy hidden within the page. This is achieved by using same text colour as background. Over a 6 month period I have acquired and Blacklisted some 16,000 IP addys who have sent univited email to the Gotcha account. It does put a smile on the diaL:-)  

---

Boris