Pegasus Mail & Mercury

David Berlind just released a video and article that is worth watching and reading

Although he claims the "big four" can fix the problem, I think it's more than just the big four - it's ISPs and others like Mercury Transport as well. If everybody continues to refuse to adopt new anti-spam standards, the problem will continue. IMHO, the reason for failure of anti-spam solutions is not so much technological, but adoption - spam cannot be stopped if nobody adopts new anti-spam technologies. This is especially true since many of these technologies work best when nearly everybody is using them - and work poorly if most people are not using them. I feel that cooperation and adoption of new technologies are the biggest hurdles in the war against spam.

David Berlind is getting sick of statistical methods to "solve" the spam problem - and quite frankly, so am I. I'd like to see something stronger than statistical methods being used. I'd like to see new technologies implemented to actually fix the problem. I'd like to see Mercury Transport with stronger, less statistical methods in the future.

AFAIK, here are the anti-spam "solutions" that come with Mercury:

Rules: While these are nice, and should be kept, they have to be constantly updated as spammers switch tactics, and will never work perfectly.

SpamHalter: Statistical solution. Can, and often does, classify emails incorrectly. 99% won't cut it. 99% means that if you're getting hundreds or more emails, you are bound to get incorrectly classified emails. Yes, some people deal with hundreds - often thousands - of emails.

GreyWall: Very effective - today. But it depends entirely on the behavior of spammers (counting on them to not resend emails that are temporarily rejected)  - if Greylisting becomes common, they'd just change their behavior.

DNSBLs: Better than rules, but still have to be maintained and updated, and will never be perfect.

I think we can do better. We do have technologies that can provide better authentication. It's being used on https servers. You know - signed certificates? AKA electronic signatures? You know, the cryptographic kind? The kind that can be used to create a web of trust?

Cryptography and some sort of implementation of the "web of trust" idea is, IMHO, the only truly effective way to combat spam. DomainKeys and other cryptographic methods are, IMHO, headed in the right direction.

There is absolutely no necessity to get complicated. The more complicated the solution, the more difficult it is to implement. And few of the "anti-spam technologies" tackle the problem where it occurs, which is actually where it must be tackled. Once it gets past the SMTP layer, i.e. at the mail clients, all the "anti-spam technologies" are just dealing with the symptoms, not the problem.

What has to be understood is that spam can be classified with 100% certainty as "that which is sent over a bot net." Thus, the onus lies with ISPs and SMTP servers alone:

* a machine without a static IP has no business running a smart mailer. Not only can direct access to anything but the ISPs own SMTP host be blocked, but dynamic IPs can be made machine-identifiable with reverse lookup. Humans can identify a dynamic IP from the host name; for machines to able to do that is only difficult because there are no standard naming conventions. For example, that all dyn addresses contain ".dyn." or "-dyn-" in their hostnames or that these hostnames be >4th level addresses (this is already true for 80% of all ISPs) or that one segment in the hostname be all numeric (+ hyphens).

*  If mail purportedly coming from a @myisp.net or @mywebmail.com service has not sent from that ISP's/webmail service's own hosts, it is 100% spam. For example: if mail from purportedly @aol addresses were checked in a smart way (which is to check if the sending smtp is one of the AOL SMTP hosts), all that supposedly "spam from AOL" would vanish. The same applies to any other ISP and webmail service. Such "SMTP client must match sender field" whitelisting is a simple common sense rule that takes no effort, no database maintainence, and is already implemented in most anti-spam heuristics.

* mail from host X that goes to a honeypot account (which includes all accounts that *were* - a long time ago - once real, but have since been closed) automatically blacklists that host X.

These are not things that can be accomplished with mail-client filter rules because they all occur at the SMTP/envelope level and are stateful activities that apply over a longer period of time. These are also not things that spammers can work around. To "work around" them would require them to cease using botnets.

The "big four" can neither "fix" the problem, nor are they responsible for it (well, hotmail and AOL once were, but that is history). David Berlind has no clue how spammers work, and many of the webmail services, including hotmail/gmail/yahoomail, are - if anything - leading lights in the way they deal with spam. And his fourth of the big four is AOL, which is an ISP, and I know of no ISP that fights spammers as AOL does. I dislike AOL and Microsoft for their business practices, but - now that they are awake - their anti-spam efforts are faultless.

 

To put it in perspective...

Hotmail (285 million email accounts, of which 130,000 honeypots) receives 4 billion pieces of mail every single day. Of that only 600 million pass the rudimentary SMTP-level filters. The other 85% is - *at the envelope level* - illegitimate for one reason or another.

To not give the spammers any hints of why their mail is being rejected, most of the rejected 85% is silently "accepted" but not actually processed beyond the recv() cycle.

Of the 600 million that lands in mailboxes, appropriately flagged using statistical methods; about 210 million are flagged > 60% likelyhood of spam and another 270 million flagged as being > 30% likelihood of spam. Of altogether 600 million messages, 130 million are deleted within 20 seconds of opening.

Related reading:

http://community.pmail.com/forums/thread/4603.aspx ("AOL is problematic")

http://community.pmail.com/forums/thread/4134.aspx ("554 Invalid HELO format")

"There is absolutely no necessity to get complicated."

Yeah - except the "simple" solutions aren't working.

 "And few of the 'anti-spam technologies' tackle the problem where it occurs, which is actually where it must be tackled."

The problem is that email is designed to blindly accept emails from new people, including those who have falsified their identities. Everything else is a symptom.

"Once it gets past the SMTP layer, i.e. at the mail clients, all the 'anti-spam technologies' are just dealing with the symptoms, not the problem."

The problem is that we don't have any way of verifying who people are (authentication), and we don't have any way of telling if they are trustworthy (trust). No amount of tweaking at the SMTP level is going to fix the root cause of the problem - which is ultimately whether or not you trust the person on the other end to send you email.

"What has to be understood is that spam can be classified with 100% certainty as 'that which is sent over a bot net.'"

Bot nets are a means to an end. If you take out the current means, they will develop a new means. You are merely dealing with the symptoms, not the problem itself. Ultimately, the problem is one of trust, and therefore must be solved at the trust level. This means that users must be authenticated and that a system of trust must be established.

People have been tweaking SMTP left and right, trying to get rid of spammers - but to what avail? We got rid of open relays, and they moved to bot nets. If we get rid of bot nets, they will move to something else. We have to solve the real problem - and the real problem is how email fundamentally works. Playing around with SMTP and IP addresses won't solve that.

"Thus, the onus lies with ISPs and SMTP servers alone"

Frankly, that's a big, huge, problem. Lots of ISPs are not fixing the problem, and many of them are even friendly to spammers. The problem must be solved in such a way that the user has more control. ISPs cannot be counted on to implement a new solution. 

>> The problem is that we don't have any way of verifying who people are (authentication), and we don't have any way of telling if they are trustworthy (trust)

IMO, thats tackling it from the wrong end. :) Instead of positive identification, use negative identification. In other words, identify spam, don't waste time trying to identify legitimate senders.

If I (SMTP server) get a connection from someone purporting to be foo@company.com and the IP address of the SMTP peer does not reverse-resolve to xyz.company.com, then the sender is a fake. I don't have to establish a "trust" relationship with the sender. I only have to identify if the sender is *unworthy* of my trust.

>> ISPs cannot be counted on to implement a new solution. 

Unfortunately true. But this is not because they intend to be friendly to spammers (the vast majority are actually very hostile to it), but because *it is thought* that implementing any solution (say blocking port 25 access to any host other than the ISP's own) would result in nightmare for the support center. Few have actually dared to try it (AOL being a notable exception).

>> If we get rid of bot nets, they will move to something else.

Perhaps so. But that doesn't mean we should give up. At present most SMTP servers are - by design - dumber than a box of rocks (that includes Mercury) and delegate spam detection to a later stage. They need to get smart. Fast. If I knew how to deal with the "does recipient exist" stuff that mercury does, I'd have written a new MercuryS ages ago.

 

> >> The problem is that we don't have any way of verifying who people
> >> are (authentication), and we don't have any way of telling if they
> >> are trustworthy (trust)
>
> IMO, thats tackling it from the wrong end. :) Instead of positive
> identification, use negative identification. In other words, identify
> spam, don't waste time trying to identify legitimate senders.

Helpful if you can do it.

>
> If I (SMTP server) get a connection from someone purporting to be
> foo@company.com and the IP address of the SMTP peer does not
> reverse-resolve to xyz.company.com, then the sender is a fake. I don't
> have to establish a "trust" relationship with the sender. I only have
> to identify if the sender is *unworthy* of my trust.

The rDNS matching will block a lot of good mail.  If you wish to do this that's up to you but I'm always surprised when companies do this and then wonder why they do not get as many RFP as they did in the past.  The rDNS is not required to match the server name, there are hundreds of servers with a fixed IP address that is registered to the host name of the connecting system where the rDNS comes back with something else.

>
> >> ISPs cannot be counted on to implement a new solution.
>
> Unfortunately true. But this is not because they intend to be friendly
> to spammers (the vast majority are actually very hostile to it), but
> because *it is thought* that implementing any solution (say blocking
> port 25 access to any host other than the ISP's own) would result in
> nightmare for the support center. Few have actually dared to try it
> (AOL being a notable exception).

Actually more and more ISPs are doing this sort of blocking based on what I see in the support of both Mercury and Pegasus Mail.  Now of course more and more spammers are using the ISPs servers to send their spam via their botnets.

>
> >> If we get rid of bot nets, they will move to something else.
>
> Perhaps so. But that doesn't mean we should give up. At present most
> SMTP servers are - by design - dumber than a box of rocks (that
> includes Mercury) and delegate spam detection to a later stage. They
> need to get smart. Fast. If I knew how to deal with the "does
> recipient exist" stuff that mercury does, I'd have written a new
> MercuryS ages ago.

Getting smart though means the SMTP host MUST be receiving all good mail before blocking the bad stuff.  If you block just 1% of the good mail on Na busy system you are going to be blocking a lot of good mail.

---

Thomas R. Stephenson
San Jose, California
Member of Pegasus Mail Support Team

"If I (SMTP server) get a connection from someone purporting to be foo@company.com and the IP address of the SMTP peer does not reverse-resolve to xyz.company.com, then the sender is a fake. I don't have to establish a "trust" relationship with the sender."

What you are describing is a style of authentication, albeit one that is flawed.. If the sender's identity fails authentication, then yes they can be immediately rejected and the trust step can be skipped.

"Perhaps so. But that doesn't mean we should give up."

I'm not saying we give up. I'm saying that we should consider solutions that are better suited to the problem. Quite frankly, a single "complicated" solution that works is far better (and simpler) than a million "simple" solutions that do not work.

If we are really serious about stopping spam, we should implement a solution that is difficult to find a workaround for - even if that means some temporary inconveniences for the developers.
 

"IMO, thats tackling it from the wrong end."

Quite frankly, both ends have to be tackled for a truly effective system. The less time the user has to look in the spam folder for legit emails, the better. This is especially useful in environments where an email may be important or even critical, such as when the boss is sending an employee information about a new meeting. Do you really want to take the chance that such an email will end up in a spam folder? Identifying legitimate senders is never a waste of time. If a legitimate sender ends up in a spam folder, your anti-spam solution just failed, because now the user has to start wading through millions of spams looking for good mail.

"I don't have to establish a "trust" relationship with the sender."

Yes, you do. The bad guys are hiding their identities, so only identifying untrustworthy senders is ineffective. Existing "anti-spam" solutions have proven that beyond a shadow of a reasonable doubt.

"that implementing any solution (say blocking port 25 access to any host other than the ISP's own)"

Bypassing a port block is Networking 101 (just use a different port). Ineffective.

I'm getting sick of these non-solutions. They've all been tried, and they've all failed. How long will it take before we learn that we can't solve the problem by ignoring the trust and authentication issues?

Trying the same thing over and over and over again hoping that it will be effective someday is not the way to proceed. If it didn't work the first dozen times, it''s no more likely to work the next dozen times. It's time we started  looking for something new that really works, rather than continuing to try old solutions that don't work.

"At present most SMTP servers are - by design - dumber than a box of rocks (that includes Mercury)"

And it looks like that's not going to change. Changes to Mercury are slow, and David seems to prefer to wait for widespread adoption before implementing a new solution. IMHO implementation has to come first, and if the idea works adoption will follow.

I understand, though, that resources are limited and that there's not much that can be done about it.

Therefore, I'm looking into implementing something myself. Maybe a daemon for Mercury (one of the reasons I set up a Mercury server on my system), and possibly a stand alone application. Probably both. What I do know is that it's going to be something that will at the very least use white lists and some form of authentication, and most likely a few other ideas.
 

Take a look at Karmasphere http://www.karmasphere.com/

Even as a DNS blocklist and whitelist it can be used with Mercury as is and appears to be effective.

A list like that helps the trust issue - but doesn't help the authentication issue. You'd check a list like this after you've checked the digital signature to ensure the identity of the sender, and after you've checked your own personal white/black lists.

Lists tend to be slow at updating, which may reduce the effectiveness of relying on such a list.

Once authentication is in place, the biggest issue is going to be how to handle new senders - that is, how handle mail from people that haven't been seen before. If implementations can be agreed upon and deployed, and if people are willing to accept it, the eventual goal of all of this is to switch from the current system of accepting all new senders by default to a system where new senders have to obtain permission of the receivers before sending messages to them. Trust networks and white list servers can help identify new senders.

CobraA1:

Changes to Mercury are slow, and David seems to prefer to wait for widespread adoption before implementing a new solution.

I want to be fairly sure that the solution will actually achieve something before I invest considerable effort into it.

Personally, I do not in any way share your optimism that PK-based solutions like DomainKeys or trust-based initiatives like eTrust will actually have any impact on spam: all they will do is make Internet access expensive for the average user (PK certs are *never* going to be cheap) without actually placing any particular restrictions on the spammers; indeed, with most spammers having control of large botnets (and hence, the PKC keys that reside on all those machines), there's an easily-imaginable scenario that PKC-based initiatives might actually make the situation *worse*. It also requires no great leap of intuition to see mail becoming vastly less useful in this brave new world, because the only reliable way you will have of contacting someone is effectively by holding a recommendation from someone they trust: if you don't happen to share a common point of reference, that's a problem.

All of this is without even beginning to consider the cost, difficulty, political wrangling and growing pains of setting up what would be the hugest single piece of trans-national technological infrastructure in history, nor the difficulty (one might even say impossibility) of keeping it secure.

I admire your enthusiasm and share your frustration, but simply maintaining that your idea is right and everyone else who doesn't agree with it is wrong isn't going to achieve much. There's much more than just idealism in play here - there is cost, complacency, ignorance, and above all, the truly vast scope of the problem. No one solution can work - indeed, I'm almost at the point of thinking that *no* solution, however constructed, really has much chance in the short- to medium-term.

CobraA1:

Bypassing a port block is Networking 101 (just use a different port). Ineffective.

A comment that suggests you haven't quite thought the matter through. SMTP listens on a well-defined port - port 25. If ISPs prevent outbound attempts to connect to port 25 from passing through their routers, SMTP becomes impossible unless you have access to specialized port translation proxies, which can be easily blocked themselves. In my 20-odd years of practical networking and mail development, I've never seen a way of bypassing a port block that I would call "networking 101": sure, you can use a different port number and the ISP might let the connection through, but the SMTP server is still (and *only*) listening on port 25, so you're wasting your time.

It's an interesting discussion, I guess - the kind I might like to have with Peter Gutmann over a chinese dinner sometime... But as a serious pragmatic blueprint for changing the world, I think it has a long way to go.

Cheers!

-- David --

"(PK certs are *never* going to be cheap)"

Who said anything about getting a certificate from a certification authority? Encryption and digital signatures are possible without a central certification authority. In fact, I'd much prefer a decentralized model over a centralized model.

"indeed, with most spammers having control of large botnets (and hence, the PKC keys that reside on all those machines), there's an easily-imaginable scenario that PKC-based initiatives might actually make the situation *worse*."

I don't see how. Once a compromised computer starts sending spam, the computer can be easily blacklisted and blocked, and the key revoked.

"It also requires no great leap of intuition to see mail becoming vastly less useful in this brave new world, because the only reliable way you will have of contacting someone is effectively by holding a recommendation from someone they trust: if you don't happen to share a common point of reference, that's a problem."

The first time you contact somebody, yeah, you'll have to get their permission to send the person email. After the first few manual contacts, however,. the web of trust could allow future contacts to be more automatic.

Yes, people may have to get used to the idea of giving permission for a new user to contact them.

This is not, however, a new concept:

-Instant messaging already does this. I have to give permission for new users to contact me, and I have to wait for permission of new users to start sending them IMs. How much has this affected the usefulness of IM?

-Many people I know have already taken the white list approach and I have to ask them to add me to their white list before contacting them for the first time. With or without authentication, white listing is already starting to happen, and people with more aggressive spam filters already are having to use white lists to tell the email program not to dump known users into a spam folder. This seems to be something most people are willing to accept.

Also, there's nothing saying that businesses and individuals can't take different approaches: An individual can create his/her own white lists, and get white lists from trusted friends & family. A business can use a single server as a centralized white list for email between employees. Email addresses intended for contacting a business can use something like Karmasphere and have more relaxed rules for accepting new emails. The great thing is that this approach can be made flexible so that it can be tailored to fit the needs of whoever is using it.

 "All of this is without even beginning to consider the cost, difficulty, political wrangling and growing pains of setting up what would be the hugest single piece of trans-national technological infrastructure in history, nor the difficulty (one might even say impossibility) of keeping it secure."

Use a decentralized P2P model, so that the infrastructure is largely self-building. I don't see why we need to stick to a server-client model.

From what I can tell, the model I'm suggesting is inheritly more secure than the existing model. Our current system has no security at all!

 "I admire your enthusiasm and share your frustration, but simply maintaining that your idea is right and everyone else who doesn't agree with it is wrong isn't going to achieve much."

I'm not trying to say everybody else is wrong - but when I look around, I see the same problem crop up again and again: It's too easy for the bad guys to hide their identities and fool the system.

The best anti-spam systems can get in the high 90%s, even 99% sometimes, but with users getting thousands of spams we need 99.99% or higher. Falsified identities are IMHO preventing us from creating a much stronger system.

"There's much more than just idealism in play here - there is cost, complacency, ignorance, and above all, the truly vast scope of the problem. No one solution can work - indeed, I'm almost at the point of thinking that *no* solution, however constructed, really has much chance in the short- to medium-term."

IMHO creating a new system is a very appealing option, and yes we may have to live with stuff breaking in the short term if we wish to have long term benefits.

 "SMTP becomes impossible unless you have access to specialized port translation proxies, which can be easily blocked themselves. In my 20-odd years of practical networking and mail development, I've never seen a way of bypassing a port block that I would call 'networking 101'"

Obtaining a proxy is easy, as far as I can tell. What do you think those zombie machines are? Fancy proxies. In addition, spammers may simply bypass getting an ISP and find a way to connect directly to the Internet - and as I've said, not all ISPs are unfriendly to spammers. The last thing we can depend on is trusting ISPs to bock their spam. I'd like to see a solution that does not depend on the cooperation of ISPs, because getting ISPs to cooperate simply does not work.
 We need inbound blocking of spam. We can never guarantee the Internet itself is going to be clean.

On 5 Nov 2007 Pegasus Mail & Mercury - Automated Email <NoReply@praktit.se> wrote:

> "(PK certs are *never* going to be cheap)"
>
> Who said anything about getting a certificate from a certification
> authority? Encryption and digital signatures are possible without a
> central certification authority. In fact, I'd much prefer a
> decentralized model over a centralized model.

And if there is no centralized authority then what says that these data are in fact valid?

>
> "indeed, with most spammers having control of large botnets (and
> hence, the PKC keys that reside on all those machines), there's an
> easily-imaginable scenario that PKC-based initiatives might actually
> make the situation *worse*."
>
> I don't see how. Once a compromised computer starts sending spam,
> the computer can be easily blacklisted and blocked, and the key
> revoked.

If this were so simply then the botnets would not be getting their spam out.  This blocking depends on the people controlling top level to block them and that's simply not happening.

>
> "It also requires no great leap of intuition to see mail becoming
> vastly less useful in this brave new world, because the only
> reliable way you will have of contacting someone is effectively by
> holding a recommendation from someone they trust: if you don't
> happen to share a common point of reference, that's a problem."
>
> The first time you contact somebody, yeah, you'll have to get their
> permission to send the person email. After the first few manual
> contacts, however,. the web of trust could allow future contacts to
> be more automatic.

Sorry about that, that simply equates to a challenge and response system and that is a non-starter.  I get mail from these garbage system all the time asking for help and then bounce my response.  I will not do their spam fighting for them.

>
> Yes, people may have to get used to the idea of giving permission
> for a new user to contact them.
>
> This is not, however, a new concept:
>
> -Instant messaging already does this. I have to give permission for
> new users to contact me, and I have to wait for permission of new
> users to start sending them IMs. How much has this affected the
> usefulness of IM?

Never use it and have never given permission for anyone to contact me via Skype either.  IM and SMTP mail are simply apples and oranges in any case.

>
> -Many people I know have already taken the white list approach and I
> have to ask them to add me to their white list before contacting
> them for the first time. With or without authentication, white
> listing is already starting to happen, and people with more
> aggressive spam filters already are having to use white lists to
> tell the email program not to dump known users into a spam folder.
> This seems to be something most people are willing to accept.

Do your thing, I've no problem with that.  If I get bounced by a anti-spam system that requires me to beg permission to send mail I simply do not send them mail.  I was flabbergasted the first time I ran into these systems when I was putting out requests for proposals for big bucks and encountered these.  The sales people often came asking why they were not getting any new business from us.  They were also surprised when I told them they were bouncing the mail.  
>
> Also, there's nothing saying that businesses and individuals can't
> take different approaches: An individual can create his/her own
> white lists, and get white lists from trusted friends & family. A
> business can use a single server as a centralized white list for
> email between employees. Email addresses intended for contacting a
> business can use something like Karmasphere and have more relaxed
> rules for accepting new emails. The great thing is that this
> approach can be made flexible so that it can be tailored to fit the
> needs of whoever is using it.

As long as you do not mind losing a significant percentage of your good mail then these system certainly will reduce the amount of spam you receive.  Personally I feel that 5%-10% false alarm rate is way too high.


>
>  "All of this is without even beginning to consider the cost,
> difficulty, political wrangling and growing pains of setting up what
> would be the hugest single piece of trans-national technological
> infrastructure in history, nor the difficulty (one might even say
> impossibility) of keeping it secure."
>
> Use a decentralized P2P model, so that the infrastructure is largely
> self-building. I don't see why we need to stick to a server-client
> model.
>
> From what I can tell, the model I'm suggesting is inheritly more
> secure than the existing model. Our current system has no security
> at all!

Sure it does.  The system is so secure that spammers are using it to conduct their business.  You simply need to be a little creative is controlling the spam so it gets to a manageable level.  I keep the spam out of the users mailboxes pretty well.  My system is currently 99.82% effective at detecting and blocking spam and my false alarm rate is zero since I go through the spam looking for the 0.035% of the mail that is falsely classified as spam.

>
>  "I admire your enthusiasm and share your frustration, but simply
> maintaining that your idea is right and everyone else who doesn't
> agree with it is wrong isn't going to achieve much."
>
> I'm not trying to say everybody else is wrong - but when I look
> around, I see the same problem crop up again and again: It's too
> easy for the bad guys to hide their identities and fool the
> system.

Not true, they are very easily spotted by any human and a Bayesian classifier.  

>
> The best anti-spam systems can get in the high 90%s, even 99%
> sometimes, but with users getting thousands of spams we need 99.99%
> or higher. Falsified identities are IMHO preventing us from creating
> a much stronger system.

Not a prayer.  If the SMTP mail system requires accurate identification of the sending system to deliver the mail then the spammers will be the first ones on board.  They will be identified properly and meet all the requirements much faster than those that cannot afford to get rid of their current systems.

>
> "There's much more than just idealism in play here - there is cost,
> complacency, ignorance, and above all, the truly vast scope of the
> problem. No one solution can work - indeed, I'm almost at the point
> of thinking that *no* solution, however constructed, really has much
> chance in the short- to medium-term."

What cost are you talking about?  We have more bandwidth for browsing in internet and playing videos that ever is used by the spammers sending mail.  Our systems are pretty much sized based on other factors.  I do not increase the size of my SMTP host because there are more spammers out there.  Most of the people running Mercury/32 (or any other SMTP host) generally use blacklists to block the spammers at the front door.  There is also other techniques like Greywall to keep the botnets at bay.  I spend little or nothing extra keeping the spam to a manageable level.  

>
> IMHO creating a new system is a very appealing option, and yes we
> may have to live with stuff breaking in the short term if we wish to
> have long term benefits.
>
>  "SMTP becomes impossible unless you have access to specialized port
> translation proxies, which can be easily blocked themselves. In my
> 20-odd years of practical networking and mail development, I've
> never seen a way of bypassing a port block that I would call
> 'networking 101'"
>
> Obtaining a proxy is easy, as far as I can tell. What do you think
> those zombie machines are? Fancy proxies. In addition, spammers may
> simply bypass getting an ISP and find a way to connect directly to
> the Internet - and as I've said, not all ISPs are unfriendly to
> spammers. The last thing we can depend on is trusting ISPs to bock
> their spam. I'd like to see a solution that does not depend on the
> cooperation of ISPs, because getting ISPs to cooperate simply does
> not work.
>  We need inbound blocking of spam. We can never guarantee the
> Internet itself is going to be clean.

Of course not and that's the whole point.  Any system that you come up with is only going to make is more difficult for the average user/system and have little or no affect on the spam.  The only thing that will stop spam is to make it unprofitable.  Since none of us want a pay per e-mail type system then the only solution is to shut it down at the source.

>


However, that said, I am never going to convince you that that the "whole system must be changed" is not a solution.  You believe that there is a way to stop spam will little or no impact on the email system.  I do not believe this; most people do not believe this; many are still coming up with "solutions" though.

 

---

Thomas R. Stephenson
San Jose, California
Member of Pegasus Mail Support Team

"And if there is no centralized authority then what says that these data are in fact valid?"

Why do we need a central authority to determine validity in the first place?

Here's how it works: 

-A shared secret is created and exchanged using either a key exchange method (such as Diffie-Hellman) and/or using asymmetric encryption. This ensures the sender cannot be spoofed in the future.

-A cryptographic hash is created from the message. This ensures the message has not been tampered with.

-The entire thing is encrypted using the shared secret. This ensures nobody except the receiver can read the message, and (assuming the shared secret is a sufficient length cryptographic key) ensures no man in the middle attack is possible.

If all of these steps are followed, we can ensure the sender's identity and the integrity of the message with an astronomical amount of confidence - well above and beyond 99.99%. After the initial exchange of the shared secrets, I can be absolutely confident that future messages from the sender are in fact from that sender and are not spoofed. Of course, that doesn't give us the trust level of the sender, but it does ensure the validity of the data.

 "If this were so simply then the botnets would not be getting their spam out."

Why would a system that is not implemented yet be effective?

"Sorry about that, that simply equates to a challenge and response system and that is a non-starter.  I get mail from these garbage system all the time asking for help and then bounce my response."

I'm not saying that it has to respond like a challenge response system. Another possibility is to simply drop untrusted emails silently, and have the user add people manually to the white list.

 

Another is to accept messages from unknown senders, but filter them using spam filters. Of course that would bring back the spam problem, although it would still be better than the filter by itself. Authentication works well to strengthen existing anti-spam solutions, assuming most of the good senders adopt the method being used to authenticate the emails.

 

Personally, though, I'm not interested in your personal vendetta against challenge/response systems. I'm interested in effectiveness.

"Never use it and have never given permission for anyone to contact me via Skype either.  IM and SMTP mail are simply apples and oranges in any case."

Yeah, but they're both fruit. Just because there are differences doesn't mean that you can't combine the advantages of both and create a new system. In fact, the possibility of a hybrid system intrigues me.

So you have a completely empty Skype list? If not, how did people end up on your list? Why would you not want your email to be the same way?

 "If I get bounced by a anti-spam system that requires me to beg permission to send mail I simply do not send them mail."

I wish all salesmen would be like you. I'd stop getting telemarketing calls and junk mail pretty quickly. Your loss, not mine. I think a lot of people would be open to the idea.

 "As long as you do not mind losing a significant percentage of your good mail then these system certainly will reduce the amount of spam you receive."

That's why I'd maintain a white list of people I know, and why a web of trust is a web - to reduce the number of dropped legitimate emails. If we were to establish a system that identifies good senders with a high degree of accuracy, blocking spam would be simple.

"Personally I feel that 5%-10% false alarm rate is way too high."

How about we work on ways to decrease the false alarm rate to a minimum? 

 "Sure it does.  The system is so secure that spammers are using it to conduct their business."

You have this backwards. The spammers are using it to conduct their business because it's so insecure. They spend a lot of time finding exploits and holes so they can get their spam out. How do you think they get zombie computers? They exploit security flaws.

"My system is currently 99.82% effective at detecting and blocking spam"

I'd like to see four or five nines, not two. I personally get very little spam, but there exists some people who receive so much that even blocking 99% means a lot gets through.

"Not true, they are very easily spotted by any human and a Bayesian classifier."

Which explains why I'm reading in the tech news that spam has doubled. I'm not so sure about "easily." Frankly, the vast majority of Bayesian classifiers are poorly designed. I've seen a lot of classifiers built upon political principles about how they should work rather than doing actual tests to see how well the principles hold up in practice.

"If the SMTP mail system requires accurate identification of the sending system to deliver the mail then the spammers will be the first ones on board."

I certainly hope so! Because as soon as they come on board, they'll be blocked. Black lists will become a lot more effective.

 "They will be identified properly and meet all the requirements"

They will be identified properly, but will not meet the requirement of already being in my white list. You really think I'd add a spammer to my white list?

"Any system that you come up with is only going to make is more difficult for the average user/system and have little or no affect on the spam."

If you can explain to me in detail how a spammer is going to end up on my personal white list or spoof the digital signature of a trusted user, feel free to tell me how this is going to have "little or no affect on the spam."

"The only thing that will stop spam is to make it unprofitable."

If the spam never gets to the receiver, it does not gain a profit.

"Since none of us want a pay per e-mail type system then the only solution is to shut it down at the source."

And David said I was "maintaining that your idea is right and everyone else who doesn't agree with it is wrong isn't going to achieve much."

"You believe that there is a way to stop spam will little or no impact on the email system."

Did I say "with little or no impact on the email system?" I think it'll have a huge impact on the email system.

"However, that said, I am never going to convince you that that the "whole system must be changed" is not a solution."

I prefer a large scale change, yes. However, it may be possible to wrap up new stuff in the old protocol - DomainKeys and similar technologies are exploring these options. Problem is, there's still a lot of email floating around that has no DomainKey or similar technology attached, so the email systems have to decide what to do with email that has no authentication attached. Over time, I hope for more widespread adoption, but it seems to be moving pretty slowly.

Your best argument seems to be that you're going to sulk in a corner by yourself not talking to people if the system is implemented.