Pegasus Mail & Mercury

> "And if there is no centralized authority then what says that these
> data are in fact valid?"
>
> Why do we need a central authority to determine validity in the first
> place?
>
> Here's how it works:

This all presupposes that there is some sort of preexisting situation between  the two systems.  What happens when this is the first time there is a connection between the two and there is no shared secret?  Many of the connections are going to be between two unknown system setup and if this is not allowed then the whole

If there is no shared secret the the encryption system does not work unless there is a trusted third party.

>
> -A shared secret is created and exchanged using either a key exchange
> method (such as Diffie-Hellman) and/or using asymmetric encryption.
> This ensures the sender cannot be spoofed in the future.
>
> -A cryptographic hash is created from the message. This ensures the
> message has not been tampered with.
>
> -The entire thing is encrypted using the shared secret. This ensures
> nobody except the receiver can read the message, and (assuming the
> shared secret is a sufficient length cryptographic key) ensures no man
> in the middle attack is possible.
>
> If all of these steps are followed, we can ensure the sender's
> identity and the integrity of the message with an astronomical amount
> of confidence - well above and beyond 99.99%. After the initial
> exchange of the shared secrets, I can be absolutely confident that
> future messages from the sender are in fact from that sender and are
> not spoofed. Of course, that doesn't give us the trust level of the
> sender, but it does ensure the validity of the data.

Again, what about the first message from the sender?

>
>  "If this were so simply then the botnets would not be getting their
> spam out."
>
> Why would a system that is not implemented yet be effective?
>
> "Sorry about that, that simply equates to a challenge and response
> system and that is a non-starter.  I get mail from these garbage
> system all the time asking for help and then bounce my response."
>
> I'm not saying that it has to respond like a challenge response
> system. Another possibility is to simply drop untrusted emails
> silently, and have the user add people manually to the white list.

Huh?  If it's not a challenge /response system then what kind of permission system will you setup?  Again, will you quietly drop all mail from any unknown sender?
>
>  
>
> Another is to accept messages from unknown senders, but filter them
> using spam filters. Of course that would bring back the spam problem,
> although it would still be better than the filter by itself.
> Authentication works well to strengthen existing anti-spam solutions,
> assuming most of the good senders adopt the method being used to
> authenticate the emails.

How is this any different than the current system?  What changes are to be made to stop the spam?


>
>
>  
>
> "Never use it and have never given permission for anyone to contact me
> via Skype either.  IM and SMTP mail are simply apples and oranges in
> any case."
>
> Yeah, but they're both fruit. Just because there are differences
> doesn't mean that you can't combine the advantages of both and create
> a new system. In fact, the possibility of a hybrid system intrigues
> me.

Yes they are both fruit in that they are both means of communications.  So is a letter and a phone call but they are entirely different in their operation.
>
> So you have a completely empty Skype list? If not, how did people end
> up on your list? Why would you not want your email to be the same way?

I have a list of people I contact via telephone and Skype.  I call them, they don't call me.
>
>  "If I get bounced by a anti-spam system that requires me to beg
> permission to send mail I simply do not send them mail."
>
> I wish all salesmen would be like you. I'd stop getting telemarketing
> calls and junk mail pretty quickly. Your loss, not mine. I think a lot
> of people would be open to the idea.

I guess you would not care if I was requesting a quote from your company for a few thousand dollars and you bounced me.  I'm not selling anything, I'm buying.

>
>  "As long as you do not mind losing a significant percentage of your
> good mail then these system certainly will reduce the amount of spam
> you receive."
>
> That's why I'd maintain a white list of people I know, and why a web
> of trust is a web - to reduce the number of dropped legitimate emails.
> If we were to establish a system that identifies good senders with a
> high degree of accuracy, blocking spam would be simple.

And how do you handle the cold calls where this is the first time anyone contacted you?

>
> "Personally I feel that 5%-10% false alarm rate is way too high."
>
> How about we work on ways to decrease the false alarm rate to a
> minimum?

I've got mine to 0.035% and I do not think that is acceptable so I look at all the mail that is sent to the spam user.

>
>  "Sure it does.  The system is so secure that spammers are using it to
> conduct their business."
>
> You have this backwards. The spammers are using it to conduct their
> business because it's so insecure. They spend a lot of time finding
> exploits and holes so they can get their spam out. How do you think
> they get zombie computers? They exploit security flaws.

What "security" flaws.  This is an open system where two people that have never communicated before can make contact.  The security flaw you are talking about where the people are being taken over by botnets is a operating system failure, not a failure of security in the mail system.


>
> "My system is currently 99.82% effective at detecting and blocking
> spam"
>
> I'd like to see four or five nines, not two. I personally get very
> little spam, but there exists some people who receive so much that
> even blocking 99% means a lot gets through.

You may like to see this level but getting to that level by breaking a systems that is working quite well is not a solution.  If you can come up with a solution that everyone can agree on more power to you.

>
> "Not true, they are very easily spotted by any human and a Bayesian
> classifier."
>
> Which explains why I'm reading in the tech news that spam has doubled.
> I'm not so sure about "easily." Frankly, the vast majority of Bayesian
> classifiers are poorly designed. I've seen a lot of classifiers built
> upon political principles about how they should work rather than doing
> actual tests to see how well the principles hold up in practice.

Spam may have double or tripled but the amount of spam getting to a users mailbox is a different matter.  There are tools currently available that bring spam under control.  
>
> "If the SMTP mail system requires accurate identification of the
> sending system to deliver the mail then the spammers will be the first
> ones on board."
>
> I certainly hope so! Because as soon as they come on board, they'll be
> blocked. Black lists will become a lot more effective.

You again forget that a blacklist requires someone to identify the bad guys before they get on the blacklist.  These spammers are using throwaway domains and IP addresses and by the time they hit the blacklists they have delivered all they want to with that setup.

>
>  "They will be identified properly and meet all the requirements"
>
> They will be identified properly, but will not meet the requirement of
> already being in my white list. You really think I'd add a spammer to
> my white list?

If you are not accepting mail from anyone except those on your whitelist then you are blocking tons of good mail.

>
> "Any system that you come up with is only going to make is more
> difficult for the average user/system and have little or no affect on
> the spam."
>
> If you can explain to me in detail how a spammer is going to end up on
> my personal white list or spoof the digital signature of a trusted
> user, feel free to tell me how this is going to have "little or no
> affect on the spam."

It will have no affect on what you receive but if you only accept mail from those on your whitelist then you are blocking both spammers and good mail in pretty much equal quantities.

>
> "The only thing that will stop spam is to make it unprofitable."
>
> If the spam never gets to the receiver, it does not gain a profit.
>
> "Since none of us want a pay per e-mail type system then the only
> solution is to shut it down at the source."
>
> And David said I was "maintaining that your idea is right and everyone
> else who doesn't agree with it is wrong isn't going to achieve much."
>
> "You believe that there is a way to stop spam will little or no impact
> on the email system."
>
> Did I say "with little or no impact on the email system?" I think
> it'll have a huge impact on the email system.
>
> "However, that said, I am never going to convince you that that the
> "whole system must be changed" is not a solution."
>
> I prefer a large scale change, yes. However, it may be possible to
> wrap up new stuff in the old protocol - DomainKeys and similar
> technologies are exploring these options. Problem is, there's still a
> lot of email floating around that has no DomainKey or similar
> technology attached, so the email systems have to decide what to do
> with email that has no authentication attached. Over time, I hope for
> more widespread adoption, but it seems to be moving pretty slowly.
>
> Your only argument is that you're going to sulk in a corner by
> yourself not talking to people if the system is implemented. Whoop de.
> I couldn't care less.
>

---

Thomas R. Stephenson
San Jose, California
Member of Pegasus Mail Support Team

"What happens when this is the first time there is a connection between the two and there is no shared secret?"

A Diffie-Hellman key exchange, or the connection is established using asymmetric key encryption.

"If there is no shared secret the the encryption system does not work unless there is a trusted third party."

Yes and no. For the most part, a Diffie-Hellman key exchange will work fine. It is, however, potentially open to a man in the middle (MITM) attack. This sounds bad at first, but it's much better than what we have now: Right now, it doesn't matter which computer(s) are compromised, they just need to have a way of sending email. If the only attack available is a MITM attack, then they have to meet a lot more conditions before they can launch a successful attack: They've got to compromise a computer that is between the sender and the receiver, and they have to do it before the sender and receiver perform the key exchange. They can't compromise random computers anymore, and if they're too late in detecting the key exchange they're out of luck. It's much more difficult and it raises the bar a lot higher.

So yes, there is a weakness, but it's a lot stronger than what we're doing right now.

In addition, if you have the opportunity to contact the person using a non-email method before you send the email, you can eliminate the possibility of a MITM entirely by exchanging the shared secret over the non-email method. I'd say the vast majority of people in my address book are people I've met before I emailed them.

And who says the trusted third party has to be Verisign or some expensive SSL key provider? David bought that up, but I'm just not buying it. There's nothing saying that you can't choose somebody else you trust. If you want to trust your uncle Joe and exchange white lists and keys with him, I don't see why we should stop that. You can even trust everybody in your address book, and the software can work out the details of handling a peer to peer network of trust. I don't see why we have to limit ourselves to a single large provider for this. You can if you want, but I don't see why we should be forced to.

 "Huh?  If it's not a challenge /response system then what kind of permission system will you setup?  Again, will you quietly drop all mail from any unknown sender?"

How about this: We let the user decide.

We give the user the option of being open, being on a challenge/response system, or just dropping unknown emails. If the user decides to be open, we perform sanity checks on the email, and hand it to a filter to decide if it's spam. That way even if the user decides to be open there's still some protection.

 "How is this any different than the current system?"

 It's different because the system would know with nearly absolute certainty that an email sent from a repeat sender is the same sender as the one who sent the first email. After somebody sends you an email for the first time, you know it's going to be the same person every time and is not spoofed. Yes, there is the first email problem, and we may have to accept a bit of uncertainty for the first email - which is really no different from our current system - but after that, the problem disappears and you'll never see that problem again from the same person. You'll know for sure that you're talking to the same person every time you receive an email with the same signature. Just the fact that I don't have to head over to my spam folder to find a person I already know is in my opinion a very nice thing to have. I really don't see why we can't have at least a way of knowing our white lists are good. You said it yourself: Even a false alarm rate of 0.035% is unacceptable. What if, even if it's just for people you have contacted before, you could add 76 more zeros before that decimal point?

"What changes are to be made to stop the spam?"

Right now, all of the fields save a couple of "Received:" fields can be spoofed - and even that is problematic because the sender can throw in a few fake fields to confuse the system. Why is this acceptable? We have the technology to say "yes, I've seen this person before, and yes, this email address matches the digital signature on my list, so yes, I know this person is who he or she says he or she is." Is this somehow wrong? Why should we deny the user the ability to do this? Okay, maybe it won't completely stop spam in its tracks - but it definitely provides the user with the ability to know with absolute certainty that nobody can spoof emails from people they've seen before. I think that's a powerful tool, and I see no good reason to deny users the chance to use it.

"I guess you would not care if I was requesting a quote from your company for a few thousand dollars and you bounced me.  I'm not selling anything, I'm buying."

So we set up the email for your business differently than what we'd set up on Grandma's computer. Different people and different businesses have different needs. We allow some flexibility so that people and businesses can choose how to handle the cases where the sender is a new sender.

"What 'security' flaws.  This is an open system where two people that have never communicated before can make contact.  The security flaw you are talking about where the people are being taken over by botnets is a operating system failure, not a failure of security in the mail system."

It doesn't really matter whether SMTP itself is flawed, because email is viewed with the same engines that we view the internet with - and we all know that none of them are perfect and all of them contain bugs than can potentially be exploited. It's ridiculous to look at only one protocol and ignore everything else. Nobody sits at their computer and types SMTP commands manually. Everybody uses some sort of software, and as far as I know all software more complex than a "hello world" program has bugs. I simply do not think that looking at only SMTP by itself is going to be productive.

"You may like to see this level but getting to that level by breaking a systems that is working quite well is not a solution."

Maybe we have to, maybe we don't. You don't seem to  be offering an explanation of why it is not a solution. Okay, maybe my first idea wasn't so hot. But you know what? I think it's good to discuss this stuff, and perhaps we can work out something that will work, and work a lot better than what we have now. So maybe my dreams of a perfect system are going to remain a dream. That's not to say, though, that the existing system can't be improved on both the server and client ends.

"Spam may have double or tripled but the amount of spam getting to a users mailbox is a different matter.  There are tools currently available that bring spam under control. "

Okay, great! Now - why should we stop there? Why not do what we can to make the system stronger? Why not give the user more power over what they want to let in? Just because we have some ways to do something at the server doesn't mean we should ignore the client. After all, it's the users' email and not ours. Ultimately the user is going to have personal tastes and preferences, and if we can allow the user to adjust the client to their preference - I don't see what the problem is.

"if you only accept mail from those on your whitelist then you are blocking both spammers and good mail in pretty much equal quantities."

In some situations, maybe. But I have to disagree that this is the normal case. Most people I know of don't talk to total strangers on a regular basis. They talk to people they already know and have contact with. If you want to talk to strangers on a regular basis, fine: We can provide settings you can adjust to make that possible. I do not think, however, that we should deny the tool to the majority of people just because there are a small number of people that use it differently and wouldn't be able to take advantage of all of the benefits. I think it would be great to make the system flexible enough to fill the needs of different people.

 

CobraA1:

David Berlind just released a video and article that is worth watching and reading

You mean for laughs?

Berlind is an idiot and/or snake-oil salesman (take your pick; the terms are not mutually exclusive), with a long and well-documented PRO-spam history.  If you think that characterization is harsh, just do a Google Groups search on his name in <news.admin.net-abuse.email>.  Here's a particularly telling nugget:

David Berlind was the ZDNet columnist who, at the 2003 MIT Spam Conference, stood up with his bare face hanging out and proposed "shutting down the blacklists" by force.  He did other pretty things, such as exhibiting only one side (guess which one) of a flamewar he had with [IIRC] Ron Guilmette, in order to make DNSBL operators look bad.  It's hard to make trade-rag wankers look any worse than they already do, but he managed. 

CobraA1:

Although he claims the "big four" can fix the problem, I think it's more than just the big four - it's ISPs and others like Mercury Transport as well. If everybody continues to refuse to adopt new anti-spam standards, the problem will continue. IMHO, the reason for failure of anti-spam solutions is not so much technological, but adoption - spam cannot be stopped if nobody adopts new anti-spam technologies. This is especially true since many of these technologies work best when nearly everybody is using them - and work poorly if most people are not using them. I feel that cooperation and adoption of new technologies are the biggest hurdles in the war against spam.

I disagree.  We don't really need much (if anything) in the way of "new technologies".  We only need a widespread resolve that the problem MUST be fixed, and the will to act on that resolve.  Implicit in that are two things:

• A willingness to accept what some folks call "collateral damage", but is really just the predictable and necessary result of insisting that mail senders keep their house(s) in order, if they expect their traffic to be accepted.
• A major amount of education (particularly of the end-users) that if their mail was rejected, it's very probably either their fault or the fault of their provider, and NOT the fault of the target domain/service; and concomitantly, they need to DEMAND that their provider do whatever it takes -- like, for example, really getting rid of their zombie/botnet problems, as opposed to just hiding the symptoms with port 25 redirection or similar -- for their outgoing mail to again be considered "acceptable" by the recipient domains
  

 It all boils down to a fundamental premise that far too few folks seem to take seriously:  Having your e-mail traffic accepted is a privilege, not a right.

Cyrus:

What has to be understood is that spam can be classified with 100% certainty as "that which is sent over a bot net."

Sorry, but no.

The definition of spam is well-established, and VERY simple:  "Unsolicited [Bulk and/or Commercial] Email" (where "Commercial" is interpreted broadly to include any sort of promotional content, including political/religious tracts, etc.).

No other definition can ever be correct or valid.

Hence, while it is true that today, the vast majority of spam is delivered by botnets, that is merely coincidental (and subject to change, without having ANY impact on whether or not a given piece of mail is or is not "spam").

Cyrus:

Thus, the onus lies with ISPs and SMTP servers alone:

Well, mostly -- but not solely (or even primarily) for the reasons you cite.

Cyrus:

* a machine without a static IP has no business running a smart mailer.

I think you mean that they have no business NOT "smarthosting" their outgoing mail traffic through a legitimate server (such as, but not necessarily only, the one(s) operated by their ISP -- remember, there are plenty of legitimate third-party e-mail providers, many of whom do a FAR better job than the typical retail ISP), as opposed to trying to mail direct-->MX from  dynamic IP space.  With that correction/clarification in place, I agree completely.

Cyrus:

Not only can direct access to anything but the ISPs own SMTP host be blocked, but dynamic IPs can be made machine-identifiable with reverse lookup. Humans can identify a dynamic IP from the host name; for machines to able to do that is only difficult because there are no standard naming conventions.

To rely on "naming conventions" is a guarantee of failure.  You will NEVER get all network operators to agree on (much less implement) any given "standard" naming scheme.

Cyrus:

For example, that all dyn addresses contain ".dyn." or "-dyn-" in their hostnames or that these hostnames be >4th level addresses (this is already true for 80% of all ISPs) or that one segment in the hostname be all numeric (+ hyphens).

There's a perfect example of what I'm talking about.  Sure, having ".dyn." or similar in the PTR record is a pretty good clue, if not exactly definitive; but the bit about hyphens is way off-base.  There are LOTS of static addresses out there with PTR records along the lines of "static-111-222-333-444.example.net".  Whether or not these IP ranges are suitable homes for MTAs is another question entirely; but if/when the Spamhaus PBL and/or <dul.dnsbl.sorbs.net> and/or other similarly chartered DNSbls get both widely adopted AND reasonably complete (sadly, both of the cited examples are far from that now), we will have a far more reliable (and efficient) means of determining that.

Cyrus:

*  If mail purportedly coming from a @myisp.net or @mywebmail.com service has not sent from that ISP's/webmail service's own hosts, it is 100% spam.

Maybe.  It depends on (among other things) what you mean by "purportedly coming from" in this context.  The only cross-comparison which is potentially useful at this point in time is the HELO/EHLO vs. the IP address.  And even then, you'd need to be VERY generous in interpreting or acting on the results of such a comparison, since (particularly with larger providers) it is very common for the outgoing mail servers to NOT be the same hosts -- or even on the same network -- as the domain's MX servers.

Cyrus:

For example: if mail from purportedly @aol addresses were checked in a smart way (which is to check if the sending smtp is one of the AOL SMTP hosts),

What makes you think you even know what (all of) the "AOL SMTP hosts" are?  Remember, the MX hosts are NOT the same thing.

This is the sort of thing that SPF was intended to address; but in practice, SPF has enough problems that I doubt it will ever be universally adopted -- and it is certainly not a FUSSP.

Cyrus:

Such "SMTP client must match sender field" whitelisting is a simple common sense rule that takes no effort, no database maintainence, and is already implemented in most anti-spam heuristics.

Given that there is no known (public) database of "legitimate SMTP client IP addresses" (and yes, in this context, the sending hosts are "clients", not "servers"), let alone ALL of them, I don't think it's anywhere near as simple as you claim.  In fact, I cannot fathom a mechanism by which this scheme could ever be implemented without reference to a (presumably, but not necessarily, third-party) DNSbl.

Cyrus:

The "big four" can neither "fix" the problem, nor are they responsible for it (well, hotmail and AOL once were, but that is history).

Your faith in that so-called "big four" is woefully misplaced.  AOL is the one possible exception, inasmuch as they have for a long time done a better job of both curtailing their outgoing spam and keeping a fairly tight reign on other spam-support services than any other similar-size provider.  But Google is at least questionable; and both Hotmail and Yahoo are STILL deliberately spam-supportive -- particularly in terms of willfully providing various spam-support services such as dropboxes and landing pages, and not having an effective "abuse@..." mailbox.  This is NOT new: see <http://www.rfc-ignorant.org/tools/lookup.php?domain=hotmail.com> and <http://www.rfc-ignorant.org/tools/lookup.php?domain=yahoo.com>, just for starters.

Cyrus:

David Berlind has no clue how spammers work,

Well, that much is for sure.

Berlind is an idiot and/or snake-oil salesman (take your pick; the terms are not mutually exclusive), with a long and well-documented PRO-spam history.  If you think that characterization is harsh, just do a Google Groups search on his name in <news.admin.net-abuse.email>.  Here's a particularly telling nugget:

Snake oil salesman or no, he makes a good point: Ham being put in the spam folder is a problem, and there's still a lot of spam ending up in people's inboxes. If you want to go bash his character, go ahead, I don't care - I'm more concerned about how we're going to fix this problem rather than your personal opinion about another man's character

Also, I don't care whether there's some gossip on some list (or lists) I don't use and don't know if I can trust. I just care about what we're going to do about the spam problem.

A willingness to accept what some folks call "collateral damage", but is really just the predictable and necessary result of insisting that mail senders keep their house(s) in order, if they expect their traffic to be accepted.

I fear this is indeed true if we are to truly have a solid solution for this problem. Unfortunately, as you may have seen in this thread so far, people are not willing to accept it.

Even if we don't adopt a strategy that produces a lot of collateral damage, IMHO we should still do some basic authentication with something like DomainKeys or something similar. It can easily serve to strengthen existing anti-spam strategies and make them more effective.

The definition of spam is well-established, and VERY simple:  "Unsolicited [Bulk and/or Commercial] Email" (where "Commercial" is interpreted broadly to include any sort of promotional content, including political/religious tracts, etc.).

I usually drop the "bulk"  and "commercial" part. I don't really care why they sent me the email - if I didn't ask for it, I almost always don't want it.

No other definition can ever be correct or valid.

Why not? I think that discussing definitions is not unreasonable.

Cyrus:

Thus, the onus lies with ISPs and SMTP servers alone:

Well, mostly -- but not solely (or even primarily) for the reasons you cite.

I'll have to continue to disagree. Nobody has given me a solid reason why the individual user should be powerless in the fight against spam. Also, many ISPs have been proven to be spam friendly. So far, I'm seeing nothing but failure if we trust ISPs to clean up their act. We must put more power in the users' hands if we are to be effective against spam.

To rely on "naming conventions" is a guarantee of failure.  You will NEVER get all network operators to agree on (much less implement) any given "standard" naming scheme.

Not to mention that spoofing legitimate names is one of the reasons why spam is so difficult to fight. A naming convension without protection from spoofing would be useless.

The only cross-comparison which is potentially useful at this point in time is the HELO/EHLO vs. the IP address.

That's the reason I'm pushing so hard for some means of authentication. If we can authenticate, we can make a LOT more reasonable comparisons.

 but in practice, SPF has enough problems that I doubt it will ever be universally adopted

The only real problems I'm really aware of are a lack of adoptation and some patent disputes. All other "problems" I know of are poorly reasoned excuses not to adopt it or a lack of understanding of how it works.

Unfortunately, a lack of adoption is a real problem - and is going to be a real problem for ANY solution on the SMTP side of things. We just can't rely on the people who run SMTP servers to adopt a solution. A real solution is going to have to include technologies on the email clients themselves. Otherwise, this problem is never going to be manageable.

Your faith in that so-called "big four" is woefully misplaced. 

This is one area where I'll have to agree with you and disagree with David Berlind. I don't think we can trust ISPs to fix the problem for us. Relying on SMTP to fix the problem and not giving users power over spam is a mistake, and as long as we make that mistake we'll still be talking about this years from now.

CobraA1:

Snake oil salesman or no, he makes a good point: Ham being put in the spam folder is a problem,

But it is a problem born solely out of ill-conceived and poorly implemented "anti-spam" measures -- namely, after-the-fact content-based filtering, which is a fundamentally wrong-headed approach.  The place to reject spam (and viruses, and anything else you do not want for policy reasons) is during the initial SMTP "conversation".  Hence, the solution is not yet another overly complex pie-in-the-sky "new technology"; it's simply adherence to the old adage, "When you find yourself in a hole, the first thing to do is Stop Digging!"

and there's still a lot of spam ending up in people's inboxes.

On this I agree.  But alas, the main reason for that is a widespread unwillingness on the part of major retail ISPs to Do What Is Necessary to police their own networks.

If you want to go bash his character, go ahead, I don't care - I'm more concerned about how we're going to fix this problem rather than your personal opinion about another man's character

It's not about character (tho' that remains a noteworthy side issue); it's about credibility -- and Berlind has none.

Also, I don't care whether there's some gossip on some list (or lists) I don't use and don't know if I can trust. I just care about what we're going to do about the spam problem.

I could be wrong; but it sounds like you're relatively new to these issues.  For starters, NANAE is not a "list"; it is a Usenet newsgroup.  Usenet predates the Internet by a significant margin; and at least most of these issues have been long settled -- after they were discussed into the ground by folks with (in aggregate) FAR more hands-on experience and expertise than you or I could ever hope to achieve.

A willingness to accept what some folks call "collateral damage", but is really just the predictable and necessary result of insisting that mail senders keep their house(s) in order, if they expect their traffic to be accepted.

I fear this is indeed true if we are to truly have a solid solution for this problem. Unfortunately, as you may have seen in this thread so far, people are not willing to accept it.

And until they have (and exhibit) that willingness, the spam problem will NEVER be solved.  Period.

Even if we don't adopt a strategy that produces a lot of collateral damage, IMHO we should still do some basic authentication with something like DomainKeys or something similar. It can easily serve to strengthen existing anti-spam strategies and make them more effective.

Maybe, in theory.  But the problem with this approach is the same as nearly all the other FUSSPs which have been proposed (and shot down) over the years:  They require universal (or close to it) acceptance and implementation in order to be effective (which is obviously near-impossible to achieve); and most operators won't implement until and unless they see effectiveness -- hence, chicken and egg.  And besides, even at best, this still only addresses a symptom of spam (i.e., the currently fashionable delivery mechanism); it does not address spam, per se.

The definition of spam is well-established, and VERY simple:  "Unsolicited [Bulk and/or Commercial] Email" (where "Commercial" is interpreted broadly to include any sort of promotional content, including political/religious tracts, etc.).

I usually drop the "bulk"  and "commercial" part. I don't really care why they sent me the email - if I didn't ask for it, I almost always don't want it.

Perhaps not; but in that case, you're talking about something else besides "spam".  Those qualifiers are a NECESSARY part of the definition.  Just because you don't like or don't want any given piece of mail does NOT make it "spam".

No other definition can ever be correct or valid.

Why not? I think that discussing definitions is not unreasonable.

Because it has already been discussed -- into the ground -- and decided, LONG ago.  The definition of spam is a done deal; so deal with it.  More to the point, the definition I quoted is the only one for which you will ever get any widespread support or agreement.

Cyrus:

Thus, the onus lies with ISPs and SMTP servers alone:

Well, mostly -- but not solely (or even primarily) for the reasons you cite.

I'll have to continue to disagree. Nobody has given me a solid reason why the individual user should be powerless in the fight against spam.

You misunderstand.  I'm not saying that individual users are (or should be) powerless.  Only that the nature of their power lies not in technological band-aids applied after-the-fact, but in simple economics.  They need to demand better performance from their ISPs in terms of both rejecting (not "tagging" or "sorting" or engaging in any other form of pointless monkey-motion) spam, AND keeping their networks clean (of not just willful and knowing spammers, but also of botnet zombies) so that other networks will be willing to accept their outgoing traffic.

Also, many ISPs have been proven to be spam friendly. So far, I'm seeing nothing but failure if we trust ISPs to clean up their act.

Who said anything about trusting the ISPs?  As you note, at least most of them have conclusively proven themselves to be wholly UNtrustworthy.  But "trustworthy" and "predictable" are two different things.  The (major retail) ISPs are predictable, if given the right incentives (namely, a credible threat to their economic bottom lines).

We must put more power in the users' hands if we are to be effective against spam.

The users already have all the power they need -- if they were but simply willing to use it.  It's called their checkbook.

To rely on "naming conventions" is a guarantee of failure.  You will NEVER get all network operators to agree on (much less implement) any given "standard" naming scheme.

Not to mention that spoofing legitimate names is one of the reasons why spam is so difficult to fight. A naming convension without protection from spoofing would be useless.

Probably true, but irrelevant.  It doesn't matter how robust a given scheme might be technologically, if it is never widely enough implemented to reach critical mass.

The only cross-comparison which is potentially useful at this point in time is the HELO/EHLO vs. the IP address.

That's the reason I'm pushing so hard for some means of authentication. If we can authenticate, we can make a LOT more reasonable comparisons.

The biggest word in the English language is "if".  And as pointed out above, until and unless you solve that "if", everything else simply doesn't matter.

but in practice, SPF has enough problems that I doubt it will ever be universally adopted

The only real problems I'm really aware of are a lack of adoptation and some patent disputes. All other "problems" I know of are poorly reasoned excuses not to adopt it or a lack of understanding of how it works.

There's more.  Think about LISTSERV-style mailing lists, for example.  That has inspired various header-rewriting schemes; but these band-aid fixes are themselves problematic, as they essentially constitute header forgery -- and they inherently undermine the effectiveness of SPF.

Unfortunately, a lack of adoption is a real problem - and is going to be a real problem for ANY solution on the SMTP side of things. We just can't rely on the people who run SMTP servers to adopt a solution. A real solution is going to have to include technologies on the email clients themselves. Otherwise, this problem is never going to be manageable.

Sorry,. but you've got that cart squarely before the horse.  As I noted above, no technological "band-aid" applied after-the-fact at the client level is or ever can be the solution, because that is not where the problem lies.  Once the client even has the opportunity to act on a given piece of (incoming) mail, the damage has already been done, for that mail has already been accepted, stored and transferred by the server -- i.e., all of those costs have already been incurred, so there's nothing to save.  And not incidentally, from the professional spammer's POV, that spam-delivery job was a success, and is thus billable to his mark^W... er... "client".  What happens to it after that doesn't really matter.

Your faith in that so-called "big four" is woefully misplaced. 

This is one area where I'll have to agree with you and disagree with David Berlind. I don't think we can trust ISPs to fix the problem for us.

Again, I never said we should trust the ISPs; quite the opposite, in fact.   Until and unless they are forced (including via economic pressure from their own customers) to act responsibly, they will continue to do whatever they myopically see as most in their short-term economic best interest -- which in general is "little or nothing".  Remember this infamous admission (the third-to-last paragraph of which contains the real "smoking gun")?  Based on that little bombshell, anyone with any ethical grounding whatsoever should and would have absolutely refused to do business with Comcast.  Had that happened, Comcast (and the rest of the world) would have gotten the message pronto; and I strongly suspect that we would have far less of a spam problem today as a result.  But the fact that the lemmings kept flocking to the sea proves rather conclusively that the end users are simply not willing to wield the power they already have.

Relying on SMTP to fix the problem and not giving users power over spam is a mistake, and as long as we make that mistake we'll still be talking about this years from now.

Sorry, but your metaphor here is so mixed as to obviate the point.  SMTP is the transport protocol by which all Internet e-mail is exchanged.  That is NOT going to change, if for no other reason than the fact that it is already far too well-entrenched.  But beyond that, it doesn't need to change; for the SMTP protocol is not the problem.  The policies of (many of) the people using SMTP is the problem; and it is those policies which need to change.  And as noted above, the end users already have all the power they need to effect that change -- they (apparently) just aren't willing to use it.

The users already have all the power they need -- if they were but simply willing to use it.  It's called their checkbook.

Problem is, that power isn't enough, especially in some areas where the ISPs are virtually monopolies or oligopolies. In addition, the ISP gets money for the Internet service whether the customer uses the ISP's email or somebody else's. The checkbook is virtually useless in many areas.

Many people have in fact voted with their checkbooks by switching to web based email instead of using what came with their ISP - but to no avail, since the ISP isn't making money from legit users' email. It's making money by being the only show in town.

The place to reject spam (and viruses, and anything else you do not want for policy reasons) is during the initial SMTP "conversation".

You can't trust your SMTP server, and your SMTP server is unaware of your personal preferences and email habits. The SMTP server often has no idea what spam is, and may even be working against you. Ultimately, the SMTP server is an inheritly untrustable party, and should be treated as such.

Hence, the solution is not yet another overly complex pie-in-the-sky "new technology"

"Pie-in-the-sky?" How do you figure? Cryptography and digital signatures are proven technologies. If you can crack a 100+ bit key for a known strong cryptographic algorithm, give it your  best shot.

Maybe, in theory.  But the problem with this approach is the same as nearly all the other FUSSPs which have been proposed (and shot down) over the years:  They require universal (or close to it) acceptance and implementation in order to be effective (which is obviously near-impossible to achieve); and most operators won't implement until and unless they see effectiveness -- hence, chicken and egg.

I'd be willing to bet that people would gladly latch on to an effective solution if given the chance. Problem is, nobody has stepped up to the plate and offered something that would solve the problem in a manner that is easy to use and easy to adopt.

Problem with server side SMTP solutions is that they're difficult to adopt, because ISPs refuse to adopt them, and end users can't adopt them because the end user generally isn't running a personal SMTP server. It's no wonder nothing has been adopted - you've just killed adoption simply by making it a requirement that the solution is adopted on a SMTP server.

Hello? Anybody home? How about we look at the adoption issue as well? If adoption is an issue, how about we take a look at what we can do to encourage adoption?

Ease of use and solutions that can be implemented on the client side are possible ways to increase the chances of adoption. Forcing the solution to be only available to SMTP servers is hurting adoption.

If you have better (and realistic) ideas to increase adoption, feel free to share them.

And besides, even at best, this still only addresses a symptom of spam (i.e., the currently fashionable delivery mechanism); it does not address spam, per se.

Really? Not being able to identify people and being able to easily spoof identities is the root of the problem, not a mere symptom. The whole reason it's a major problem is because it's too easy to fool current systems.

Perhaps not; but in that case, you're talking about something else besides "spam".  Those qualifiers are a NECESSARY part of the definition.  Just because you don't like or don't want any given piece of mail does NOT make it "spam".

Fine, whatever, maybe I should give it a new name.

Or maybe, just to annoy you, I'll still call it spam :P.

I never really recognized any ultimate authority when it comes to word definitions - and I frankly don't care if I have "widespread support" or not. Problem is, there's a lot of concepts that just don't translate to a single word, and often the meaning of the word detracts from the conversation. It doesn't matter what word you want to use to describe it, I want to address the problem of unwanted email. Want to call it spam? Fine. Want to call it something else? Fine. I couldn't care less if you want to have hostile feelings simply because I'm not using a word the "right" way. I want to fix a problem, not spend a thesis writing about the "true" definition of a word.

Not to mention that spoofing legitimate names is one of the reasons why spam is so difficult to fight. A naming convension without protection from spoofing would be useless.

Probably true, but irrelevant.  It doesn't matter how robust a given scheme might be technologically, if it is never widely enough implemented to reach critical mass.

Not irrelevant. It's the whole point. If they can keep spoofing their identity, and there's no way to positively identify people you already know, that allows them to get past anything you will throw up at them. Even extermely harsh systems that only accept emails from known email addresses can be fooled if the email address from a known sender can be easily forged.

I'm not saying that individual users are (or should be) powerless.

Oh, but you are! You say that users should have no say in deciding what's spam or how to filter it, and that some elitist (but more realistically malicious) people running SMTP servers should have the power.

Again, I never said we should trust the ISPs;

Who exactly is running the SMTP servers? Oh, yeah, that's right - the ISPs. Yeah, there's a few more people running SMTP servers out there as well - but how do I know they are any more trustworthy than an ISP?

SMTP is the transport protocol by which all Internet e-mail is exchanged.

It's a protocol you do not want users to have access to - and that takes power away from the user. If the user is to truly have the power to fight spam, the protocol - be it an old protocol or  new one - should be end to end, peer to peer, or something else - just not server/client. The server/client protocol has failed to be effective. It's time we moved to something more effective.

You can even piggyback an end to end protocol on your precious SMTP protocol, if that makes you feel all warm and fuzzy inside.

Because then, if it's piggybacked, you can accuse it of being a "band-aid" instead of a real solution. Works very conveniently for you! I must say, you are a master of manipulation of logic. By removing the idea of a new protocol and replacing it with a piggybacked protocol, you can now apply a metaphor of bandages!

How convenient. Needless to say, I want PROOF that it's a band-aid. Simply whining about how adding onto an existing protocol is a "band-aid" is not going to cut it anymore - because a protocol can be developed without piggybacking as well.

So, I'm "wrong" if I want a new protocol, and "wrong" if I want to modify or piggyback onto SMTP. I think you're just being stubborn. Quite frankly, even with the potential problems my proposed solutions may have, I still think they'll be effective.

The policies of (many of) the people using SMTP is the problem; and it is those policies which need to change.

Right now, the ISP is the one with the SMTP server, and the one dictating the policy. And as I've said, the ISP is not to be trusted. It's time we stopped relying on people who have shown that they cannot be relied on.

And as noted above, the end users already have all the power they need to effect that change

False. They have very little power because A, they often live in areas controlled by monopolies who make money regardless of email usage and B because they depend on unknown (and often untrustworthy) people using "SMTP servers" to manage their email.

The user needs to be the ultimate authority of their inbox, and not to be a slave of whatever SMTP server they decided to connect to.

Anyways - I was thinking - why just rely on only one method for stopping spam anyways? Security is usually layered, with several layers - why not spam filtering as well? Instead of arguing over which method is best, perhaps several methods should be implemented - as I have pointed out, cryptography and digital signatures can be used to supplement existing anti-spam technologies to make them more effective. We're not talking about anything that is mutually exclusive here.

You can even apply anti-spam methods to both the users' email clients and the SMTP servers, if it makes you feel warm and fuzzy inside to keep the SMTP system.

Decided to sell something on ebay, got a question from a potential customer, which ebay forwarded to my email. Of course I went to ebay by typing in the URL manually and logged in and checked to see if it was legit.

 But I realized something: A lot of phishing scams try to look like they come from ebay and banks.

At the very least, I'd like some way to authenticate legitimate emails. Because I know that if I white list ebay without authentication, it will let some phishing scams through that spoof the "From: " address.

Even if we disagree about authentication stopping spam - how about phishing? Certainly this could be effective against phishing.
 

Well, it seems like Comodo is offering free certificates for email, for those who want to go with the PK scheme. Personally, I still prefer a web of trust, but it does seem that all we need now is a good implementation and some way to encourage adoption. Turns out David's assertion that "PK certs are *never* going to be cheap" was way off target.

http://www.comodo.com/products/certificate_services/email_certificate.html

Frankly, though, all I see here is stubbornness. Yes, I know that preventing forgery of an identity is only a small part of the battle, and will not by itself stop spam. But IMHO, it is an important part of the battle and it can make already strong anti-spam solutions even stronger. And with phishing being a bigger problem than ever, I think that positively identifying legitimate emails is more important than ever. I'm sorry, but after hearing all of the arguments I've come to the conclusion that such a system would greatly benefit our email system.

Yes, it may add some complexity to email systems, but I think the benefits greatly outweigh the added complexity. Yes, the ISPs and other email providers should do a better job at ensuring email being relayed through them is legit. But the unfortunate truth is that many are not doing this and it is doubtful that we will ever be able to rely on them to clean our email.

Yes, ideally spam should be stopped at the source, because that places the least burden on the clients and the email system as a whole. Unfortunately, the people at the source are not cooperating to stop spam. Stopping spam at the destination may not be the best way - but it is the only way.

Just to throw a curve ball - I've come up with yet another idea. A crazy sounding one, but considering current trends, it may be the future. You know what happened with the Internet in general as viruses spread all over the place and the Internet essentially got filled with a lot of junk packets? Well, businesses started using firewalls and routers, and eventually even the consumers started buying routers. Linksys and Belkin routers are cheap and can be found in almost everybody's home these days. So, what I'm thinking is that spam filtering may eventually stop being a software thing, and may eventually be built directly into the hardware routers most people have. This may be a way to cut down on spam before it ever reaches the client - and it would also do it without relying on the user's ISP to do it.

I think it may happen. We saw routers go from being expensive units found only in big businesses to cheap units found in people's homes. I don't think it's a stretch to think that future routers may become full security gateways.

In any case, if some other email client does a better job at determining the legitimacy of emails in the future, I would be open to changing email clients. Pmail's filters are very nice - but useless if they are easily fooled.
 

I definately would not want Linksys or Belkin deciding what email I am "allowed" to receive.

I don't like my ISP doing it either, which is why I run Mercury so I can decide for myself  

Well, one of the biggest complaints people have on this forum is that we shouldn't be running our own servers. We simply can't have our own personal solutions, according to them. We have to stop it at its source, which will realistically never happen.

It's been awhile since I checked in here; and having done so, I found that this thread is apparently still going.  So I thought I'd make one more attempt to get you to understand some of the points I was making...

CobraA1:

The users already have all the power they need -- if they were but simply willing to use it.  It's called their checkbook.

Problem is, that power isn't enough, especially in some areas where the ISPs are virtually monopolies or oligopolies. In addition, the ISP gets money for the Internet service whether the customer uses the ISP's email or somebody else's. The checkbook is virtually useless in many areas.

Only if the customer *willingly* concedes it to them -- and in most cases, there is no real need to do that.  There are at least a handful (usually many more than that) of competing service providers available to the VAST majority of users, at least in the U.S. (which is all I have experience with; but I sincerely doubt that most of the "civilized" world is all that different).  Some cities are even implementing wide-area WiFi/WiMAX networks, available to all within (radio) earshot.  Sure, there can be exceptions in some really isolated areas; but if we're that far out in the boondocks, how many users are we really talking about?  You might also have to compromise in some other aspect, such as speed, or cost, or what-have-you.  But that is strictly a matter of your personal priorities; it is *not* a case of "no choice".

Many people have in fact voted with their checkbooks by switching to web based email instead of using what came with their ISP

I doubt that the increasing popularity of WebMail has much of anything to do with "voting with your checkbook".  It has more to do with (perceived, if not actual) convenience, ubiquity, and quasi-anonymity.  It's not for nothing that the two largest "WebMail" providers (Hotmail and Yahoo) are perennially two of the largest spam-source/spam-support "problems" to the rest of the world.

- but to no avail, since the ISP isn't making money from legit users' email. It's making money by being the only show in town.

Well, I can cite at least one example with complete authority:  Several years ago, I dropped a major regional ISP who is/was widely reputed to be "better than average" in terms of their anti-spam policies, and I took a handful of my clients with me.  I did this precisely *because* that "anti-spam" rep was gained solely with respect to their *outgoing* mail.  Through it all, they continued to drop ever-larger mountains of *incoming* crap into their users' mailboxes, and steadfastly refused to implement even the most basic and obvious blocking/filtering such as against even the most egregious well-known spamhauses or such obviously illegitimate sources as the individual classrooms of the Korean public school system (which, at least at that time, was near-100% "zombied").  Through several "rounds" of discussion with the upper management of said ISP, it became clear that this dichotomy was due at least in part to one particularly recalcitrant "Lead E-Mail Administrator".  At that time, I let them know in no uncertain (but polite to a fault) terms *exactly* why they were losing several long-time loyal customers.  How much impact that had on their corporate policies, I cannot know; but I do note that said recalcitrant Administrator left their employ a short time later.

The place to reject spam (and viruses, and anything else you do not want for policy reasons) is during the initial SMTP conversation".

You can't trust your SMTP server,

Sure I can.  I administer it.  And before I did that, I paid someone I trust (and with whom I mostly agree about basic anti-spam issues) to do it for me.  There are *very* few folks who cannot do at least one or the other of these things, despite your apparent insistence that everyone is dependant on large retail ISPs; that presumption is just plain baseless.

But beyond that, you have missed the point of my earlier comment.  Regardless of whether or not you feel you can "trust" your local SMTP server, the fact remains:  That is the ONLY place where spam can be properly rejected.  If a given piece of mail traffic is *not* stopped there, there is little or nothing more which can be done to combat it, because the lion's share of the damage is already done as soon as the server issues the "250 OK" message.  At that point, there is nothing else for the server to do except deliver it, or silently drop it on the floor (which is problematic in a number of ways), or generate and issue an NDR (which is usually a VERY bad idea, as this most often constitutes "backscatter" spam -- which can get you blacklisted in a hurry).

The "Bottom Line" here is, no client-side "solution" is or can ever be a *real* solution, simply because it comes too late in the mail-delivery process.

Hence, the solution is not yet another overly complex pie-in-the-sky "new technology"

"Pie-in-the-sky?" How do you figure?

Simple...  You have yet to show how such a "solution" could be workably implemented.  In that respect, it's just like every other FUSSP that has ever been put forth -- it's a pipedream.

Cryptography and digital signatures are proven technologies. If you can crack a 100+ bit key for a known strong cryptographic algorithm, give it your  best shot.

Yeah....  So?  Show me *specifically* how that applies to this discussion?  What does it matter if you, as an individual end-user, can determine with 100% accuracy that a given message which has already been delivered to your inbox is or is not actually from whomever it purports to be from?  How has that lightened your spam load one iota?

Maybe, in theory.  But the problem with this approach is the same as nearly all the other FUSSPs which have been proposed (and shot down) over the years:  They require universal (or close to it) acceptance and implementation in order to be effective (which is obviously near-impossible to achieve); and most operators won't implement until and unless they see effectiveness -- hence, chicken and egg.

I'd be willing to bet that people would gladly latch on to an effective solution if given the chance.

But you have yet to show that your "solution" really is effective -- or even that it is a "solution", for that matter.

Problem is, nobody has stepped up to the plate and offered something that would solve the problem in a manner that is easy to use and easy to adopt.

Problem with server side SMTP solutions is that they're difficult to adopt, because ISPs refuse to adopt them, and end users can't adopt them because the end user generally isn't running a personal SMTP server. It's no wonder nothing has been adopted - you've just killed adoption simply by making it a requirement that the solution is adopted on a SMTP server.

*I* am not making that requirement; it is simply a fact of life given the way internet e-mail works -- a process which, frankly, I am becoming ever-more convinced you simply do not understand.

And besides, even at best, this still only addresses a symptom of spam (i.e., the currently fashionable delivery mechanism); it does not address spam, per se.

Really? Not being able to identify people and being able to easily spoof identities is the root of the problem,

No, it isn't.

The root of the problem is that spammers send spam (cue reference to "The Scorpion and the Frog" http://allaboutfrogs.org/stories/scorpion.html).  Whether or not they properly identify themselves in the process (some do, some don't) is merely a side issue -- and one which doesn't even apply much of the time.

The whole reason it's a major problem is because it's too easy to fool current systems.

But notably, the "systems" you're talking about are at the SMTP server level, *not* the end-user's POP3/IMAP client.  Hence, a "solution" applied at the end-user level CANNOT work.  Is that not obvious to you?