[Coco] Re: Someone might come to complain

[John E. Malmberg]

John R. Hogerhuis wrote:
> On Fri, 2005-04-29 at 13:39 -0500, Roger Taylor wrote:
> 
>>I've had this problem trying to talk to a few Portal-9 customers to send 
>>them their key.  SPAM filters just plain suck. My own ISP catches a 
>>tremendous amount of real SPAM, which is great, but they also catch real 
>>e-mails and make me have to weed through their report e-mails to see which 
>>e-mails are not junk.  It takes less time to just hit delete on the obvious 
>>junk letters as they come in.

That is just indicating that your ISP has not implemented anything close 
to a state of the art anti-spam system and has just implemented one that 
is close to just being a placebo either because they really do not know 
what the state of the art is or they do not really care.

State of the art is that over 80% of the spam can be detected with no 
false positives from confirmed spam sources before the spam enters the 
mail server.  This was also state of the art well over 5 years ago.

You can get close to 99% with most people seeing no false positives if 
you refuse e-mail from known DHCP assigned addresses in addition to the 
above.

This was also state of the art over 10 years ago.

The above has been state of the art for so long, that it is built into 
just about all commercial mail server software.  All the mail server 
operator needs to do is set the configuration options, and if they are a 
large operation, some of the DNSbl services need compensation for the feed.

And doing so significantly reduces the cost of operating a medium to 
large mail system.  In some case we are easily talking thousands of 
dollars a month just for bandwidth costs.

If you have a bit more time when you are sorting though the tagged spam,
and especially the mis-tagged mail, plug the I.P. address that your ISP 
mail server accepted the e-mail into the box at:

http://www.moensted.dk/spam

It will not take long until you notice a clear trend.  One of CBL, XBL, 
DSBL, NJABL, and SPAMHAUS.ORG or SORBSDUL will show up on the majority 
of the spam, and if they are not, what ever one is missing or rarely 
showing up, your ISP is rejecting based on them before they tag the 
mail.  Of the real mail there is only a small chance that you will get a 
non-spam e-mail from a SORBSDUL listed address.

And of what spam gets through the above checks, most can be detected by 
checking the I.P. of the URLs that the spammer needs the spam victim to 
click on, which SpamAssasin 3.0 is able to do.

This is state of the art as of 1 year ago.  Only some mail servers can 
take advantage of this before the SMTP transaction is over.

A properly implemented state of the art spam filtering system does not 
suck, it is just the multitude of bad systems and the large amount of 
people who thing that the bad systems are anything close to state of the 
art that is giving spam filtering a bad reputation in some areas.

It amazes me to see NEW anti-spam systems being deployed that only use 
methods that for at least the last 5 years have been proven not to work.

With a state of the art system, the average user will not see much if 
any spam, and their senders will not get non-delivery notices.

There are many mail servers operating that way.

By the way, for qsl.net to have me "Just Hit Delete" instead of having 
them reject the spam, would increase their cash operating costs over 
300%.   qsl.net is a non-profit hobby operation that runs on donations. 
  Such an increase in costs would shutdown the service.

> I get a lot of spam mails on my jhoger-e+AXbWqSrlAAvxtiuMwx3w(at)public.gmane.org
 > account, since I give it out to everyone. It's my universal email address
 > so I refuse to give in and try to hide it.

It is quite hidden from the gmane users :-)

If I had not removed the at-sign, gmane would have munged it yet again.

> I have my pobox filters set up to either bounce the mail immediately, or
> let it through.
> 
> That way, at least the sender *knows* that I didn't get the email since
> it is immediately bounced back to them, never stored/delayed anywhere.

Please use the term "reject" instead of "bounce".  With an reject, your 
server has simply refused the message and it is the sending server that 
generates the bounce.

The difference is significant because mail servers that bounce instead 
of rejects end up participating in a DDOS on the domains that spammers 
and viruses have forged.  Test.com and others have had their mail 
servers brought down because of the volume of those bounces.

Some mail server operators still do not understand the difference and 
why having their servers ever generate a bounce after the SMTP 
transaction is over is now extremely bad practice.

With spam and viruses, there is no real sending server, so no bounce 
message is ever generated, unless an open relay is used.  And most mail 
server operators will not accept e-mail from open relays, so those 
bounces are never seen.

> If you run your own MTA you can do the same thing.

Yes, unfortunately it is not an option for my area at this time.  There 
are no ISPs that will provide me with such service to my home unless I 
pay them to run a dedicated line to it.  From what I can see, most of 
the major home broadband ISPs prohibit such servers.  A few will allow 
it for an extra cost, but not the one that has a monopoly in my area.

However qsl.net and encompasserve.org will deliver a message to me or 
they will use an SMTP reject to indicate why it can not be delivered. 
Because they are just a mail forwarding service, that reject can also 
indicate that the system that I do get the mail on is unreachable for 
some reason.

The bounce message will come from the sender's mail server not from 
either of my mail servers.

So if someone forges your e-mail address on a spam or a virus and tries 
to send it to my e-mail address, your e-mail boxes will not get any 
backscatter from either of my primary mail servers.

Most commercial mail servers reject on non-delivery.  If that were not 
the case, internet e-mail would have collapsed a long time ago.

> Some people even think that the spammers remove bouncing addresses from
> lists, so it may even help to reduce the volume of spam.

In general though there is no evidence that a spammer ever removes an 
address that they get non-delivery notices on.  Look up on google for 
the tale of "nadine", and e-mail address that has never accepted e-mail 
and is still continuing to get spam.

When people inactivate an account that gets spam, and then reactivated 
it some time later and find it spam free, they think the bounces are the 
reason.  It is more likely that the mail server operator improved the 
spam filtering even if they do not admit to it than the spammer removed 
their e-mail address from their list.

Because of people complaining about censorship and related, many mail 
server operators have decided to pretend that they are not filtering 
even when they are, and if they are blackholing what they detect as spam 
by source I.P, there is no way for most users to know what is going on.

When someone complains, and the mail server operator sees that it is an 
address they have blackholed, they will whitelist that address and tell 
the user to have the sender retry, at which point the problem will be 
closed as not reproducible.  I could easily do this on a corporate or 
ISP mail server with no one catching on to it that I did not want to, 
because I could set up scripting to whitelist domains based on watching 
outgoing e-mail.

It seems many people prefer that type of service than knowing what is 
really being done.  Of course it is more expensive rejecting, and it 
takes longer to get the address whitelisted when the script misses it, 
and some lost messages would never be noticed.

> But there's no way I could filter through it all manually. Way too much
> work...

Too high of a chance of error.  Once the SMTP transaction is over, there 
is no way to reliably automatically notify the sender of real e-mail it 
was not received with out being abusive to the rest of the internet.

With quarantine or tagging system, by the time that a mis-classified 
message is detected if it is ever detected, substantial time may have 
occurred.

-John
wb8tyw at qsl.network
Personal Opinion Only