Milter-greylist

Hi all,

I have just returned from CiscoExpo exhibition - I was quite curious to
see what technology they offer to fight spam. It turned out that they
acquired company Ironport which is looking after senderbase, the most
successful black/whitelist sender database (so they say).

Google says the usage of the senderbase portal should be free so
question is:
- Is it possible to use www.senderbase.org to gather reputation of the
sender and set greylisting constants accordingly to that reputation?
(i.e. the similar way we can do with dnswl or dnsbl)
- Is here anyone who managed to do it? How?
- Any other comments, suggestions?

Thanks,
Ondrej

Ondrej Valousek wrote:
> I have just returned from CiscoExpo exhibition - I was quite
> curious to see what technology they offer to fight spam. It turned
> out that they acquired company Ironport which is looking after
> senderbase, the most successful black/whitelist sender database (so
> they say).
> 
> Google says the usage of the senderbase portal should be free so 
> question is:
> - Is it possible to use www.senderbase.org to gather reputation of
>   the sender and set greylisting constants accordingly to that
>   reputation? (i.e. the similar way we can do with dnswl or dnsbl)
> - Is here anyone who managed to do it? How?
> - Any other comments, suggestions?

Short answer:  The Spamcop DNSBL represents the data from Senderbase.
 However, it requires 100% accuracy, and we don't care so much for
greylisting.  I wrote a script to delay members of bad networks.


Ironport owns and operates Spamcop (senderbase's big sister), which is
one of the best of the DNSBLs.  I correlated a significant boost to
spam filtering to the fact that I started reporting spam to them.
Some spam botnet/relays stay below the radar by limiting who they
spam, so it is your duty to report them; DNSBLs can only go so far
with their honeynets.  Take a look at http://stats.dnsbl.com/ for
DNSBL stats and reviews.

My greylisting time is increased for hits in DNSBLs and whitelisted
for DNSWLs.  I bumped up the SpamAssassin score on trusted DNSBLs
rather than rejecting mail outright due to not fully trusting DNSBLs.

Spamcop/Senderbase is GREAT.  I regularly scrape their top offending
/24 blocks to add to milter-greylist for extra delays in hopes that it
delays spam long enough to get reported (this helps fight growing
botnets and the like).

I've attached my update/install script, which should be very portable,
though it might require GNU sed, and systems with non-fully
POSIX-compliant /bin/sh should run it with /bin/bash (it works with
ash/dash but possibly not with jsh (Solaris /bin/sh)).  It's extremely
user-friendly and well documented, even if my code is a bit dense.

I have a similar non-updating rule defined in SpamAssassin, which
seems to verify that this helps a lot.

Tere.
>
> I've attached my update/install script, which should be very portable,
> though it might require GNU sed, and systems with non-fully
> POSIX-compliant /bin/sh should run it with /bin/bash (it works with
> ash/dash but possibly not with jsh (Solaris /bin/sh)).  It's extremely
> user-friendly and well documented, even if my code is a bit dense.
>
>   
Great script but why to use DNSBL with milter-greylist instead of using 
it with MTA? I'm using milter-greylist for greylisting and sendmail as 
MTA with Spamcop DNSBL.

-- 
Mart

On Thu, Apr 17, 2008 at 2:53 PM, Mart Pirita <sysadmin@...> wrote:
>  Great script but why to use DNSBL with milter-greylist instead of using
>  it with MTA? I'm using milter-greylist for greylisting and sendmail as
>  MTA with Spamcop DNSBL.

Because with the MTA you can only block mail with RBL:s. With
greylisting you can use RBL:s to block *or* to have a longer
greylisting period for example. Also much of the access rules of
Sendmail can be done in milter-greylist with much more ease.

-- 
/peter

Hi Adam,

Thank you for this comprehensive explanation.
Just a few notes:
1) Your script only gathers info about the botnets from the spamcop
site, but you also mentioned dnswl to whitelist the "promising" senders.
How did you implemented this? senderbase is meant to be more like
"scoring" system ranking the sender's IP from -10 (a spammer) to 10
(innocent guy) -> so I thought we could use it for whitelisting as well.

I have almost persuaded my boss to engage greylisting to protect the
whole company, but he is still sort of concerned about the possible
delays it could cause (and I must admit, I understand that concern). So
far, I am whitelisting all senders who passes the SPF check or are able
to do the TLS encryption, but it would be nice to have some other
judgement as well (SPF and TLS technologies are gaining the popularity
quite slowly)

2) Before I went to the CiscoExpo, I was sort of careful about dnsbls or
dnswls. How could we trust them? If one starts to be too popular,
spammers might try to DOS it (it is a single point of failure). You say,
report the spam to make it better, but how is it protected from the
spammers filling it with a number if innocent senders?

Ok, now I think, Cisco & Ironport use it and so do all their customers
so it is probably working. But still, I feel a bit uneasy about it.

3) I have installed your script to cron on our MTA (looks like it works
fine) to give it a try. I am sure others would be interested as well.
What about putting it into our wiki so it won't get lost?

Thanks,
Ondrej
Adam Katz wrote:

>
> Ondrej Valousek wrote:
> > I have just returned from CiscoExpo exhibition - I was quite
> > curious to see what technology they offer to fight spam. It turned
> > out that they acquired company Ironport which is looking after
> > senderbase, the most successful black/whitelist sender database (so
> > they say).
> >
> > Google says the usage of the senderbase portal should be free so
> > question is:
> > - Is it possible to use www.senderbase.org to gather reputation of
> > the sender and set greylisting constants accordingly to that
> > reputation? (i.e. the similar way we can do with dnswl or dnsbl)
> > - Is here anyone who managed to do it? How?
> > - Any other comments, suggestions?
>
> Short answer: The Spamcop DNSBL represents the data from Senderbase.
> However, it requires 100% accuracy, and we don't care so much for
> greylisting. I wrote a script to delay members of bad networks.
>
> Ironport owns and operates Spamcop (senderbase's big sister), which is
> one of the best of the DNSBLs. I correlated a significant boost to
> spam filtering to the fact that I started reporting spam to them.
> Some spam botnet/relays stay below the radar by limiting who they
> spam, so it is your duty to report them; DNSBLs can only go so far
> with their honeynets. Take a look at http://stats.dnsbl.com/
> <http://stats.dnsbl.com/> for
> DNSBL stats and reviews.
>
> My greylisting time is increased for hits in DNSBLs and whitelisted
> for DNSWLs. I bumped up the SpamAssassin score on trusted DNSBLs
> rather than rejecting mail outright due to not fully trusting DNSBLs.
>
> Spamcop/Senderbase is GREAT. I regularly scrape their top offending
> /24 blocks to add to milter-greylist for extra delays in hopes that it
> delays spam long enough to get reported (this helps fight growing
> botnets and the like).
>
> I've attached my update/install script, which should be very portable,
> though it might require GNU sed, and systems with non-fully
> POSIX-compliant /bin/sh should run it with /bin/bash (it works with
> ash/dash but possibly not with jsh (Solaris /bin/sh)). It's extremely
> user-friendly and well documented, even if my code is a bit dense.
>
> I have a similar non-updating rule defined in SpamAssassin, which
> seems to verify that this helps a lot.
>
>

Mart Pirita wrote:
>> Great script but why to use DNSBL with milter-greylist instead of
>> using it with MTA? I'm using milter-greylist for greylisting and
>> sendmail as MTA with Spamcop DNSBL.

shuttlebox wrote:
> Because with the MTA you can only block mail with RBL:s. With 
> greylisting you can use RBL:s to block *or* to have a longer 
> greylisting period for example. Also much of the access rules of 
> Sendmail can be done in milter-greylist with much more ease.

I should further elaborate.  Spamcop already operates an RBL at
bl.spamcop.com.  However, it doesn't work with newly activated
zombies.  By taking a look at the top offending IP blocks, you can
discriminate against systems on networks known to be easily
compromised (like a lab whose machines are slowly getting infected).
This is a buffer-zone kind of RBL, which is not (to my knowledge)
served via DNS in the same manner as Spamhaus and Spamcop.

However, you would never want to block them outright, since they're
not known spammers (yet).  By delaying their mail, you give the
blacklists time to receive reports, and then Spamassassin can kill
them when it checks those blacklists.

Oops, I sent with the wrong email address.  Looks like it was eaten by
yahoo, or it awaits moderator approval.  In the latter case, it can be
deleted as I am resending it from the correct account.

Ondrej Valousek wrote:
> 1) Your script only gathers info about the botnets from the spamcop
> site, but you also mentioned dnswl to whitelist the "promising" senders.
> How did you implemented this? senderbase is meant to be more like
> "scoring" system ranking the sender's IP from -10 (a spammer) to 10
> (innocent guy) -> so I thought we could use it for whitelisting as well.

We certainly can.  In fact, it would probably only take a small tweak
to my script to do that; offer a different URL and create a different
link and you're done.  That's a really good idea, actually.  I'll
implement it when I have a spare moment.

I currently implement this (as previously mentioned on this list):

dnsrbl "DNS Whitelist"  list.dnswl.org  127.0.0.0/16
racl whitelist dnsrbl "DNS Whitelist"

> I have almost persuaded my boss to engage greylisting to protect the
> whole company, but he is still sort of concerned about the possible
> delays it could cause (and I must admit, I understand that concern). So
> far, I am whitelisting all senders who passes the SPF check or are able
> to do the TLS encryption, but it would be nice to have some other
> judgement as well (SPF and TLS technologies are gaining the popularity
> quite slowly)

Yeah, I get complaints from co-workers on this every once in a while.
 Since spammers who don't forge sender domains can control their own
SPF records and thus bless their own botnets, I turned off SPF.  This
was the source of my request in early-mid March.

I've found "lazyaw" to be indispensable.  Without it, the most common
user complaint was "I didn't get an email from customer X, but
coworker Y did."  This solves that issue right off.  It also mostly
solves the issue of larger companies having a bazillion SMTP servers
(since the greylisting only examines the server rather than the server
plus the sender plus the recipient).  Its only real weakness is
against ISPs, free webmail providers, universities, and other giants,
which is why I proposed lazyaw thresholds and awspf (on March 11).

> 2) Before I went to the CiscoExpo, I was sort of careful about dnsbls or
> dnswls. How could we trust them?

Yeah, some admins just don't trust them.  I've been very happy with my
usage, and the numbers at http://stats.dnsbl.com are very heartening
(though that's only one source).  I do NOT block a single email based
on listing in DNSBLs.  Items that hit spamcop, spamhaus zen, PSBL, my
hacked spamcop /24 blocks, and SORBS DUN are all greylisted (with
SORBS DUN staying whitelisted for a longer time due to the massive
number of false positives), but nothing is blocked.

In Spamassassin, I've added PSBL and bumped up spamcop and SBL.  I
also created an adjuster to ensure a combination of DNSBLs doesn't
single-handedly auto-delete an email.  Here's a snippet from my
local.cf file; note I mark spam at 5.0 and reject it (at
connection-time via spamass-milter) at 8.0.  The "meta" line wraps:

# all but implicit trust for zen/spamcop/psbl, http://stats.dnsbl.com
# psbl doesn't exist - copied from http://psbl.surriel.com/howto/
header   RCVD_IN_PSBL eval:check_rbl('psbl', 'psbl.surriel.com.')
describe RCVD_IN_PSBL Received via a relay in PSBL Spamikaze trap
tflags   RCVD_IN_PSBL net
#     SCORE_NAME              -  n  b n+b     defaults
score RCVD_IN_PSBL            0 1.0 0 2.3   # suggested: 0 1 0 1
      score RCVD_IN_BL_SPAMCOP_NET  0 2.2 0 3.2   # 0 2.188 0 1.960
score RCVD_IN_SBL             0 2.5 0 2.5   # 0 2.810 0 1.551
#score RCVD_IN_XBL            0 2.5 0 3.0   # 0 2.896 0 3.033
#score RCVD_IN_PBL            0 0.5 0 1.0   # 0 0.509 0 0.905

meta ICS_DNSBL_ADJ      ( RCVD_IN_PSBL + RCVD_IN_BL_SPAMCOP_NET +
RCVD_IN_SBL + RCVD_IN_XBL ) > 2
describe ICS_DNSBL_ADJ  Undo autokill from possibly incestuous DNSBLs
score ICS_DNSBL_ADJ     0 -0.3 0 -1.8
# min:         7.8->6.0 / max: 8.7->6.9
# nobayes min: 5.7->5.4 / max: 7.2->5.4

> If one starts to be too popular, spammers might try to DOS it (it
> is a single point of failure).

Yup.  PSBL is down right now, perhaps due to an attack, perhaps
because it is a poorly funded free service.  None of these are
"single" points of failure.  They are all helpers in a larger picture
of fighting spam.  If they ALL went away, spam would continue to be
filtered.  My ADJ meta rul ensures delivery if some (or all!) of them
were compromised (i.e. filled with false positives).

> You say, report the spam to make it better, but how is it protected
> from the spammers filling it with a number if innocent senders?

Few (if any) reporting facilities share their data.  If you fill our
manual reports with Spamcop, they won't even store your email address
(though sometimes they're not so great about censoring it).  They
don't report headers to the public, and the automated system doesn't
report headers when they report spammers; that's for real-person dialogs.

As to "innocent senders," that's what reporting is all about; Spamcop
(et al) actually contacts the ISP or IT dept in charge and reports the
spam attempting to ebb the flow from the source (e.g. deny internet
access to a zombie system).  Reporting functions on relays rather than
apparent email addresses, so you need not worry about forged senders
(or even forged relays, since it only goes as far as the most recent
untrusted relay).  Peruse the Spamcop website for more detail.

> 3) I have installed your script to cron on our MTA (looks like it works
> fine) to give it a try. I am sure others would be interested as well.

Don't forget to install it; the output (emailed to you via cron) will
tell you that on the first run.

> What about putting it into our wiki so it won't get lost?

I did that yesterday :-D
http://milter-greylist.wikidot.com/local--files/downloads/greylist-spamcop-offenders
which is linked from the files section a the bottom of
http://milter-greylist.wikidot.com/downloads (could they hide it any
more?)

On Thu, Apr 17, 2008 at 9:15 PM, Adam Katz <yegsa-yahoo@...> wrote:
>  > What about putting it into our wiki so it won't get lost?
>
>  I did that yesterday :-D
>  http://milter-greylist.wikidot.com/local--files/downloads/greylist-spamcop-offenders
>  which is linked from the files section a the bottom of
>  http://milter-greylist.wikidot.com/downloads (could they hide it any
>  more?)

They sure are not easy to spot. I took the liberty to add a link to
your script so it's easier to find.

-- 
/peter

Ondrej wrote:
>> I have almost persuaded my boss to engage greylisting to protect the
>> whole company, but he is still sort of concerned about the possible
>> delays it could cause (and I must admit, I understand that concern).

I responded:
> Yeah, I get complaints from co-workers on this every once in a while.

... I completely forgot to mention my workaround:

racl whitelist rcpt /\+nogrey@(example\.com|example\.net)\b/

I tell my users to use +nogrey in their username, so you could mail
bob+nogrey@... and the message won't be delayed by
greylisting.  This is insanely useful for online registrations.

I also let users opt out of greylisting altogether.  Our support queue
does this.  Usernames that are both non-guessable and non-published
(e.g. ondrej.valousek@... instead of andrew@...)
rarely get hit by bots, so the delay is more annoying than useful.

Simple usernames are very sexy, but they are giant spam targets.  For
example, basic username construction on my name would give me
adam@... or akatz@..., but those are both heavy
targets for spamming via name-dictionaries; I'm better off with
adam.katz@....  This would also solve the problem of
determining what usernames Adam Kant and Adrianne Katz should get.
Short usernames are also brute-forced, so you also don't want 1-4
letter usernames (gmail doesn't even allow them!).

Along this line of thought, it is actually advantageous for the
anti-spam administrator of a company to have a very sexy username
--uh, I mean, very guessable username.  Users with problems can just
guess it, and spammers will also guess it.  This means anti-spam
admins get more spam than anybody else, and lo and behold, more
attention is paid to fixing the issue.

>> senderbase is meant to be more like "scoring" system ranking the
>> sender's IP from -10 (a spammer) to 10 (innocent guy) -> so I
>> thought we could use it for whitelisting as well.
> 
> We certainly can.  In fact, it would probably only take a small
> tweak to my script to do that; offer a different URL and create a
> different link and you're done.  That's a really good idea,
> actually.  I'll implement it when I have a spare moment.

I changed my mind.  Senderbase is an index of reported or otherwise
collected spam.  They do not have reliable numbers for non-spam email.

If you click on the [SB] link at the bottom end of the "spam reports
vs email volume" chart, you'll see that every IP in that block is
listed in 2-3 RBLs.  Using the opposite end of chart I pull my data
from, you can see there is almost no data, and one of the three IPs is
listed in two RBLs.  I see no useful data in any configuration here.

Best to stick with DNSWL.  Maybe also use some other whitelists too,
but be careful; some whitelists are rather generous (ASPATH and RDNS
simply verify real live servers rather than trustworthiness.  See
http://openrbl.org/client/#213.144.132.251 (good entry) versus
http://openrbl.org/client/#212.76.37.150 (bad entry) and click on the
hits' corresponding wiki pages.

I won't use whitelisting services that charge entrants for listing
based on the simple principle that it's akin to extortion: "pay us for
whitelisting or your mail won't get through!"

On Fri, 18 Apr 2008, Adam Katz wrote:

> I won't use whitelisting services that charge entrants for listing
> based on the simple principle that it's akin to extortion: "pay us for
> whitelisting or your mail won't get through!"

Or, to look at it from another perspective, it penalizes the poor.  If you 
believe that the poor should have just as much access or ability to be 
whitelisted, then it wouldn't make sense to support a system that charges 
for whitelisting.


-- 
While anyone can admit to themselves they were wrong, the true test is
admission to someone else.

finger://ephemeron.org/bigby
http://www.ephemeron.org/~bigby/
irc://irc.ephemeron.org/#the_pub
news://news.ephemeron.org/alt.lemurs

Tere.
>
> Because with the MTA you can only block mail with RBL:s. With
> greylisting you can use RBL:s to block *or* to have a longer
> greylisting period for example. Also much of the access rules of
> Sendmail can be done in milter-greylist with much more ease.
>
>   
Why waste bandwidth, cpu, ram etc for spammers instead of rejecting them 
in first attempt (MTA)? Ok, You give them longer greylisting period. How 
many who did pass, it was not sending spam? Yes milter-greylist is much 
easier to config, but this doesn't actually mean, that You have to 
config it all the time:).

And btw, if milter-greylist dies (it happens sometimes) and You are away 
from computer for long time, all spam passes true?

-- 
Mart

Tere.
>
> However, you would never want to block them outright, since they're
> not known spammers (yet).  By delaying their mail, you give the
> blacklists time to receive reports, and then Spamassassin can kill
> them when it checks those blacklists.
>
>   
How many legal email have You received from these not know spammers yet? 
I presume none. Usually, if they appear in blacklist, their 
administrator tries soon to release from there. It can take less time 
then greylisting them in Your server.

-- 
Mart

On Sat, Apr 19, 2008 at 8:10 AM, Mart Pirita <sysadmin@...> wrote:
> Tere.
>  >
>  > Because with the MTA you can only block mail with RBL:s. With
>  > greylisting you can use RBL:s to block *or* to have a longer
>  > greylisting period for example. Also much of the access rules of
>  > Sendmail can be done in milter-greylist with much more ease.
>  >
>  >
>  Why waste bandwidth, cpu, ram etc for spammers instead of rejecting them
>  in first attempt (MTA)? Ok, You give them longer greylisting period. How
>  many who did pass, it was not sending spam? Yes milter-greylist is much
>  easier to config, but this doesn't actually mean, that You have to
>  config it all the time:).

Since I tempfail them before the DATA phase I don't really waste much.
I haven't found a single RBL I trust to block outright, I regularly
see servers from the largest ISP:s listed and it causes problems even
when scoring in SpamAssassin.

>  And btw, if milter-greylist dies (it happens sometimes) and You are away
>  from computer for long time, all spam passes true?

Not really an issue when on Solaris 10 with its Service Management
Facility that takes care of things.

-- 
/peter

I wrote:
>> However, you would never want to block them outright, since
>> they're not known spammers (yet).  By delaying their mail, you
>> give the blacklists time to receive reports, and then
>> Spamassassin can kill them when it checks those blacklists.

Mart Pirita wrote:
> How many legal email have You received from these not know spammers
> yet? I presume none. Usually, if they appear in blacklist, their 
> administrator tries soon to release from there. It can take less
> time then greylisting them in Your server.

My script extrapolates data, creating a buffer around spammer IPs.
The IP blocks are not *entirely* composed of blacklisted addresses,
therefore this is not a blacklist.  I'm not going to block somebody
because they are in China or Poland and their neighbors are spammers.

Plus, as shuttlebox/peter points out, the time/resources it requires
to greylist is roughly equivalent to a rejection, which I typically
regard as a free operation.  It's running through spamassassin and
clamav that consume resources.