Milter-greylist

Hi,

I've been using 2.0.* for quite a while now. It's efficient but yet, I'd like
to propose something.

If I look at the logs of my MXs (10k+ avg incoming emails per day), I can
easily identify two types of heavy spam tools. The 1st one is blocked by
the grey list feature, the 2nd retries then end-up auto-whitelisted.
Problem is, it fills up the database with hundreds of tuples (in a couple
of seconds), with no added values (as we know it's the same source of spam,
one entry should be enough).

Basically, an ip addr sends 30 to 200 emails in a row.
It's often a dozen of "from" mapped to a list of "to" (rcpts in my domain).
Some of those "to" are valid, some are not. From those that are not, I can
identify some emails wrongly harvested from usenet, mailing lists or
web pages. My users often use fake emails as spam protection. I prefer
distinct pseudo-emails per usage (as for this email) that I drop when they get
spammed (and I can trace where they've been harvested/sold).
Anyway, I'd like to propose an auto-blacklist mechanism.

I see that you've just introduced blacklists in the devel branch.
What I would like to have is not blacklist but auto-blacklist.

Something like that:

I define a rule saying that if an email is sent to someuser@...
(the honey pot), and if the IP addr of the sender is not whitelisted (*),
then:
- the IP addr is immediately auto-blacklisted for a certain amount of time,
even if the recipient or the from are whitelisted
- all entries/tuples from the DB matching this IP are immediately
removed, both the greylisted ones and the auto-whitelisted ones.

(*) this is useful if a relay or a secondary MX already accepted the emails,
then it's no longer possible to distinguish spam from real emails that way.

The auto-blacklist timer should be reset after each match.

In my case, that would remove ~50% of the (bogus) whitelist and ~90% of the
greylist.
That's far less work that manually blacklisting the ip addresses and
thanks to the greylist, that should catch 100% of this kind of spam.

It would be nice to have a counter for the number of emails received by
the auto-blacklisted IP addrs, but that's the cherry on the shortcake.

Thoughts ?
Can you see a flaw ?

/Fabien

Fabien Tassin <fta+miltergreylist@...> wrote:

> Thoughts ?
> Can you see a flaw ?

What is the advantage of doing this inside milter-greylist instead of
using other existing tools? I'm all in favor of improving this software,
but I wouldn't want to see it killed by bloat...

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

According to manu@...:
> 
> > Thoughts ?
> > Can you see a flaw ?
> 
> What is the advantage of doing this inside milter-greylist instead of
> using other existing tools? I'm all in favor of improving this software,
> but I wouldn't want to see it killed by bloat...

I agree that bloat is no good but here, I don't think it is.

Well, the point is to stop spam as early as possible, right?
Greylisting is good but it has to store all tuples and then treats all emails
separately. I mean, spammers make sure the tuples are always different
so you have to store everything to be effective. honeypots are no good
here as you have no way to kill everything else coming from the same source.
By mixing the advantage of grey list on one side and black list of those honey
pots on the other side, you can stop the whole flow coming from an ip that
you don't know beforehand (and that you will probably never see again),
even the emails that came before the one toward the honeypot, just because
they are on hold in the greylist.

If you split the features, let say, grey 1st, then black, you loose the
advantage of grey. Same thing, if you black 1st, grey has no more ways to
take advantage of the honeypots. It has to be at the same level to kill
the whole flow.

Do you get my point?

I don't know how your own db looks like but mine has plenty of consecutive
duplicates for the Sender IP field. Just got a row of 369 emails from the
same sender in a few seconds, all are sitting in the greylist now, with
thousands of others. With the auto-blacklist feature I've described before,
at the 3rd email (that happened to be a honeypot addr), I would have stopped
all that, even if the spammer retry later (no longer matters).

/Fabien

Fabien Tassin <fta+miltergreylist@...> wrote:

> If you split the features, let say, grey 1st, then black, you loose the
> advantage of grey. Same thing, if you black 1st, grey has no more ways to
> take advantage of the honeypots. It has to be at the same level to kill
> the whole flow.

We can immagine that the honeypot is managed by another program that
tells milter-greylist to drop entries from the database. That way you
keep the flexibility of doing anything you want with the honeypot tool,
such as archiving spam messages, communicating with other antispam
tools, and so on.

How to tell milter-greylist? We can just have a small feature addition:

acl blacklist could have a flush statement, which could tell
milter-greylist to drop anything about the IP in its databases.
Something like this:

acl blacklist dnsrbl "local-blacklist" flush

Your honeypotware would just have to feed a local DNSRBL to have
milter-greylist forgetting about the IP's tupples and blacklisting it.

If you don't want to use an external tool, then just do this:

acl blacklist rcpt spamtrap@... flush


I wonder if it is interesting to have a syntax for selecting the flushed
tuples:
 
flush (greylist|autowhite|*) (addr|*) (from|*) (rcpt|*)

Examples:   
flush greylist addr * rcpt 
flush autowhite addr * *

Or if we should just flush everything related to the IP address.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

--- In milter-greylist@yahoogroups.com, manu@... wrote:
>
> Fabien Tassin <fta+miltergreylist@...> wrote:
> 
> > If you split the features, let say, grey 1st, then black, you
loose the
> > advantage of grey. Same thing, if you black 1st, grey has no more
ways to
> > take advantage of the honeypots. It has to be at the same level to
kill
> > the whole flow.
> 
> We can immagine that the honeypot is managed by another program that
> tells milter-greylist to drop entries from the database. That way you
> keep the flexibility of doing anything you want with the honeypot tool,
> such as archiving spam messages, communicating with other antispam
> tools, and so on.
> 
> How to tell milter-greylist? We can just have a small feature addition:
> 
> acl blacklist could have a flush statement, which could tell
> milter-greylist to drop anything about the IP in its databases.
> Something like this:
> 
> acl blacklist dnsrbl "local-blacklist" flush
> 
> Your honeypotware would just have to feed a local DNSRBL to have
> milter-greylist forgetting about the IP's tupples and blacklisting it.
> 

I think this sounds great.
I would feed this from another milter we have that is running
Spamassassin and antivirus from F-Secure.

On Tue, Aug 01, 2006 at 06:24:11AM -0000, strandkjell wrote:
> I think this sounds great.
> I would feed this from another milter we have that is running
> Spamassassin and antivirus from F-Secure.

I started a patch, if someone can start testing it and maybe fixing it...
http://ftp.espci.fr/shadow/manu/flush.patch

-- 
Emmanuel Dreyfus
manu@...

Tere.
>
> I started a patch, if someone can start testing it and maybe fixing it...
> http://ftp.espci.fr/shadow/manu/flush.patch
>   

I'd like to test it, but first I need to compile milter-greylist with 
--enable-dnsrbl and it gives still errors.

-- 
Sysadmin

On Tue, Aug 01, 2006 at 12:10:59PM +0300, Sysadmin wrote:
> > I started a patch, if someone can start testing it and maybe fixing it...
> > http://ftp.espci.fr/shadow/manu/flush.patch
> I'd like to test it, but first I need to compile milter-greylist with 
> --enable-dnsrbl and it gives still errors.

You should be able to build with this patch and without --enable-dnsrbl

-- 
Emmanuel Dreyfus
manu@...

Tere.
> On Tue, Aug 01, 2006 at 12:10:59PM +0300, Sysadmin wrote:
>   
>>> I started a patch, if someone can start testing it and maybe fixing it...
>>> http://ftp.espci.fr/shadow/manu/flush.patch
>>>       
>> I'd like to test it, but first I need to compile milter-greylist with 
>> --enable-dnsrbl and it gives still errors.
>>     
>
> You should be able to build with this patch and without --enable-dnsrbl
>
>   
Ok, I'll give a try. Btw, I found in server logs interesting error, 
seems someone already is using this kind of software:

mail.kristjanjarvi.com.: SMTP DATA-2 protocol error: 500 Message 
rejected because it contains links to an IP address that is blacklisted


-- 
Sysadmin

Sysadmin wrote:
> Tere.
>   
>> On Tue, Aug 01, 2006 at 12:10:59PM +0300, Sysadmin wrote:
>>   
>>     
>>>> I started a patch, if someone can start testing it and maybe fixing it...
>>>> http://ftp.espci.fr/shadow/manu/flush.patch
>>>>       
>>>>         
>>> I'd like to test it, but first I need to compile milter-greylist with 
>>> --enable-dnsrbl and it gives still errors.
>>>     
>>>       
>> You should be able to build with this patch and without --enable-dnsrbl
>>
>>   
>>     
> Ok, I'll give a try. Btw, I found in server logs interesting error, 
> seems someone already is using this kind of software:
>
> mail.kristjanjarvi.com.: SMTP DATA-2 protocol error: 500 Message 
> rejected because it contains links to an IP address that is blacklisted
>   

It appears to be a commercial product for Windows -- BitDefender Mail
Protection for Enterprises (licensed according to number of users and
number of years you want it -- gobs of $,\ufffd,\ufffd....) The web connection to
kristjanjarvi certainly appears to be Windows.


---------------

Chris Hoogendyk

-
   O__  ---- Systems Administrator
  c/ /'_ --- Biology & Geology Departments
 (*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst 

<hoogendyk@...>

--------------- 

Erd\ufffds 4