Milter-greylist

After a couple of days, this seems to be the magic bullet.  I hope it lasts.

The delay reported on the message below appears to be over 23 hours but 
calculating from the received headers, the real delay was only about an 
hour and that was internally at gmail.


-- 
Alan

---------- Forwarded message ----------
Return-Path: <alanclifford@...>
Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.202])
     by nard.clifford.ac (8.12.11/8.12.11) with ESMTP id j2B32JX4025112
     for <alan@...>; Fri, 11 Mar 2005 03:02:19 GMT
Received: by wproxy.gmail.com with SMTP id 69so774280wri
         for <alan@...>; Thu, 10 Mar 2005 19:02:18 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
         s=beta; d=gmail.com;
         h=received:message-id:date:from:reply-to:to:subject:mime-version:content
     -type:content-transfer-encoding;
         b=NOn83pHXygSEaoRmR/hBg8BR/otbR0vWYyBkX5/c6g3lKQ9gZIT3ivuepO5E7hg5BGLN/z
     lsnP7RwMGAb6XH8wLD4grqW2sTqLFHgEKL73annnPAOMh9RKGROH+ZPVOSl2bzKvopf3Pe5Rv/HY
     p7ABN9gUuYMXE/oKxY+vX7HoI=
Received: by 10.54.4.30 with SMTP id 30mr357563wrd;
         Thu, 10 Mar 2005 18:02:16 -0800 (PST)
Received: by 10.54.29.3 with HTTP; Thu, 10 Mar 2005 18:02:15 -0800 (PST)
Message-ID: <83e6414a0503101802409e7d07@...>
Date: Fri, 11 Mar 2005 02:02:15 +0000
From: alan clifford <alanclifford@...>
Reply-To: alan clifford <alanclifford@...>
To: alan@...
Subject: grey list test 6
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Greylist: Delayed for 23:40:36 by milter-greylist-1.6 (nard.clifford.ac
     [81.187.211.34]); Fri, 11 Mar 2005 03:02:19 +0000 (GMT)
X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on nard.clifford.ac
X-Spam-Level: X-Spam-Status: No, score=-2.4 required=10.0 
tests=AWL,BAYES_00,RCVD_BY_IP,
     SPF_HELO_PASS,SPF_PASS autolearn=ham version=3.0.2
X-MARP-Status: checked (25115)

test

Alan Clifford <lists@...> wrote:

> After a couple of days, this seems to be the magic bullet.  I hope it lasts.
> 
> The delay reported on the message below appears to be over 23 hours but
> calculating from the received headers, the real delay was only about an
> hour and that was internally at gmail.

Usually, this is caused by that: 
Mail A arrives from (IP, from, rcpt) at time t
Mail B arrives from (IP, from, rcpt) at time t+23  
Mail A arrives from (IP, from, rcpt) at time t+24
 
B is tagged as being 23 hours late. 
A is tagged as being not delayed

There is nothing we can do about that. Except maybe adding the
Message-Id in the tuple?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Sat, 12 Mar 2005 manu@... wrote:

> Usually, this is caused by that:
> Mail A arrives from (IP, from, rcpt) at time t
> Mail B arrives from (IP, from, rcpt) at time t+23
> Mail A arrives from (IP, from, rcpt) at time t+24
>
> B is tagged as being 23 hours late.
> A is tagged as being not delayed
>
> There is nothing we can do about that. Except maybe adding the
> Message-Id in the tuple?
>

Thanks for the explanation.  I wouldn't like to comment on the message-id 
proposal as I haven't had much experience using greylisting yet.  Or maybe 
I will.  The example came from gmail which uses lots of ip addresses.  If 
one put in the message-id, gmail's cycle through many ip numbers would 
have to be done for each email.  Would that be a good thing?


-- 
Alan

( Please do not email me AS WELL as replying to the list.  Please
   address personal email to alan+1@ as lists@ is not read. A
   password autoresponder may be invoked if this email is very old. )

Alan Clifford <lists@...> wrote:

> Thanks for the explanation.  I wouldn't like to comment on the message-id
> proposal as I haven't had much experience using greylisting yet.  Or maybe
> I will.  The example came from gmail which uses lots of ip addresses.  If
> one put in the message-id, gmail's cycle through many ip numbers would
> have to be done for each email.  Would that be a good thing?

The right way of dealing with mail farms is to whitelist them. A message
coming from Google pool will always hit your mailbox, regardless if it
is spam or not. greylisting buys you nothing in this situation and it
causes delays.

-- 
Emmanuel Dreyfus
Un bouquin en français sur BSD:
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...

On Sat, 12 Mar 2005 manu@... wrote:

>
> Alan Clifford <lists@...> wrote:
>
>> Thanks for the explanation.  I wouldn't like to comment on the message-id
>> proposal as I haven't had much experience using greylisting yet.  Or maybe
>> I will.  The example came from gmail which uses lots of ip addresses.  If
>> one put in the message-id, gmail's cycle through many ip numbers would
>> have to be done for each email.  Would that be a good thing?
>
> The right way of dealing with mail farms is to whitelist them. A message
> coming from Google pool will always hit your mailbox, regardless if it
> is spam or not. greylisting buys you nothing in this situation and it
> causes delays.
>

That moves the subject away from the original subject but no matter - it 
is interesting.  It does seem sensible but I wouldn't want to get into 
maintaining a list of ip's by hand.  So could this be done automatically? 
I guess it would be too liberal to white list an ip number on the basis of 
of one tuple being successful after one temporary reject.  But what about 
if, for example, an ip a.b.c.d were whitelisted for 'm' days if 'n' 
different tuples with that ip were successfully whitelisted?

-- 
Alan


( Please do not email me AS WELL as replying to the list.  Please
   address personal email to alan+1@ as lists@ is not read. A
   password autoresponder may be invoked if this email is very old. )

Alan,

> is interesting.  It does seem sensible but I wouldn't want to get into 
> maintaining a list of ip's by hand.  So could this be done automatically? 

I currently maintain a whitelist for my own use, a couple of other 
people use it and provide info for it.

http://www.ntjl.net/whitelist/

It's fairly straight forward to write something up to work with whatever 
version/install you have.

I'm happy to maintain the list, if anyone has any ranges they have good 
reason to whitelist please email the address listed on the site.

Regards,

Joseph

Joseph Burford <joseph@...> wrote:

> I currently maintain a whitelist for my own use, a couple of other 
> people use it and provide info for it.
> 
> http://www.ntjl.net/whitelist/

Centralized whitelist would be a nice feature. I see two ways of
implementing that in milte-rgreylist:

1) URL inclusion: you'd say in greylist.conf that you want to include
config from an URL, with a refresh time:
include http://www.example.net/whitelist.txt refresh 1d

2) DNS based: you'd tell in greylist.conf that hosts in a given DNS
revserse list should be whitelisted.

acl whitelist dnsrbl whitelist.example.com

3) Don't add any code to milter-greylist, and have a cron launched
shell-script that fetches Joseph's file and rebuild a config file from
it. 

Solution 1 would require linking with libcurl. It would have other
advantages than centralized whitelisting: centralized configuration for
MX pools. But that can also be done externally by a cron task that just
load the file with wget or ftp and overwrite the config file. I'm not
sure it's worth implenting it. OTOH inclusion of local files is usefull
too.

Solution 2 is nice because it can also be used for greylisting hosts
that appear in various DNSRBL (dial-up DNSRBL for instance). 

Of course both can be done. As usual, IMO that's good features, but
that's not features I need, so if someone wants to work on it,
contributions are welcome.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Tue, Mar 15, 2005 at 07:16:45AM +0100, Emmanuel Dreyfus wrote:
> 3) Don't add any code to milter-greylist, and have a cron launched
> shell-script that fetches Joseph's file and rebuild a config file from
> it. 

4) Add a "include" feature to the configuration parser which would make
   3) a lot easier because one could simply overwrite the file with
   the automatically created whitelist.

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

On Tue, Mar 15, 2005 at 11:48:26AM +0000, Matthias Scheler wrote:
> 4) Add a "include" feature to the configuration parser which would make
>    3) a lot easier because one could simply overwrite the file with
>    the automatically created whitelist.

I cat several files to overwrite my config at the moment.

If we do an include option, I think it would be nice to have the 
ability to include remote files. That would help maintaining 
centralized configs

-- 
Emmanuel Dreyfus
manu@...

> Centralized whitelist would be a nice feature. I see two ways of
> implementing that in milte-rgreylist:
> 
> 1) URL inclusion: you'd say in greylist.conf that you want to include
> config from an URL, with a refresh time:
> include http://www.example.net/whitelist.txt refresh 1d

I see this as being a nice feature, to me I would probably prefer 1 over 
the DNS based option 2. Option 4 sounded good as well. Whatever the 
general consensus is :)

I'm happy to keep building and providing the whitelist, I would like to 
automate it where sys admins can register their mail server IP space, 
however at present it hardly seems worth it.

Regards,

Joseph