Milter-greylist

I have received little response from the author on this list, 
although I greatly appreciate the quality of his code. I suppose the 
French and Americans think "Vous est merde", although I suspect we 
really appreciate each other more than that (I hope).

Anyway, I will continue to offer ideas, however poorly they might be 
received...

milter-greylist can offer invaluable data regarding what the "magic" 
number of delay minutes is before purging a greylist entry. By 
recording the time when an entry enters the the greylist, latest time 
it was accessed, versus those entries that were auto-whitelisted; a 
histogram might be constructed to optimize the greylist timeout 
period. I would assume this would offend no one, and would give each 
user an insight into how long it takes to get spam-bot's to give up 
versus the annoying delay joining new BBS's, etc.

Comments (and flames) are welcomed.

Tom

On Wed, Feb 21, 2007 at 12:00:33AM -0800, Tom Hedges wrote:
> I have received little response from the author on this list, 
> although I greatly appreciate the quality of his code. I suppose the 
> French and Americans think "Vous est merde", although I suspect we 
> really appreciate each other more than that (I hope).

I'm really surprised you could feel that way about our exchanges. While
it's true I could endlessly rant about US administration's behavior
(though I never did that on that list), I never confused US administration 
with US individuals. And I did not even noticed you were an amercian citizen,
anyway. 

> Anyway, I will continue to offer ideas, however poorly they might be 
> received...

They are not poorly received, it's just that I'm working on something else
right now, and I have little time for implementing new stuff. Your idea
makes sense and I think we can have it as a global setting disabled by 
default. Provide a patch against CVS for that (don't forget documentation),
and I will integrate it. If you did it already, then I have let it slipped
through and I own you apologies, just resend it until I handle it. 

Alternatively you can wait I get some free time for doing that work, but 
if you go that way, don't hold your breath on it. 

> milter-greylist can offer invaluable data regarding what the "magic" 
> number of delay minutes is before purging a greylist entry. By 
> recording the time when an entry enters the the greylist, latest time 
> it was accessed, versus those entries that were auto-whitelisted; a 
> histogram might be constructed to optimize the greylist timeout 
> period. I would assume this would offend no one, and would give each 
> user an insight into how long it takes to get spam-bot's to give up 
> versus the annoying delay joining new BBS's, etc.

If I understand correctly what you need, then perhaps the stat feature
can output the information you need:

stat ">>/var/log/greylist.stat" "%S %i %f %r %E %R\n"

For each handled message, you'll get a line with the following substitution:
%S the action (accept, tempfail, reject)
%i sender IP address
%f sender e-mail address
%r recipient e-mail address
%E Elapsed time since first delivery attempt
%R Remaining time before the message gets accepted

See greylist.conf(5) for other possible substitutions. There is a lot of
information you can gather that way.

-- 
Emmanuel Dreyfus
manu@...

> milter-greylist can offer invaluable data regarding what the "magic" 
> number of delay minutes is before purging a greylist entry. By 
> recording the time when an entry enters the the greylist, latest time 
> it was accessed, versus those entries that were auto-whitelisted; a 
> histogram might be constructed to optimize the greylist timeout 
> period. I would assume this would offend no one, and would give each 
> user an insight into how long it takes to get spam-bot's to give up 
> versus the annoying delay joining new BBS's, etc.

Can't you get all this info from the standard maillog output of
milter-greylist-3.0?

Cheers,

	- Joel

ignore this guy he's afs

i am going to try the line
stat ">>/var/log/greylist.stat" "%S %i %f %r %E %R\n"

just so you know the ex) stat line in the readme was printing out %s
(literally) to the log, i am guessing it is typo ? should be %S instead ?? 


 
____________________________________________________________________________________
Never Miss an Email
Stay connected with Yahoo! Mail on your mobile.  Get started!
http://mobile.yahoo.com/services?promote=mail

George Elgin <gelgin@...> wrote:

> ignore this guy he's afs

afs? 

> i am going to try the line
> stat ">>/var/log/greylist.stat" "%S %i %f %r %E %R\n"
> 
> just so you know the ex) stat line in the readme was printing out %s
> (literally) to the log, i am guessing it is typo ? should be %S instead ??

That was a documentation error. The example in README and
greylist.conf(5) showed a %s that does not really exist. The sender
e-mail address is %f. I just fixed it in CVS.

%S is the SMTP status (accept, tempfail, reject)

The list of format string substitutions in greylist.conf(5) was correct.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

what i have found is that many spammers will try something once a day
so if you want real results; presume a real server retry's more often
than that so drop things off the grey list in 

timeout 23h  # default is 5d !?

i only autowhite for 2-3d, but that is my preference. again making
these shorter is  agressive AND it should help milter perform better
by keeping the db smaller

it is pretty easy to build a histogram of acl rules hit using
emanual's %A formatting option something like

cut -d: -f6,7  /var/milter-greylist/milter-greylist.log | grep -c
184:tempfail
325

for me this %A option is marvelous (tnx emanual) a script below might
shed insight. i am NOT sure analysis of %E , %R histograms would give
any more insight, but i would welcome you to prove me wrong...

# %E Elapsed time since first delivery attempt
# %R Remaining time before the message gets accepted


#!/usr/bin/perl -w
# for linenos on the command line print out the acl rule from
greylist.conf
# example usage:
#  cut -d: -f6 /var/milter-greylist/milter-greylist.log | sort -u  | 
xargs -n1 mrule
#
# where
#  stat ">>/var/milter-greylist/milter-greylist.log" "%T{%T}
%i:%f:%r:%A:%S\n"

if ( 0 > $#ARGV ) {
        print STDERR "usage: $0 lineno [...]\n";
        exit;
}

open P, "/etc/mail/greylist.conf" || die "$!";

for $lin ( sort {$a <=> $b}  @ARGV ) {
   while (<P>) {
       $rul = $_;
       if ( $lin == $. ) {
           print "$lin: $rul";
           last;
       }
   }
}


--- In milter-greylist@yahoogroups.com, Tom Hedges <tom@...> wrote:

>
> milter-greylist can offer invaluable data regarding what the "magic" 
> number of delay minutes is before purging a greylist entry. By 
> recording the time when an entry enters the the greylist, latest time 
> it was accessed, versus those entries that were auto-whitelisted; a 
> histogram might be constructed to optimize the greylist timeout 
> period. I would assume this would offend no one, and would give each 
> user an insight into how long it takes to get spam-bot's to give up 
> versus the annoying delay joining new BBS's, etc.
> 
> Comments (and flames) are welcomed.
> 
> Tom
>

george <gelgin@...> wrote:

> what i have found is that many spammers will try something once a day
> so if you want real results; presume a real server retry's more often
> than that so drop things off the grey list in 
> 
> timeout 23h  # default is 5d !?

It's not impossible for a real site to get offline for a whole day. The
23h timeout seems a bit too short to me.

> for me this %A option is marvelous (tnx emanual)

Brought to you by the fstring_expand() function. It took me some time to
write it, but I'm sometimes suprised to discover usages I never thought
about. 

We should run a poll on the stat function: what do people use it for?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

well `tombo` asked for histograms ?

[gelgin@mail5 ~]$ ./hist.pl
    0:accept (  2) *
  123:accept ( 15) **
123:tempfail ( 27) ***
126:tempfail (  3) *
178:tempfail ( 19) **
181:tempfail (  1)
  184:accept ( 53) *****
184:tempfail (325) ******************************
   51:accept ( 92) *********
   86:accept (  1)
   87:accept ( 10) *
 87:tempfail ( 23) **
   91:accept (  1)
 91:tempfail (662)
*************************************************************
 93:tempfail ( 23) **
      <zero>  -10 /

these are my `hits` on ACL lineno (courtesy Text::BarGraph)
here are the ACLs it corresponds to ...

51: acl whitelist list "my network"
86: acl greylist dnsrbl "SCORE" delay 2h
87: acl greylist dnsrbl "MCFADDEN" delay 2h
91: acl greylist dnsrbl "DYNABLOCK" delay 4h
93: acl greylist dnsrbl "DNSBLNJABL" delay 4h
123: acl greylist dnsrbl "SORBS-SPAM" delay 1h
126: acl greylist dnsrbl "SORBS-WEB" delay 1h
178: acl greylist dnsrbl "NOMOREFUN" delay 2h
181: acl greylist dnsrbl "unconfirmed" delay 2h
184: acl greylist sm_macro "maybe_forged" delay 8h

51 is actually decieving. as i discussed (outside group) i use
spamassassin as the backend to greylist. that milter forwards spam
(i use 4.2 cutoff VERY agressive) to a local /dev/null mail drop
which show up as most 51's (a good thing actually)

you say "timeout 23h" too tight ? i agree, but note that even
Yahoo!Mail doesn't honor the traditional 5 dys of retries anymore (
i think 24 hrs and `thats all folks` ) ReturnToSender full stop.
if they get a bounce and are too lazy to resend probably not
something worth reading anyway -;

i would encourage folks to continue along the original thread and
`discover` in an analytic way to the magic autowhite time. 



 
____________________________________________________________________________________
8:00? 8:25? 8:40? Find a flick in no time 
with the Yahoo! Search movie showtime shortcut.
http://tools.search.yahoo.com/shortcuts/#news

George Elgin <gelgin@...> wrote:

> 51: acl whitelist list "my network"
> 86: acl greylist dnsrbl "SCORE" delay 2h
> 87: acl greylist dnsrbl "MCFADDEN" delay 2h
> 91: acl greylist dnsrbl "DYNABLOCK" delay 4h
> 93: acl greylist dnsrbl "DNSBLNJABL" delay 4h
> 123: acl greylist dnsrbl "SORBS-SPAM" delay 1h
> 126: acl greylist dnsrbl "SORBS-WEB" delay 1h
> 178: acl greylist dnsrbl "NOMOREFUN" delay 2h
> 181: acl greylist dnsrbl "unconfirmed" delay 2h
> 184: acl greylist sm_macro "maybe_forged" delay 8h

I'm interested by the dnsrbl definitions. 

It seems there is a missing data in order to discover what rule is
efficient. For instance, acl 184 gets 53/325 accept/tempfail, which
suggests a high rate of spam. But it could be a few legitimate messages
that are often retried. 

Do you think it would buy us something to keep record the the number of
delivery attempts before a message gets accepted?


-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

According to manu@...:
> Do you think it would buy us something to keep record the the number of
> delivery attempts before a message gets accepted?

I've coded that a while ago in 2.* (but lost it when I recently upgraded to
3.1.5+cvs).

My patch added an additionnal field in the db that was reported in the logs.
I was useful to fine tune the delays. I was also able to classify the
botnets based on their retry mechanisms.

I never posted the patch as it made the db backward incompatible
and I know you hate that.

/Fabien

> > Do you think it would buy us something to keep record the the
number of
> > delivery attempts before a message gets accepted?
> 
> I've coded that a while ago in 2.* (but lost it when I recently
upgraded to
> 3.1.5+cvs).


actually if stat is working right shouldn't i see repeats of the same
ip with tempfail, tempfail ... [maybe accept ?]

well i am 

[gelgin@mail5 ~]$ cut -c10- /var/milter-greylist/milter-greylist.log |
cut -d: -f 1,5 | sort | uniq -c | sort -n | tail -10
      7 71.16.122.51:tempfail
      8 64.235.57.104:tempfail
      9 124.82.8.129:tempfail
      9 64.235.57.101:tempfail
      9 64.235.57.102:tempfail
     10 83.9.88.7:tempfail
     13 204.15.59.195:tempfail
     15 64.235.57.106:tempfail
     92 127.0.0.1:accept
    535 83.237.213.139:tempfail

then of course one can do this kind of analysis:

http://www.completewhois.com/cgi2/rbl_lookup.cgi?query=64.235.57.106&display=webtable

hm... time to start using spews again -;

> > 91: acl greylist dnsrbl "DYNABLOCK" delay 4h
> > 184: acl greylist sm_macro "maybe_forged" delay 8h

funny how these are some 90% of MY acl hits :|

i forget why i use 184 i think it came from your man page ?
i am guessing it comes via this

[gelgin@mail5 mail]$ grep -i forge sendmail.cf
R<FORGED>               $#error $@ 5.7.1 $: "550 Relaying denied. IP
name possibly forged " $&{client_name}

oops do i have a config bug ::|

> 
> I'm interested by the dnsrbl definitions. 

i could send you the list i am sure mine is the BIGGEST.

> 
> It seems there is a missing data in order to discover what rule is
> efficient. For instance, acl 184 gets 53/325 accept/tempfail, which
> suggests a high rate of spam. But it could be a few legitimate messages
> that are often retried.

i shouldn't say this publicly but i set my grey time REALLY small
if someone is trying more than 1/15m they are 91% (ah... j'aim those
spam assassin heuristics) probably spammer.
 
> 
> Do you think it would buy us something to keep record the the number of
> delivery attempts before a message gets accepted?
> 

not really since a key composed of %f%i%S already has this info
besides it might NEVER get accepted (which is why false +s are 
such a problem with SORBS).