Milter-greylist

We're running milter-greylist 4.0 out of Sendmail 8.14.2 (we also have a 
number of other tools in the mix) on Solaris 9, SPARC.

Last Friday, I uncommented the line in greylist.conf for the 
"subnetmatch /24". We didn't think we needed to do a restart. Then over 
the weekend, we got lots of complaints about people not getting mail 
from off campus.

On one department's server, milter-greylist crashed early Sunday 
(haven't seen this since we upgraded from a very old version some months 
back). This is the department that has the heaviest load by a good 
factor. It also happens to be the department that is my own primary mail 
server. I got an alert that it wasn't running, and the alert system 
couldn't restart it. So, I restarted it manually. It was later Sunday 
that complaints started accumulating from the other department that they 
hadn't seen any outside mail over Saturday or Sunday. I didn't really 
put it together -- I was getting outside mail and the same changes are 
almost always tracked together on these two systems.

Finally, Monday, my boss restarted sendmail on the other system (in 
Massachusetts it was a 3 day weekend for Patriot's Day :-) ). Anyway, 
then people in that department started saying things were working and 
they were getting pent up mail from the last couple of days. Then I put 
2 and 2 together and guessed that maybe we caused some problems by 
uncommenting the "subnetmatch /24" and not restarting milter-greylist.

I should also note that this 3 day weekend for us showed the highest 
volume of spam traffic attempts of any we have seen. Lots of entries in 
the sendmail logs for various kinds of rejects including connection rate 
limits.

So, from this I have a couple of questions for the list.

1) Am I right in guessing that I needed to restart milter-greylist? Is 
this just proper SOP? Or might there be a bug where it isn't 
straightening out what it has in memory when it gets a change like this 
in the greylist.conf on the fly?

2) Should I continue looking for other causes of our problems? (i.e. is 
the change in subnetmatch just a red herring?) Or is it probable that 
this really was the cause of our problems?

3) My boss is talking like milter-greylist may have outlived its 
usefulness. It does cause headaches for lots of users who don't want 
mail delayed even if it is attributable to the other side being 
misconfigured. And, over the past year or so, it seems like more and 
more spammers have implemented resending (anyone tracking this have 
statistics?). Meanwhile, we have adopted a number of other mechanisms, 
including settings in sendmail.cf (by way of config.mc), that seem to be 
more effective, blocking things before they even get to milter-greylist. 
I know I'm asking the choir to commit blasphemy ;-) , but I thought 
perhaps others on this list might have comments on this.


TIA


---------------

Chris Hoogendyk

-
   O__  ---- Systems Administrator
  c/ /'_ --- Biology & Geology Departments
 (*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst 

<hoogendyk@...>

--------------- 

Erd\ufffds 4

On Tue, Apr 22, 2008 at 5:51 PM, Chris Hoogendyk
<hoogendyk@...> wrote:
> We're running milter-greylist 4.0 out of Sendmail 8.14.2 (we also have a
>  number of other tools in the mix) on Solaris 9, SPARC.
>
>  1) Am I right in guessing that I needed to restart milter-greylist? Is
>  this just proper SOP? Or might there be a bug where it isn't
>  straightening out what it has in memory when it gets a change like this
>  in the greylist.conf on the fly?

I use milter-greylist from Blastwave on Solaris 10 with SMF support so
it takes care of restarting milter-greylist if needed. I also use a
small cron script to check for excessive memory use (it always leaks
memory) and restarts it if needed, use the pmap command if you want to
check the memory use. You can trim overall memory use by looking over
the time you keep tuples and how long you autowhitelist. I only keep
tuples for 1d instead of the default 5d.

http://www.blastwave.org/packages.php/miltergreylist (note that it
runs on Solaris 8 and higher both Sparc and i386)

>  3) My boss is talking like milter-greylist may have outlived its
>  usefulness. It does cause headaches for lots of users who don't want
>  mail delayed even if it is attributable to the other side being
>  misconfigured. And, over the past year or so, it seems like more and
>  more spammers have implemented resending (anyone tracking this have
>  statistics?). Meanwhile, we have adopted a number of other mechanisms,
>  including settings in sendmail.cf (by way of config.mc), that seem to be
>  more effective, blocking things before they even get to milter-greylist.
>  I know I'm asking the choir to commit blasphemy ;-) , but I thought
>  perhaps others on this list might have comments on this.

I also used to get a lot of complaints and requests to statically
whitelist this and that until I started to use dnswl.org, after that
it's a lot smoother. I also greylist hits from SORBS DUL for 3h to
combat resends. Look in the wiki for a few setup examples:
http://milter-greylist.wikidot.com/. I have a lot more filtering after
the greylisting but I can't think of going back to not using it, after
all it reduces the load with something like 80%.

-- 
/peter

3 things:
1. I also saw milter-greylist crashed on Sunday morning. I did not know
what was wrong, and then I noticed log rotating took place at that time
- affecting the log file of milter-greylist. Maybe just a coincidence.... ?
2. Sometimes I also feel like it is loosing its effectiveness so I have
written a script to analyze sendmail logs - it turned out that its
effectiveness is still >85% so yes, I still do receive a spam, but if it
weren't for greylisting.... :-)
3. It also seems to me that it is leaking memory, but I am not sure.
Maybe it is just OS thing - the daemon is doing lots of malloc/free of
small pieces of memory an thus fragmenting the free pool....?
Ondrej

shuttlebox wrote:

>
> On Tue, Apr 22, 2008 at 5:51 PM, Chris Hoogendyk
> <hoogendyk@... <mailto:hoogendyk%40bio.umass.edu>> wrote:
> > We're running milter-greylist 4.0 out of Sendmail 8.14.2 (we also have a
> > number of other tools in the mix) on Solaris 9, SPARC.
> >
> > 1) Am I right in guessing that I needed to restart milter-greylist? Is
> > this just proper SOP? Or might there be a bug where it isn't
> > straightening out what it has in memory when it gets a change like this
> > in the greylist.conf on the fly?
>
> I use milter-greylist from Blastwave on Solaris 10 with SMF support so
> it takes care of restarting milter-greylist if needed. I also use a
> small cron script to check for excessive memory use (it always leaks
> memory) and restarts it if needed, use the pmap command if you want to
> check the memory use. You can trim overall memory use by looking over
> the time you keep tuples and how long you autowhitelist. I only keep
> tuples for 1d instead of the default 5d.
>
> http://www.blastwave.org/packages.php/miltergreylist
> <http://www.blastwave.org/packages.php/miltergreylist> (note that it
> runs on Solaris 8 and higher both Sparc and i386)
>
> > 3) My boss is talking like milter-greylist may have outlived its
> > usefulness. It does cause headaches for lots of users who don't want
> > mail delayed even if it is attributable to the other side being
> > misconfigured. And, over the past year or so, it seems like more and
> > more spammers have implemented resending (anyone tracking this have
> > statistics?). Meanwhile, we have adopted a number of other mechanisms,
> > including settings in sendmail.cf (by way of config.mc), that seem to be
> > more effective, blocking things before they even get to milter-greylist.
> > I know I'm asking the choir to commit blasphemy ;-) , but I thought
> > perhaps others on this list might have comments on this.
>
> I also used to get a lot of complaints and requests to statically
> whitelist this and that until I started to use dnswl.org, after that
> it's a lot smoother. I also greylist hits from SORBS DUL for 3h to
> combat resends. Look in the wiki for a few setup examples:
> http://milter-greylist.wikidot.com/.
> <http://milter-greylist.wikidot.com/.> I have a lot more filtering after
> the greylisting but I can't think of going back to not using it, after
> all it reduces the load with something like 80%.
>
> -- 
> /peter
>
>

On Tue, Apr 22, 2008 at 6:36 PM, Ondrej Valousek <webserv@...> wrote:
>  3. It also seems to me that it is leaking memory, but I am not sure.
>  Maybe it is just OS thing - the daemon is doing lots of malloc/free of
>  small pieces of memory an thus fragmenting the free pool....?

It's not an OS thing in my case, my Solaris servers stay up for years
without problems. Isn't it strange that the database dump on disk is
always 11-12 MB in my case but milter-greylist keeps growing beyond
512 MB in a week or so? It's like Firefox to me, I have just accepted
that it will eat my memory until there's none left. :-)

-- 
/peter

Chris Hoogendyk <hoogendyk@...> wrote:

> 1) Am I right in guessing that I needed to restart milter-greylist? Is
> this just proper SOP? Or might there be a bug where it isn't 
> straightening out what it has in memory when it gets a change like this
> in the greylist.conf on the fly?

You should not need to restart it to get changes taken into account. But
indeed sone change don't affect what you already have in memory.

> 2) Should I continue looking for other causes of our problems? (i.e. is
> the change in subnetmatch just a red herring?) Or is it probable that
> this really was the cause of our problems?

I have trouble to understand why your had difficulties to automatically
restart it.

> 3) My boss is talking like milter-greylist may have outlived its 
> usefulness. It does cause headaches for lots of users who don't want 
> mail delayed even if it is attributable to the other side being 
> misconfigured. 

You can make greylisting user-configurable. My users have a web form
where they can choose various settings, including no greylisting at all.
It's their spam, after all.

I use a LDAP directory to store the user settings.

> And, over the past year or so, it seems like more and 
> more spammers have implemented resending (anyone tracking this have 
> statistics?).

I use 15mn for the average server and several hours for anyone listed in
a pool of cable/xDSL addresses. It's still very satisfying. 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

manu@... wrote:
> Chris Hoogendyk <hoogendyk@...> wrote:
>   
>> 1) Am I right in guessing that I needed to restart milter-greylist? Is
>> this just proper SOP? Or might there be a bug where it isn't 
>> straightening out what it has in memory when it gets a change like this
>> in the greylist.conf on the fly?
>>     
> You should not need to restart it to get changes taken into account. But
> indeed sone change don't affect what you already have in memory.
>
>   
>> 2) Should I continue looking for other causes of our problems? (i.e. is
>> the change in subnetmatch just a red herring?) Or is it probable that
>> this really was the cause of our problems?
>>     
>
> I have trouble to understand why your had difficulties to automatically
> restart it.
>   

Not milter-greylist's fault at all. We used to have some custom code 
embedded in milter-greylist that checked the popip.db to see if someone 
had checked their mail from that IP, and if so to bypass greylisting. 
This involved adding a call paramater "-b popip.db" to point to the 
database. When we upgraded from the very old 1.6.3 of milter-greylist to 
the newest 4.0, we ditched our custom code. By this time we had 
implemented other procedures that simplified access for our users. 
However, I forgot to remove the "-b" parameter from my greycheck script. 
It had never been invoked since the upgrade on Nov. 28, which speaks 
well for the new milter-greylist. It came up this time, because I messed up.

>> 3) My boss is talking like milter-greylist may have outlived its 
>> usefulness. It does cause headaches for lots of users who don't want 
>> mail delayed even if it is attributable to the other side being 
>> misconfigured. 
>>     
>
> You can make greylisting user-configurable. My users have a web form
> where they can choose various settings, including no greylisting at all.
> It's their spam, after all.
>
> I use a LDAP directory to store the user settings.
>
>   
>> And, over the past year or so, it seems like more and 
>> more spammers have implemented resending (anyone tracking this have 
>> statistics?).
>>     
>
> I use 15mn for the average server and several hours for anyone listed in
> a pool of cable/xDSL addresses. It's still very satisfying. 
>   

Well, satisfying is subjective. I was looking for hard statistics, sort 
of like what shuttlebox and Ondrej posted.


---------------

Chris Hoogendyk

-
   O__  ---- Systems Administrator
  c/ /'_ --- Biology & Geology Departments
 (*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst 

<hoogendyk@...>

--------------- 

Erd\ufffds 4

shuttlebox wrote:
> On Tue, Apr 22, 2008 at 5:51 PM, Chris Hoogendyk
> <hoogendyk@...> wrote:
>   
<snip>
>>  3) My boss is talking like milter-greylist may have outlived its
>>  usefulness. It does cause headaches for lots of users who don't want
>>  mail delayed even if it is attributable to the other side being
>>  misconfigured. And, over the past year or so, it seems like more and
>>  more spammers have implemented resending (anyone tracking this have
>>  statistics?). Meanwhile, we have adopted a number of other mechanisms,
>>  including settings in sendmail.cf (by way of config.mc), that seem to be
>>  more effective, blocking things before they even get to milter-greylist.
>>  I know I'm asking the choir to commit blasphemy ;-) , but I thought
>>  perhaps others on this list might have comments on this.
>>     
>
> I also used to get a lot of complaints and requests to statically
> whitelist this and that until I started to use dnswl.org, after that
> it's a lot smoother. I also greylist hits from SORBS DUL for 3h to
> combat resends. Look in the wiki for a few setup examples:
> http://milter-greylist.wikidot.com/. I have a lot more filtering after
> the greylisting but I can't think of going back to not using it, after
> all it reduces the load with something like 80%

AND

Ondrej Valousek wrote:

<snip>
> 2. Sometimes I also feel like it is loosing its effectiveness so I have
> written a script to analyze sendmail logs - it turned out that its
> effectiveness is still >85% so yes, I still do receive a spam, but if it
> weren't for greylisting.... :-)
<snip>

Can either of you give specifics for how you determined those 
percentages and what they mean? Share your script? Have you been 
tracking it over time? And has it changed? Or is it resting on its laurels?

Thanks,



---------------

Chris Hoogendyk

-
   O__  ---- Systems Administrator
  c/ /'_ --- Biology & Geology Departments
 (*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst 

<hoogendyk@...>

--------------- 

Erd\ufffds 4

Ondrej Valousek wrote:

<snip>
> 3. It also seems to me that it is leaking memory, but I am not sure.
> Maybe it is just OS thing - the daemon is doing lots of malloc/free of
> small pieces of memory an thus fragmenting the free pool....?
>   
<snip>


AND

shuttlebox replied:
> On Tue, Apr 22, 2008 at 6:36 PM, Ondrej Valousek <webserv@...> wrote:
>   
>>  3. It also seems to me that it is leaking memory, but I am not sure.
>>  Maybe it is just OS thing - the daemon is doing lots of malloc/free of
>>  small pieces of memory an thus fragmenting the free pool....?
>>     
>
> It's not an OS thing in my case, my Solaris servers stay up for years
> without problems. Isn't it strange that the database dump on disk is
> always 11-12 MB in my case but milter-greylist keeps growing beyond
> 512 MB in a week or so? It's like Firefox to me, I have just accepted
> that it will eat my memory until there's none left. :-)
>   

I thought that in previous versions there had been issues, and some 
specifically with Solaris. I recall exchanges on the list going over 
coding specifics with respect to Solaris, and then manu indicating that 
he thought it was fixed in the 4.0 betas. That is at least in part what 
lead me to make the jump. And, as I said in my reply to manu's post on 
this thread, milter-greylist has not gone down (and thereby caused my 
greycheck script to take action) since I upgraded on Nov. 28.

That's not to say it hasn't been restarted. I've upgraded spamassassin 
and sendmail during that time, and we've also been tweaking a lot of 
other things that involve restarting sendmail. But, I no longer have to 
worry about milter-greylist bombing out just out of the blue, so to speak.

So, shuttlebox, are you up on 4.0? Ondrej didn't say whether he was on 
Solaris or not.



---------------

Chris Hoogendyk

-
   O__  ---- Systems Administrator
  c/ /'_ --- Biology & Geology Departments
 (*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst 

<hoogendyk@...>

--------------- 

Erd�s 4

On Wed, Apr 23, 2008 at 9:40 PM, Chris Hoogendyk
<hoogendyk@...> wrote:
>  Can either of you give specifics for how you determined those
>  percentages and what they mean? Share your script? Have you been
>  tracking it over time? And has it changed? Or is it resting on its laurels?

Since a resend will come as a new message it's hard to track absolute
numbers at the greylist host. One can use the last line in the db as
an indicator with the number of greylisted tuples vs autowhitelisted
but the last number is highly affected by your static and dynamic
whitelisting.

I have a setup with two gateways running milter-greylist, they deliver
to another pair of servers running MailScanner with ClamAV and
SpamAssassin. I compare how much incoming connections each pair gets.
When I restart milter-greylist it's also very noticable on the
MailScanner servers, all graphs (Vispan) spike immediately.

-- 
/peter

On Wed, Apr 23, 2008 at 9:50 PM, Chris Hoogendyk
<hoogendyk@...> wrote:
>  I thought that in previous versions there had been issues, and some
>  specifically with Solaris. I recall exchanges on the list going over
>  coding specifics with respect to Solaris, and then manu indicating that
>  he thought it was fixed in the 4.0 betas. That is at least in part what
>  lead me to make the jump. And, as I said in my reply to manu's post on
>  this thread, milter-greylist has not gone down (and thereby caused my
>  greycheck script to take action) since I upgraded on Nov. 28.
>
>  That's not to say it hasn't been restarted. I've upgraded spamassassin
>  and sendmail during that time, and we've also been tweaking a lot of
>  other things that involve restarting sendmail. But, I no longer have to
>  worry about milter-greylist bombing out just out of the blue, so to speak.
>
>  So, shuttlebox, are you up on 4.0? Ondrej didn't say whether he was on
>  Solaris or not.

The issue wasn't with memory leaks but with file handles. A lot of
work was done to combat that and Johann Klasek has written a summary
of it on the wiki.

Yes, I'm on 4.0. I'm actually the Blastwave maintainer of
milter-greylist (and MailScanner and some more packages). I also
started the wiki so I was mainly plugging my own stuff. :-)

-- 
/peter

As long as someone make me a member of the wiki, I ham happy to upload
my script which analyses the sendmail logs and produces some statistics.
The script takes a care of the resends so the results are pretty much
accurate.
Ondrej

shuttlebox wrote:

>
> On Wed, Apr 23, 2008 at 9:40 PM, Chris Hoogendyk
> <hoogendyk@... <mailto:hoogendyk%40bio.umass.edu>> wrote:
> > Can either of you give specifics for how you determined those
> > percentages and what they mean? Share your script? Have you been
> > tracking it over time? And has it changed? Or is it resting on its
> laurels?
>
> Since a resend will come as a new message it's hard to track absolute
> numbers at the greylist host. One can use the last line in the db as
> an indicator with the number of greylisted tuples vs autowhitelisted
> but the last number is highly affected by your static and dynamic
> whitelisting.
>
> I have a setup with two gateways running milter-greylist, they deliver
> to another pair of servers running MailScanner with ClamAV and
> SpamAssassin. I compare how much incoming connections each pair gets.
> When I restart milter-greylist it's also very noticable on the
> MailScanner servers, all graphs (Vispan) spike immediately.
>
> -- 
> /peter
>
>

On Thu, Apr 24, 2008 at 11:38 AM, Ondrej Valousek <webserv@...> wrote:
> As long as someone make me a member of the wiki, I ham happy to upload
>  my script which analyses the sendmail logs and produces some statistics.
>  The script takes a care of the resends so the results are pretty much
>  accurate.
>  Ondrej

Sorry for not noticing your application, the wiki doesn't seem to mail
me a notification when someone applies. You're approved now though.

If you upload your script as a file please make a link to it since the
wiki makes files pretty hard to find otherwise. Thanks for
contributing.

-- 
/peter

File uploaded & link created.
The only unfortunate thing is, that I can not delete it. So if I wanted
to upload some updated file (replacing the old one) in the future,
I would have to bother the site admin :-(
Ondrej

shuttlebox wrote:

>
> On Thu, Apr 24, 2008 at 11:38 AM, Ondrej Valousek <webserv@...
> <mailto:webserv%40s3group.cz>> wrote:
> > As long as someone make me a member of the wiki, I ham happy to upload
> > my script which analyses the sendmail logs and produces some statistics.
> > The script takes a care of the resends so the results are pretty much
> > accurate.
> > Ondrej
>
> Sorry for not noticing your application, the wiki doesn't seem to mail
> me a notification when someone applies. You're approved now though.
>
> If you upload your script as a file please make a link to it since the
> wiki makes files pretty hard to find otherwise. Thanks for
> contributing.
>
> -- 
> /peter
>
>

On Thu, Apr 24, 2008 at 3:01 PM, Ondrej Valousek <webserv@...> wrote:
> File uploaded & link created.
>  The only unfortunate thing is, that I can not delete it. So if I wanted
>  to upload some updated file (replacing the old one) in the future,
>  I would have to bother the site admin :-(
>  Ondrej

I have tried to modify to permissions to allow that but it was
actually for page owners/creators, not for file owners/creators so I
don't know if it had the desired effect. Maybe you could look again
and see if the delete option is there?

-- 
/peter