Milter-greylist

I have a system with about 200K e-mails and at least 700K rejected connections a day. I dump the DB every 48h, but I have also made the decision not retain tuples for more than 5h since imo a server that have not returned after 5 hours should again be delayed. This decreased my db-dumps from about 2GB down to 2-300MB also all problems related to db-dump disappeared. Also I had to increase the milter-timeout. With the default value (5 sec if not misstaken) I notices about 10% of the total greylist-lookups failed. After increasing the value to 60 sec (reccomended by someone in the list) I have only seen a few timeout errors.

Rgds Jonas Israelsson

Eduardo Casarero (03/19/2008 16:27):

>Hi, i'd like to comment some experiences we detected today with
>milter-greylist in dumping db activities.
>
>In big systems (we have more than 10 servers that handles 80k of emails
>and rejects 300k of connections) we notice that the dump needs lots of
>resources, but that is not the only issue the problem is that during the
>dump process all sendmail traffic get queued in the box without
>greylisting, we see in the log lots of this message.
>
>Mar 19 09:12:49 XXXXXXX sm-mta[765]: m2JECaMo000765: Milter (greylist):
>timeout before data read
>Mar 19 09:12:49 XXXXXXX sm-mta[765]: m2JECaMo000765: Milter (greylist):
>to error state
>
>So a lot of traffic gets queued generating a quick grow of our inbound
>mqueue.
>
>We solved this increasing dumpfreq to 1 day so in case of crash we dont
>lose too much, also we enable dump_no_time_translation to do a fastest
>dump.
>
>Note: our greylist.db size is aprox 250 Mb (milter-greylist v 4.0)
>
>Is this a normal behaviour of milter-greylist not to answer (or answer
>fast) when dumping? is de memory data locked when dumping, so no answers
>can be generated?
>
>This data may help to other admins, if what i'm saying is true can it be
>on the official man page? this gave us several headaches. Also i can
>provide more info if anybody wants.
>
>Regards,
>
>----
>Eduardo Casarero
>Informatica Avanzada SRL
>Email: eduardo.casarero@....ar
>Tel: (054) 011-52353995 int. 207
>

On 20 Mar 2008 12:55:00 +0200, Jonas Israelsson <jonas@...> wrote:
> I have a system with about 200K e-mails and at least 700K rejected connections a day. I dump the DB every 48h, but I have also made the decision not retain tuples for more than 5h since imo a server that have not returned after 5 hours should again be delayed. This decreased my db-dumps from about 2GB down to 2-300MB also all problems related to db-dump disappeared. Also I had to increase the milter-timeout. With the default value (5 sec if not misstaken) I notices about 10% of the total greylist-lookups failed. After increasing the value to 60 sec (reccomended by someone in the list) I have only seen a few timeout errors.

There seems to be different opinions whether to dump frequently (say
10m) or seldom (say once a day or so). Is it a necessity to dump the
whole db from memory to disk every time?

I've read on the list a few times about support for SQL, I guess that
would make it possible to do incremental dumps or maybe
milter-greylist should access the SQL db directly and not use a db in
memory any longer?

Any news on that?

-- 
/peter

shuttlebox <shuttlebox@...> wrote:

> I've read on the list a few times about support for SQL, I guess that
> would make it possible to do incremental dumps or maybe
> milter-greylist should access the SQL db directly and not use a db in
> memory any longer?
> 
> Any news on that?

Nobody has been annoyed enough by dumps to develop a SQL backend.

Note that you can also completely disable dumps. If milter-greylist is
stopped, you loose your database, and mails will just be delayed a bit
longer.
 
-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...