Milter-greylist

> We've implemented milter-greylist at our site last monday and a lot seems to
> be going well. Unfortunately, sometimes the milter dies when the machine
> seems to be on heavy load.

I'm getting the same problem.

I haven't looked at it closely yet, but the presence of files like
-rw-------  1 smmsp  wheel      0 Dec  8 16:14 greylist.db-012920ab

created at the exact moment of the crash make me think it's the database
dump code that's doing it.

I've just set dumpfreq to 0 and turned off time translation, so let's
see if the problem gets better or worse. :)

Cheers,

	- Joel

Joel Reicher wrote:
 > 
 > > We've implemented milter-greylist at our site last monday and a lot seems to
 > > be going well. Unfortunately, sometimes the milter dies when the machine
 > > seems to be on heavy load.
 > 
 > I'm getting the same problem.

That sounds like a resource problem.  Are you sure that
your milter-greylist process doesn't hit any resource
limits?  (i.e. data size, number of file descriptors etc.)

 > I haven't looked at it closely yet, but the presence of files like
 > -rw-------  1 smmsp  wheel      0 Dec  8 16:14 greylist.db-012920ab
 > created at the exact moment of the crash make me think it's the database
 > dump code that's doing it.

When milter-greylist terminates it tries to perform a
final dump of the database, but sometimes it fails under
certain conditions.  I think that's a known issue and
has nothing to do with your actual problem.

 > I've just set dumpfreq to 0 and turned off time translation, so let's
 > see if the problem gets better or worse. :)

I bet it'll still crash when it's running out of resources,
but then it won't try to dump the database.  :-)

Best regards
   Oliver

-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

$ dd if=/dev/urandom of=test.pl count=1
$ file test.pl
test.pl: perl script text executable

> Joel Reicher wrote:
>  > 
>  > > We've implemented milter-greylist at our site last monday and a lot seem
> s to
>  > > be going well. Unfortunately, sometimes the milter dies when the machine
>  > > seems to be on heavy load.
>  > 
>  > I'm getting the same problem.
> 
> That sounds like a resource problem.  Are you sure that
> your milter-greylist process doesn't hit any resource
> limits?  (i.e. data size, number of file descriptors etc.)

No, I'm not sure. I put the limits up to the hard limits, but not beyond.
I don't want to make this unlimited.

The weird thing is that I'm getting a segmentation fault but no
core dump.

I'm trying to get more debugging info at the moment.

Cheers,

	- Joel

Jasper Aukes wrote:
 > Hi Oliver,
 > > > > We've implemented milter-greylist at our site last monday and a lot seems to
 > > > > be going well. Unfortunately, sometimes the milter dies when the machine
 > > > > seems to be on heavy load.
 > > > 
 > > > I'm getting the same problem.
 > > 
 > > That sounds like a resource problem.  Are you sure that
 > > your milter-greylist process doesn't hit any resource
 > > limits?  (i.e. data size, number of file descriptors etc.)
 > 
 > The only limits that are set are these two:
 > 
 > descriptors  1024
 > stacksize    8192 kbytes
 > 
 > Should I increase some numbers here? (Not sure how to debug this)

The stacksize shouldn't be a problem, but the descriptors
might be.  You can use a tool like lsof to check how many
descriptors are currently in use.  If the number approaches
your limit during times of heavy load, then you should
definitely increase it.

(Of course, if milter-greylist crashes in such a situation,
then that's a bug.  It should handle the situation more
gracefully.  But maybe sendmail's milter API makes that
difficult, I don't know.)

Best regards
   Oliver

-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

"We, the unwilling, led by the unknowing,
are doing the impossible for the ungrateful.
We have done so much, for so long, with so little,
we are now qualified to do anything with nothing."
\ufffd \ufffd \ufffd \ufffd -- Mother Teresa

Joel Reicher wrote:
 > The weird thing is that I'm getting a segmentation fault but no
 > core dump.

Usually one of two reaons:
-1- The milter-greylist user doesn't have permission to
    write a core dump to its current working directory.
-2- The core dump size limit is set to 0, effectively
    disabling core dumps.

Some operating systems also have other ways to control core
dumps.  For example, FreeBSD has a number of "sysctl" MIB
values (see "sysctl kern | grep core") which can be used to
enable/disable core dumps globally, and to specify the
location and name of core dumps.

Some programs also prevent proper core dumps themselves by
installing a signal handler for SIGSEGV (which is usually a
bad idea because it makes debugging more difficult, even if
the signal handler then performs a core dump "manually").
But I think milter-greylist doesn't do that, and sendmail's
libmilter only traps SIGPIPE, SIGHUP, SIGTERM and SIGINT.

Best regards
   Oliver


-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

'Instead of asking why a piece of software is using "1970s technology,"
start asking why software is ignoring 30 years of accumulated wisdom.'

Oliver Fromme <olli@...> wrote:

> (Of course, if milter-greylist crashes in such a situation,
> then that's a bug.  It should handle the situation more
> gracefully.  But maybe sendmail's milter API makes that
> difficult, I don't know.)

Well, code review is welcome, but I don't expect you'll find such issues
in milter-greylist itself: memory allocation and file descriptors should
be always checked.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

> Joel Reicher wrote:
>  > The weird thing is that I'm getting a segmentation fault but no
>  > core dump.
> 
> Usually one of two reaons:
> -1- The milter-greylist user doesn't have permission to
>     write a core dump to its current working directory.

Doh. I'm running it as smmsp. Thanks for pointing this out.

Now to change the working directory...

Cheers,

	- Joel