Milter-greylist

aNguyen wrote:
 > Is there a command to force db dump while milter-greylist is
 > running?  I have a script to restart greylist when it takes up
 > over 35% cpu.  It would be nice to have a command to dump the
 > database before killing it.

It does that automatically when you kill it.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart
Any opinions expressed in this message are personal to the author and may
not necessarily reflect the opinions of secnetix GmbH & Co KG in any way.
FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"File names are infinite in length, where infinity is set to 255 characters."
        -- Peter Collinson, "The Unix File System"

On Fri, Feb 23, 2007 at 09:23:21AM +0100, Oliver Fromme wrote:
>  > Is there a command to force db dump while milter-greylist is
>  > running?  I have a script to restart greylist when it takes up
>  > over 35% cpu.  It would be nice to have a command to dump the
>  > database before killing it.
> It does that automatically when you kill it.

On some OSes, that does not work properly. In an ideal world, we could
have milter-greylist listening on a control socket, whith a command line
tool to query and manage the database.

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:
 > Oliver Fromme wrote:
 > > > Is there a command to force db dump while milter-greylist is
 > > > running?  I have a script to restart greylist when it takes up
 > > > over 35% cpu.  It would be nice to have a command to dump the
 > > > database before killing it.
 > > 
 > > It does that automatically when you kill it.
 > 
 > On some OSes, that does not work properly.

Oh, really?  It works perfectly fine for me on FreeBSD 6.
I can't imagine what the problem could be with simply
installing a handler for SIGTERM (and possibly SIGHUP
and other signals) which sets a flag so the program can
perform a controlled shutdown.  I've done such things in
my own programs a million times, and it works on any
POSIX-compliant system.

 > In an ideal world, we could
 > have milter-greylist listening on a control socket, whith a command line
 > tool to query and manage the database.

Yes, that might be a useful addition.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart
Any opinions expressed in this message are personal to the author and may
not necessarily reflect the opinions of secnetix GmbH & Co KG in any way.
FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

cat man du : where Unix geeks go when they die

On Fri, Feb 23, 2007 at 09:44:31AM +0100, Oliver Fromme wrote:
> Oh, really?  It works perfectly fine for me on FreeBSD 6.
> I can't imagine what the problem could be with simply
> installing a handler for SIGTERM (and possibly SIGHUP
> and other signals) which sets a flag so the program can
> perform a controlled shutdown.  I've done such things in
> my own programs a million times, and it works on any
> POSIX-compliant system.

The milter API forbids installing a signal handler. 

-- 
Emmanuel Dreyfus
manu@...

On Fri, Feb 23, 2007 at 01:07:19AM -0800, An.H.Nguyen wrote:
> When I copy the 64bit milter-greylist binary to the other server, it runs for two days without crashing... Unfortunately, it crashes in about an hour on the box that I compiled tthe 64bit libmilter and 64bit milter-greylist binary... I notice that the 64bit milter-greylist runs fine wih the 32bit libmilter on the other box...

I think you are confused. A 32 bit binary cannot link with a 64 bit library.
-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:
 > Oliver Fromme wrote:
 > > Oh, really?  It works perfectly fine for me on FreeBSD 6.
 > > I can't imagine what the problem could be with simply
 > > installing a handler for SIGTERM (and possibly SIGHUP
 > > and other signals) which sets a flag so the program can
 > > perform a controlled shutdown.  I've done such things in
 > > my own programs a million times, and it works on any
 > > POSIX-compliant system.
 > 
 > The milter API forbids installing a signal handler. 

You are right, I've just had a look at the documentation.
That's a serious design flaw in the milter API.  :-(
Brain-damaged, if you ask me.

However, it seems like the xxfi_abort callback is called
when the milter is about to terminate.  Is that correct?
Can't it be used for dumping the DB?

After all, the dumping upon SIGTERM _is_ working fine on
FreeBSD 6, so there must already be a way to do it.
I just wonder why it doesn't work on some other operating
systems ...  What's the problem with those OS?  Do they
have a buggy POSIX threads implementation that doesn't
handle signals correctly?

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart
Any opinions expressed in this message are personal to the author and may
not necessarily reflect the opinions of secnetix GmbH & Co KG in any way.
FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"If you aim the gun at your foot and pull the trigger, it's
UNIX's job to ensure reliable delivery of the bullet to
where you aimed the gun (in this case, Mr. Foot)."
        -- Terry Lambert, FreeBSD-hackers mailing list.

On Fri, Feb 23, 2007 at 01:23:04AM -0800, aNguyen wrote:
> When I check with ldd, milter-greylist does not link to either libmilter.a or libmilter.so ... I wonder if something is missing here. thx

ldd show you dynamic linking. A .a is a static library which is embedded in 
the executable. If ldd does nto shows you libmilter, it means the static
libmilter.a was used at build time.

-- 
Emmanuel Dreyfus
manu@...

On Fri, Feb 23, 2007 at 10:24:57AM +0100, Oliver Fromme wrote:
> You are right, I've just had a look at the documentation.
> That's a serious design flaw in the milter API.  :-(
> Brain-damaged, if you ask me.

Well, I'd just say that there is a missing feature: libmilter
should call a handler in the milter when receiving kill -1 
(for reloading config), or kill -15 (for terminating)

> However, it seems like the xxfi_abort callback is called
> when the milter is about to terminate.  Is that correct?
> Can't it be used for dumping the DB?

Is it called? That'seasy to check, can you give it a try?

> After all, the dumping upon SIGTERM _is_ working fine on
> FreeBSD 6, so there must already be a way to do it.

I use an atexit() hook. It works on some systems, but not all. 
Feedbacks have been mixed.

> I just wonder why it doesn't work on some other operating
> systems ...  What's the problem with those OS?  Do they
> have a buggy POSIX threads implementation that doesn't
> handle signals correctly?

The question is: is the atexit hooks called when the process is killed
by a signal? I'm not sure POSIX says anything about it.

-- 
Emmanuel Dreyfus
manu@...

> I use an atexit() hook. It works on some systems, but not all. 
> Feedbacks have been mixed.
> 
> > I just wonder why it doesn't work on some other operating
> > systems ...  What's the problem with those OS?  Do they
> > have a buggy POSIX threads implementation that doesn't
> > handle signals correctly?
> 
> The question is: is the atexit hooks called when the process is killed
> by a signal? I'm not sure POSIX says anything about it.
>

Emmanuel,

I have submitted a patch earlier
that fixes the missing dump at program exit.
You wrote that you are not going to put into the 3.0 stable release,
but only into the new development releases. Have you done that?

If it is done, I recommend to use the development release
for those who is having the missing dump problem.

For everybody, just read my previous postings
on the semantics of the asynchronous signal
delivery mechanism to the POSIX threads.
I am not going to repeat myself here.

Bests,
Attila

On Fri, Feb 23, 2007 at 10:59:36AM +0100, attila.bruncsak@... wrote:
> I have submitted a patch earlier
> that fixes the missing dump at program exit.
> You wrote that you are not going to put into the 3.0 stable release,
> but only into the new development releases. Have you done that?

I hope I did, but it may have sliped through. ChaneLog does not says
about it. Can you check 3.1.6 to see if your code was included, and if it 
was not, can you repost it?

Sorry if I missed it.

-- 
Emmanuel Dreyfus
manu@...

> > I have submitted a patch earlier
> > that fixes the missing dump at program exit.
> > You wrote that you are not going to put into the 3.0 stable release,
> > but only into the new development releases. Have you done that?
> 
> I hope I did, but it may have sliped through. ChaneLog does not says
> about it. Can you check 3.1.6 to see if your code was 
> included, and if it 
> was not, can you repost it?
> 
> Sorry if I missed it.
> 

You did not missed it, the patch is in it.
I did not verified for this release,
but if there is no other glitch it should work.

Emmanuel Dreyfus wrote:
 > Oliver Fromme wrote:
 > > However, it seems like the xxfi_abort callback is called
 > > when the milter is about to terminate.  Is that correct?
 > > Can't it be used for dumping the DB?
 > 
 > Is it called? That'seasy to check, can you give it a try?

According to the source (signal.c and engine.c), that call-
back is called upon SIGINT, but not on SIGTERM or SIGHUP.
I don't understand why.  Additionally, the callback is only
called during a mail transaction.  So it probably cannot be
used for reliably dumping the DB.  :-(

 > > After all, the dumping upon SIGTERM _is_ working fine on
 > > FreeBSD 6, so there must already be a way to do it.
 > 
 > I use an atexit() hook. It works on some systems, but not all. 
 > Feedbacks have been mixed.
 > 
 > > I just wonder why it doesn't work on some other operating
 > > systems ...  What's the problem with those OS?  Do they
 > > have a buggy POSIX threads implementation that doesn't
 > > handle signals correctly?
 > 
 > The question is: is the atexit hooks called when the process is killed
 > by a signal? I'm not sure POSIX says anything about it.

Let me quote:

"The atexit() function shall register the function [...] to
be called [...] at normal program termination.  At normal
program termination, all functions registered by the atexit()
function shall be called [...].  Normal termination occurs
either by a call to exit() or a return from main()."

Since libmilter catches the signal and performs a controlled
shutdown, the process is terminated normally by return from
main().  Therefore the atexit() function should be executed.

In fact, it should be sufficient to simply put the final
dump function after the call to smfi_main(), i.e. before
returning normally from main().  I don't think it is
necessary to use atexit() at all.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart
Any opinions expressed in this message are personal to the author and may
not necessarily reflect the opinions of secnetix GmbH & Co KG in any way.
FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"FreeBSD is Yoda, Linux is Luke Skywalker"
        -- Daniel C. Sobral

On Fri, Feb 23, 2007 at 11:25:04AM +0100, Oliver Fromme wrote:
> "The atexit() function shall register the function [...] to
> be called [...] at normal program termination.  At normal
> program termination, all functions registered by the atexit()
> function shall be called [...].  Normal termination occurs
> either by a call to exit() or a return from main()."

Which means that if you get a SIGSEGV, you loose.

> Since libmilter catches the signal and performs a controlled
> shutdown, the process is terminated normally by return from
> main().  Therefore the atexit() function should be executed.
> 
> In fact, it should be sufficient to simply put the final
> dump function after the call to smfi_main(), i.e. before
> returning normally from main().  I don't think it is
> necessary to use atexit() at all.

I thought smfi_main() was never returning. Does it?

-- 
Emmanuel Dreyfus
manu@...

> > In fact, it should be sufficient to simply put the final
> > dump function after the call to smfi_main(), i.e. before
> > returning normally from main().  I don't think it is
> > necessary to use atexit() at all.
> 
> I thought smfi_main() was never returning. Does it?
> 

Yes it does. RTFM. Absolutely no need for the use of atexit() here.

On Fri, Feb 23, 2007 at 11:32:20AM +0100, attila.bruncsak@... wrote:
> > I thought smfi_main() was never returning. Does it?
> Yes it does. RTFM. Absolutely no need for the use of atexit() here.

I just looked at the code, and we already dump when smfi_main exits.
atexit serves just as a safety net if libmilter calls exit(), but I'm not
sure if it can heppen.

-- 
Emmanuel Dreyfus
manu@...

> I just looked at the code, and we already dump when smfi_main exits.
> atexit serves just as a safety net if libmilter calls exit(), 
> but I'm not
> sure if it can heppen.
> 

I library should never decide to exit on its own, it is supposed to return
error code if it has any trouble. Exiting is the privilege of the main program.
I do not know the libmilter is conforming, but if not it must be fixed.

Emmanuel Dreyfus wrote:
 > Oliver Fromme wrote:
 > > "The atexit() function shall register the function [...] to
 > > be called [...] at normal program termination.  At normal
 > > program termination, all functions registered by the atexit()
 > > function shall be called [...].  Normal termination occurs
 > > either by a call to exit() or a return from main()."
 > 
 > Which means that if you get a SIGSEGV, you loose.

Right, but you shouldn't get a SIGSEGV during normal
operation, because that would be a bug, wouldn't it.  ;-)

And you _can_ install your own signal handler for SIGSEGV.
libmilter doesn't forbid that.  It only ever touches the
signals SIGPIPE, SIGHUP, SIGTERM and SIGINT.

 > > Since libmilter catches the signal and performs a controlled
 > > shutdown, the process is terminated normally by return from
 > > main().  Therefore the atexit() function should be executed.
 > > 
 > > In fact, it should be sufficient to simply put the final
 > > dump function after the call to smfi_main(), i.e. before
 > > returning normally from main().  I don't think it is
 > > necessary to use atexit() at all.
 > 
 > I thought smfi_main() was never returning. Does it?

Yes it does, and it even returns a value:  MI_SUCCESS or
MI_FAILURE (if the network initialization failed).  Note
that, if it is stopped because of a signal, it still
returns MI_SUCCESS.

I.e. if your maqin() function ends like this, you should
be fine:

        int result;

        ...

        result = smfi_main();
        final_dump();
        if (result != MI_SUCCESS) {
                /* Maybe log an error message here. */
                return 1;
        }
        /* Maybe log successful termination here. */
        return 0;   /* success */
}

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart
Any opinions expressed in this message are personal to the author and may
not necessarily reflect the opinions of secnetix GmbH & Co KG in any way.
FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"Unix gives you just enough rope to hang yourself --
and then a couple of more feet, just to be sure."
        -- Eric Allman

Emmanuel Dreyfus wrote:
 > attila.bruncsak@... wrote:
 > > > I thought smfi_main() was never returning. Does it?
 > > Yes it does. RTFM. Absolutely no need for the use of atexit() here.
 > 
 > I just looked at the code, and we already dump when smfi_main exits.
 > atexit serves just as a safety net if libmilter calls exit(), but I'm not
 > sure if it can heppen.

There are no calls to exit() or _exit() in the libmilter
sources.  Easy to check with "grep -w".  If there were any,
that would be a serious bug.  A library may never take
away process termination control from the application.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart
Any opinions expressed in this message are personal to the author and may
not necessarily reflect the opinions of secnetix GmbH & Co KG in any way.
FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"UNIX was not designed to stop you from doing stupid things,
because that would also stop you from doing clever things."
        -- Doug Gwyn

>  > > "The atexit() function shall register the function [...] to
>  > > be called [...] at normal program termination.  At normal
>  > > program termination, all functions registered by the atexit()
>  > > function shall be called [...].  Normal termination occurs
>  > > either by a call to exit() or a return from main()."
>  > 
>  > Which means that if you get a SIGSEGV, you loose.
> 
> Right, but you shouldn't get a SIGSEGV during normal
> operation, because that would be a bug, wouldn't it.  ;-)
> 
And probably you do not want to destroy the previous dump
with some junk/garbage you have in the memory.
(since SIGSEGV warns you about something horrible).

On Fri, 23 Feb 2007 01:07:19 -0800 "An.H.Nguyen" <AnNguyen251@...> wrote:

> I have two Sparc Solaris 8. In order to build 64bit milter-greylist, I had to build 64bit libmilter.a and libmilter.so on this box.
> When I copy the 64bit milter-greylist binary to the other server, it runs for two days without crashing... Unfortunately, it crashes in about an hour on the box that I compiled tthe 64bit libmilter and 64bit milter-greylist binary...

Please use a debugger (gdb or some Solaris one), run milter-greylist under it with a -D
switch and post backtrace here when it crashes. You can try the latest development
version too, maybe the bug is fixed there.

Regards,
Nerijus

On Fri, 23 Feb 2007 15:56:02 -0800 "An.H.Nguyen" <AnNguyen251@...> wrote:

> I downloaded 3.1.6 and now it dumps correctly when it is terminated on Solaris 8. But it still crashes every hour.
> I use mimedefang and spamassassin. What's the correct way to start milter-greylist in gdb while starting mimedefang/sendmail?

Hmm, I told you already in a private mail, didn't I?
cd /where milter-greylist is
gdb ./milter-greylist
r -P /var/milter-greylist/milter-greylist.pid -p /var/milter-greylist/milter-greylist.sock -D   (it's an example,
adjust for your options)
wait until it crashes
bt
post output to the list.

Regards,
Nerijus

An.H.Nguyen <AnNguyen251@...> wrote:

> I downloaded 3.1.6 and now it dumps correctly when it is terminated on
> Solaris 8. But it still crashes every hour.

That can probably be fixed, but I need a backtrace (see Nerijus
instructions)

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Tue, 27 Feb 2007 01:04:45 -0800 "An.H.Nguyen" <AnNguyen251@...> wrote:

> [2]+  Stopped                 gdb ./milter-greylist 4477

Does command fg 2 bring it back?

Regards,
Nerijus

On Tue, 27 Feb 2007 01:29:33 -0800 "An.H.Nguyen" <AnNguyen251@...> wrote:

> Here's the output for another try. milter-greylist does not crash yet so I am unable to get bt when it actually crashes...
> 
> bash-2.03# gdb ./milter-greylist 5885

> (gdb) bt
> #0  0xff19da3c in _poll () from /usr/lib/libc.so.1
> #1  0xff14d618 in select () from /usr/lib/libc.so.1
> #2  0xff0dab30 in select () from /usr/lib/libthread.so.1
> #3  0xff2e4ae0 in smfi_main () from /usr/lib/libmilter.so
> (gdb)

Type c (continue), wait until it crashes and then post bt output.

Regards,
Nerijus