Milter-greylist

Hello,

I figured out that sometimes it takes me ages
to reload the configuration file. The last time it
took 109 seconds. (I am still investigating the
reason, I suspect that it was due to an overloaded
MX peer. The system which was loading the new config file
wasn't under heavy load.)

The problem is that during the long reload time a lot
of sendmail processes went into error state timing out
on the milter-greylist.

The patch I sent months ago are blocking all the threads
during the config reload. I am attaching now an other patch
which fixes this behaviour. One thread will be used to load
the config file. The other threads may go ahead returning
immediately from conf_update() and use the old in memory 
configuration during the new config load. Actually it would be
possible to have more than one config load to run simultaneously if the
config load takes too long and the config file has been modified
again during this time. This would be a problem due to some unprotected
external
variable use from multiple threads. To avoid this I introduced a new
variable
to limit the number of threads to one loading the new config file.

I suspect that the problem with SIGSEGV (see comments in the code)
may be rather related to multiple simultaneous config loads.

I tried to minimize the critical code section as much as I could.
The lock protects now the access of the conffile_modified.tv_sec
and the numb_of_conf_update_threads variables instead of the full
conf_update() function.
The conffile_modified.tv_sec variable is going to be only initialized
for the first time from the conf_load(),
not as before modified on each new config reload in conf_load().

I have tested the patch it is working fine to me.
It should be applied to the 1.5.10 release.
Probably the problem is not so big that it would require
an immediate new release.

Bests,
Attila

> I figured out that sometimes it takes me ages
> to reload the configuration file. The last time it
> took 109 seconds. (I am still investigating the
> reason, I suspect that it was due to an overloaded
> MX peer. The system which was loading the new config file
> wasn't under heavy load.)

What takes so long? Do we have DNS resolutions at config file reload
time?
 
> The problem is that during the long reload time a lot
> of sendmail processes went into error state timing out
> on the milter-greylist.
> 
> The patch I sent months ago are blocking all the threads during the config
> reload. I am attaching now an other patch which fixes this behaviour. One
> thread will be used to load the config file. The other threads may go
> ahead returning immediately from conf_update() and use the old in memory
> configuration during the new config load.

That's nice. This could be improved a bit by letting the thread that
calls conf_update return and use the old config. It can be done easily
by tweaking conf_load:
- Remove pthread_join
- Handle a structure containing the attribute and the config file stream
to the spawned thread that reloads the config, and free the attribute
and close the config file from that thread. 

What do you think?

> Actually it would be possible to
> have more than one config load to run simultaneously if the config load
> takes too long and the config file has been modified again during this
> time. This would be a problem due to some unprotected external variable
> use from multiple threads. To avoid this I introduced a new variable to
> limit the number of threads to one loading the new config file.
> 
> I suspect that the problem with SIGSEGV (see comments in the code)
> may be rather related to multiple simultaneous config loads.

No I tracked it down on a test machine, with no activity at all except
what I produced. The SIGSEGV was produced by a stack overflow. The way
the yacc parser was made caused huge memory usage. This has been fixed,
but we still book 2MB just in case.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...