Milter-greylist

Hi folks!  I have some (heavily loaded) FreeBSD 4.11 boxes on which
milter-greylist 4.0b1 is crashing frequently with a "milter-greylist:
malloc() failed: Cannot allocate memory" error being logged in the
syslog.  Often, but not always, there are
/var/milter-greylist/greylist.db-somethingorother files being left
around.  The crash also happens less frequently on a less loaded
FreeBSD 6.2 box.

Relevant facts:
 * Recently upgraded from 4.0a6 (which also crashed often, but not
quite so much as now)
 * Running mxsync between four servers that contain identical config files
 * Bumped up a couple of sync.h parameters to try to handle things better:
      SYNC_MAXQLEN from 1024 to 8192 (was 4096 in 4.0a6)
      MXGLSYNC_BACKLOG from 5 to 8
 * Collecting lots of tuples, since I whitelist for two weeks:
      acl greylist default delay 4m autowhite 14d msg "Please come
back later."

Thanks in advance for any advice or assistance!

        Stuart
---
Stuart Freedman        stuart@...
Carroll-Net, Inc.      (201)488-1332
905 Main St.           (201)488-5108 (direct)
Hackensack, NJ 07601   (201)488-1093 (FAX)

On Tue, Aug 28, 2007 at 02:11:13PM -0000, Stuart Freedman wrote:
> Hi folks!  I have some (heavily loaded) FreeBSD 4.11 boxes on which
> milter-greylist 4.0b1 is crashing frequently with a "milter-greylist:
> malloc() failed: Cannot allocate memory" error being logged in the
> syslog.  

Raise memory limits (ulimit with sh/ksh/bash, limit with csh/tcsh).

-- 
Emmanuel Dreyfus
manu@...

--- In milter-greylist@yahoogroups.com, Emmanuel Dreyfus <manu@...> wrote:
>
> On Tue, Aug 28, 2007 at 02:11:13PM -0000, Stuart Freedman wrote:
> > Hi folks!  I have some (heavily loaded) FreeBSD 4.11 boxes on which
> > milter-greylist 4.0b1 is crashing frequently with a "milter-greylist:
> > malloc() failed: Cannot allocate memory" error being logged in the
> > syslog.  
> 
> Raise memory limits (ulimit with sh/ksh/bash, limit with csh/tcsh).
> 
> -- 
> Emmanuel Dreyfus
> manu@...
>

I don't think that ulimit is the problem, as the memory limit is
supposedly "unlimited" (both soft & hard)...

Current ulimit –aS output is:
cpu time               (seconds, -t)  unlimited
file size           (512-blocks, -f)  unlimited
data seg size           (kbytes, -d)  524288
stack size              (kbytes, -s)  65536
core file size      (512-blocks, -c)  unlimited
max memory size         (kbytes, -m)  unlimited
locked memory           (kbytes, -l)  unlimited
max user processes              (-u)  5547
open files                      (-n)  24000
virtual mem size        (kbytes, -v)  unlimited
sbsize                   (bytes, -b)  unlimited

Current ulimit –aH output is:
cpu time               (seconds, -t)  unlimited
file size           (512-blocks, -f)  unlimited
data seg size           (kbytes, -d)  524288
stack size              (kbytes, -s)  65536
core file size      (512-blocks, -c)  unlimited
max memory size         (kbytes, -m)  unlimited
locked memory           (kbytes, -l)  unlimited
max user processes              (-u)  5547
open files                      (-n)  24000
virtual mem size        (kbytes, -v)  unlimited
sbsize                   (bytes, -b)  unlimited

One thing I noticed yesterday was that it appeared that mxsync
sessions got disconnected & reconnected a lot.  Is this supposed to
happen?  Could there be an issue related to not freeing resources
before re-initiating such connections?  Should I try to get a core
dump/stack trace?

Thanks again,

        Stuart

Stuart Freedman <stuart@...> wrote:

> One thing I noticed yesterday was that it appeared that mxsync
> sessions got disconnected & reconnected a lot.  Is this supposed to
> happen?  Could there be an issue related to not freeing resources
> before re-initiating such connections?  Should I try to get a core
> dump/stack trace?

You can enhance the debug message you get when malloc fails. Make a
sweep change, adding something such as:

"%s:%d malloc falied: %s", __FILE__, __LINE__, strerror(errno)

That way you'll see if the problem happen at the same place or not. Next
step is to display the size requested by malloc to see if there is a
miscalculation somewhere.
-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Stuart Freedman wrote:
 > I don't think that ulimit is the problem, as the memory limit is
 > supposedly "unlimited" (both soft & hard)...
 > [...]
 > data seg size           (kbytes, -d)  524288

So your DSIZE is limited to 512 MB (hard limit).  How
big does your milter-greylist process get?  It is quite
possible that you're hitting the limit.

If that's the case, try increasing the hard limit (it's
a kernel variable that can be changed statically in the
kernel config, or changed via a loader tunable).

The following two lines in your /boot/loader.conf file
will set both hard and soft limits for the data segment
size to 1 GB:

kern.maxdsiz="1073741824"
kern.dfldsiz="1073741824"

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"Documentation is like sex; when it's good, it's very, very good,
and when it's bad, it's better than nothing."
        -- Dick Brandon

--- In milter-greylist@yahoogroups.com, Oliver Fromme <olli@...> wrote:
> So your DSIZE is limited to 512 MB (hard limit).  How
> big does your milter-greylist process get?  It is quite
> possible that you're hitting the limit.
> 
> If that's the case, try increasing the hard limit (it's
> a kernel variable that can be changed statically in the
> kernel config, or changed via a loader tunable).
> 
> The following two lines in your /boot/loader.conf file
> will set both hard and soft limits for the data segment
> size to 1 GB:
> 
> kern.maxdsiz="1073741824"
> kern.dfldsiz="1073741824"
> 
> Best regards
>    Oliver

I saw the milter-greylist process hanging right around/above the 512MB
value for a while before it would crash.  I have taken your suggestion
re the data size definition, and so far so good on the servers I've
rebooted to take the values.  (The real test will not come 'til the
load starts rising again tomorrow morning, though!)

BTW, after sticking in the additional debug output, my crash error
message now looks like this:
milter-greylist: milter-greylist.c:770 malloc failed: Cannot allocate
memory

This line number corresponds to this chunk of code (770 being the
mg_log function), given the way the line numbers shifted when I stuck
the error messages in the ugly way :-) :

                i++; /* Include the \n in this chunk */
                linelen = priv->priv_buflen + i;
 
                if ((b->b_lines = malloc(linelen + 1)) == NULL) {
/*                      mg_log(LOG_ERR, "malloc() failed: %s",
strerror(errno));
*/
mg_log(LOG_ERR, "%s:%d malloc failed: %s", __FILE__, __LINE__,
strerror(errno));
                        exit(EX_OSERR);
                }

Thank you, and best regards to you both, Oliver & Emmanuel!

        Stuart

Stuart Freedman <stuart@...> wrote:

> This line number corresponds to this chunk of code (770 being the
> mg_log function), given the way the line numbers shifted when I stuck
> the error messages in the ugly way :-) :

Well, feel free to make a cleaner change and to send a patch. The
__FILE__, __LINE__ will be useful for other debuggings.


>                 if ((b->b_lines = malloc(linelen + 1)) == NULL) {
> mg_log(LOG_ERR, "%s:%d malloc failed: %s", __FILE__, __LINE__,
> strerror(errno));

Is it always there? Next step is to display linelen + 1 and see if the
value gets odd.

I see it happens in the code that records the message body. You have
dacl statements? Did you use the maxpeek option to limit the size of
message seen by milter-greylist. If you did not, it's no surprise it may
explode if it is getting multiple big messages at the same time.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...