Milter-greylist

I have a problem with a terminating daemon after a flood via SMTP. I'm
afraid I can not give much debug output but just the mail server log
of this happening (see below). If I can turn on some switches to give
more output please let me know. The system is a debian one running
sendmail 8.14.2-3. After restart the greylist service it is working
again. This happens more often these days, I'm afraid spammer know
this already?

Thank you
Thomas

PS: This software rocks like hell. Please keep going.

Package: milter-greylist
Priority: extra
Section: mail
Installed-Size: 296
Maintainer: Cord Beermann <cord@...>
Architecture: i386
Version: 3.0-3
Depends: libc6 (>= 2.5-5), libmilter1, libspf2-2 (>= 1.2.5), adduser
(>= 3.11)
Recommends: sendmail
Filename: pool/main/m/milter-greylist/milter-greylist_3.0-3_i386.deb
Size: 85748
MD5sum: e5d18e06984a3a1dc263dad6061db5bb


Mar 14 12:39:18 delphin sm-mta[12124]: m2EBdG5R012124:
dyn-89.136.83.248.upcnet.ro [89.136.83.248] (may be forged): Possible
SMTP RCPT flood, thro
ttling.
Mar 14 12:39:19 delphin sm-mta[12115]: m2EBdAWN012115:
milter_sys_read(greylist): cmd read returned 0, expecting 5
Mar 14 12:39:19 delphin sm-mta[12115]: m2EBdAWN012115: Milter
(greylist): to error state
Mar 14 12:39:19 delphin sm-mta[12124]: m2EBdG5R012124: Milter
(greylist): write(D) returned -1, expected 6: Broken pipe
Mar 14 12:39:19 delphin sm-mta[12124]: m2EBdG5R012124: Milter
(greylist): to error state
Mar 14 12:39:19 delphin sm-mta[12120]: m2EBdIpm012120: Milter
(greylist): write(D) returned -1, expected 6: Broken pipe
Mar 14 12:39:19 delphin sm-mta[12120]: m2EBdIpm012120: Milter
(greylist): to error state
Mar 14 12:39:19 delphin sm-mta[12123]: m2EBdJOB012123: Milter
(greylist): error connecting to filter: Connection refused by
/var/run/milter-greyli
st/milter-greylist.sock

netbrain2003 <netbrain2003@...> wrote:

> I have a problem with a terminating daemon after a flood via SMTP. I'm
> afraid I can not give much debug output but just the mail server log
> of this happening (see below). If I can turn on some switches to give
> more output please let me know. The system is a debian one running
> sendmail 8.14.2-3. After restart the greylist service it is working
> again. This happens more often these days, I'm afraid spammer know
> this already?

Perhaps you hit a system limit? memory or processes?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Sat, Mar 15, 2008 at 7:22 PM,  <manu@...> wrote:
> netbrain2003 <netbrain2003@...> wrote:
>
>  > I have a problem with a terminating daemon after a flood via SMTP. I'm
>  > afraid I can not give much debug output but just the mail server log
>  > of this happening (see below). If I can turn on some switches to give
>  > more output please let me know. The system is a debian one running
>  > sendmail 8.14.2-3. After restart the greylist service it is working
>  > again. This happens more often these days, I'm afraid spammer know
>  > this already?
>
>  Perhaps you hit a system limit? memory or processes?

It looked like the throttling in Sendmail was activated and that
milter-greylist got an unexpected value from it and crashed???

-- 
/peter

I have no special limits adjusted and I can not imagine the daemon
itself is running out of resources. Anyway that are the system
defaults, I don't know the software needs, perhaps you see a bottleneck:

$: ulimit -a                                                         
                                           
-t: cpu time (seconds)         unlimited
-f: file size (blocks)         unlimited
-d: data seg size (kbytes)     unlimited
-s: stack size (kbytes)        8192
-c: core file size (blocks)    0
-m: resident set size (kbytes) unlimited
-u: processes                  8063
-n: file descriptors           1024
-l: locked-in-memory size (kb) 32
-v: address space (kb)         unlimited
-x: file locks                 unlimited
-i: pending signals            8063
-q: bytes in POSIX msg queues  819200
-e: max nice                   0
-r: max rt priority            0

Thomas

--- In milter-greylist@yahoogroups.com, manu@... wrote:

>
> netbrain2003 <netbrain2003@...> wrote:
> 
> > I have a problem with a terminating daemon after a flood via SMTP. I'm
> > afraid I can not give much debug output but just the mail server log
> > of this happening (see below). If I can turn on some switches to give
> > more output please let me know. The system is a debian one running
> > sendmail 8.14.2-3. After restart the greylist service it is working
> > again. This happens more often these days, I'm afraid spammer know
> > this already?
> 
> Perhaps you hit a system limit? memory or processes?
> 
> -- 
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz
> manu@...
>

netbrain2003 <netbrain2003@...> wrote:

> I have no special limits adjusted and I can not imagine the daemon
> itself is running out of resources. Anyway that are the system
> defaults, I don't know the software needs, perhaps you see a bottleneck:
>
> -n: file descriptors           1024

That could be the culprit, but it's hard to tell.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Sat, Mar 15, 2008 at 07:46:45PM -0000, netbrain2003 wrote:
> I have no special limits adjusted and I can not imagine the daemon
> itself is running out of resources. Anyway that are the system
> defaults, I don't know the software needs, perhaps you see a bottleneck:
> 
> $: ulimit -a                                                         
>                                            
> -t: cpu time (seconds)         unlimited
> -f: file size (blocks)         unlimited
> -d: data seg size (kbytes)     unlimited
> -s: stack size (kbytes)        8192

Depending on which OS you have, memory consumption in due to multi-
threading can be considerably. E.g. Multithreading libraries in current
Linux distributions allocates stacksize for *each* thread that memory
every new process usually gets. Calculating this with your stacksize
above, 8192 kbytes and assuming a 32bit system which is usally limited
to say 2 GByte (virtual) address space then just around 256 threads
working in the milter-greylist process are filling up the address space.
In addition, milter-greylist needs a plenty of memory for its greylist
DB too, which resides in the same address space and is not counted so
far.

At our site we have an average of 128 current milter-greylist threads
(ps -eL|grep milter-greylist). In heavy times 256 and more are easily
reached. We are facing this situation by reducing the stacksize to 256K
even we are on a 64bit system (with larger address space, but only if
the application was compiled for 64bit). Sometimes I encountered
packages (e.g. amavis-milter or other thread-based things) which were
architecture independent but alas, they have the usual 32bit limits ...

> -c: core file size (blocks)    0
> -m: resident set size (kbytes) unlimited
> -u: processes                  8063
> -n: file descriptors           1024

As Manu already mentioned, the maximum descriptor cound should be raised
too.


Johann K.

Well, I only hope there is no reason for milter-greylist to allocate
8192Kb stack size, right?
Both greylist and whitelist databases are in the main memory.

Ondrej

Johann E. Klasek wrote:

>
> On Sat, Mar 15, 2008 at 07:46:45PM -0000, netbrain2003 wrote:
> > I have no special limits adjusted and I can not imagine the daemon
> > itself is running out of resources. Anyway that are the system
> > defaults, I don't know the software needs, perhaps you see a bottleneck:
> >
> > $: ulimit -a
> >
> > -t: cpu time (seconds) unlimited
> > -f: file size (blocks) unlimited
> > -d: data seg size (kbytes) unlimited
> > -s: stack size (kbytes) 8192
>
> Depending on which OS you have, memory consumption in due to multi-
> threading can be considerably. E.g. Multithreading libraries in current
> Linux distributions allocates stacksize for *each* thread that memory
> every new process usually gets. Calculating this with your stacksize
> above, 8192 kbytes and assuming a 32bit system which is usally limited
> to say 2 GByte (virtual) address space then just around 256 threads
> working in the milter-greylist process are filling up the address space.
> In addition, milter-greylist needs a plenty of memory for its greylist
> DB too, which resides in the same address space and is not counted so
> far.
>
> At our site we have an average of 128 current milter-greylist threads
> (ps -eL|grep milter-greylist). In heavy times 256 and more are easily
> reached. We are facing this situation by reducing the stacksize to 256K
> even we are on a 64bit system (with larger address space, but only if
> the application was compiled for 64bit). Sometimes I encountered
> packages (e.g. amavis-milter or other thread-based things) which were
> architecture independent but alas, they have the usual 32bit limits ...
>
> > -c: core file size (blocks) 0
> > -m: resident set size (kbytes) unlimited
> > -u: processes 8063
> > -n: file descriptors 1024
>
> As Manu already mentioned, the maximum descriptor cound should be raised
> too.
>
> Johann K.
>
>

On Tue, Mar 18, 2008 at 11:04:55AM +0100, Ondrej Valousek wrote:
> Well, I only hope there is no reason for milter-greylist to allocate
> 8192Kb stack size, right?

IIRC, some systems had trouble at config file parse time. Increasing
stack size helped.

-- 
Emmanuel Dreyfus
manu@...

On Tue, Mar 18, 2008 at 11:04:55AM +0100, Ondrej Valousek wrote:
> Well, I only hope there is no reason for milter-greylist to allocate
> 8192Kb stack size, right?

Not really ;)
Just to point out:
It is not the matter of allocating, the stack area is reserved by the
thread library, no matter if the threaded code is using it or not.
Anyway, the address space for the reserved stack area is consumed, no
matter how the stack is filled. Each thread instance in a process
context needs (gets) this space sharing all the same (virtual) memory
address space (which is limited depending on the address architecture of
the OS).

> Both greylist and whitelist databases are in the main memory.

What is main memory? What you probably want to say is, that the DB
resides on the heap of the process (virtual) memory space. Actually, the
stack area for each newly started thread is also taken from this process
heap ...

However, in virtual memory context, the address space (per process) is
limited anyway to the range an 32-bit addresses could reach.


Johann K.

OOops - my greylist-milter died after a week of functioning.
It is the first time I saw it down.
I do not see anything strange in the logs - could it be the same reason?
My system:
libspf2-1.2.5 (no patches applied), RHEL-3, 32-bit, 2Gb RAM
Thanks,

Ondrej

Emmanuel Dreyfus wrote:

>
> On Tue, Mar 18, 2008 at 11:04:55AM +0100, Ondrej Valousek wrote:
> > Well, I only hope there is no reason for milter-greylist to allocate
> > 8192Kb stack size, right?
>
> IIRC, some systems had trouble at config file parse time. Increasing
> stack size helped.
>
> -- 
> Emmanuel Dreyfus
> manu@... <mailto:manu%40netbsd.org>
>
>

Hi Johann,

Ok I see the point.
I was trying to figure out how much memory is the milter-greylist using
on my system using the top command and I am not sure whether it gives me
the accurate results.
Do I have the feed top command with some additional switches to find out
the total amount of resources it consumes?

I am just trying to find out why it has died last time - was it just a
lack of system resources, not enough file descriptors or broken spf
library (saw the last discussion).
At the moment I have nothing to start with.
Maybe I could enable the verbose logging or enable core files (not sure
what is the best).

So far, this software just rocks and from what I have seen, it is also
well written (every memory allocation is checked, bullet-proof code,
pity it is not C++...) so there must be some ways to find out more. I
would like to keep using it.

Ondrej

Johann E. Klasek wrote:

>
> On Tue, Mar 18, 2008 at 11:04:55AM +0100, Ondrej Valousek wrote:
> > Well, I only hope there is no reason for milter-greylist to allocate
> > 8192Kb stack size, right?
>
> Not really ;)
> Just to point out:
> It is not the matter of allocating, the stack area is reserved by the
> thread library, no matter if the threaded code is using it or not.
> Anyway, the address space for the reserved stack area is consumed, no
> matter how the stack is filled. Each thread instance in a process
> context needs (gets) this space sharing all the same (virtual) memory
> address space (which is limited depending on the address architecture of
> the OS).
>
> > Both greylist and whitelist databases are in the main memory.
>
> What is main memory? What you probably want to say is, that the DB
> resides on the heap of the process (virtual) memory space. Actually, the
> stack area for each newly started thread is also taken from this process
> heap ...
>
> However, in virtual memory context, the address space (per process) is
> limited anyway to the range an 32-bit addresses could reach.
>
> Johann K.
>
>

Just in relation to my previous post, I have found out that the milter
crashed right after this:

Mar 16 05:05:20 mailsrv milter-greylist: (local): addr 118.232.8.153
from \ufffd^9   \ufffd=6      rcpt <grzegorz.kus@...>: autowhitelisted
entry expired

Look at the from address - it is obviously incorrect. And it is
obviously related to this one:

Mar 15 03:07:01 mailsrv milter-greylist: m2F370Hr012369: addr
118.232.8.153 from <sypak@...> rcpt
<grzegorz.kus@...>: autowhitelisted for 24:00:00

Here is the mail from address correct. It looks like the mail from
address has been recorded in the autowhitelist database incorrectly.

Ondrej


Ondrej Valousek wrote:

>
> Hi Johann,
>
> Ok I see the point.
> I was trying to figure out how much memory is the milter-greylist using
> on my system using the top command and I am not sure whether it gives me
> the accurate results.
> Do I have the feed top command with some additional switches to find out
> the total amount of resources it consumes?
>
> I am just trying to find out why it has died last time - was it just a
> lack of system resources, not enough file descriptors or broken spf
> library (saw the last discussion).
> At the moment I have nothing to start with.
> Maybe I could enable the verbose logging or enable core files (not sure
> what is the best).
>
> So far, this software just rocks and from what I have seen, it is also
> well written (every memory allocation is checked, bullet-proof code,
> pity it is not C++...) so there must be some ways to find out more. I
> would like to keep using it.
>
> Ondrej
>
> Johann E. Klasek wrote:
> >
> > On Tue, Mar 18, 2008 at 11:04:55AM +0100, Ondrej Valousek wrote:
> > > Well, I only hope there is no reason for milter-greylist to allocate
> > > 8192Kb stack size, right?
> >
> > Not really ;)
> > Just to point out:
> > It is not the matter of allocating, the stack area is reserved by the
> > thread library, no matter if the threaded code is using it or not.
> > Anyway, the address space for the reserved stack area is consumed, no
> > matter how the stack is filled. Each thread instance in a process
> > context needs (gets) this space sharing all the same (virtual) memory
> > address space (which is limited depending on the address architecture of
> > the OS).
> >
> > > Both greylist and whitelist databases are in the main memory.
> >
> > What is main memory? What you probably want to say is, that the DB
> > resides on the heap of the process (virtual) memory space. Actually, the
> > stack area for each newly started thread is also taken from this process
> > heap ...
> >
> > However, in virtual memory context, the address space (per process) is
> > limited anyway to the range an 32-bit addresses could reach.
> >
> > Johann K.
> >
> >
>
>

Ondrej Valousek <webserv@...> wrote:

> Just in relation to my previous post, I have found out that the milter
> crashed right after this:
> 
> Mar 16 05:05:20 mailsrv milter-greylist: (local): addr 118.232.8.153
> from ?^9   ?=6      rcpt <grzegorz.kus@...>: autowhitelisted
> entry expired

It looks like memory corruption. What version is this? Can you reproduce
the problem?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Hi Emmanuel,

I am using version 4.1.1.
I have seen it only once so far.... :-(

Ondrej

manu@... wrote:

>
> Ondrej Valousek <webserv@... <mailto:webserv%40s3group.cz>> wrote:
>
> > Just in relation to my previous post, I have found out that the milter
> > crashed right after this:
> >
> > Mar 16 05:05:20 mailsrv milter-greylist: (local): addr 118.232.8.153
> > from ?^9 ?=6 rcpt <grzegorz.kus@...
> <mailto:grzegorz.kus%40s3group.com>>: autowhitelisted
> > entry expired
>
> It looks like memory corruption. What version is this? Can you reproduce
> the problem?
>
> -- 
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz <http://hcpnet.free.fr/pubz>
> manu@... <mailto:manu%40netbsd.org>
>
>

Ondrej Valousek <webserv@...> wrote:

> I am using version 4.1.1.
> I have seen it only once so far.... :-(

Okay, let's hope it was a cosmic ray, then :-)

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

manu@... wrote:
> Ondrej Valousek <webserv@...> wrote:
>
>   
>> I am using version 4.1.1.
>> I have seen it only once so far.... :-(
>>     
>
> Okay, let's hope it was a cosmic ray, then :-)
>   

Isn't that what ECC memory is for?  
http://en.wikipedia.org/wiki/ECC_memory#Error-correcting_memory

Then again, too many people using desktop class hardware as servers 
these days.


---------------

Chris Hoogendyk

-
   O__  ---- Systems Administrator
  c/ /'_ --- Biology & Geology Departments
 (*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst 

<hoogendyk@...>

--------------- 

Erd\ufffds 4

Chris Hoogendyk <hoogendyk@...> wrote:

> > Okay, let's hope it was a cosmic ray, then :-)
> Isn't that what ECC memory is for?  

That does not screen you from data been altered in transit, on the
buses.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Wed, Mar 19, 2008 at 10:22:22AM +0100, Ondrej Valousek wrote:
> Hi Johann,
> 
> Ok I see the point.
> I was trying to figure out how much memory is the milter-greylist using
> on my system using the top command and I am not sure whether it gives me
> the accurate results.
> Do I have the feed top command with some additional switches to find out
> the total amount of resources it consumes?

Just look at the "VIRT" column.
You can also use the command "pmap PID" available under Linux or Solaris.
The latter serves a more detailed view of allocated chunks in virutal 
memory.

> 
> I am just trying to find out why it has died last time - was it just a
> lack of system resources, not enough file descriptors or broken spf
> library (saw the last discussion).
> At the moment I have nothing to start with.
> Maybe I could enable the verbose logging or enable core files (not sure
> what is the best).

Just enable core dumping.

Note to Fedora users: write into the appropriate sysconfig file

DAEMON_COREFILE_LIMIT=unlimited

to enable core dump creation.

Use gdb to see the backtrace ...

gdb -d milter-greylist-source-dir Binary Corefile
(gdb) backtrace


I hope you catch the beast ;)

Johann

Hi Johann,

Thanks for the advice.
Actually - I have already found the cause of the last problem - the main
database in memory got corrupted during the milter run which caused that
some pointer holding the sender's email address (i.e. one part of the
triplet) was overwritten.
Later on, when the triplet was about to be expired, milter-greylist
released the resources and bailed out when freeing the invalid memory
where the pointer pointed.
This is kind of programmer's nightmare, I would say - even core dump
would not help you here.

The only thing I am wondering about is whether maybe the spf dynamic
library might have caused this. Is it possible for a dynamic library to
overwrite memory that the calling process is using?
I am asking because I am using the unpatched version of the libspf2 -
and I read about the problems it caused to other guys...

Thanks,
Ondrej

Johann E. Klasek wrote:

>
> On Wed, Mar 19, 2008 at 10:22:22AM +0100, Ondrej Valousek wrote:
> > Hi Johann,
> >
> > Ok I see the point.
> > I was trying to figure out how much memory is the milter-greylist using
> > on my system using the top command and I am not sure whether it gives me
> > the accurate results.
> > Do I have the feed top command with some additional switches to find out
> > the total amount of resources it consumes?
>
> Just look at the "VIRT" column.
> You can also use the command "pmap PID" available under Linux or Solaris.
> The latter serves a more detailed view of allocated chunks in virutal
> memory.
>
> >
> > I am just trying to find out why it has died last time - was it just a
> > lack of system resources, not enough file descriptors or broken spf
> > library (saw the last discussion).
> > At the moment I have nothing to start with.
> > Maybe I could enable the verbose logging or enable core files (not sure
> > what is the best).
>
> Just enable core dumping.
>
> Note to Fedora users: write into the appropriate sysconfig file
>
> DAEMON_COREFILE_LIMIT=unlimited
>
> to enable core dump creation.
>
> Use gdb to see the backtrace ...
>
> gdb -d milter-greylist-source-dir Binary Corefile
> (gdb) backtrace
>
> I hope you catch the beast ;)
>
> Johann
>
>

Ondrej Valousek <webserv@...> wrote:

> The only thing I am wondering about is whether maybe the spf dynamic
> library might have caused this. Is it possible for a dynamic library to
> overwrite memory that the calling process is using?

Yes, it is possible.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...