Yahoo Groups archive

Milter-greylist

Index last updated: 2026-04-28 23:32 UTC

Thread

Frequent crashes when dumping db

Frequent crashes when dumping db

2006-11-24 by shuttlebox

I have been running milter-greylist 2.0.2 for a few months and I'm
very pleased with how it performs, it really eases the job for my
MailScanner servers. However, I've had a few restarts (I monitor the
milter via a cron job) every other day or so, no big deal and I tuned
the time to keep the tuples from 5d to 1d and that shrunk the db
considerably. The last few days though it has restarted like crazy
(10-20 times a day) and spam pours through. I upgraded yesterday to
3.0 but there's no difference in stability. Here's the log when it
crashes:

Nov 24 11:20:04 viola milter-greylist: [ID 653641 mail.error]
mkstemp("/var/milter-greylist/greylist.db-XX.caqCI") failed: Too many
open files
Nov 24 11:20:04 viola milter-greylist: [ID 421540 mail.info] Final
database dump: no change to dump
Nov 24 11:20:04 viola milter-greylist: [ID 649572 mail.info] Exiting

About the "Too many open files" - that file does not exist in the
/var/milter-greylist directory. The only files there are greylist.db,
milter-greylist.pid and milter-greylist.sock.

The frequent crashes of one peer (I have two) also gives me lots of these:

Nov 24 14:45:01 viola milter-greylist: [ID 615005 mail.error] cannot
sync with peer xxx.xxx.xxx.xxx: Too many open files (905 entries
queued)

or these:

Nov 24 03:35:23 viola milter-greylist: [ID 615005 mail.error] cannot
sync with peer xxx.xxx.xxx.xxx: Connection refused (46 entries queued)

-- 
/peter

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-24 by Oliver Fromme

shuttlebox wrote:
 > Nov 24 11:20:04 viola milter-greylist: [ID 653641 mail.error]
 > mkstemp("/var/milter-greylist/greylist.db-XX.caqCI") failed: Too many
 > open files

There's your problen:  "Too many open files".
You need to increase the limit on file descriptors.
(The exact way to do that depends on your OS and shell.)

Most shells have a "ulimit" or "limit" built-in command,
some systems also have /usr/bin/limits (see the manual
pages).  They can be used to lift the soft limit on
file descriptors, among other things.

If you need to increase the hard limit of your OS, well,
that's OS-dependant.  For example, FreeBSD has a sysctl
variable (kern.maxfiles and kern.maxfilesperproc), and
the limits can be tuned per login-class in /etc/login.conf
(for system daemons like milter-greylist the login class
would be "daemon", unless a different one is created.)

Best regards
   Oliver

-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

'Instead of asking why a piece of software is using "1970s technology,"
start asking why software is ignoring 30 years of accumulated wisdom.'

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-24 by shuttlebox

On 11/24/06, Oliver Fromme <olli@...> wrote:
>  shuttlebox wrote:
>   > Nov 24 11:20:04 viola milter-greylist: [ID 653641 mail.error]
>   > mkstemp("/var/milter-greylist/greylist.db-XX.caqCI") failed: Too many
>   > open files
>
>  There's your problen:  "Too many open files".
>  You need to increase the limit on file descriptors.
>  (The exact way to do that depends on your OS and shell.)
>
>  Most shells have a "ulimit" or "limit" built-in command,
>  some systems also have /usr/bin/limits (see the manual
>  pages).  They can be used to lift the soft limit on
>  file descriptors, among other things.
>
>  If you need to increase the hard limit of your OS, well,
>  that's OS-dependant.  For example, FreeBSD has a sysctl
>  variable (kern.maxfiles and kern.maxfilesperproc), and
>  the limits can be tuned per login-class in /etc/login.conf
>  (for system daemons like milter-greylist the login class
>  would be "daemon", unless a different one is created.)

But the limit for open files is 256 for both root and the milter user,
I can't see how the milter user needs more than that when I can't find
more than three files and none of them are actively open. Does the
process use a lot of temporary files outside the milter directory?

I raised the ulimit to 1024 files before I wrote the first mail and it
kept crashing. How can I know how many files I need? How many do you
use?

Another thing, the init.d script starts milter-greylist as root and
then drops privileges to the milter user I assume, where do I put the
ulimit change then to make it count? I made /var/milter-greylist/ the
home directory of the milter user and created a .profile containing
"ulimit -n 1024" there. Is that the correct way to make
milter-greylist pick it up? Can I check that it uses the new limit?

Sorry for all the questions but I have a lot of load problems when
greylisting doesn't work. Thanks for the help. :-)

-- 
/peter

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-24 by Oliver Fromme

shuttlebox wrote:
 > Oliver Fromme wrote:
 > > shuttlebox wrote:
 > > > Nov 24 11:20:04 viola milter-greylist: [ID 653641 mail.error]
 > > > mkstemp("/var/milter-greylist/greylist.db-XX.caqCI") failed: Too many
 > > > open files
 > > 
 > >  There's your problen:  "Too many open files".
 > >  You need to increase the limit on file descriptors.
 > >  (The exact way to do that depends on your OS and shell.)
 > > 
 > >  Most shells have a "ulimit" or "limit" built-in command,
 > >  some systems also have /usr/bin/limits (see the manual
 > >  pages).  They can be used to lift the soft limit on
 > >  file descriptors, among other things.
 > > 
 > >  If you need to increase the hard limit of your OS, well,
 > >  that's OS-dependant.  For example, FreeBSD has a sysctl
 > >  variable (kern.maxfiles and kern.maxfilesperproc), and
 > >  the limits can be tuned per login-class in /etc/login.conf
 > >  (for system daemons like milter-greylist the login class
 > >  would be "daemon", unless a different one is created.)
 > 
 > But the limit for open files is 256 for both root and the milter user,
 > I can't see how the milter user needs more than that when I can't find
 > more than three files and none of them are actively open. Does the
 > process use a lot of temporary files outside the milter directory?

File descriptors are not only used for plain files, but
also for network connections and other things.  You can
view the current descriptors with the "lsof" tool.

On my personal machine (which has very low traffic), the
milter-greylist process currently uses 21 descriptors.

 > I raised the ulimit to 1024 files before I wrote the first mail and it
 > kept crashing. How can I know how many files I need? How many do you
 > use?

I think 1024 should be plenty.  As I said above, use the
"lsof" tool to check how many descriptors are in use by
the process.

Actually, looking at the error message again, it might be
the case that the mkstemp() function opens a large number
of files while trying to find a unique filename.  If that's
the case, then the implementation of mkstemp() of your OS
(which you still didn't mention) is broken.

 > Another thing, the init.d script starts milter-greylist as root and
 > then drops privileges to the milter user I assume, where do I put the
 > ulimit change then to make it count? I made /var/milter-greylist/ the
 > home directory of the milter user and created a .profile containing
 > "ulimit -n 1024" there. Is that the correct way to make
 > milter-greylist pick it up?

No, .profile is read only by login shells.  You should put
the ulimit command inside the init.d script right before
milter-greylist is started.  The limits are inherited by
child processes.

 > Can I check that it uses the new limit?

Depends on your OS.  When you use the "lsof" utility (see
above) and it reports more than 256 descriptors in use for
your milter-greylist process, then it obviously uses the
new limit.  ;-)

Best regards
   Oliver

-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

"Clear perl code is better than unclear awk code; but NOTHING
comes close to unclear perl code"  (taken from comp.lang.awk FAQ)

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-25 by shuttlebox

On 11/24/06, Oliver Fromme <olli@...> wrote:
>  File descriptors are not only used for plain files, but
>  also for network connections and other things.  You can
>  view the current descriptors with the "lsof" tool.
>
>  On my personal machine (which has very low traffic), the
>  milter-greylist process currently uses 21 descriptors.

Ok, thanks. I will install lsof.

>  I think 1024 should be plenty.  As I said above, use the
>  "lsof" tool to check how many descriptors are in use by
>  the process.
>
>  Actually, looking at the error message again, it might be
>  the case that the mkstemp() function opens a large number
>  of files while trying to find a unique filename.  If that's
>  the case, then the implementation of mkstemp() of your OS
>  (which you still didn't mention) is broken.

Sorry, it's Solaris 9. But it shouldn't be hard to find a unique
filename in a directory containing only three files..?

>  No, .profile is read only by login shells.  You should put
>  the ulimit command inside the init.d script right before
>  milter-greylist is started.  The limits are inherited by
>  child processes.

I tried both :-) and it still crashes all the time.

>  Depends on your OS.  When you use the "lsof" utility (see
>  above) and it reports more than 256 descriptors in use for
>  your milter-greylist process, then it obviously uses the
>  new limit.  ;-)

I'll install lsof first thing Monday morning and see what it tells me.
Thanks again.

-- 
/peter

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-27 by eclark

Im with the rest here re use of sockets and the like eating up your limit. 
Frankly, I think 256 is a bit low for this user, and would probably set it to 
512 or 1024. It isnt a generic user after all, and has to support up to ~260 
descriptors open at once. 
Show quoted textHide quoted text
On Saturday 25 November 2006 11:03 am, shuttlebox wrote:
> On 11/24/06, Oliver Fromme <olli@...> wrote:
> >  File descriptors are not only used for plain files, but
> >  also for network connections and other things.  You can
> >  view the current descriptors with the "lsof" tool.
> >
> >  On my personal machine (which has very low traffic), the
> >  milter-greylist process currently uses 21 descriptors.
>
> Ok, thanks. I will install lsof.
>
> >  I think 1024 should be plenty.  As I said above, use the
> >  "lsof" tool to check how many descriptors are in use by
> >  the process.
> >
> >  Actually, looking at the error message again, it might be
> >  the case that the mkstemp() function opens a large number
> >  of files while trying to find a unique filename.  If that's
> >  the case, then the implementation of mkstemp() of your OS
> >  (which you still didn't mention) is broken.
>
> Sorry, it's Solaris 9. But it shouldn't be hard to find a unique
> filename in a directory containing only three files..?
>
> >  No, .profile is read only by login shells.  You should put
> >  the ulimit command inside the init.d script right before
> >  milter-greylist is started.  The limits are inherited by
> >  child processes.
>
> I tried both :-) and it still crashes all the time.
>
> >  Depends on your OS.  When you use the "lsof" utility (see
> >  above) and it reports more than 256 descriptors in use for
> >  your milter-greylist process, then it obviously uses the
> >  new limit.  ;-)
>
> I'll install lsof first thing Monday morning and see what it tells me.
> Thanks again.

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-27 by shuttlebox

On 11/27/06, eclark <eclark@...> wrote:
> Im with the rest here re use of sockets and the like eating up your limit.
>  Frankly, I think 256 is a bit low for this user, and would probably set it to
>  512 or 1024. It isnt a generic user after all, and has to support up to ~260
>  descriptors open at once.

I tried lsof on the greylist process and counted 268 lines in the
output. I use a 4 second greet pause delay and that means more open
connections with Sendmail and it seems as if the milter is involved in
all those open connections even if most are probably just idling for 4
seconds. I'm not saying that is wrong or anything but in my case it
means I use up a lot of file descriptors.

How can I get it to use more? It doesn't seem to bother with my
inserted ulimit command in the init.d script. Can I check what the
running process uses? This is the start section of my init.d script,
basically what is included with milter-greylist and I added the ulimit
command.

        # Start daemon.
        echo "Starting milter-greylist: ... \c"
        chown $user $mildir
        /usr/bin/ulimit -n 4096
        $binpath/milter-greylist $OPTIONS
        echo "done."
        ;;

I also ran truss on the process and it complains about too many open
files of course but I have a hard time evaluating the output. Any
pointers what to look for?

-- 
/peter

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-27 by eclark

Are you on a linux box? Using bash? If so, whats your output of ulimit -a as 
your milter user? If you are doing linux, is sysctl overriding ulimit? 
Show quoted textHide quoted text
On Monday 27 November 2006 07:42 am, shuttlebox wrote:
> On 11/27/06, eclark <eclark@...> wrote:
> > Im with the rest here re use of sockets and the like eating up your
> > limit. Frankly, I think 256 is a bit low for this user, and would
> > probably set it to 512 or 1024. It isnt a generic user after all, and has
> > to support up to ~260 descriptors open at once.
>
> I tried lsof on the greylist process and counted 268 lines in the
> output. I use a 4 second greet pause delay and that means more open
> connections with Sendmail and it seems as if the milter is involved in
> all those open connections even if most are probably just idling for 4
> seconds. I'm not saying that is wrong or anything but in my case it
> means I use up a lot of file descriptors.
>
> How can I get it to use more? It doesn't seem to bother with my
> inserted ulimit command in the init.d script. Can I check what the
> running process uses? This is the start section of my init.d script,
> basically what is included with milter-greylist and I added the ulimit
> command.
>
>         # Start daemon.
>         echo "Starting milter-greylist: ... \c"
>         chown $user $mildir
>         /usr/bin/ulimit -n 4096
>         $binpath/milter-greylist $OPTIONS
>         echo "done."
>         ;;
>
> I also ran truss on the process and it complains about too many open
> files of course but I have a hard time evaluating the output. Any
> pointers what to look for?

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-27 by Oliver Fromme

shuttlebox wrote:
 > How can I get it to use more? It doesn't seem to bother with my
 > inserted ulimit command in the init.d script. Can I check what the
 > running process uses? This is the start section of my init.d script,
 > basically what is included with milter-greylist and I added the ulimit
 > command.
 >         [...]
 >         /usr/bin/ulimit -n 4096
 >         $binpath/milter-greylist $OPTIONS

That won't work.  /usr/bin/ulimit cannot be use in that way,
and even if it could, the modified limit wouldn't take effect
because the command runs in a subshell.  It doesn't affect
the parent shell which then spawns milter-greylist.

You need to use "ulimit -n 4096" (without "/usr/bin"!) which
is a shell-builtin command.

Best regards
   Oliver

-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

"C++ is the only current language making COBOL look good."
        -- Bertrand Meyer

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-27 by shuttlebox

On 11/27/06, Oliver Fromme <olli@...> wrote:
>  That won't work.  /usr/bin/ulimit cannot be use in that way,
>  and even if it could, the modified limit wouldn't take effect
>  because the command runs in a subshell.  It doesn't affect
>  the parent shell which then spawns milter-greylist.
>
>  You need to use "ulimit -n 4096" (without "/usr/bin"!) which
>  is a shell-builtin command.

Thanks for spotting that. I changed it and found a way to see a
running processes limits in Solaris:

/usr/proc/bin/pfiles 3716 | grep rlimit
  Current rlimit: 1024 file descriptors

I tested with 1024 this time and now it uses it (after the change
above) but it still crashes all the time. I seem to have another error
message though.

Nov 27 17:05:01 viola milter-greylist: [ID 525239 mail.error] cannot
write dumpfile "/var/milter-greylist/greylist.db-XXKqaGyg": No such
file or directory
Nov 27 17:05:01 viola milter-greylist: [ID 421540 mail.info] Final
database dump: no change to dump
Nov 27 17:05:01 viola milter-greylist: [ID 649572 mail.info] Exiting

That file (temporary db?) does in fact exist.

# ls -l /var/milter-greylist/greylist.db-XXKqaGyg
-rw-------   1 milter   milter         0 Nov 27 17:05
/var/milter-greylist/greylist.db-XXKqaGyg

I also have the permissions to create files in that directory.

# su - milter
Sun Microsystems Inc.   SunOS 5.9       Generic May 2002
$ id
uid=5501(milter) gid=5501(milter)
$ pwd
/var/milter-greylist
$ ls -la
total 8
drwxr-x---   2 milter   milter       512 Nov 27 17:05 .
drwxr-xr-x  35 root     sys         1024 Sep 13 18:55 ..
-rw-------   1 milter   milter         0 Nov 27 16:50 greylist.db-XX7CaGTa
-rw-------   1 milter   milter         0 Nov 27 17:05 greylist.db-XXKqaGyg
-rw-------   1 milter   milter         0 Nov 27 16:35 greylist.db-XXuZaaM1
-rw-r--r--   1 root     other          5 Nov 27 16:55 milter-greylist.pid
srwxr-xr-x   1 milter   milter         0 Nov 27 16:55 milter-greylist.sock
$ touch testing
$ ls -l
total 4
-rw-r--r--   1 root     other          5 Nov 27 16:05 fdlimit
-rw-------   1 milter   milter         0 Nov 27 16:50 greylist.db-XX7CaGTa
-rw-------   1 milter   milter         0 Nov 27 17:05 greylist.db-XXKqaGyg
-rw-------   1 milter   milter         0 Nov 27 16:35 greylist.db-XXuZaaM1
-rw-r--r--   1 root     other          5 Nov 27 16:55 milter-greylist.pid
srwxr-xr-x   1 milter   milter         0 Nov 27 16:55 milter-greylist.sock
-rw-r--r--   1 milter   milter         0 Nov 27 17:09 testing


-- 
/peter

Re: [milter-greylist] Frequent crashes when dumping db

2006-11-30 by shuttlebox

On 11/27/06, shuttlebox <shuttlebox@...> wrote:
> I tested with 1024 this time and now it uses it (after the change
> above) but it still crashes all the time. I seem to have another error
> message though.
>
> Nov 27 17:05:01 viola milter-greylist: [ID 525239 mail.error] cannot
> write dumpfile "/var/milter-greylist/greylist.db-XXKqaGyg": No such
> file or directory
> Nov 27 17:05:01 viola milter-greylist: [ID 421540 mail.info] Final
> database dump: no change to dump
> Nov 27 17:05:01 viola milter-greylist: [ID 649572 mail.info] Exiting

I have actually gone back to using 2.0.2 and it works much better even
though it also crashes sometimes. Kind of ironic when 3.0 claims to
have fixed the db dump bug.

Is there anyone who can help with more debugging, I could post some
truss (similar to strace) output if you want?

-- 
/peter

Move to quarantaine

This moves the raw source file on disk only. The archive index is not changed automatically, so you still need to run a manual refresh afterward.