crashing under heavy load
2005-11-09 by Ken Serrine
Yahoo Groups archive
Index last updated: 2026-04-28 23:32 UTC
Thread
2005-11-09 by Ken Serrine
Anyone else having this issue? I'm using version 2.0.2 on Redhat Enterprise Linux, with sendmail 8.13.1, and under heavy load, the milter crashes. By heavy load, I mean about 44 new connections per second. I have another milter that I run that is fine. I don't have the multple MX enabled.
2005-11-09 by Emmanuel Dreyfus
On Wed, Nov 09, 2005 at 09:38:03AM -0600, Ken Serrine wrote: > Anyone else having this issue? > I'm using version 2.0.2 on Redhat Enterprise Linux, with sendmail > 8.13.1, and under heavy load, the milter crashes. By heavy load, I mean > about 44 new connections per second. I have another milter that I run > that is fine. I don't have the multple MX enabled. Do you hit a system limit? -- Emmanuel Dreyfus manu@...
2005-11-09 by Ken Serrine
On Wed, Nov 09, 2005 at 09:38:03AM -0600, Ken Serrine wrote: >>Anyone else having this issue? >>I'm using version 2.0.2 on Redhat Enterprise Linux, with sendmail >>8.13.1, and under heavy load, the milter crashes. By heavy load, I mean >>about 44 new connections per second. I have another milter that I run >>that is fine. I don't have the multple MX enabled. >> >> > >Emmanuel Dreyfus wrote: > >Do you hit a system limit? > > > >No limit that I'm aware of. If it crashes again tonight, I will try to >use Inet Sockets instead of Unix sockets. I have a memory of a milter >long ago that seemed to have that problem, but I wasn't able to prove to myself that that was the problem for sure. It may be a Linux specific (or, redhat linux specific) problem. That's why I was wondering if >anyone else ran into this. > > >------------------------------------------------------------------------ > > >
2005-11-09 by Hajimu UMEMOTO
Hi,
>>>>> On Wed, 09 Nov 2005 11:07:48 -0600
>>>>> Ken Serrine <kserrine@...> said:
kserrine> On Wed, Nov 09, 2005 at 09:38:03AM -0600, Ken Serrine wrote:
>>Anyone else having this issue?
>>I'm using version 2.0.2 on Redhat Enterprise Linux, with sendmail
>>8.13.1, and under heavy load, the milter crashes. By heavy load, I mean
>>about 44 new connections per second. I have another milter that I run
>>that is fine. I don't have the multple MX enabled.
>>
>>
>
>Emmanuel Dreyfus wrote:
>
>Do you hit a system limit?
>
>
>
>No limit that I'm aware of. If it crashes again tonight, I will try to
>use Inet Sockets instead of Unix sockets. I have a memory of a milter
>long ago that seemed to have that problem, but I wasn't able to prove to myself that that was the problem for sure. It may be a Linux specific (or, redhat linux specific) problem. That's why I was wondering if
>anyone else ran into this.
Are you using SPF? If so, I suspect thread-safeness of your resolver.
Sincerely,
--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@... ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/2005-11-09 by Raul Dias
I had this problem twice last week. It was under heavy (mail) load. I still had plenty of cpu power and memory. The machine is SMP Linux (non redhat). Sendmail 8.13.5 []'s Raul Dias
On Wed, 2005-11-09 at 11:07 -0600, Ken Serrine wrote: > On Wed, Nov 09, 2005 at 09:38:03AM -0600, Ken Serrine wrote: > > >>Anyone else having this issue? > >>I'm using version 2.0.2 on Redhat Enterprise Linux, with sendmail > >>8.13.1, and under heavy load, the milter crashes. By heavy load, I mean > >>about 44 new connections per second. I have another milter that I run > >>that is fine. I don't have the multple MX enabled. > >> > >> > > > >Emmanuel Dreyfus wrote: > > > >Do you hit a system limit? > > > > > > > >No limit that I'm aware of. If it crashes again tonight, I will try to > >use Inet Sockets instead of Unix sockets. I have a memory of a milter > >long ago that seemed to have that problem, but I wasn't able to prove to myself that that was the problem for sure. It may be a Linux specific (or, redhat linux specific) problem. That's why I was wondering if > >anyone else ran into this.
2005-11-10 by Ken Serrine
Anyone else having this issue? >>>I'm using version 2.0.2 on Redhat Enterprise Linux, with sendmail >>>8.13.1, and under heavy load, the milter crashes. By heavy load, I mean >>>about 44 new connections per second. I have another milter that I run >>>that is fine. I don't have the multple MX enabled. >>> >>> >Hajimu UMEMOTO wrote: > > >Are you using SPF? If so, I suspect thread-safeness of your resolver. > > > > Hajimu, I'm not using SPF. Raul, are you using Unix-sockets (the default for 2.0.2)?
2005-11-18 by Ken Serrine
The milter crashed twice more on two different machines, so I switched to using internet sockets. There have not been anymore crashes. I've had this issue with a different milter in the past, so I'm sure it's nothing to do with the milter. Also, I figured out why the milter was crashing on reloading the greylist.conf. The timing was such that the milter was dumping the memory at the same time it noticed that I made changes to greylist.conf. In that case, the milter crashes. Ken > >
2005-11-19 by Emmanuel Dreyfus
On Fri, Nov 18, 2005 at 04:14:16PM -0600, Ken Serrine wrote: > The milter crashed twice more on two different machines, so I switched > to using internet sockets. There have not been anymore crashes. I've > had this issue with a different milter in the past, so I'm sure it's > nothing to do with the milter. Probably a bug in libmilter. > Also, I figured out why the milter was crashing on reloading the > greylist.conf. The timing was such that the milter was dumping the > memory at the same time it noticed that I made changes to > greylist.conf. In that case, the milter crashes. This suggests we miss a lock somewhere. I can't see how a config reload can break a dump. Can you post a backtrace? -- Emmanuel Dreyfus manu@...
2005-11-21 by Ken Serrine
On Fri, Nov 18, 2005 at 04:14:16PM -0600, Ken Serrine wrote: >>Also, I figured out why the milter was crashing on reloading the >>greylist.conf. The timing was such that the milter was dumping the >>memory at the same time it noticed that I made changes to >>greylist.conf. In that case, the milter crashes. >> >> > >This suggests we miss a lock somewhere. I can't see how a config >reload can break a dump. Can you post a backtrace? > > > Yes, but I need some help with running the milter under gdb. I compiled it with debugging info, and tried running it 2 different ways, but sendmail complains that it cannot connect to the filter. I can see the filter running, but maybe I need to give certain gdb commands? The first way I tried was: gdb milter-greylist then, I ran it with (gdb) run <my arguments>, where <my arguments> were the arguments that I normally use. The second way was that I started the milter-greylist like normal, then ran gdb with the pid number like this: gdb milter-greylist 2265 Both ways seemed to work fine, but then when I started sendmail, it could not use the milter as usual.
2005-11-21 by Emmanuel Dreyfus
On Mon, Nov 21, 2005 at 10:30:26AM -0600, Ken Serrine wrote: > >Can you post a backtrace? > Yes, but I need some help with running the milter under gdb. Add -g to CFLAGS in Makefile, make clean && make && make install # gdb milter-greylist (gdb) run -Dv (and other args you usually use) -- Emmanuel Dreyfus manu@...
2005-11-21 by Ken Serrine
Emmanuel Dreyfus wrote: >On Mon, Nov 21, 2005 at 10:30:26AM -0600, Ken Serrine wrote: > > >>>Can you post a backtrace? >>> >>> >>Yes, but I need some help with running the milter under gdb. >> >> > >Add -g to CFLAGS in Makefile, make clean && make && make install ># gdb milter-greylist >(gdb) run -Dv (and other args you usually use) > > > Yep, tried that. sendmail won't attach to the milter.
2005-11-21 by Emmanuel Dreyfus
On Mon, Nov 21, 2005 at 10:52:58AM -0600, Ken Serrine wrote: > >(gdb) run -Dv (and other args you usually use) > Yep, tried that. sendmail won't attach to the milter. UNIX or INET sockets? If UNIX, wrong permissions? Any error message in sendmaill logs? -- Emmanuel Dreyfus manu@...
2005-11-21 by Ken Serrine
Emmanuel Dreyfus wrote: >On Mon, Nov 21, 2005 at 10:52:58AM -0600, Ken Serrine wrote: > > >>>(gdb) run -Dv (and other args you usually use) >>> >>> >>Yep, tried that. sendmail won't attach to the milter. >> >> > > >UNIX or INET sockets? If UNIX, wrong permissions? Any error message in >sendmaill logs? > > INET. Here's the error I get, even though I can see the milter running in my process list. >Nov 21 10:25:05 mail3 sendmail[2642]: jALGP5m2002642: Milter (greylist): error c >onnecting to filter: Connection refused by localhost > > Also, I've been trying to get the failure to occur, but I can't seem to get it to fail when using internet sockets. The greylist.db isn't as big as it was when the crash was occuring before, so I'll wait until it gets larger and keep trying. It might be the case that it only crashes when using UNIX sockets. I'll test that theory.
2005-11-21 by Emmanuel Dreyfus
On Mon, Nov 21, 2005 at 12:05:50PM -0600, Ken Serrine wrote: > >UNIX or INET sockets? If UNIX, wrong permissions? Any error message in > >sendmaill logs? > > > INET. > Here's the error I get, even though I can see the milter running in my > process list. > > >Nov 21 10:25:05 mail3 sendmail[2642]: jALGP5m2002642: Milter (greylist): error c > >onnecting to filter: Connection refused by localhost Does netstat -na show an open port? -- Emmanuel Dreyfus manu@...
2005-11-22 by Ken Serrine
Emmanuel Dreyfus wrote:
>On Mon, Nov 21, 2005 at 12:05:50PM -0600, Ken Serrine wrote: > > >>>UNIX or INET sockets? If UNIX, wrong permissions? Any error message in >>>sendmaill logs? >>> >>> >>> >>INET. >>Here's the error I get, even though I can see the milter running in my >>process list. >> >> >> >>>Nov 21 10:25:05 mail3 sendmail[2642]: jALGP5m2002642: Milter (greylist): error c >>>onnecting to filter: Connection refused by localhost >>> >>> > >Does netstat -na show an open port? > > > >The port does not show open. Is there a special (gdb) command that I need to use? >------------------------------------------------------------------------ > > >
2005-11-22 by Ken Serrine
Emmanuel Dreyfus wrote: >On Mon, Nov 21, 2005 at 12:05:50PM -0600, Ken Serrine wrote: > > >>>UNIX or INET sockets? If UNIX, wrong permissions? Any error message in >>>sendmaill logs? >>> >>> >>> >>INET. >>Here's the error I get, even though I can see the milter running in my >>process list. >> >> >> >>>Nov 21 10:25:05 mail3 sendmail[2642]: jALGP5m2002642: Milter (greylist): error c >>>onnecting to filter: Connection refused by localhost >>> >>> > >Does netstat -na show an open port? > > I tried running the milter under gdb using UNIX sockets, and at first it seems like it's going to work, but then it quickly goes into an error state: Nov 22 15:46:22 mail3 sendmail[21377]: jAMLkME9021377: Milter (greylist): error connecting to filter: Connection refused by /var/milter-greylist/milter-greylist.sock Nov 22 15:46:22 mail3 sendmail[21377]: jAMLkME9021377: Milter (greylist): to error state If I run the milter under INET sockets first, and then attach gdb to it, the port shows open with netstat -na, but sendmail can no longer talk to the milter. I get the following Nov 22 15:28:57 mail3 sendmail[20361]: jAMLRv75020361: Milter (greylist): to error state Nov 22 15:28:57 mail3 sendmail[20361]: jAMLRv75020361: Milter (greylist): init failed to open I can't get it to crash under INET sockets anyway, so that seems to solve my problem, but I won't be able to provide a backtrace of the crash under UNIX sockets unless I can get the milter running under gdb and sendmail using it.