Yahoo Groups archive

Milter-greylist

Index last updated: 2026-04-28 23:32 UTC

Thread

SIGSEV upon conf reload

SIGSEV upon conf reload

2004-05-27 by Cyril Guibourg

Hi list,

Did someone experienced the same behaviour ?

milter-greylist running fine for a while, updates done to greylist.conf
and then process killed after some memory access:


May 27 18:14:28 sulfateuse milter-greylist: reloading
 "/usr/local/etc/mail/greylist.conf"
May 27 18:14:28 sulfateuse sm-mta[68434]: i4RGENbH068434: 
milter_read(greylist): cmd read returned 0, expecting 5
May 27 18:14:28 sulfateuse sm-mta[68434]: i4RGENbH068434: Milter (greylist): 
to error state
May 27 18:22:20 sulfateuse sm-mta[68724]: i4RGMJQe068724: Milter (greylist): 
error connecting to filter: Connection refused by 
/var/milter-greylist/milter-greylist.sock

May 27 18:14:28 blackbox /kernel: pid 66307 (milter-greylist), uid 25:
exited on signal 11


Context:
FreeBSD 4.10-RC3, milter-greylist 1.2.2 running into a jail.

I have no core, perhaps it is due to the jail (to be checked). I'll try to
reproduce this behaviour in order to get a core.

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-27 by manu@netbsd.org

Cyril Guibourg <cg+milter-greylist@...> wrote:

> milter-greylist running fine for a while, updates done to greylist.conf
> and then process killed after some memory access:
(snip) 
> I have no core, perhaps it is due to the jail (to be checked). I'll try to
> reproduce this behaviour in order to get a core.

If you can't get a core, you can run it under gdb, that should do it.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by manu@netbsd.org

Cyril Guibourg <cg+milter-greylist@...> wrote:

> Well, I am able to reproduce the error very easily:
> 
>       - start milter greylist
>       - one tupple processing
>       - update of config file mtime
>       - one tupple processing
> 
> You don't even need to alter the content of the config file.

It's crashing in the parser. I think I'll need your config file to
reproduce that. The file generated by yacc might be usefull (just in
case it is different from what I get)

Will it crash if you don't do the first tuple processing? 

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Cyril Guibourg

manu@... writes:

> If you can't get a core, you can run it under gdb, that should do it.

Well, I am able to reproduce the error very easily:

      - start milter greylist
      - one tupple processing
      - update of config file mtime
      - one tupple processing

You don't even need to alter the content of the config file.

[Switching to process 95023, thread 6]

Program received signal SIGSEGV, Segmentation fault.
0x804e915 in conf_parse () at y.tab.c:751
751     y.tab.c: No such file or directory.
(gdb) where
#0  0x804e915 in conf_parse () at y.tab.c:751
#1  0x8052009 in conf_load () at conf.c:104
#2  0x805211e in conf_update () at conf.c:136
#3  0x8049a68 in mlfi_envfrom (ctx=0x8066d80, envfrom=0x805f080) at milter-greylist.c:167
#4  0x2807ba76 in mi_clr_macros () from /usr/lib/libmilter.so.2
#5  0x2807b248 in mi_engine () from /usr/lib/libmilter.so.2
#6  0x2807aec1 in mi_handle_session () from /usr/lib/libmilter.so.2
#7  0x2807a676 in mi_thread_handle_wrapper () from /usr/lib/libmilter.so.2
#8  0x2809b240 in _thread_start () from /usr/lib/libc_r.so.4
#9  0x0 in ?? ()


If you need more details let me know.

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Cyril Guibourg

manu@... writes:

> It's crashing in the parser. I think I'll need your config file to
> reproduce that. The file generated by yacc might be usefull (just in
> case it is different from what I get)

OK,  I send them to you in a separate message.

> Will it crash if you don't do the first tuple processing? 

Yes.

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by manu@netbsd.org

Cyril Guibourg <cg+milter-greylist@...> wrote:

> > Will it crash if you don't do the first tuple processing?  
> Yes.

So you have a setup where:
s
1) startup milter-greylist
2) touch the config file
3) handle a tuple

Will crash reliabily, right?

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by manu@netbsd.org

Cyril Guibourg <cg+milter-greylist@...> wrote:

> > So you have a setup where:
> >
> > 1) startup milter-greylist
> > 2) touch the config file
> > 3) handle a tuple
> >
> > Will crash reliabily, right?
> 
> right

Anyone else has this problem? I can't reproduce it at mine, even with
downgrading bison to the same version as Cyril (1.75)

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Cyril Guibourg

manu@... writes:

> So you have a setup where:
>
> 1) startup milter-greylist
> 2) touch the config file
> 3) handle a tuple
>
> Will crash reliabily, right?

right

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by manu@netbsd.org

Cyril Guibourg <cg+milter-greylist@...> wrote:

> If you wish I can dig deeper on it this weekend. I will just need some
> guidance for being able to trace into the parser with gdb. It's a long
> time I don't play with exotic utilities ;-)

The first step would be to build with latest bison and check if the
problem is still there. No need to rediscover something that has already
been fixed months ago.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Cyril Guibourg

manu@... writes:

> Anyone else has this problem? I can't reproduce it at mine, even with
> downgrading bison to the same version as Cyril (1.75)

If you wish I can dig deeper on it this weekend. I will just need some
guidance for being able to trace into the parser with gdb. It's a long
time I don't play with exotic utilities ;-)

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Cyril Guibourg

manu@... writes:

> The first step would be to build with latest bison and check if the
> problem is still there. No need to rediscover something that has already
> been fixed months ago.

Aaaargh !  Same issue with bison 1.875

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by manu@netbsd.org

Cyril Guibourg <cg+milter-greylist@...> wrote:

> > The first step would be to build with latest bison and check if the
> > problem is still there. No need to rediscover something that has already
> > been fixed months ago.
> Aaaargh !  Same issue with bison 1.875

Try upgrading flex as well?

Build the beast with -g, make sure the binary is not stripped at install
step, run it under gdb, and we'll have the source code where it crashes.

I hope we can see some code I wrote in the yacc specification file,
because if the problem occurs in the parser internals, it will be a
nightmare to debug.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Cyril Guibourg

manu@... writes:

> Try upgrading flex as well?

No, simply forgot to do it.

> Build the beast with -g, make sure the binary is not stripped at install
> step, run it under gdb, and we'll have the source code where it crashes.

This is what I did when I was trying to reproduce with gdb `attached' to
the running process. gdb claimed he was unable to open y.tab.c ...

> I hope we can see some code I wrote in the yacc specification file,
> because if the problem occurs in the parser internals, it will be a
> nightmare to debug.

I am afraid I will have to add some debug print statements in order
to figure out where the code crashes.

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by manu@netbsd.org

Cyril Guibourg <cg+milter-greylist@...> wrote:

> > Build the beast with -g, make sure the binary is not stripped at install
> > step, run it under gdb, and we'll have the source code where it crashes.
> This is what I did when I was trying to reproduce with gdb `attached' to
> the running process. gdb claimed he was unable to open y.tab.c ...

Copy conf_yacc.c to y.tab.c and gdb should be happy with that.
 
> > I hope we can see some code I wrote in the yacc specification file,
> > because if the problem occurs in the parser internals, it will be a
> > nightmare to debug. 
> I am afraid I will have to add some debug print statements in order
> to figure out where the code crashes.

You can add printf at the beginning and the end of command blocks in the
conf_yacc.y file, that could help. But source debugging should work with
the trick described above.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Cyril Guibourg

manu@... writes:

> You can add printf at the beginning and the end of command blocks in the
> conf_yacc.y file, that could help. But source debugging should work with
> the trick described above.

I've put conf_yacc.y in debug mode it didn't help.

This is very strange: the very first time conf_parse is called, I see all
state machine debug output. On reload, I go directly to the sigsev. My
understanding is that some return address in the stack is incorrect.

Right ?


Breakpoint 1 at 0x8051f7a: file conf.c, line 91.
(gdb) info address conf_parse
Symbol "conf_parse" is a function at address 0x804e90c.
(gdb) r -f /milter/greylist.conf
Starting program: /home/cyril/ports/milter-greylist/work/milter-greylist-1.2.2/milter-greylist -f /milter/greylist.conf

Breakpoint 1, conf_load () at conf.c:91
91              memcpy(&conf, &defconf, sizeof(conf));
(gdb) c
Continuing.
milter-greylist: reloading "/milter/greylist.conf"
[Switching to process 10467, thread 2]

Breakpoint 1, conf_load () at conf.c:91
91              memcpy(&conf, &defconf, sizeof(conf));
(gdb) info address conf_parse
Symbol "conf_parse" is a function at address 0x804e90c.
(gdb) info frame
Stack level 0, frame at 0xbfabadfc:
 eip = 0x8051f7a in conf_load (conf.c:91); saved eip 0x805211e
 called by frame at 0xbfabae8c
 source language c.
 Arglist at 0xbfabadfc, args: 
 Locals at 0xbfabadfc, Previous frame's sp is 0x0
 Saved registers:
  ebp at 0xbfabadfc, eip at 0xbfabae00
(gdb) info register 
eax            0x2811d65c       672257628
ecx            0x15     21
edx            0x28138018       672366616
ebx            0x2807f598       671610264
esp            0xbfabade4       0xbfabade4
ebp            0xbfabadfc       0xbfabadfc
esi            0x805f090        134606992
edi            0xbfabaf4c       -1079267508
eip            0x8051f7a        0x8051f7a
eflags         0x286    646
cs             0x1f     31
ss             0x2f     47
ds             0x2f     47
es             0x2f     47
fs             0x2f     47
gs             0x2f     47
(gdb) info threads
  6 process 10467, thread 6  0x280d7653 in _thread_kern_sched () from /usr/lib/libc_r.so.4
  5 process 10467, thread 5  0x280d7653 in _thread_kern_sched () from /usr/lib/libc_r.so.4
  4 process 10467, thread 4  0x280d7653 in _thread_kern_sched () from /usr/lib/libc_r.so.4
  3 process 10467, thread 3  0x280d7653 in _thread_kern_sched () from /usr/lib/libc_r.so.4
* 2 process 10467, thread 2  conf_load () at conf.c:91
  1 process 10467, thread 1  0x280d7653 in _thread_kern_sched () from /usr/lib/libc_r.so.4
(gdb) step
96              if ((stream = fopen(conffile, "r")) == NULL) {
(gdb) 
103             conf_in = stream;
(gdb) 
104             conf_parse();
(gdb) 

Program received signal SIGSEGV, Segmentation fault.
0x804e915 in conf_parse () at y.tab.c:751
751     }
(gdb) info address conf_parse
Symbol "conf_parse" is a function at address 0x804e90c.
(gdb) p yystate
$1 = 671610264
(gdb) l 730
725       YYFPRINTF (yyout, ")");
726     }
727     #endif /* YYDEBUG. */
728
729
730     /*-----------------------------------------------.
731     | Release the memory associated to this symbol.  |
732     `-----------------------------------------------*/
733
734     static void
(gdb) l
735     #if defined (__STDC__) || defined (__cplusplus)
736     yydestruct (int yytype, YYSTYPE yyvalue)
737     #else
738     yydestruct (yytype, yyvalue)
739         int yytype;
740         YYSTYPE yyvalue;
741     #endif
742     {
743       /* Pacify ``unused variable'' warnings.  */
744       (void) yyvalue;
(gdb) l
745
746       switch (yytype)
747         {
748           default:
749             break;
750         }
751     }
752
753     ^L
754
(gdb) info frame
Stack level 0, frame at 0xbfabaddc:
 eip = 0x804e915 in conf_parse (y.tab.c:751); saved eip 0x8052009
 called by frame at 0xbfabadfc
 source language c.
 Arglist at 0xbfabaddc, args: 
 Locals at 0xbfabaddc, Previous frame's sp is 0x0
 Saved registers:
  ebx at 0xbfa884c4, ebp at 0xbfabaddc, esi at 0xbfa884c8, edi at 0xbfa884cc, eip at 0xbfabade0
(gdb) info register
eax            0x28131d20       672341280
ecx            0x2811b874       672249972
edx            0x8      8
ebx            0x2807f598       671610264
esp            0xbfa884d0       0xbfa884d0
ebp            0xbfabaddc       0xbfabaddc
esi            0x805f090        134606992
edi            0xbfabaf4c       -1079267508
eip            0x804e915        0x804e915
eflags         0x10282  66178
cs             0x1f     31
ss             0x2f     47
ds             0x2f     47
es             0x2f     47
fs             0x2f     47
gs             0x2f     47

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by manu@netbsd.org

Cyril Guibourg <cg+milter-greylist@...> wrote:

> This is very strange: the very first time conf_parse is called, I see all
> state machine debug output. On reload, I go directly to the sigsev. My
> understanding is that some return address in the stack is incorrect.

Yes, this is probably something like that (check with the bt command).
This is probably caused by some local variable overflow, but
yydestruct() is so uterly void that I can't understand what could have
gone wrong.

Maybe in the calling function? Who's calling yydestruct()?

We should probably move that discussion off-list, as most subscribers
are probably not very interested (tell us if you want to help debugging
that) 

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Cyril Guibourg

manu@... writes:

> We should probably move that discussion off-list, as most subscribers
> are probably not very interested (tell us if you want to help debugging
> that) 

Agree with you.

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Ethan Burnside

Yea, I grabbed 1.3.4 last night and put it on our cluster of 5 or so
mail servers.  (somewhat of a forced upgrade... we were in desperate
need of the dumpfreq setting)  We have the greylist.conf auto generated
from a database every time someone changes their settings from a
web-based interface.  We also monitor the processes on all 5 of these
mail servers to make sure all the necessary processes are running all of
the time.  I can reliably go into the web interface, make a change, and
minutes later all of our mail servers are red on the monitor and paging
us, angry that milter-greylist is no longer running.  ;-)  While I don't
recall 1.2.2 crashing all at once like this, it hadn't been particularly
reliable either, so we've just kind of got used to having a cron job
setup to restart it periodically.

We're running on slack linux, 8.1.  Somewhat dated distro, but we
maintain our own packages and update as security flaws are discovered. 
I believe the version of bison we're using is 1.35, but we've never had
any problems related to the bison version before.

Let me know if there's anything I can do to help... I've never done much
with gdb for debugging and we don't have the systems setup to dump
cores, so I'm not sure how much help I can be, but I'll do what I can.

Also, props on a very nice piece of software.  It's very effective at
what it does, when it's running.  ;-)

Cheers,

~Ethan B.





-- 
--------------------------
Ethan Burnside - Founder
Kattare Internet Services
http://www.kattare.com
--------------------------



Quoting manu@...:
Show quoted textHide quoted text
> Cyril Guibourg <cg+milter-greylist@...> wrote:
> 
> > > So you have a setup where:
> > >
> > > 1) startup milter-greylist
> > > 2) touch the config file
> > > 3) handle a tuple
> > >
> > > Will crash reliabily, right?
> > 
> > right
> 
> Anyone else has this problem? I can't reproduce it at mine, even
> with
> downgrading bison to the same version as Cyril (1.75)
> 
> -- 
> Emmanuel Dreyfus
> Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
> le binaire et ceux qui ne le comprennent pas.
> manu@...
> 
> 
> ------------------------ Yahoo! Groups Sponsor
> --------------------~--> 
> Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
> Now with Pop-Up Blocker. Get it for free!
> http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/W4wwlB/TM
> --------------------------------------------------------------------~->
> 
> 
>  
> Yahoo! Groups Links
> 
> 
> 
>  
>

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by manu@netbsd.org

Ethan Burnside <burnside@...> wrote:

> Yea, I grabbed 1.3.4 last night and put it on our cluster of 5 or so
> mail servers.  (somewhat of a forced upgrade... we were in desperate
> need of the dumpfreq setting)  We have the greylist.conf auto generated
> from a database every time someone changes their settings from a
> web-based interface.  We also monitor the processes on all 5 of these
> mail servers to make sure all the necessary processes are running all of
> the time.  I can reliably go into the web interface, make a change, and
> minutes later all of our mail servers are red on the monitor and paging
> us, angry that milter-greylist is no longer running.  ;-)  While I don't
> recall 1.2.2 crashing all at once like this, it hadn't been particularly
> reliable either, so we've just kind of got used to having a cron job
> setup to restart it periodically.

Obviously there is something horribly wrong with the configuration
parser. The temporary workaround is simple: restart milter-greylist once
you changed the configuration file.

Is someone able to provide me an access to a system where the crash can
be reproduced reliabily? It's hard for me to track down bugs that I
can't see.

Anyway... Nice setup you have. Would you like to write a web page about
it? We could have a milter-greylist success-stories page.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Re: [milter-greylist] SIGSEV upon conf reload

2004-05-28 by Ethan Burnside

Sure, I have one of my coworkers setting up a dedicated virtual server
right now for you.  I'll send you hostname/login/pass details in a
seperate mail when it's all ready to go.

As for writing a webpage, I'm up for that.  Let's get this bug stuff
fixed first though, that way I can rave about the stability.  ;-)

Cheers,

~Ethan B.

-- 
--------------------------
Ethan Burnside - Founder
Kattare Internet Services
http://www.kattare.com
--------------------------



Quoting manu@...:
Show quoted textHide quoted text
> Ethan Burnside <burnside@...> wrote:
> 
> > Yea, I grabbed 1.3.4 last night and put it on our cluster of 5 or
> so
> > mail servers.  (somewhat of a forced upgrade... we were in
> desperate
> > need of the dumpfreq setting)  We have the greylist.conf auto
> generated
> > from a database every time someone changes their settings from a
> > web-based interface.  We also monitor the processes on all 5 of
> these
> > mail servers to make sure all the necessary processes are running
> all of
> > the time.  I can reliably go into the web interface, make a change,
> and
> > minutes later all of our mail servers are red on the monitor and
> paging
> > us, angry that milter-greylist is no longer running.  ;-)  While I
> don't
> > recall 1.2.2 crashing all at once like this, it hadn't been
> particularly
> > reliable either, so we've just kind of got used to having a cron
> job
> > setup to restart it periodically.
> 
> Obviously there is something horribly wrong with the configuration
> parser. The temporary workaround is simple: restart milter-greylist
> once
> you changed the configuration file.
> 
> Is someone able to provide me an access to a system where the crash
> can
> be reproduced reliabily? It's hard for me to track down bugs that I
> can't see.
> 
> Anyway... Nice setup you have. Would you like to write a web page
> about
> it? We could have a milter-greylist success-stories page.
> 
> -- 
> Emmanuel Dreyfus
> Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
> le binaire et ceux qui ne le comprennent pas.
> manu@...
> 
> 
> ------------------------ Yahoo! Groups Sponsor
> --------------------~--> 
> Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
> Now with Pop-Up Blocker. Get it for free!
> http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/W4wwlB/TM
> --------------------------------------------------------------------~->
> 
> 
>  
> Yahoo! Groups Links
> 
> 
> 
>  
>

Move to quarantaine

This moves the raw source file on disk only. The archive index is not changed automatically, so you still need to run a manual refresh afterward.