Yahoo Groups archive

Milter-greylist

Index last updated: 2026-04-28 23:32 UTC

Thread

4.0beta2 is available

4.0beta2 is available

2007-09-18 by manu@netbsd.org

Here it is:

http://ftp.espci.fr/pub/milter-greylist/milter-greylist.4.0n2.tgz
MD5 (milter-greylist-4.0b2.tgz) = 7f75a4109db5031d3db31ca968bb9c6b

Changes since 4.0beta1:
        Treat protocol errors in urlcheck clauses as temporary failures
        Report missing SPF reasons in X-Greylist (Benoit Branciard)
        Allow building objects outside of source directory (Mattheu Herrb)
        Fix configure LDFLAG generation, -R was missing (Mattheu Herrb)
        Fix MX sync on Solaris (Mattheu Herrb)

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] 4.0beta2 is available

2007-09-18 by shuttlebox

On 9/18/07, manu@... <manu@...> wrote:
>          Fix MX sync on Solaris (Mattheu Herrb)

What was the problem? I've been having problems with MX sync on
Solaris and would be very pleased to see it fixed because many servers
seems to try the other MX and then give up since it will be greylisted
there too if the sync doesn't work. I don't remember it being
discussed on the list either.

-- 
/peter

Re: [milter-greylist] 4.0beta2 is available

2007-09-18 by Seth Mos

> On 9/18/07, manu@... <manu@...> wrote:
>>          Fix MX sync on Solaris (Mattheu Herrb)
>
> What was the problem? I've been having problems with MX sync on
> Solaris and would be very pleased to see it fixed because many servers
> seems to try the other MX and then give up since it will be greylisted
> there too if the sync doesn't work. I don't remember it being
> discussed on the list either.

I am questioning the validity of the MX sync feature, in my personal
experience with it I ran into quite a number of mail runs which in quick
sequential order process all MX records for the domain in question.

This resulted in the 70% of the spam coming through in my setup. Perhaps
we should add a warning into the man page warning of this.

After disabling the MX communication I see over 50% of the email
greylisting once on the primary and once on the secondary, never to return
again.

Cheers,

Seth

Re: [milter-greylist] 4.0beta2 is available

2007-09-19 by manu@netbsd.org

shuttlebox <shuttlebox@...> wrote:

> >          Fix MX sync on Solaris (Mattheu Herrb)
> What was the problem? 

On Solaris 10, in local_addr(), bind() was returning -1 without setting
errno, thus causing MX sync to just not work. Perhaps the contributor
can tell us more...

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] 4.0beta2 is available

2007-09-19 by shuttlebox

On 9/18/07, Seth Mos <seth.mos@...> wrote:
>  After disabling the MX communication I see over 50% of the email
>  greylisting once on the primary and once on the secondary, never to return
>  again.

And you will see the same with legit mail which I don't need, there's
enough questions/complaints about greylisting as there is. I need the
MX sync to work, otherwise I will have to spend all day whitelisting
stuff.

-- 
/peter

Re: [milter-greylist] 4.0beta2 is available

2007-09-19 by Seth Mos

shuttlebox schreef:
> 
> 
> On 9/18/07, Seth Mos <seth.mos@... <mailto:seth.mos%40xs4all.nl>> 
> wrote:
>  > After disabling the MX communication I see over 50% of the email
>  > greylisting once on the primary and once on the secondary, never to 
> return
>  > again.
> 
> And you will see the same with legit mail which I don't need, there's
> enough questions/complaints about greylisting as there is. I need the
> MX sync to work, otherwise I will have to spend all day whitelisting
> stuff.

If you are focusing on accepting email, I suggest you have a look at the 
UrlCheck feature I wrote that accepts mail from valid mail server.

It made a huge improvement on my greylisting effort, a lot of email 
comes through without issues and delay, I get less complaints from the 
people around me, and I still have a low amount of spam coming through.

Search the list on "Optimistic MX host whitelisting" or something along 
those lines. That should give you enough information to go on.

Cheers,

Seth

Re: 4.0beta2 is available

2007-09-19 by Jim Hermann

--- In milter-greylist@yahoogroups.com, manu@... wrote:

> Changes since 4.0beta1:
>         Report missing SPF reasons in X-Greylist (Benoit Branciard)

What is FORMAT STRING for the missing SPF reasons for use in a 
CUSTOM REPORT?

I currently use these configuration lines at the end of 
greylist.conf:

racl greylist spf rcpt /.*@.*/ report "SPF_OK From %f at IP %i 
Delayed for %E by %V"
racl greylist rcpt /.*@.*/ report "SPF_NOT_OK From %f at IP %i 
Delayed for %E by %V"

Thanks.

Jim

Re: [milter-greylist] Re: 4.0beta2 is available

2007-09-19 by manu@netbsd.org

Jim Hermann <hostmaster@...> wrote:

> > Changes since 4.0beta1:
> >         Report missing SPF reasons in X-Greylist (Benoit Branciard)
> 
> What is FORMAT STRING for the missing SPF reasons for use in a 
> CUSTOM REPORT?
> 
> I currently use these configuration lines at the end of 
> greylist.conf:
> 
> racl greylist spf rcpt /.*@.*/ report "SPF_OK From %f at IP %i 
> Delayed for %E by %V"
> racl greylist rcpt /.*@.*/ report "SPF_NOT_OK From %f at IP %i 
> Delayed for %E by %V"

You mean you'd like a format string that is substituted by the SPF
status (ie: pass, fail, none)?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: 4.0beta2 is available

2007-09-20 by Jim Hermann

--- In milter-greylist@yahoogroups.com, manu@... wrote:
>
> Jim Hermann <hostmaster@...> wrote:
> 
> > > Changes since 4.0beta1:
> > >         Report missing SPF reasons in X-Greylist (Benoit 
Branciard)
> > 
> > What is FORMAT STRING for the missing SPF reasons for use in a 
> > CUSTOM REPORT?
> > 
> 
> You mean you'd like a format string that is substituted by the SPF
> status (ie: pass, fail, none)?

I thought that's what you meant by "Report missing SPF reasons in X-
Greylist."  I see now that the maillog contains "Sender is SPF-
compliant" when they pass SPF.

I can wait for the SPF upgrade to all the possible statuses.  I 
don't think it is a good idea to lump several statuses into one 
status called fail.  A SOFTFAIL is entirely different than a hard 
FAIL.

Jim

Re: [milter-greylist] Re: 4.0beta2 is available

2007-09-20 by manu@netbsd.org

Jim Hermann <hostmaster@...> wrote:

> I can wait for the SPF upgrade to all the possible statuses.  I 
> don't think it is a good idea to lump several statuses into one 
> status called fail.  A SOFTFAIL is entirely different than a hard 
> FAIL.

What is a softfail? Is it just that you failed to obtain the SPF record?
(ie: DNS failure)

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: {Disarmed} Re: [milter-greylist] Re: 4.0beta2 is available

2007-09-20 by Benoit Branciard

manu@... a \ufffdcrit :
> 
> 
> Jim Hermann <hostmaster@... <mailto:hostmaster%40uuism.net>> wrote:
> 
>  > I can wait for the SPF upgrade to all the possible statuses. I
>  > don't think it is a good idea to lump several statuses into one
>  > status called fail. A SOFTFAIL is entirely different than a hard
>  > FAIL.
> 
> What is a softfail? Is it just that you failed to obtain the SPF record?
> (ie: DNS failure)

This would be "TempError".
The "SoftFail" result indicates the sender host should not be 
authorized, but not as categorically as a "Fail". This is matched bu a 
"~" entry in the SPF record. Cf. http://www.openspf.org/RFC_4408#op-result.

Re: {Disarmed} Re: [milter-greylist] Re: 4.0beta2 is available

2007-09-20 by Emmanuel Dreyfus

On Thu, Sep 20, 2007 at 09:17:41AM +0200, Benoit Branciard wrote:
> The "SoftFail" result indicates the sender host should not be 
> authorized, but not as categorically as a "Fail". This is matched bu a 
> "~" entry in the SPF record. Cf. http://www.openspf.org/RFC_4408#op-result.

So SPF is "neutral" for the message, right?

-- 
Emmanuel Dreyfus
manu@...

Re: {Disarmed} Re: {Disarmed} Re: [milter-greylist] Re: 4.0beta2 is available

2007-09-20 by Benoit Branciard

Emmanuel Dreyfus a \ufffdcrit :
> 
> 
> On Thu, Sep 20, 2007 at 09:17:41AM +0200, Benoit Branciard wrote:
>  > The "SoftFail" result indicates the sender host should not be
>  > authorized, but not as categorically as a "Fail". This is matched bu a
>  > "~" entry in the SPF record. Cf. 
> http://www.openspf.org/RFC_4408#op-result. 
> 
> So SPF is "neutral" for the message, right?

No, "Neutral" (?) and "SoftFail" (~) are different.

"Neutral" is merely "I can't tell" and must be treated the same way as 
"None" (no SPF record at all), whereas "SoftFail" is more repressive 
("it shouldn't")

{Disarmed} Re: [milter-greylist] Re: 4.0beta2 is available

2007-09-20 by Jim Hermann

--- In milter-greylist@yahoogroups.com, Benoit Branciard 
<benoit.branciard@...> wrote:
>
> manu@... a écrit :
> > 
> > 
> > Jim Hermann <hostmaster@... <mailto:hostmaster%40uuism.net>> 
wrote:
> > 
> >  > I can wait for the SPF upgrade to all the possible statuses. I
> >  > don't think it is a good idea to lump several statuses into 
one
> >  > status called fail. A SOFTFAIL is entirely different than a 
hard
> >  > FAIL.
> > 
> > What is a softfail? Is it just that you failed to obtain the SPF 
record?
> > (ie: DNS failure)
> 
> This would be "TempError".
> The "SoftFail" result indicates the sender host should not be 
> authorized, but not as categorically as a "Fail". This is matched 
bu a 
> "~" entry in the SPF record. Cf. 
http://www.openspf.org/RFC_4408#op-result.
>

The possible results are:

2.5. Interpreting the Result
2.5.1. None
2.5.2. Neutral
2.5.3. Pass - We also want PassAll for spammers who use +all
2.5.4. Fail
2.5.5. SoftFail - is often used for testing
2.5.6. TempError
2.5.7. PermError

2.5.5. SoftFail
A "SoftFail" result should be treated as somewhere between a "Fail" 
and a "Neutral". The domain believes the host is not authorized but 
is not willing to make that strong of a statement. Receiving 
software SHOULD NOT reject the message based solely on this result, 
but MAY subject the message to closer scrutiny than normal.

The domain owner wants to discourage the use of this host and thus 
desires limited feedback when a "SoftFail" result occurs. For 
example, the recipient's Mail User Agent (MUA) could highlight 
the "SoftFail" status, or the receiving MTA could give the sender a 
message using a technique called "greylisting" whereby the MTA can 
issue an SMTP reply code of 451 (4.3.0 DSN code) with a note the 
first time the message is received, but accept it the second time.

Jim

{Disarmed} Re: {Disarmed} Re: [milter-greylist] Re: 4.0beta2 is available

2007-09-20 by Jim Hermann

--- In milter-greylist@yahoogroups.com, Benoit Branciard 
<benoit.branciard@...> wrote:
>
> Emmanuel Dreyfus a écrit :
> > 
> > 
> > On Thu, Sep 20, 2007 at 09:17:41AM +0200, Benoit Branciard wrote:
> >  > The "SoftFail" result indicates the sender host should not be
> >  > authorized, but not as categorically as a "Fail". This is 
matched bu a
> >  > "~" entry in the SPF record. Cf. 
> > http://www.openspf.org/RFC_4408#op-result. 
> > 
> > So SPF is "neutral" for the message, right?
> 
> No, "Neutral" (?) and "SoftFail" (~) are different.
> 
> "Neutral" is merely "I can't tell" and must be treated the same 
way as 
> "None" (no SPF record at all), whereas "SoftFail" is more 
repressive 
> ("it shouldn't")
>

SOFTFAIL is often used during testing.

Jim

Re: [milter-greylist] 4.0beta2 is available

2007-09-26 by shuttlebox

On 9/18/07, manu@... <manu@...> wrote:
>  Changes since 4.0beta1:
>          Treat protocol errors in urlcheck clauses as temporary failures
>          Report missing SPF reasons in X-Greylist (Benoit Branciard)
>          Allow building objects outside of source directory (Mattheu Herrb)
>          Fix configure LDFLAG generation, -R was missing (Mattheu Herrb)
>          Fix MX sync on Solaris (Mattheu Herrb)

Another thing, the patch for Solaris file descriptor problems never
made it? There was talk about an option, disabled by default.

-- 
/peter

Re: [milter-greylist] 4.0beta2 is available

2007-09-26 by manu@netbsd.org

shuttlebox <shuttlebox@...> wrote:

> Another thing, the patch for Solaris file descriptor problems never
> made it? There was talk about an option, disabled by default.

I've been too busy for that one, but I have to look into it, since
Solaris status is really bad.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Fixing Solaris troubles

2007-09-26 by manu@netbsd.org

shuttlebox <shuttlebox@...> wrote:

> Another thing, the patch for Solaris file descriptor problems never
> made it? There was talk about an option, disabled by default.

I just integrated 4 out of the 5 patches from Johann E. Klasek:

        Cleanup temporary file after DB dump failure
        Handle libc that fails stdio without setting errno 
        Fixes the usage of the thread-proof resolver library
        Do not quit on non fatal errors

The last one is the biggest, it's the workaround for Solaris' limitation
to file descriptors < 256. We had a positive report and a negative one.
It would be nice if more people could give it a try:
http://jk.kom.tuwien.ac.at/~jklasek/software/milter-greylist/
mg.stdio-solaris.patch

I'm also somewhat puzzled about the Solaris fiasco: it seems
milter-greylist used to work on Solaris. What change broke it? Who is
using Solaris with a 32 bit milter-greylist? What is milter-greylist
version?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] Fixing Solaris troubles

2007-09-26 by Chris Hoogendyk

manu@... wrote:
> shuttlebox <shuttlebox@...> wrote:
>
>   
>> Another thing, the patch for Solaris file descriptor problems never
>> made it? There was talk about an option, disabled by default.
>>     
>
> I just integrated 4 out of the 5 patches from Johann E. Klasek:
>
>         Cleanup temporary file after DB dump failure
>         Handle libc that fails stdio without setting errno 
>         Fixes the usage of the thread-proof resolver library
>         Do not quit on non fatal errors
>
> The last one is the biggest, it's the workaround for Solaris' limitation
> to file descriptors < 256. We had a positive report and a negative one.
> It would be nice if more people could give it a try:
> http://jk.kom.tuwien.ac.at/~jklasek/software/milter-greylist/
> mg.stdio-solaris.patch
>
> I'm also somewhat puzzled about the Solaris fiasco: it seems
> milter-greylist used to work on Solaris. What change broke it? Who is
> using Solaris with a 32 bit milter-greylist? What is milter-greylist
> version?


hmm. milter-greylist-1.6 compiled 32bit on Solaris 9 with gcc 3.3.2. 
SPARC (Sun E250).

I have no idea if this is the same issue. We periodically find 
milter-greylist not running. This happens more often on our more heavily 
loaded mail system. We have a cron job that checks whether it is running 
and restarts it if it is not. It appears that it may happen at times 
when we are really being hammered by spammers.

Major caveat: our milter-greylist-1.6 is patched with a code segment 
that reads popip.db (berkeley db created and maintained by poprelayd) 
and bypasses greylisting if someone reads their mail from that IP before 
trying to send mail. We're ditching this now, since we have moved to 
requiring our users to use smtp-auth, among other things. But, it is in 
the code we have been running.



---------------

Chris Hoogendyk

-
   O__  ---- Systems Administrator
  c/ /'_ --- Biology & Geology Departments
 (*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst 

<hoogendyk@...>

--------------- 

Erd\ufffds 4

Re: Fixing Solaris troubles

2007-09-26 by mloewen16823

--- In milter-greylist@yahoogroups.com, manu@... wrote:

> I'm also somewhat puzzled about the Solaris fiasco: it seems
> milter-greylist used to work on Solaris. What change broke it? Who is
> using Solaris with a 32 bit milter-greylist? What is milter-greylist
> version?

   I'm running Solaris 9 (SPARC), 64-bit kernel, with 32-bit
milter-greylist-3.1.6 compiled with gcc-3.4.2.  I haven't run into any
problems with this configuration, but my mail volumne isn't very high.


Mike Loewen                     Information Technology Services
Email:  mcl8@...            The Pennsylvania State University
Phone:  (814) 863-7245          University Park, PA  16802
Office: 214E Computer Bldg.     http://sturgeon.css.psu.edu/mloewen.html

Re: [milter-greylist] Fixing Solaris troubles

2007-09-26 by Johann Klasek

On Wed, Sep 26, 2007 at 04:36:27PM -0400, Chris Hoogendyk wrote:
> 
> 
> manu@... wrote:
> > shuttlebox <shuttlebox@...> wrote:
> >
> >   
> >> Another thing, the patch for Solaris file descriptor problems never
> >> made it? There was talk about an option, disabled by default.
> >>     
> >
> > I just integrated 4 out of the 5 patches from Johann E. Klasek:
> >
> >         Cleanup temporary file after DB dump failure
> >         Handle libc that fails stdio without setting errno 
> >         Fixes the usage of the thread-proof resolver library
> >         Do not quit on non fatal errors
> >
> > The last one is the biggest, it's the workaround for Solaris' limitation
> > to file descriptors < 256. We had a positive report and a negative one.
> > It would be nice if more people could give it a try:
> > http://jk.kom.tuwien.ac.at/~jklasek/software/milter-greylist/
> > mg.stdio-solaris.patch
> >
> > I'm also somewhat puzzled about the Solaris fiasco: it seems
> > milter-greylist used to work on Solaris. What change broke it? Who is
> > using Solaris with a 32 bit milter-greylist? What is milter-greylist
> > version?
> 
> 
> hmm. milter-greylist-1.6 compiled 32bit on Solaris 9 with gcc 3.3.2. 
> SPARC (Sun E250).
> 
> I have no idea if this is the same issue. We periodically find 
> milter-greylist not running. This happens more often on our more heavily 
> loaded mail system. We have a cron job that checks whether it is running 
> and restarts it if it is not. It appears that it may happen at times 
> when we are really being hammered by spammers.

A far as I have experienced in Solaris this may have one of following
reasons:

  - Up to Solaris 9 the default file descriptor limit is set to 256. As
	long this is not increased the basic libc stdio file handle
	problem does not come into play. Milter-greylist will just exit
	because the file *descriptor* limit hits. This rapidly happens
	on higher request rates (spam attack) when concurrent sessions
	are going to lift up.
	In low traffic environments this will not happen often.

 - If someone increases the file descriptor limit to above 256, someone
	will observe that milter-greylist will terminate even at times
	not related with higher request rates. This is because the
	file descriptor number (not the count of used file descriptors) is
	counted up and up exceeding 255 - Solaris libc's stdio
	implementation has in its file handle structure only a 8-bit
	component only capable to hold file descriptor values from 0 to
	255. fopen() or fdopen() are failing forcing milter-greylist to
	exit ...

I tend to connect Chris problem with the first reason.
To overcome this problem you need to raise the file descriptor limit to
above 256 *and* take either the Solaris workaround patch mentioned in
the discussion before or using a stdio replacement like SFIO (not tested
yet - speaking for myself). Or you could switch to Solaris 10 which has
easier ways to overcome this file handle issue.


Johann

Re: [milter-greylist] Fixing Solaris troubles

2007-09-27 by manu@netbsd.org

Johann Klasek <johann@...> wrote:

>   - Up to Solaris 9 the default file descriptor limit is set to 256. As
>       long this is not increased the basic libc stdio file handle
>       problem does not come into play. Milter-greylist will just exit
>       because the file *descriptor* limit hits. This rapidly happens
>       on higher request rates (spam attack) when concurrent sessions
>       are going to lift up.
>       In low traffic environments this will not happen often.

Ok, so this problem is just popping up now because of traffic raise? If
that's the case, then people using older releases of milter-greylist
should have been burnt too, and if they have not yet, then it means they
can safely upgrade, right?

Do I understand the problem correctly: when you hit this Solaris
limitation, fopen/fdopen fails, returning NULL with errno == 0, right?

A simple way of fixing the issue would just be to issue a TEMPFAIL in
this situation, which would reflect the real problem: your system cannot
function proprely because of resource exhaustion, and the sender should
retry later.

Of course, if there is a file descriptor leak in milter-greylist, we are
screwed, because you won't ever be able to cope with the situation. But
nobody reported that kind of problem so far.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] Fixing Solaris troubles

2007-09-27 by manu@netbsd.org

Emmanuel Dreyfus <manu@...> wrote:

> A simple way of fixing the issue would just be to issue a TEMPFAIL in
> this situation, which would reflect the real problem: your system cannot
> function proprely because of resource exhaustion, and the sender should
> retry later.

Wait, wait, wait... fopen/fdopen usage is quite limited:
1) dump.c, for duming the database
2) sync.c, for MX sync
3) milter-greylist.c, for the PID file
4) stat.c, for the custom stat output

#3 and #4 are not a problem: it happens on startup and you'll never use
a file descriptor above 256 here

#2 can happen if the MX sync connection breaks during operation
#1 is what Solaris users experience

The TEMPFAIL solution won't help here, but we can make sure we keep a
file descriptor < 256 open for dumps: after a dumps succeeds, we open
/dev/null and use dup2 to associate it to the low file descriptor we
just used. On the next dump, we open the new file, use dup2 and here we
are. I quickly wrote some code for that idea just below (not tested, not
built)

Opinions?

diff -U2 -r1.32 dump.c
--- dump.c      27 Sep 2007 03:47:25 -0000      1.32
+++ dump.c      27 Sep 2007 03:35:02 -0000
@@ -202,6 +202,7 @@
        int final;
 {
+       static int dumpfd = -1;
+       int newdumpfd;
        FILE *dump;
-       int dumpfd;
        struct timeval tv1, tv2, tv3;
        char newdumpfile[MAXPATHLEN + 1];
@@ -234,4 +235,12 @@
        }
 
+       if (dumpfd == -1) {
+               if ((dumpfd = open(_PATH_DEVNULL, O_RDONLY, 0)) == -1) {
+                       mg_log(LOG_ERR, "cannot open \"%s\", 
+                           _PATH_DEVNULL, strerror(errno));
+                       exit(EX_OSERR);
+               }
+       }
+               
        /* 
         * Dump the database in a temporary file and 
@@ -244,12 +253,14 @@
            "%s-XXXXXXXX", conf.c_dumpfile);
 
-       if ((dumpfd = mkstemp(newdumpfile)) == -1) {
+       if ((newdumpfd = mkstemp(newdumpfile)) == -1) {
                mg_log(LOG_ERR, "mkstemp(\"%s\") failed: %s", 
                    newdumpfile, strerror(errno));
-               close(dumpfd);
+               close(newdumpfd);
                unlink(newdumpfile);            /* clean up ... */
                exit(EX_OSERR);
        }
 
+       dumpfd = dup2(newdumpfd, dumpfd);
+
        errno = 0;
        if ((dump = fdopen(dumpfd, "w")) == NULL) {
@@ -278,4 +289,10 @@
            "whitelisted\n#\n", done, greylisted_count,
whitelisted_count);
 
+       fflush(dump);
+       if ((dumpfd = open(_PATH_DEVNULL, O_RDONLY, 0)) == -1) {
+               mg_log(LOG_ERR, "cannot open \"%s\", 
+                   _PATH_DEVNULL, strerror(errno));
+               exit(EX_OSERR);
+       }
        fclose(dump);
        if (s_buffer)


-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] Fixing Solaris troubles

2007-09-27 by Johann Klasek

On Thu, Sep 27, 2007 at 05:17:38AM +0200, manu@... wrote:
> Johann Klasek <johann@...> wrote:
> 
> >   - Up to Solaris 9 the default file descriptor limit is set to 256. As
> >       long this is not increased the basic libc stdio file handle
> >       problem does not come into play. Milter-greylist will just exit
> >       because the file *descriptor* limit hits. This rapidly happens
> >       on higher request rates (spam attack) when concurrent sessions
> >       are going to lift up.
> >       In low traffic environments this will not happen often.
> 
> Ok, so this problem is just popping up now because of traffic raise? If
> that's the case, then people using older releases of milter-greylist
> should have been burnt too, and if they have not yet, then it means they
> can safely upgrade, right?
> 
> Do I understand the problem correctly: when you hit this Solaris
> limitation, fopen/fdopen fails, returning NULL with errno == 0, right?

To be exactly, Solaris libc leaves errno untouched (the latest value is
preserved ...). Therefore it is necessary to explicit clear errno before
calling fdopen/fopen to distinguish file handle error from "normal"
errors ...

> 
> A simple way of fixing the issue would just be to issue a TEMPFAIL in
> this situation, which would reflect the real problem: your system cannot
> function proprely because of resource exhaustion, and the sender should
> retry later.
> 
> Of course, if there is a file descriptor leak in milter-greylist, we are
> screwed, because you won't ever be able to cope with the situation. But
> nobody reported that kind of problem so far.

I don't think there is no general file descriptor leak problem - this
had been revealed during the work on my Solaris work around.

The only leaking problem regarding file descriptor comes with configure
option --enable-dnsrbl which uses the resolver routines and the
threadsafe interface recognition in dnsbl.c (in plain 3.0 and at least
up to 4.0a6) module fails (at least for Solaris, but do work for Linux).
Multiple threads are then calling the not reentrant resolver routines
which leads to an overwritten static data condition and "lost"
descriptors which the resolver will never see again (staying around
unclosed ...).


Johann

Re: [milter-greylist] Fixing Solaris troubles

2007-09-27 by Johann Klasek

On Thu, Sep 27, 2007 at 05:38:59AM +0200, manu@... wrote:
> Emmanuel Dreyfus <manu@...> wrote:
> 
> > A simple way of fixing the issue would just be to issue a TEMPFAIL in
> > this situation, which would reflect the real problem: your system cannot
> > function proprely because of resource exhaustion, and the sender should
> > retry later.
> 
> Wait, wait, wait... fopen/fdopen usage is quite limited:
> 1) dump.c, for duming the database
> 2) sync.c, for MX sync
> 3) milter-greylist.c, for the PID file
> 4) stat.c, for the custom stat output

you forget
 5) conf.c 
	which is used on startup *and* on every change on greylist.conf

> 
> #3 and #4 are not a problem: it happens on startup and you'll never use
> a file descriptor above 256 here
> 
> #2 can happen if the MX sync connection breaks during operation
> #1 is what Solaris users experience

The problem is not the frequency of passing the above code, but the 
fact, that in multithreading the descriptor numbers are counting quickly
for beyond 256 even if free descriptor below 256 exists (im fd limit is
above 256). There maybe only a small time slot at startup (as long the
milter engine is not running) where one can rely on file descriptor values
below 256 ...

> 
> The TEMPFAIL solution won't help here, but we can make sure we keep a

I agree.

> file descriptor < 256 open for dumps: after a dumps succeeds, we open
> /dev/null and use dup2 to associate it to the low file descriptor we
> just used. On the next dump, we open the new file, use dup2 and here we
> are. I quickly wrote some code for that idea just below (not tested, not
> built)
> 
> Opinions?

There are serveral problem in this particular proposal (I have already
running into while coping which this problem in the past):

1. dumpfd initialisation (using /dev/null) is to late - until the first
	call to dump the probability to get a descriptor below 256 my be
	fairly bad ...

2. If we assuming dumpfd containes a value of less than 256, duplication
	to the reserved desciptor (dup2) works fine the first time. But
	with fclose() this descriptor is closed also! It is *not*
	possible to keep the descriptor, because fdopen() and fclose()
	are not symmetric in this manner. There would be a function
	needed, say fdclose(), which gives the descriptor back (only
	detach the descriptor from the stream) but alas no such thing is
	existing :(

	So after one run of dump the low valued desriptor is lost and
	you cannot guarantee to reliably get a low descriptor again.
	Maybe you can loop a with short waits on trying to get a low
	valued descriptor - but this is not deterministic. When you
	catch one and if this is done in dump.c context maybe you miss
	the next dump intervall :(

	BTW this has been discussed in some ways in
	the thread around message
	http://tech.groups.yahoo.com/group/milter-greylist/message/3444

That are the reason why I put the file_ext.c module (as called the
Solaris workaround) into live (based on the suggestion in the above
mentioned thread), which seems to be complicated - but it has real
technical reasons why it has to be like it is (someone may proof or show
it to me or us to make it simpler). This module is already available -
not necessary to reinvent the wheel - and it replaces the
fopen/fdopen/fclose interface which let drop in drop in the code nearly
transparently:

 - replace all fopen/fdopen/fclose with Fopen/Fdopen/Fclose,
 - call a initialisation routine at startup, 
 - add one #include in a include-file which a read in by every module, 
 - set a define during compile, 
 - add the file_ext module for linking

and your done. ;) This way solves all file handle issues in whichever
module these are located - you will not have to insert workarounds again
and again depending on how much danger a file handle code is causing.
This is just a matter of maintainance ... (you have only to ensure to
use F* calls insteat of f* calls - should be easy)


Johann

Re: [milter-greylist] Fixing Solaris troubles

2007-09-27 by Emmanuel Dreyfus

On Thu, Sep 27, 2007 at 10:19:43AM +0200, Johann Klasek wrote:
> 	BTW this has been discussed in some ways in
> 	the thread around message
> 	http://tech.groups.yahoo.com/group/milter-greylist/message/3444

Yes, I'm trying to catch up with that problem after having been 
swamped by an OpenLDAP setup.

> That are the reason why I put the file_ext.c module (as called the
> Solaris workaround) into live (based on the suggestion in the above
> mentioned thread), which seems to be complicated - but it has real
> technical reasons why it has to be like it is (someone may proof or show
> it to me or us to make it simpler).

My only concern here is that we only have two reports on using this patch,
one positive, and one negative. Indeed an efficient way to gather more 
reports is to include it in another beta. Perhaps we should do that.

-- 
Emmanuel Dreyfus
manu@...

RE: [milter-greylist] 4.0beta2 is available

2007-09-27 by attila.bruncsak@itu.int

> http://ftp.espci.fr/pub/milter-greylist/milter-greylist.4.0n2.tgz
> MD5 (milter-greylist-4.0b2.tgz) = 7f75a4109db5031d3db31ca968bb9c6b
> 
> Changes since 4.0beta1:
>         Treat protocol errors in urlcheck clauses as 
> temporary failures
>         Report missing SPF reasons in X-Greylist (Benoit Branciard)
>         Allow building objects outside of source directory 
> (Mattheu Herrb)
>         Fix configure LDFLAG generation, -R was missing 
> (Mattheu Herrb)
>         Fix MX sync on Solaris (Mattheu Herrb)
> 

Hello,

In the LDFLAG generation of configure the changes
of the rpath variable value
has broken the configure script on Tru64 UNIX.
It does not even create Makefile any more.

3033c3032
< rpath="-R"; ldrpath=no
---
> rpath="-L"; ldrpath=no

With -L flag it was just working fine (V4.0a6):

checking for pthread_create in -lpthread... no
checking for pthread_create in -lc_r... no
checking for pthread_create in -lpthreads -lpthread... yes

With -R flag I got the following (V4.0b2):

checking for pthread_create in -lpthread... no
checking for pthread_create in -lc_r... no
checking for pthread_create in -lpthreads -lpthread... no
Required libpthread not found. Use --with-libpthread

I even tried with --with-libpthread=/usr/lib as the error message says:

$ ls -lL /usr/lib/lipth*
-rw-r--r--   1 bin      bin       556202 Jan 24  2003 /usr/lib/libpthread.a
-rw-r--r--   1 bin      bin       125634 Aug  1  2001 /usr/lib/libpthreads.a

I got the same error.

May I ask somebody to fix this? Sorry, I am not a configure specialist.
Probably an additional check what to use: -L or -R depending on platform?

Bests,
Attila

Re: [milter-greylist] 4.0beta2 is available

2007-09-27 by Emmanuel Dreyfus

On Thu, Sep 27, 2007 at 12:41:01PM +0200, attila.bruncsak@... wrote:
> In the LDFLAG generation of configure the changes
> of the rpath variable value
> has broken the configure script on Tru64 UNIX.
> It does not even create Makefile any more.
> 
> 3033c3032
> < rpath="-R"; ldrpath=no
> ---
> > rpath="-L"; ldrpath=no

cvs log says that:
revision 1.217
date: 2007/08/23 10:54:13;  author: manu;  state: Exp;  lines: +2 -2
Fix configure LDFLAG generation, -R was missing (Mattheu Herrb)
Documentation typo fix

What about this fix?

Index: configure.ac
===================================================================
RCS file: /cvsroot/milter-greylist/configure.ac,v
retrieving revision 1.217
diff -U2 -r1.217 configure.ac
--- configure.ac        23 Aug 2007 10:54:13 -0000      1.217
+++ configure.ac        27 Sep 2007 23:43:22 -0000
@@ -4,5 +4,5 @@
 
 AC_PREREQ(2.57)
-AC_INIT(milter-greylist, 4.0b1, manu@...)
+AC_INIT(milter-greylist, 4.0b2, manu@...)
 AC_CONFIG_SRCDIR([milter-greylist.c])
 AC_CONFIG_HEADER([config.h])
@@ -32,4 +32,13 @@
 NORPSAVEDLDFLAGS=$LDFLAGS
 
+# Check if the linker accepts -R 
+AC_MSG_CHECKING([if ld accepts -R])
+SAVEDLDFLAGS=$LDFLAGS
+LDFLAGS=$LDFLAGS" -Wl,-R=/"
+AC_LINK_IFELSE([AC_LANG_PROGRAM([],[])], 
+    [rpath="-R"; ldrpath=yes], [rpath="-L"; ldrpath=no])
+LDFLAGS=$SAVEDLDFLAGS
+AC_MSG_RESULT([$ldrpath])
+
 # Check if the linker accepts --rpath (for Darwin)
 AC_MSG_CHECKING([if ld accepts --rpath])
@@ -37,5 +46,5 @@
 LDFLAGS=$LDFLAGS" -Wl,--rpath=/"
 AC_LINK_IFELSE([AC_LANG_PROGRAM([],[])], 
-    [rpath="--rpath="; ldrpath=yes], [rpath="-R"; ldrpath=no])
+    [rpath="--rpath="; ldrpath=yes], [ldrpath=no])
 LDFLAGS=$SAVEDLDFLAGS
 AC_MSG_RESULT([$ldrpath])


-- 
Emmanuel Dreyfus
manu@...

Re: [milter-greylist] Fixing Solaris troubles

2007-09-27 by manu@netbsd.org

Johann Klasek <johann@...> wrote:

> That are the reason why I put the file_ext.c module (as called the
> Solaris workaround) into live 

Speaking about this one, could you add a copyright notice? 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Move to quarantaine

This moves the raw source file on disk only. The archive index is not changed automatically, so you still need to run a manual refresh afterward.