Milter-greylist

Here is our alpha 5.

http://ftp.espci.fr/pub/milter-greylist/milter-greylist-3.0a5.tgz
MD5 (milter-greylist-3.0a5.tgz) = 1ba9fdf98971066ce039115e79b8783c

In this release, we have some important fixes around thread race 
conditions, done by AIDA Shinra. The changes are important, so testing
will not hurt. Please report any problem you find.

-- 
Emmanuel Dreyfus
manu@...

Hi,

>>>>> On Thu, 31 Aug 2006 08:26:33 +0000
>>>>> Emmanuel Dreyfus <manu@...> said:

manu> Here is our alpha 5.

manu> http://ftp.espci.fr/pub/milter-greylist/milter-greylist-3.0a5.tgz
manu> MD5 (milter-greylist-3.0a5.tgz) = 1ba9fdf98971066ce039115e79b8783c

manu> In this release, we have some important fixes around thread race 
manu> conditions, done by AIDA Shinra. The changes are important, so testing
manu> will not hurt. Please report any problem you find.

Umm, it seems eat more CPU than before.  The following are top output
when no mail is arrived.  And, when killing milter-greylist, load
average goes to almost zero:

ume@ameno:ports/milter-greylist:1261% top -U smmsp -b
last pid: 87958;  load averages:  2.00,  2.07,  1.86  up 17+10:39:26    18:26:28
111 processes: 2 running, 109 sleeping, 1537 MHz

Mem: 601M Active, 614M Inact, 216M Wired, 61M Cache, 112M Buf, 7516K Free
Swap: 2048M Total, 396K Used, 2047M Free


  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
29776 smmsp       5 132    0  4340K  2344K RUN     16:07 25.68% milter-greylist
57862 smmsp       1  20    0  3700K  1528K pause    0:10  0.00% sendmail

ume@ameno:ports/milter-greylist:1262% top -U smmsp -b -H
last pid: 16227;  load averages:  2.00,  2.07,  1.86  up 17+10:39:30    18:26:32
111 processes: 2 running, 109 sleeping, 1537 MHz

Mem: 601M Active, 614M Inact, 216M Wired, 61M Cache, 112M Buf, 7516K Free
Swap: 2048M Total, 396K Used, 2047M Free


  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
29776 smmsp    125    0  4340K  2344K select  16:11 21.88% milter-greylist
29776 smmsp     20    0  4340K  2344K ksesig  16:11  0.00% milter-greylist
29776 smmsp      4    0  4340K  2344K accept  16:11  0.00% milter-greylist
29776 smmsp      4    0  4340K  2344K accept  16:11  0.00% milter-greylist
29776 smmsp    131    0  4340K  2344K RUN     16:11  0.00% milter-greylist
57862 smmsp     20    0  3700K  1528K pause    0:10  0.00% sendmail

--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@...  ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/

Building on Solaris (10 x86) doesn't work anymore. 3.0a4 works fine...

# make
gcc -g -O2 -Wall -D_REENTRANT -D__EXTENSIONS__ -DUSE_DNSRBL
-DDUMPFILE=\"/var/spool/milter-greylist\" -D_BSD_SOURCE  -c
milter-greylist.c
milter-greylist.c: In function `main':
milter-greylist.c:745: error: `LOG_PERROR' undeclared (first use in this
function)
milter-greylist.c:745: error: (Each undeclared identifier is reported
only once
milter-greylist.c:745: error: for each function it appears in.)
*** Error code 1
make: Fatal error: Command failed for target `milter-greylist.o'

/P

-----Original Message-----
From: milter-greylist@yahoogroups.com
[mailto:milter-greylist@yahoogroups.com] On Behalf Of Emmanuel Dreyfus
Sent: Thursday, August 31, 2006 10:27 AM
To: milter-greylist@yahoogroups.com
Subject: [milter-greylist] milter-greylist 3.0 alpha 5 is out

Here is our alpha 5.

http://ftp.espci.fr/pub/milter-greylist/milter-greylist-3.0a5.tgz
MD5 (milter-greylist-3.0a5.tgz) = 1ba9fdf98971066ce039115e79b8783c

In this release, we have some important fixes around thread race 
conditions, done by AIDA Shinra. The changes are important, so testing
will not hurt. Please report any problem you find.

-- 
Emmanuel Dreyfus
manu@...


 
Yahoo! Groups Links

On Thu, Aug 31, 2006 at 03:12:58PM +0200, fredrik.pettai@... wrote:
> Building on Solaris (10 x86) doesn't work anymore. 3.0a4 works fine...
> 
> # make
> gcc -g -O2 -Wall -D_REENTRANT -D__EXTENSIONS__ -DUSE_DNSRBL
> -DDUMPFILE=\"/var/spool/milter-greylist\" -D_BSD_SOURCE  -c
> milter-greylist.c
> milter-greylist.c: In function `main':
> milter-greylist.c:745: error: `LOG_PERROR' undeclared (first use in this
> function)

Outch, I missed that one during the patch merge. 
Just delete the line 745 and everything will be alright.

-- 
Emmanuel Dreyfus
manu@...

At Thu, 31 Aug 2006 18:32:47 +0900,
Hajimu UMEMOTO wrote:
> 
> Hi,
> 
> >>>>> On Thu, 31 Aug 2006 08:26:33 +0000
> >>>>> Emmanuel Dreyfus <manu@...> said:
> 
> manu> Here is our alpha 5.
> 
> manu> http://ftp.espci.fr/pub/milter-greylist/milter-greylist-3.0a5.tgz
> manu> MD5 (milter-greylist-3.0a5.tgz) = 1ba9fdf98971066ce039115e79b8783c
> 
> manu> In this release, we have some important fixes around thread race 
> manu> conditions, done by AIDA Shinra. The changes are important, so testing
> manu> will not hurt. Please report any problem you find.
> 
> Umm, it seems eat more CPU than before.  The following are top output
> when no mail is arrived.  And, when killing milter-greylist, load
> average goes to almost zero:
> 
(snip)

It is on my bug. In addition, sm_macro feature is not working at all.
Please try this:
http://www.j10n.org/files/milter-greylist-3.0a5-misc.patch

The 3.0a5 is sensitive to another bug in libmilter. You may need the
following patch to get final dump correctly. Otherwise, I recommend
dumpfreq 0.
http://www.j10n.org/files/libmilter-8.13.8-signal.patch

Hi,

>>>>> On Fri, 01 Sep 2006 00:12:01 +0900
>>>>> AIDA Shinra <shinra@...> said:

shinra> It is on my bug. In addition, sm_macro feature is not working at all.
shinra> Please try this:
shinra> http://www.j10n.org/files/milter-greylist-3.0a5-misc.patch

It seems fixed the problem.  Thank you!

shinra> The 3.0a5 is sensitive to another bug in libmilter. You may need the
shinra> following patch to get final dump correctly. Otherwise, I recommend
shinra> dumpfreq 0.
shinra> http://www.j10n.org/files/libmilter-8.13.8-signal.patch

I believe 8.13.8 is the latest release of sendmail.  Are you mean all
version of sendmail are affected?
Did you send your patch to sendmail.org, already?

Sincerely,

--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@...  ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/

On Fri, Sep 01, 2006 at 12:12:01AM +0900, AIDA Shinra wrote:
> It is on my bug. In addition, sm_macro feature is not working at all.
> Please try this:
> http://www.j10n.org/files/milter-greylist-3.0a5-misc.patch
>
> The 3.0a5 is sensitive to another bug in libmilter. You may need the
> following patch to get final dump correctly. Otherwise, I recommend
> dumpfreq 0.
> http://www.j10n.org/files/libmilter-8.13.8-signal.patch

I must confess I'm not sure I took the right decision about integrating
the thread race condition fixes before the 3.0 release instead of doing
it afterwards.

Your work is very valuable, and I really appreciate your contribution, 
but it is a huge change set. I fear it needs a lot of testing before 
being ready for a release. Moreover, it fixes problems that never 
appeared at mine, so that raise the question whether this huge fix was 
critical enough to be merged at alpha stage. 

If I made the wrong decision, it's still time to roll back. I'd like input
from people that have tested previous 3.0 alpha releases: do we need the
thread race condition fixes in the 3.0 release, or shall we back it out
from the current sources and integrate it in the next developement release?

If the answer to this question is the second alternative, then I'll 
immediatly make two releases available:
1) 3.0a6, which will be 3.0a4 plus Attila's fixes (MX sync and Tru64 build)
2) 3.1.1, which will be 3.0a5 plus your new patches.

Emmanuel Dreyfus
manu@...

Hi,

>>> Fri, 01 Sep 2006 00:50:06 +0900,
>>> Hajimu UMEMOTO <ume@...> said:

>>>>> On Fri, 01 Sep 2006 00:12:01 +0900
>>>>> AIDA Shinra <shinra@...> said:

shinra> It is on my bug. In addition, sm_macro feature is not working at all.
shinra> Please try this:
shinra> http://www.j10n.org/files/milter-greylist-3.0a5-misc.patch

ume> It seems fixed the problem.  Thank you!

Umm, the problem is not fixed.  After running for a certain time,
milter-greylist became to eat CPU.

Sincerely,

--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@...  ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/

According to Emmanuel Dreyfus:
> 
> If the answer to this question is the second alternative, then I'll 
> immediatly make two releases available:
> 1) 3.0a6, which will be 3.0a4 plus Attila's fixes (MX sync and Tru64 build)
> 2) 3.1.1, which will be 3.0a5 plus your new patches.

I vote for that choice, similar to so many projects.

[ just to let you know...
I'm still running the last 2.1 because I've started a patch implementing
autoblack but it quickly turned into a beast when I changed the db format,
then added per tuple stats (number of hits), then I found out that my
blacklist idea could be turned into a botnet-autoblacklist, which seems
an even better idea but still needs some refinements.
I'm still too far from public release and september being back, I have zero
free time to spend on that.
I pass for now, I'll eventually merge into 3.1, or forget the whole thing
if it gets nowhere. ]

/Fabien

At Fri, 01 Sep 2006 01:02:36 +0900,
Hajimu UMEMOTO wrote:
> shinra> It is on my bug. In addition, sm_macro feature is not working at all.
> shinra> Please try this:
> shinra> http://www.j10n.org/files/milter-greylist-3.0a5-misc.patch
> 
> ume> It seems fixed the problem.  Thank you!
> 
> Umm, the problem is not fixed.  After running for a certain time,
> milter-greylist became to eat CPU.

Sorry, try this patch in addition to the previous one.
http://www.j10n.org/files/milter-greylist-3.0a5-misc2.patch

Hi,

>>>>> On Thu, 31 Aug 2006 15:56:29 +0000
>>>>> Emmanuel Dreyfus <manu@...> said:

manu> I must confess I'm not sure I took the right decision about integrating
manu> the thread race condition fixes before the 3.0 release instead of doing
manu> it afterwards.

manu> Your work is very valuable, and I really appreciate your contribution, 
manu> but it is a huge change set. I fear it needs a lot of testing before 
manu> being ready for a release. Moreover, it fixes problems that never 
manu> appeared at mine, so that raise the question whether this huge fix was 
manu> critical enough to be merged at alpha stage. 

manu> If I made the wrong decision, it's still time to roll back. I'd like input
manu> from people that have tested previous 3.0 alpha releases: do we need the
manu> thread race condition fixes in the 3.0 release, or shall we back it out
manu> from the current sources and integrate it in the next developement release?

I don't see any thread related problem.  So, I'm not sure if it is
urgent.  However, since it changes the usage of mutex, it seems the
memory usage is greatly reduced.

manu> If the answer to this question is the second alternative, then I'll 
manu> immediatly make two releases available:
manu> 1) 3.0a6, which will be 3.0a4 plus Attila's fixes (MX sync and Tru64 build)
manu> 2) 3.1.1, which will be 3.0a5 plus your new patches.

It sounds bad choice to me.  When we have two alpha releases, it'll
make testing harder; someone will try 3.0a6 and others will try 3.1.1.
We should consider what should be in 3.0, and should be focused on
releasing 3.0, IMHO.

Sincerely,

--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@...  ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/

Hi,

>>>>> On Fri, 01 Sep 2006 03:11:35 +0900
>>>>> AIDA Shinra <shinra@...> said:

shinra> Sorry, try this patch in addition to the previous one.
shinra> http://www.j10n.org/files/milter-greylist-3.0a5-misc2.patch

No problem.
The milter-greylist with your two patches seems running well over two
hours.

Sincerely,

--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@...  ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/

At Thu, 31 Aug 2006 15:56:29 +0000,
Emmanuel Dreyfus wrote:
> 
> Your work is very valuable, and I really appreciate your contribution, 
> but it is a huge change set. I fear it needs a lot of testing before 
> being ready for a release. Moreover, it fixes problems that never 
> appeared at mine, so that raise the question whether this huge fix was 
> critical enough to be merged at alpha stage. 
> 
> If I made the wrong decision, it's still time to roll back. I'd like input
> from people that have tested previous 3.0 alpha releases: do we need the
> thread race condition fixes in the 3.0 release, or shall we back it out
> from the current sources and integrate it in the next developement release?
> 
> If the answer to this question is the second alternative, then I'll 
> immediatly make two releases available:
> 1) 3.0a6, which will be 3.0a4 plus Attila's fixes (MX sync and Tru64 build)
> 2) 3.1.1, which will be 3.0a5 plus your new patches.

3.0a4 minus sm_macro, commenting out them from conf_lex.l and removing
from the document. priv->priv_whitelist & EXF_XXX behaviour is too
complicated to handle sm_macro properly. We should clarify the
semantics and probably write a new code before sm_macro feature in
production releases.

Another bug fix and debug aid at the macro.c:

--- macro.c.orig	2006-08-28 15:33:14.000000000 +0900
+++ macro.c	2006-09-02 19:28:10.000000000 +0900
@@ -82,7 +82,7 @@
 
 	priv = (struct mlfi_priv *)smfi_getpriv(ctx);
 
-	value = smfi_getsymval(ctx, me->m_name);
+	value = smfi_getsymval(ctx, me->m_macro);
 							 
 	switch (me->m_type) {
 	case M_UNSET:
@@ -109,7 +109,9 @@
 	}
 
 	if (conf.c_debug) {
-		mg_log(LOG_DEBUG, "sm_macro \"%s\" match", me->m_name);
+		mg_log(LOG_DEBUG, "sm_macro \"%s\" %s=%s %s", me->m_name,
+				me->m_macro, value ? value : "(null)",
+				retval ? "nomatch" : "match");
 	}
 
 	return retval;

Tere.
>
> Umm, the problem is not fixed.  After running for a certain time,
> milter-greylist became to eat CPU.
>   

Without patch, same thing, milter-greylist takes all the cpu.

-- 
Mart

> > It is on my bug. In addition, sm_macro feature is not 
> working at all.
> > Please try this:
> > http://www.j10n.org/files/milter-greylist-3.0a5-misc.patch
> >
> > The 3.0a5 is sensitive to another bug in libmilter. You may need the
> > following patch to get final dump correctly. Otherwise, I recommend
> > dumpfreq 0.
> > http://www.j10n.org/files/libmilter-8.13.8-signal.patch
> 
> I must confess I'm not sure I took the right decision about 
> integrating
> the thread race condition fixes before the 3.0 release 
> instead of doing
> it afterwards.
> 
> Your work is very valuable, and I really appreciate your 
> contribution, 
> but it is a huge change set. I fear it needs a lot of testing before 
> being ready for a release. Moreover, it fixes problems that never 
> appeared at mine, so that raise the question whether this 
> huge fix was 
> critical enough to be merged at alpha stage. 
> 
> If I made the wrong decision, it's still time to roll back. 
> I'd like input
> from people that have tested previous 3.0 alpha releases: do 
> we need the
> thread race condition fixes in the 3.0 release, or shall we 
> back it out
> from the current sources and integrate it in the next 
> developement release?
> 
> If the answer to this question is the second alternative, then I'll 
> immediatly make two releases available:
> 1) 3.0a6, which will be 3.0a4 plus Attila's fixes (MX sync 
> and Tru64 build)
> 2) 3.1.1, which will be 3.0a5 plus your new patches.
> 

Hello,

This release does not look very stable, the daemon frequently dies without
any notice. Maybe the patches can be cut into smaller chunks and apply them
release by release? 

Bests,
Attila

On Mon, Sep 04, 2006 at 03:55:13PM +0200, attila.bruncsak@... wrote:
> This release does not look very stable, the daemon frequently dies without
> any notice. Maybe the patches can be cut into smaller chunks and apply them
> release by release? 

Yes, I think I'll roll back the thread race fix fro the 3.0 alpha releases
and merge it back in 3.1.x.

The question remain about what should be done for the sm_macto feature. 
It seems buggy, but
1) one do not have to use it, and if it's not used, it does not break 
2) the bugs that were found seems to be trivial to fix.

-- 
Emmanuel Dreyfus
manu@...