2008-10-22 by Petar Bogdanovic
Hi,
I just did some stress-testing on milter-greylist in order to test its
memory consumption and it seems that there are some leaks to be found.
The test was done with smtp-source(1), had a duration of about 30
minutes, started with:
smtp-source -4cA -s 100 (...)
at some point continued with:
smtp-source -4cA -s 500 (...)
and ended with no mails being sent at all.
Take a look at the top-output (printed every 15 seconds):
,---- Total size of the process (text, data, and stack) given in kilobytes.
/ ,-- Resident memory: current amount of process memory that resides in physical memory, given in kilobytes.
/ |
/ |
SIZE RES STATE TIME WCPU CPU
3000K 1248K sigwait 0:00 0.00% 0.00% <-- milter-greylist started
3072K 1740K sigwait 0:00 0.14% 0.10% <-- begin: 100sc
3192K 1864K sigwait 0:00 0.00% 0.00%
3312K 2068K sigwait 0:00 0.21% 0.20%
3432K 2196K sigwait 0:00 0.50% 0.49%
3560K 2336K sigwait 0:00 0.45% 0.44%
3680K 2472K sigwait 0:01 0.74% 0.73%
3804K 2596K sigwait 0:01 1.03% 1.03%
3940K 2736K sigwait 0:01 0.93% 0.93%
4072K 2868K RUN 0:01 0.95% 0.93%
4184K 2980K sigwait 0:01 1.03% 1.03%
4312K 3108K sigwait 0:01 0.88% 0.88%
4440K 3236K sigwait 0:02 1.07% 1.07%
4568K 3364K sigwait 0:02 0.98% 0.98%
4640K 3436K sigwait 0:02 0.63% 0.63%
4744K 3568K sigwait 0:02 0.59% 0.59%
4880K 3704K sigwait 0:02 0.54% 0.54%
4984K 3808K sigwait 0:02 0.88% 0.88%
5104K 3928K sigwait 0:03 0.93% 0.93%
5200K 4024K sigwait 0:03 0.49% 0.49%
5328K 4152K sigwait 0:03 0.24% 0.24%
5460K 4284K sigwait 0:03 0.34% 0.34%
5580K 4404K sigwait 0:03 0.54% 0.54%
5716K 4540K sigwait 0:04 0.39% 0.39%
5836K 4684K sigwait 0:04 0.63% 0.63%
5948K 4800K sigwait 0:04 0.68% 0.68%
6064K 4916K sigwait 0:04 0.83% 0.83%
6136K 4988K sigwait 0:04 0.54% 0.54%
6260K 5112K sigwait 0:04 0.98% 0.98%
6392K 5244K RUN 0:05 1.03% 1.03%
6472K 5328K sigwait 0:05 0.98% 0.98%
6588K 5444K sigwait 0:05 1.17% 1.17%
6708K 5564K sigwait 0:05 1.17% 1.17%
6816K 5672K sigwait 0:05 0.93% 0.93%
6912K 5768K sigwait 0:05 0.83% 0.83%
7036K 5892K sigwait 0:06 0.83% 0.83%
7172K 6028K sigwait 0:06 0.93% 0.93%
7292K 6148K sigwait 0:06 0.98% 0.98%
7420K 6276K sigwait 0:06 1.03% 1.03%
7556K 6412K sigwait 0:06 0.93% 0.93%
7640K 6496K sigwait 0:07 0.98% 0.98%
7744K 6600K sigwait 0:07 0.88% 0.88%
7884K 6740K sigwait 0:07 0.63% 0.63%
7964K 6848K sigwait 0:07 0.59% 0.59%
8084K 6968K sigwait 0:07 0.59% 0.59%
8292K 7252K sigwait 0:08 1.46% 1.46%
8640K 7968K RUN 0:08 68.00% 3.32% <-- begin: 500sc
9172K 8680K RUN 0:10 11.13% 8.25%
9660K 9168K sigwait 0:11 10.01% 10.01%
10M 9784K sigwait 0:13 11.72% 11.72%
11M 10M sigwait 0:15 13.87% 13.87%
11M 11M RUN 0:16 12.72% 12.35%
11M 11M sigwait 0:17 13.09% 13.09%
12M 12M sigwait 0:18 12.11% 12.11%
12M 12M sigwait 0:19 10.79% 10.79%
13M 13M sigwait 0:21 10.99% 10.99%
13M 13M RUN 0:22 10.27% 10.25%
14M 14M sigwait 0:23 9.96% 9.96%
14M 14M sigwait 0:24 9.23% 9.23%
15M 15M sigwait 0:25 11.33% 11.33%
15M 15M RUN 0:27 12.13% 11.82%
16M 15M sigwait 0:28 9.42% 9.42%
16M 16M sigwait 0:28 6.98% 6.98%
16M 16M RUN 0:29 5.86% 5.86%
16M 16M sigwait 0:30 4.64% 4.64%
17M 17M sigwait 0:31 5.13% 5.13%
17M 17M sigwait 0:31 3.37% 3.37%
17M 17M sigwait 0:31 1.37% 1.37% <-- end
17M 17M sigwait 0:31 0.44% 0.44%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
17M 17M sigwait 0:31 0.00% 0.00%
(many minutes later)
17M 17M sigwait 0:31 0.00% 0.00%
Content of the (striped-down) greylist.conf:
user "smmsp:postfix"
pidfile "/var/milter/mgrey.pid"
dumpfile "/var/milter/mgrey.db"
socket "/var/milter/mgrey.sock"
quiet
lazyaw
timeout 1d
autowhite 10d
report delays
racl whitelist addr 127.0.0.0/8
racl whitelist default nolog
dacl whitelist default nolog
The stress tested environment was NetBSD 4.0.1 (no swap configured),
milter-greylist 4.1.6, built through pkgsrc using the following
configure arguments:
--with-user=smmsp
--enable-dnsrbl
--with-thread-safe-resolver
--disable-drac
--with-libspf_alt=/usr/pkg
--enable-postfix
--enable-spamassassin
--without-libintl-prefix
--without-libiconv-prefix
--prefix=/usr/pkg
--host=i386--netbsdelf
--mandir=/usr/pkg/man
Any ideas?
Thanks,
Petar2008-10-22 by Oliver Fromme
Petar Bogdanovic wrote:
> I just did some stress-testing on milter-greylist in order to test its
> memory consumption and it seems that there are some leaks to be found.
milter-greylist keeps all data (all triples) in memory.
So it is expected that it will grow if new triples come
in faster than old triples are expired, which was certainly
the case in your 30-minutes test when the timeout was set
to 1d.
Best regards
Oliver
--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606, Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758, Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart
FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd
"To this day, many C programmers believe that 'strong typing'
just means pounding extra hard on the keyboard."
-- Peter van der Linden2008-10-22 by shuttlebox
On Wed, Oct 22, 2008 at 10:49 PM, Oliver Fromme <olli@...> wrote:
> milter-greylist keeps all data (all triples) in memory.
> So it is expected that it will grow if new triples come
> in faster than old triples are expired, which was certainly
> the case in your 30-minutes test when the timeout was set
> to 1d.
I have tuned my settings so the dumped database file always is around
20 MB which to me means the database in memory is the same size as
well. Still milter-greylist just keeps growing. I restart it
automatically when it reaches 512 MB which is about weekly.
Suggestions have been made that it's Solaris libc that is at fault but
another daemon that is known on other platforms to leak memory is Clam
Antivirus and it's rock solid on the my machines, I can run it for
months with the same memory usage all the time.
--
/peter
2008-10-22 by Kai Schaetzl
Petar Bogdanovic wrote on Wed, 22 Oct 2008 21:03:44 +0200:
> I just did some stress-testing on milter-greylist in order to test its
> memory consumption and it seems that there are some leaks to be found.
You mean it should have released that memory after test? The whole db is
hold in memory. So, the more triplets it holds the bigger the process
grows. You will find that it needs *a lot* of RAM for a high traffic site.
Kai
--
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
2008-10-22 by Petar Bogdanovic
On Wed, Oct 22, 2008 at 11:31:16PM +0200, Kai Schaetzl wrote:
> Petar Bogdanovic wrote on Wed, 22 Oct 2008 21:03:44 +0200:
>
> > I just did some stress-testing on milter-greylist in order to test its
> > memory consumption and it seems that there are some leaks to be found.
>
> You mean it should have released that memory after test? The whole db is
> hold in memory. So, the more triplets it holds the bigger the process
> grows. You will find that it needs *a lot* of RAM for a high traffic site.
There is no way to create a greylist tuple without using at least one
`[rd]acl greylist' statement in the configuration file.
2008-10-22 by Mark Walker
Is the decision to keep data in memory as opposed to some sort of
database a speed issue? It seems that there would be a lot of benefit
to keeping the active data in an sql database accessible from other apps.
Oliver Fromme wrote:
Show quoted textHide quoted text
>
>
> Petar Bogdanovic wrote:
> > I just did some stress-testing on milter-greylist in order to test its
> > memory consumption and it seems that there are some leaks to be found.
>
> milter-greylist keeps all data (all triples) in memory.
> So it is expected that it will grow if new triples come
> in faster than old triples are expired, which was certainly
> the case in your 30-minutes test when the timeout was set
> to 1d.
>
> Best regards
> Oliver
>
> --
> Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
> Handelsregister: Registergericht Muenchen, HRA 74606, Gesch\ufffdftsfuehrung:
> secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
> chen, HRB 125758, Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart
>
> FreeBSD-Dienstleistungen, -Produkte und mehr:
> http://www.secnetix.de/bsd <http://www.secnetix.de/bsd>
>
> "To this day, many C programmers believe that 'strong typing'
> just means pounding extra hard on the keyboard."
> -- Peter van der Linden
>
>
2008-10-23 by Patrick Domack
I use milter-greylist on my personal email, and it's been great, and
the 2.x code, and the current development code both don't leak memory
on my system, holds steady around 12megs.
Now another system I admin, we have used several different database
methods, mainly cause I knew I was going have a memory issue with
using milter-greylist, and the systems when I got them hardly had
enough memory to run as they where.
I used gpsd, some other g thing, and now policyd. The biggest issue I
have with them, is anytime the database is slow, locked, or otherwise
busy, everything comes to a dead halt, as it waits for a response from
the database, and this causes more and more threads to build up. It
just gets to be a issue back and forth. I finally have managed to keep
the greylisting table smaller than 6gigs now, using 1.4million
blacklisted ip's based on the failure of passing greylisting tests,
and this keeps it around 1gig in size, and fits in memory good, so
speed is better.
Now I suppose all these issues could be overcome, using an intergrated
db, and making it so unlimited number of threads can read from it at
will, and do limited updating. This could be achieved, but it would
take a great amount of time, design, ...
And in the end, I believe, the more widely used, and better
greylisting gets, the more likely it is for the spammers to work
around it. It shouldn't be hard for them to add a retry logic into the
virus's, or make more compliant smtp servers.
I'm just a milter-greylist user, and love it. But everything will have
design limitations.
Quoting Mark Walker <furface@...m>:
Show quoted textHide quoted text
> Is the decision to keep data in memory as opposed to some sort of
> database a speed issue? It seems that there would be a lot of benefit
> to keeping the active data in an sql database accessible from other apps.
>
>
> Oliver Fromme wrote:
>>
>>
>> Petar Bogdanovic wrote:
>> > I just did some stress-testing on milter-greylist in order to test its
>> > memory consumption and it seems that there are some leaks to be found.
>>
>> milter-greylist keeps all data (all triples) in memory.
>> So it is expected that it will grow if new triples come
>> in faster than old triples are expired, which was certainly
>> the case in your 30-minutes test when the timeout was set
>> to 1d.
>>
>> Best regards
>> Oliver
>>
>> --
>> Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
>> Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung:
>> secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
>> chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart
>>
>> FreeBSD-Dienstleistungen, -Produkte und mehr:
>> http://www.secnetix.de/bsd <http://www.secnetix.de/bsd>
>>
>> "To this day, many C programmers believe that 'strong typing'
>> just means pounding extra hard on the keyboard."
>> -- Peter van der Linden
>>
>>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>
2008-10-23 by manu@netbsd.org
Mark Walker <furface@...> wrote:
> Is the decision to keep data in memory as opposed to some sort of
> database a speed issue?
It was just to keep it simple: no external dependency, and there is no
obvious speed tradeoff. I am not even sure the memory leaks are caused
by the in-memory database storage.
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2008-10-23 by Benoit Branciard
Petar Bogdanovic a \ufffdcrit :
> Hi,
>
> I just did some stress-testing on milter-greylist in order to test its
> memory consumption and it seems that there are some leaks to be found.
>
Yes, milter-greylist has some memory leaks, at least with some
compilation options.
I have set up a cron to restart it every time it hits the 1GB VSZ limit.
This orrcurs at various rate depending on inbound traffic but roughly at
least once a week. Once restarted (and database reloaded), VSZ drops to
about 100MB, so the tuple database is not the reason.
I'm on linux Debian (Sarge with Etch libc6/kernel) and configure
milter-greylist 4.0 (with some additional patches, but this didn't
impact the memory leak problem) with:
./configure --prefix=/usr/local --bindir=/usr/local/sbin
--sysconfdir=/etc --localstatedir=/var/lib --enable-dnsrbl --with-libspf2_
10=/usr/lib --with-conffile=/etc/mail/greylist.conf
--with-dumpfile=/var/lib/milter-greylist/greylist.db
--with-libcurl=/usr/lib
Since I have this "automatic diet" cron job and the machine has plenty
of RAM, I never worried too much about this memory problem, but it still
exists...
--
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.
2008-10-23 by Emmanuel Dreyfus
FWIW, at mine, milter-greylist memory consumption raise up to about
50 megabytes. It is difficult to say if that is a memory leak, As I
restart it about once a week to test various things. So there may be
a leak, but it is not obvious.
Setup: milter-greylist-4.1.6 on NetBSD-4.0,
./configure --with-user=smmsp --enable-dnsrbl --with-libGeoIP=/usr/pkg \
--with-libmilter=/usr/pkg --with-openldap=/usr/pkg --enable-p0f
Um... I don't use SPF, perhaps there is something to dig here.
--
Emmanuel Dreyfus
manu@...
2008-10-23 by Petar Bogdanovic
On Wed, Oct 22, 2008 at 03:52:24PM -0700, Mark Walker wrote:
> Is the decision to keep data in memory as opposed to some sort of
> database a speed issue? It seems that there would be a lot of benefit
> to keeping the active data in an sql database accessible from other apps.
I'm not sure why we've shifted the focus on the in-memory db: The test I
did will never cause any greylist tuples being created in the database
since no greylist statement is present in the configuration file..
2008-10-23 by Petar Bogdanovic
On Thu, Oct 23, 2008 at 07:14:54AM +0000, Emmanuel Dreyfus wrote:
> FWIW, at mine, milter-greylist memory consumption raise up to about
> 50 megabytes. It is difficult to say if that is a memory leak, As I
> restart it about once a week to test various things. So there may be
> a leak, but it is not obvious.
>
> Setup: milter-greylist-4.1.6 on NetBSD-4.0,
> ./configure --with-user=smmsp --enable-dnsrbl --with-libGeoIP=/usr/pkg \
> --with-libmilter=/usr/pkg --with-openldap=/usr/pkg --enable-p0f
>
> Um... I don't use SPF, perhaps there is something to dig here.
Thanks for the hint, I forgot that nospf is not default. Take a look at
the new results:
************************************************
with spf
************************************************
user "smmsp:postfix"
pidfile "/var/milter/mgrey.pid"
dumpfile "/var/milter/mgrey.db"
socket "/var/milter/mgrey.sock"
racl whitelist default nolog
dacl blacklist body /jo28de928ufdosid9g8fr30fdp/
dacl whitelist default nolog
************************************************
3008K 1248K sigwait 0:00 0.00% 0.00% <-- mg started
3584K 3648K sigwait 0:03 29.87% 15.77% <-- 900 sim. connections
4008K 4100K sigwait 0:04 15.45% 12.01%
4568K 4676K sigwait 0:06 14.30% 12.79%
5112K 5220K sigwait 0:08 14.95% 14.21%
5656K 5764K sigwait 0:09 14.00% 13.67%
6180K 6296K sigwait 0:11 14.96% 14.79%
6704K 6824K sigwait 0:12 14.14% 14.06%
7272K 7392K RUN 0:14 14.63% 14.50%
7752K 7872K sigwait 0:16 13.59% 13.57%
8308K 8428K sigwait 0:17 14.17% 14.16%
8876K 8996K sigwait 0:19 14.90% 14.89%
9408K 9528K sigwait 0:21 15.87% 15.87%
9960K 10M sigwait 0:22 15.63% 15.62%
10M 10M sigwait 0:24 16.16% 16.16%
************************************************
without spf
************************************************
nospf
user "smmsp:postfix"
pidfile "/var/milter/mgrey.pid"
dumpfile "/var/milter/mgrey.db"
socket "/var/milter/mgrey.sock"
racl whitelist default nolog
dacl blacklist body /jo28de928ufdosid9g8fr30fdp/
dacl whitelist default nolog
************************************************
3008K 1248K sigwait 0:00 0.00% 0.00% <-- mg started
3168K 2860K RUN 0:00 12.43% 4.10% <-- 900 sim. connections
3172K 2924K sigwait 0:02 12.75% 9.91%
3172K 2928K sigwait 0:03 13.70% 12.26%
3172K 2928K sigwait 0:05 15.67% 14.89%
3172K 2928K sigwait 0:06 14.60% 14.26%
3168K 2924K sigwait 0:07 11.21% 11.08%
3184K 2964K RUN 0:09 20.30% 16.41%
3184K 2964K RUN 0:12 23.97% 23.49%
3200K 3000K RUN 0:24 57.61% 57.47%
3200K 3000K sigwait 0:29 44.12% 44.09%
3200K 3000K RUN 0:29 24.72% 24.71%
3200K 3000K RUN 0:31 19.30% 19.29%
3200K 3000K sigwait 0:32 15.72% 15.72%
3200K 3000K sigwait 0:34 14.26% 14.26%
3096K 2896K sigwait 0:35 12.50% 12.50%
************************************************
Interesting!2008-10-23 by Emmanuel Dreyfus
On Thu, Oct 23, 2008 at 11:17:15AM +0200, Petar Bogdanovic wrote:
[SPF may be the culprit for memory leaks]
Which libspf do you use? We support 3 of them.
The leak may be in milter-greylist interface code to libspf, or
in libspf itself...
--
Emmanuel Dreyfus
manu@...
2008-10-23 by Petar Bogdanovic
On Thu, Oct 23, 2008 at 09:46:35AM +0000, Emmanuel Dreyfus wrote:
> On Thu, Oct 23, 2008 at 11:17:15AM +0200, Petar Bogdanovic wrote:
> [SPF may be the culprit for memory leaks]
>
> Which libspf do you use? We support 3 of them.
libspf_alt
I will try mail/libspf2 from pkgsrc but which one is the third one?
2008-10-23 by Emmanuel Dreyfus
On Thu, Oct 23, 2008 at 11:53:10AM +0200, Petar Bogdanovic wrote:
> On Thu, Oct 23, 2008 at 09:46:35AM +0000, Emmanuel Dreyfus wrote:
> > On Thu, Oct 23, 2008 at 11:17:15AM +0200, Petar Bogdanovic wrote:
> > [SPF may be the culprit for memory leaks]
> >
> > Which libspf do you use? We support 3 of them.
>
> libspf_alt
> I will try mail/libspf2 from pkgsrc but which one is the third one?
libspf (neither alt, nor 2)
Reading the documentation, it seems that for libspf, SPF_close does not
free memory,and free() must be called on the peer_info_t obtained from
SPF_init(). Anyone can confirm?
If this is true, then we have a first memory leak here.
--
Emmanuel Dreyfus
manu@...
2008-10-23 by Petar Bogdanovic
On Thu, Oct 23, 2008 at 10:07:32AM +0000, Emmanuel Dreyfus wrote:
> On Thu, Oct 23, 2008 at 11:53:10AM +0200, Petar Bogdanovic wrote:
> > On Thu, Oct 23, 2008 at 09:46:35AM +0000, Emmanuel Dreyfus wrote:
> > > On Thu, Oct 23, 2008 at 11:17:15AM +0200, Petar Bogdanovic wrote:
> > > [SPF may be the culprit for memory leaks]
> > >
> > > Which libspf do you use? We support 3 of them.
> >
> > libspf_alt
> > I will try mail/libspf2 from pkgsrc but which one is the third one?
>
> libspf (neither alt, nor 2)
>
> Reading the documentation, it seems that for libspf, SPF_close does not
> free memory,and free() must be called on the peer_info_t obtained from
> SPF_init(). Anyone can confirm?
It apparently has nothing to do with libspf since I get similar results
with libspf_alt and libspf2 (1.2.8). The situation even got worse with
libspf2 from another point of view -- milter-greylist using libspf_alt
claimed about 15% of the CPU when it had to deal with 900 simultaneous
connections while the same situation ate every second of the available
CPU-time when linked against libspf2. That's why I wasn't able to drop
the same amounth of mails through milter-greylist+libspf2: the CPU was
the bottleneck.
milter-greylist + libspf2
3008K 1248K sigwait 0:00 0.00% 0.00% <-- mg started
3404K 3272K RUN 0:07 65.73% 27.83% <-- 900 sim. connections
3520K 3388K RUN 0:21 191.17% 63.09%
3624K 3492K RUN 0:35 96.25% 80.37%
3736K 3604K RUN 0:49 103.25% 89.31%
3848K 3716K RUN 1:03 99.97% 93.60%
3964K 3832K RUN 1:17 98.97% 95.51%
4072K 3940K RUN 1:31 97.95% 96.34%
4188K 4056K RUN 1:45 98.26% 97.12%
4304K 4172K RUN 1:59 97.56% 97.12%
4400K 4268K RUN 2:13 97.41% 96.97%
4520K 4388K RUN 2:27 97.54% 97.12%
4620K 4488K RUN 2:41 97.28% 97.12%
4728K 4596K RUN 2:56 97.58% 97.31%
4840K 4708K RUN 3:10 96.79% 96.73%
4936K 4804K RUN 3:24 96.84% 96.73%
5056K 4924K RUN 3:38 96.48% 96.44%
5188K 5052K RUN 3:52 95.69% 95.65%
5288K 5156K RUN 4:06 94.99% 94.97%
5416K 5284K RUN 4:20 94.83% 94.82%
5520K 5388K RUN 4:35 95.27% 95.26%
5652K 5520K RUN 4:49 114.61% 95.70%
5768K 5636K RUN 5:03 96.19% 96.19%
5872K 5744K RUN 5:17 103.06% 97.12%
milter-greylist + libspf_alt
3008K 1248K sigwait 0:00 0.00% 0.00% <-- mg started
3584K 3648K sigwait 0:03 29.87% 15.77% <-- 900 sim. connections
4008K 4100K sigwait 0:04 15.45% 12.01%
4568K 4676K sigwait 0:06 14.30% 12.79%
5112K 5220K sigwait 0:08 14.95% 14.21%
5656K 5764K sigwait 0:09 14.00% 13.67%
6180K 6296K sigwait 0:11 14.96% 14.79%
6704K 6824K sigwait 0:12 14.14% 14.06%
7272K 7392K RUN 0:14 14.63% 14.50%
7752K 7872K sigwait 0:16 13.59% 13.57%
8308K 8428K sigwait 0:17 14.17% 14.16%
8876K 8996K sigwait 0:19 14.90% 14.89%
9408K 9528K sigwait 0:21 15.87% 15.87%
9960K 10M sigwait 0:22 15.63% 15.62%
10M 10M sigwait 0:24 16.16% 16.16%2008-10-23 by Hajimu UMEMOTO
Hi,
>>>>> On Thu, 23 Oct 2008 14:31:15 +0200
>>>>> Petar Bogdanovic <petar@...> said:
petar> It apparently has nothing to do with libspf since I get similar results
petar> with libspf_alt and libspf2 (1.2.8). The situation even got worse with
petar> libspf2 from another point of view -- milter-greylist using libspf_alt
petar> claimed about 15% of the CPU when it had to deal with 900 simultaneous
petar> connections while the same situation ate every second of the available
petar> CPU-time when linked against libspf2. That's why I wasn't able to drop
petar> the same amounth of mails through milter-greylist+libspf2: the CPU was
petar> the bottleneck.
The BIND9's resolver requires res_ndestroy() for cleanup res_state.
However, libspf2 doesn't issue res_ndestroy() but issues res_nclose().
It causes memory leak. The FreeBSD port of libspf2 has a patch to
address this issue.
Sincerely,
--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@... ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/2008-10-23 by Petar Bogdanovic
On Fri, Oct 24, 2008 at 02:07:58AM +0900, Hajimu UMEMOTO wrote:
> Hi,
>
> >>>>> On Thu, 23 Oct 2008 14:31:15 +0200
> >>>>> Petar Bogdanovic <petar@...> said:
>
> petar> It apparently has nothing to do with libspf since I get similar results
> petar> with libspf_alt and libspf2 (1.2.8). The situation even got worse with
> petar> libspf2 from another point of view -- milter-greylist using libspf_alt
> petar> claimed about 15% of the CPU when it had to deal with 900 simultaneous
> petar> connections while the same situation ate every second of the available
> petar> CPU-time when linked against libspf2. That's why I wasn't able to drop
> petar> the same amounth of mails through milter-greylist+libspf2: the CPU was
> petar> the bottleneck.
>
> The BIND9's resolver requires res_ndestroy() for cleanup res_state.
> However, libspf2 doesn't issue res_ndestroy() but issues res_nclose().
> It causes memory leak. The FreeBSD port of libspf2 has a patch to
> address this issue.
Thanks, I applied all patches (~10) from the FreeBSD Ports tree and the
problem is gone. The memory usage of milter-greylist is actually frozen
now, no matter how hard I try. Also, the claimed cpu-time is much lower.
2008-10-23 by Fredrik Pettai
On Oct 23, 2008, at 11:55 PM, Petar Bogdanovic wrote:
> On Fri, Oct 24, 2008 at 02:07:58AM +0900, Hajimu UMEMOTO wrote:
> > >>>>> On Thu, 23 Oct 2008 14:31:15 +0200
> > >>>>> Petar Bogdanovic <petar@...> said:
> >
> > petar> It apparently has nothing to do with libspf since I get
> similar results
> > petar> with libspf_alt and libspf2 (1.2.8). The situation even got
> worse with
> > petar> libspf2 from another point of view -- milter-greylist using
> libspf_alt
> > petar> claimed about 15% of the CPU when it had to deal with 900
> simultaneous
> > petar> connections while the same situation ate every second of
> the available
> > petar> CPU-time when linked against libspf2. That's why I wasn't
> able to drop
> > petar> the same amounth of mails through milter-greylist+libspf2:
> the CPU was
> > petar> the bottleneck.
> >
> > The BIND9's resolver requires res_ndestroy() for cleanup res_state.
> > However, libspf2 doesn't issue res_ndestroy() but issues
> res_nclose().
> > It causes memory leak. The FreeBSD port of libspf2 has a patch to
> > address this issue.
>
> Thanks, I applied all patches (~10) from the FreeBSD Ports tree and
> the
> problem is gone. The memory usage of milter-greylist is actually
> frozen
> now, no matter how hard I try. Also, the claimed cpu-time is much
> lower.
>
Did you really used libspf2 from pkgsrc?
I only find the current version to be 1.2.5 in there (but it seems to
miss res_ndestroy() patch(es))
I'll see if I can convince the maintainer put those patches in the
pkgsrc tree as well.
Regards,
/P
2008-10-23 by Petar Bogdanovic
On Fri, Oct 24, 2008 at 12:37:36AM +0200, Fredrik Pettai wrote:
> On Oct 23, 2008, at 11:55 PM, Petar Bogdanovic wrote:
> > On Fri, Oct 24, 2008 at 02:07:58AM +0900, Hajimu UMEMOTO wrote:
> > > >>>>> On Thu, 23 Oct 2008 14:31:15 +0200
> > > >>>>> Petar Bogdanovic <petar@...> said:
> > >
> > > petar> It apparently has nothing to do with libspf since I get
> > similar results
> > > petar> with libspf_alt and libspf2 (1.2.8). The situation even got
> > worse with
> > > petar> libspf2 from another point of view -- milter-greylist using
> > libspf_alt
> > > petar> claimed about 15% of the CPU when it had to deal with 900
> > simultaneous
> > > petar> connections while the same situation ate every second of
> > the available
> > > petar> CPU-time when linked against libspf2. That's why I wasn't
> > able to drop
> > > petar> the same amounth of mails through milter-greylist+libspf2:
> > the CPU was
> > > petar> the bottleneck.
> > >
> > > The BIND9's resolver requires res_ndestroy() for cleanup res_state.
> > > However, libspf2 doesn't issue res_ndestroy() but issues
> > res_nclose().
> > > It causes memory leak. The FreeBSD port of libspf2 has a patch to
> > > address this issue.
> >
> > Thanks, I applied all patches (~10) from the FreeBSD Ports tree and
> > the
> > problem is gone. The memory usage of milter-greylist is actually
> > frozen
> > now, no matter how hard I try. Also, the claimed cpu-time is much
> > lower.
> >
>
> Did you really used libspf2 from pkgsrc?
> I only find the current version to be 1.2.5 in there
I did use the current version (1.2.5). See LOCALPATCHES.
> I'll see if I can convince the maintainer put those patches in the
> pkgsrc tree as well.
Yes, please!
2008-10-24 by manu@netbsd.org
Petar Bogdanovic <petar@...> wrote:
> > The BIND9's resolver requires res_ndestroy() for cleanup res_state.
> > However, libspf2 doesn't issue res_ndestroy() but issues res_nclose().
> > It causes memory leak. The FreeBSD port of libspf2 has a patch to
> > address this issue.
>
> Thanks, I applied all patches (~10) from the FreeBSD Ports tree and the
> problem is gone. The memory usage of milter-greylist is actually frozen
> now, no matter how hard I try. Also, the claimed cpu-time is much lower.
I have nothing to add about libspf2, I just changed the title so that
other people do not miss that.
People using --enable-dnsrbl may also have a problem on some systems.
dnsrbl.c contains:
#if (defined(res_ninit) || (__RES >= 19991006) )
#define HAVE_RESN 1
#ifndef res_ndestroy
#define res_ndestroy(res) res_nclose(res)
#endif
#else
#define res_ninit(res) \
((_res.options & RES_INIT) == 0 && res_init())
#define res_nquery(res, req, class, type, ans, anslen) \
res_query(req, class, type, ans, anslen)
#define res_ndestroy(res)
#endif
At mine (NetBSD 4.0, built-in BIND 9), this does not touch
res_ndestroy(), so I am safe.
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...2008-10-24 by Hajimu UMEMOTO
Hi,
>>>>> On Fri, 24 Oct 2008 05:27:18 +0200
>>>>> manu@... said:
manu> People using --enable-dnsrbl may also have a problem on some systems.
manu> dnsrbl.c contains:
manu> #if (defined(res_ninit) || (__RES >= 19991006) )
manu> #define HAVE_RESN 1
manu> #ifndef res_ndestroy
manu> #define res_ndestroy(res) res_nclose(res)
manu> #endif
manu> #else
manu> #define res_ninit(res) \
manu> ((_res.options & RES_INIT) == 0 && res_init())
manu> #define res_nquery(res, req, class, type, ans, anslen) \
manu> res_query(req, class, type, ans, anslen)
manu> #define res_ndestroy(res)
manu> #endif
I contributed the chunk to milter-greylist.
Where there is res_ndestroy(), we do use it.
Where there is no res_ndestroy(), we use res_nclose().
Where there is no res_n* API, we use old res_init API, and we don't
need to issue res_nclose() nor res_ndestroy().
I believe it covers all variations of the BIND resolver.
If I'm thinking in a wrong way, please let me know.
Sincerely,
--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@... ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/2008-10-24 by manu@netbsd.org
Hajimu UMEMOTO <ume@...> wrote:
> I believe it covers all variations of the BIND resolver.
> If I'm thinking in a wrong way, please let me know.
Well, the logic seems good, but there may be some corner cases we did
not think about. This is why I said there *may* be a problem on some
setups...
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2008-10-24 by Petar Bogdanovic
On Fri, Oct 24, 2008 at 05:27:18AM +0200, manu@... wrote:
> Petar Bogdanovic <petar@...> wrote:
>
> > > The BIND9's resolver requires res_ndestroy() for cleanup res_state.
> > > However, libspf2 doesn't issue res_ndestroy() but issues res_nclose().
> > > It causes memory leak. The FreeBSD port of libspf2 has a patch to
> > > address this issue.
> >
> > Thanks, I applied all patches (~10) from the FreeBSD Ports tree and the
> > problem is gone. The memory usage of milter-greylist is actually frozen
> > now, no matter how hard I try. Also, the claimed cpu-time is much lower.
>
> I have nothing to add about libspf2, I just changed the title so that
> other people do not miss that.
milter-greylist using libspf-alt leaks too but I didn't try to patch it.
A quick check reveals that they don't use res_ndestroy() either:
[petar@pintail ~/tmp/libspf_alt-0.4.0/lib/spf_alt]
$ grep -r res_ .
./spf_dns_resolv.c: struct __res_state res_state;
./spf_dns_resolv.c:#define SPF_h_errno spfhook->res_state.res_h_errno
./spf_dns_resolv.c: dns_len = res_nquery( &spfhook->res_state, domain, ns_c_in, rr_type,
./spf_dns_resolv.c: dns_len = res_query( domain, ns_c_in, rr_type,
./spf_dns_resolv.c: if ( res_ninit( &spfhook->res_state ) != 0 )
./spf_dns_resolv.c: if ( res_init() != 0 )
./spf_dns_resolv.c: res_nclose( &spfhook->res_state );
./spf_dns_resolv.c: res_close();
2008-10-24 by Emmanuel Dreyfus
On Fri, Oct 24, 2008 at 11:05:25AM +0200, Petar Bogdanovic wrote:
> milter-greylist using libspf-alt leaks too but I didn't try to patch it.
> A quick check reveals that they don't use res_ndestroy() either:
Anyone voluntering for reporitng the bugs to various libspf maintainers?
--
Emmanuel Dreyfus
manu@...
2008-10-24 by Fredrik Pettai
On Oct 24, 2008, at 11:06 AM, Emmanuel Dreyfus wrote:
> On Fri, Oct 24, 2008 at 11:05:25AM +0200, Petar Bogdanovic wrote:
> > milter-greylist using libspf-alt leaks too but I didn't try to
> patch it.
> > A quick check reveals that they don't use res_ndestroy() either:
>
> Anyone voluntering for reporitng the bugs to various libspf
> maintainers?
>
Sure, I can do that (if somebody else hasn't done that already?)
Re,
/P
2008-10-24 by Petar Bogdanovic
On Fri, Oct 24, 2008 at 11:22:35AM +0200, Fredrik Pettai wrote:
> On Oct 24, 2008, at 11:06 AM, Emmanuel Dreyfus wrote:
> > On Fri, Oct 24, 2008 at 11:05:25AM +0200, Petar Bogdanovic wrote:
> > > milter-greylist using libspf-alt leaks too but I didn't try to
> > patch it.
> > > A quick check reveals that they don't use res_ndestroy() either:
> >
> > Anyone voluntering for reporitng the bugs to various libspf
> > maintainers?
> >
>
> Sure, I can do that (if somebody else hasn't done that already?)
Nope, I didn't do it yet so please go ahead! :)
2008-10-25 by Johann E. Klasek
On Fri, Oct 24, 2008 at 01:34:42PM +0900, Hajimu UMEMOTO wrote:
> Hi,
>
> >>>>> On Fri, 24 Oct 2008 05:27:18 +0200
> >>>>> manu@... said:
>
> manu> People using --enable-dnsrbl may also have a problem on some systems.
> manu> dnsrbl.c contains:
>
> manu> #if (defined(res_ninit) || (__RES >= 19991006) )
> manu> #define HAVE_RESN 1
> manu> #ifndef res_ndestroy
> manu> #define res_ndestroy(res) res_nclose(res)
> manu> #endif
> manu> #else
> manu> #define res_ninit(res) \
> manu> ((_res.options & RES_INIT) == 0 && res_init())
> manu> #define res_nquery(res, req, class, type, ans, anslen) \
> manu> res_query(req, class, type, ans, anslen)
> manu> #define res_ndestroy(res)
> manu> #endif
>
> I contributed the chunk to milter-greylist.
> Where there is res_ndestroy(), we do use it.
That is only true on systems (at least Linux and NetBSD AFAIK) where
res_* consists of a macro layer which can easily checked with
preprocessor #ifdef but *not* on such systems where res_ndestroy is a
real library function! (e.g. Solaris).
For the 4.0 branch I contributed the __RES hack above (which addresses
the previous mentioned problem in a simple way), but this seems not to
help much if we want to know if res_ndestroy exists (Solaris 9 has
res_ndestroy, Solaris 8 not but both have the same __RES value ...).
> Where there is no res_ndestroy(), we use res_nclose().
> Where there is no res_n* API, we use old res_init API, and we don't
> need to issue res_nclose() nor res_ndestroy().
> I believe it covers all variations of the BIND resolver.
> If I'm thinking in a wrong way, please let me know.
What we need is an additional autoconf rule which checks the
res_ndestroy existence on a platform ...
BTW, the existence of res_ninit could be also checked (so we can
drop the somekind unreliable __RES hack).
To Manu: Yes, I know, you are expecting something that looks like
a patch ... I will try ASAP.
Johann Klasek
2008-10-25 by manu@netbsd.org
Johann E. Klasek <johann@...> wrote:
> To Manu: Yes, I know, you are expecting something that looks like
> a patch ... I will try ASAP.
That will not be that difficult: you just have to use AC_HAVE_FUNCS, and
HAVE_* will be generated in config.h (or not)
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2010-02-16 by Dietmar Rieder
Hi,
we are running milter-greylist-4.2.3 on Solaris 10 x86, it works great
so far, but I'm a little concerned by its memory consumption. Right
after starting the service it uses < 30Mb of RAM but after running it
about 48h it uses already ~1Gb of RAM.
Is this normal?
Is there something that could be done to limit the memory consumption?
Here is some additional information:
The greylist.db is about 14Mb and gets dumped every 15 minutes.
We used the following configure option for building the package.
./configure --enable-dnsrbl --sysconfdir=/etc/mail --with-user=smmsp
The milter is used together with sendmail-8.14.4 and linked against its
libmilter.
I hope someone can help us with this issue, and please let me non if
there is more information that I can provide...
Thanks
Didi
2010-02-16 by Petar Bogdanovic
On Tue, Feb 16, 2010 at 03:12:41PM +0100, Dietmar Rieder wrote:
>
> we are running milter-greylist-4.2.3 on Solaris 10 x86, it works great
> so far, but I'm a little concerned by its memory consumption. Right
> after starting the service it uses < 30Mb of RAM but after running it
> about 48h it uses already ~1Gb of RAM.
Are you using libspf2 <= 1.2.8?
Petar Bogdanovic
2010-02-16 by Peter Bonivart
On Tue, Feb 16, 2010 at 3:12 PM, Dietmar Rieder <adrieder@...> wrote:
> Hi,
>
> we are running milter-greylist-4.2.3 on Solaris 10 x86, it works great
> so far, but I'm a little concerned by its memory consumption. Right
> after starting the service it uses < 30Mb of RAM but after running it
> about 48h it uses already ~1Gb of RAM.
>
> Is this normal?
> Is there something that could be done to limit the memory consumption?
I package miltergreylist for www.opencsw.org and it's "normal" that it
leaks memory. I use a restart script to keep it in check.
With commands like "pmap -x <miltergreylist-pid>" you can see how much
memory it's consuming. From there you can easily build a script to
restart it before it goes to insane values.
Funny thing is that some Linux users (including myself) have no issues
with this on miltergreylist but with ClamAV it's the other way round,
ClamAV seems to leek more memory on Linux than on Solaris.
--
/peter
2010-02-16 by Dietmar Rieder
On 02/16/2010 03:36 PM, Petar Bogdanovic wrote:
>
>
> On Tue, Feb 16, 2010 at 03:12:41PM +0100, Dietmar Rieder wrote:
> >
> > we are running milter-greylist-4.2.3 on Solaris 10 x86, it works great
> > so far, but I'm a little concerned by its memory consumption. Right
> > after starting the service it uses < 30Mb of RAM but after running it
> > about 48h it uses already ~1Gb of RAM.
>
> Are you using libspf2 <= 1.2.8?
No, it is not compiled with spf support.
Didi
2010-02-16 by Dietmar Rieder
On 02/16/2010 04:08 PM, Peter Bonivart wrote:
>
>
> On Tue, Feb 16, 2010 at 3:12 PM, Dietmar Rieder <adrieder@...
> <mailto:adrieder%40sbox.tugraz.at>> wrote:
> > Hi,
> >
> > we are running milter-greylist-4.2.3 on Solaris 10 x86, it works great
> > so far, but I'm a little concerned by its memory consumption. Right
> > after starting the service it uses < 30Mb of RAM but after running it
> > about 48h it uses already ~1Gb of RAM.
> >
> > Is this normal?
> > Is there something that could be done to limit the memory consumption?
>
> I package miltergreylist for www.opencsw.org and it's "normal" that it
> leaks memory. I use a restart script to keep it in check.
that's sad
> With commands like "pmap -x <miltergreylist-pid>" you can see how much
> memory it's consuming. From there you can easily build a script to
> restart it before it goes to insane values.
ok thanks, sounds "hackish" though
It is something I wanted to avoid :-(
> Funny thing is that some Linux users (including myself) have no issues
> with this on miltergreylist but with ClamAV it's the other way round,
> ClamAV seems to leek more memory on Linux than on Solaris.
Strange, and you are right I did not experience memory leaks with other
milters (MIMEDefang, milter-sender, clamav) on Solaris.
Didi
2010-02-16 by Oliver Fromme
Peter Bonivart wrote:
> Dietmar Rieder wrote:
> > we are running milter-greylist-4.2.3 on Solaris 10 x86, it works great
> > so far, but I'm a little concerned by its memory consumption. Right
> > after starting the service it uses < 30Mb of RAM but after running it
> > about 48h it uses already ~1Gb of RAM.
> >
> > Is this normal?
> > Is there something that could be done to limit the memory consumption?
>
> I package miltergreylist for www.opencsw.org and it's "normal" that it
> leaks memory. I use a restart script to keep it in check.
>
> With commands like "pmap -x <miltergreylist-pid>" you can see how much
> memory it's consuming. From there you can easily build a script to
> restart it before it goes to insane values.
>
> Funny thing is that some Linux users (including myself) have no issues
> with this on miltergreylist but with ClamAV it's the other way round,
> ClamAV seems to leek more memory on Linux than on Solaris.
It's not necessarily milter-greylist (or clamav) that leaks
memory. It could be a matter of the malloc implementation
in libc on the respective systems.
The malloc implementations are usually optimized for "typical"
allocation patterns, but sometimes there are programs that
use memory allocations in different patterns which leads to
fragmentation and inefficient use of actual memory. Threaded
programs seem to be especially prone to this kind of problem.
On FreeBSD, the allocation algorithm can be tuned with about
a dozen knobs (globally via /etc/malloc.conf, or per-process
via environment variables). This is sometimes worthwhile if
a program seems to leak memory. I don't know if Solaris has
something similar, but it might be worth investigating.
By the way, on my FreeBSD machines milter-greylist doesn't
leak. It grows up to a certain size and then stabilizes,
depending on the amount of mail traffic. (I could probably
shrink that size down somewhat via malloc.conf, but I didn't
bother so far because there's enough memory available.)
Best regards
Oliver
--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606, Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758, Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart
FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd
We're sysadmins. To us, data is a protocol-overhead.
2010-02-16 by Christian Pélissier
Le mar. 16/02/2010 à 15:12, Dietmar Rieder a écrit :
>
>
> Hi,
>
> we are running milter-greylist-4.2.3 on Solaris 10 x86, it works great
> so far, but I'm a little concerned by its memory consumption. Right
> after starting the service it uses < 30Mb of RAM but after running it
> about 48h it uses already ~1Gb of RAM.
Here is a bad solution for milter-greylist under Solaris 10.
The above ksh script launched by cron every 15 minutes.
Why milter-greylist doesn't free memory under Solaris ?
Same bug under Linux and BSD ?
==============
#!/bin/ksh
export PATH=/usr/bin:/usr/sbin
maxsize=1048576 # 1 Go
#maxsize=2097152 # 2 Go
#maxsize=4194304 # 4 Go
function psmgl
{
ps -eo vsz,rss,comm | fgrep milter-greylist
}
if pgrep -u smmsp milter-greylist > /dev/null
then
psmgl | read vsz rss comm etime
if (( $vsz >= $maxsize ))
then
(psmgl ; ps -fely ; swap -s; df -h ) | \
mailx -s "Restart greycheck" root
/etc/init.d/greymilter stop
sleep 10
/etc/init.d/greymilter start
logmsg="milter-greylist restarted (too big)"
logger -t greycheck -p mail.notice -i "$logmsg"
fi
else
( ps -fely ; swap -s; df -h ) | mailx -s "Start greycheck" root
/etc/init.d/greymilter start
logmsg="milter-greylist started (not running)"
logger -t greycheck -p mail.notice -i "$logmsg"
fi
==================
>
> Is this normal?
> Is there something that could be done to limit the memory consumption?
>
> Here is some additional information:
>
> The greylist.db is about 14Mb and gets dumped every 15 minutes.
>
> We used the following configure option for building the package.
> ./configure --enable-dnsrbl --sysconfdir=/etc/mail --with-user=smmsp
>
> The milter is used together with sendmail-8.14.4 and linked against
> its
> libmilter.
>
> I hope someone can help us with this issue, and please let me non if
> there is more information that I can provide...
>
> Thanks
> Didi
>
>
>
>
>
--
Christian Pélissier
Office National d'Études et de Recherches Aérospatiales
BP 72 92322 Chatillon
Tel: 33 1 46 73 44 19, Fax: 33 1 46 73 41 502010-02-16 by Bob Friesenhahn
On Tue, 16 Feb 2010, Dietmar Rieder wrote:
> we are running milter-greylist-4.2.3 on Solaris 10 x86, it works great
> so far, but I'm a little concerned by its memory consumption. Right
> after starting the service it uses < 30Mb of RAM but after running it
> about 48h it uses already ~1Gb of RAM.
As a potentially useful data point, I am running milter-greylist-4.2.2
on Solaris 10 U6 (+patches) SPARC with libspf2 1.2.9 and after 126
days of uptime, it is consuming only 36MB of RAM, with no sign of a
leak.
> The greylist.db is about 14Mb and gets dumped every 15 minutes.
My greylist.db is only 1.5MB. When greylisting was first employed,
the file was much larger but it seems that over time, the amount of
spam attempts has considerably diminished.
It could be that the connection rate is overflowing an arbitrary limit
on 32-bit Solaris. I recall something in the notes regarding a file
descriptor limit. Unfortunately, libmilter is not provided as a
64-bit build under Solaris and it is more work to build your own.
Bob
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
2010-02-16 by Peter Bonivart
Can you tell us something about your build environment?
How do you call configure?
Do you use gcc or Sun Studio?
Sent from my iPhone
On 16 feb 2010, at 18.26, Bob Friesenhahn <
bfriesen@...> wrote:
Show quoted textHide quoted text
As a potentially useful data point, I am running milter-greylist-4.2.2
on Solaris 10 U6 (+patches) SPARC with libspf2 1.2.9 and after 126
days of uptime, it is consuming only 36MB of RAM, with no sign of a
leak.
2010-02-16 by Dietmar Rieder
On 02/16/2010 03:12 PM, Dietmar Rieder wrote:
>
>
> Hi,
>
> we are running milter-greylist-4.2.3 on Solaris 10 x86, it works great
> so far, but I'm a little concerned by its memory consumption. Right
> after starting the service it uses < 30Mb of RAM but after running it
> about 48h it uses already ~1Gb of RAM.
>
> Is this normal?
> Is there something that could be done to limit the memory consumption?
First of all, THANKS to all of you for your valuable input...
I think I found a solution that works cleanly an Solaris 10. Olivers
hint to look more closely at malloc put me on the right track:
Solaris 10 provides several libraries with implementation of malloc
different from the standard malloc shipped with libc. So if you link
milter-greylist to libumem (or libbsdmalloc) the memory consumption
stays at a normal level. (see: man libumem [man libbsdmalloc])
Here is basically what I'm using to build milter-greylist:
#!/bin/sh
CC=gcc
LIBS="-lumem"
LDFLAGS="-L/usr/local/lib"
LDFLAGS="$LDFLAGS -R/usr/local/lib"
CFLAGS="-I/usr/local/include"
export LIBS
export LDFLAGS
export CC
export CFLAGS
./configure --enable-dnsrbl --sysconfdir=/etc/mail --with-user=smmsp
gmake
I hope this of help also to someone else,
Didi
2010-02-16 by Dietmar Rieder
On 02/16/2010 07:27 PM, Peter Bonivart wrote:
>
>
> Can you tell us something about your build environment?
>
> How do you call configure?
> Do you use gcc or Sun Studio?
gcc
But it seems that I found the solution, by linking to libumem..... (see
my other post)
Thanks
Didi
2010-02-16 by Bob Friesenhahn
On Tue, 16 Feb 2010, Peter Bonivart wrote:
>
>
> Can you tell us something about your build environment?
>
> How do you call configure?
> Do you use gcc or Sun Studio?
For my successful case, I used GCC 4.3.4 and with these configure
options:
'--with-libspf2=/usr/local' '--enable-dnsrbl' '--with-user=smmsp'
I didn't use a special memory allocator like libumem.
Whenever a recommended patch for libc or one of the other base runtime
libraries appears, I go ahead and apply it.
Bob
>
> Sent from my iPhone
>
> On 16 feb 2010, at 18.26, Bob Friesenhahn <bfriesen@...> wrote:
>
> As a potentially useful data point, I am running milter-greylist-4.2.2
> on Solaris 10 U6 (+patches) SPARC with libspf2 1.2.9 and after 126
> days of uptime, it is consuming only 36MB of RAM, with no sign of a
> leak.
>
>
>
>
>
>
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
2010-02-16 by Bob Friesenhahn
On Tue, 16 Feb 2010, Dietmar Rieder wrote:
>
> Solaris 10 provides several libraries with implementation of malloc
> different from the standard malloc shipped with libc. So if you link
> milter-greylist to libumem (or libbsdmalloc) the memory consumption
> stays at a normal level. (see: man libumem [man libbsdmalloc])
Under Solaris, is not even necessary to explicitly link to libumem in
order to use it. The malloc functions can be overloaded by setting
this in the environment before starting the program:
LD_PRELOAD=libumem.so
Regardless, if switching to libumem solves problems, there is surely a
bug somewhere. Libumem supports some built in debugging features
which might help to discover the location of the problem.
For those who are interested, libumem is available for use on other
systems such as Linux and FreeBSD. I have used it under FreeBSD.
Bob
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
2010-02-16 by Dietmar Rieder
On 02/16/2010 09:37 PM, Bob Friesenhahn wrote:
>
>
> On Tue, 16 Feb 2010, Dietmar Rieder wrote:
> >
> > Solaris 10 provides several libraries with implementation of malloc
> > different from the standard malloc shipped with libc. So if you link
> > milter-greylist to libumem (or libbsdmalloc) the memory consumption
> > stays at a normal level. (see: man libumem [man libbsdmalloc])
>
> Under Solaris, is not even necessary to explicitly link to libumem in
> order to use it. The malloc functions can be overloaded by setting
> this in the environment before starting the program:
>
> LD_PRELOAD=libumem.so
Right, I know, but then I should never forget to set it ;-)
That's why it linked it at compile time.
> Regardless, if switching to libumem solves problems, there is surely a
> bug somewhere. Libumem supports some built in debugging features
> which might help to discover the location of the problem.
...I suspected that, but unfortunately I do not have the skills nor
resources for debugging this problem, but hopefully there is someone out
there who does...
Didi
2010-02-16 by Bob Friesenhahn
On Tue, 16 Feb 2010, Dietmar Rieder wrote:
>
>> Regardless, if switching to libumem solves problems, there is surely a
>> bug somewhere. Libumem supports some built in debugging features
>> which might help to discover the location of the problem.
>
> ...I suspected that, but unfortunately I do not have the skills nor
> resources for debugging this problem, but hopefully there is someone out
> there who does...
Consult 'man umem_debug'. This documents a UMEM_DEBUG environment
variable and some other features which can be used to enable debugging
and reporting of issues.
It seems that
UMEM_DEBUG=default
is useful. With this enabled, the software may run a bit slower, but
it should report any issues. Allocated data is non-zero so if the
software somehow depended on this, then the problem should soon become
evident.
A portable version of libumem is available from
"https://labs.omniti.com/trac/portableumem".
Bob
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
2010-02-17 by Christian Pélissier
Le mar. 16/02/2010 à 19:55, Dietmar Rieder a écrit :
> Solaris 10 provides several libraries with implementation of malloc
> different from the standard malloc shipped with libc. So if you link
> milter-greylist to libumem (or libbsdmalloc) the memory consumption
> stays at a normal level. (see: man libumem [man libbsdmalloc])
>
Take care with Solaris 10 libumem is threads Safe and libbsdmalloc is MT
unsafe (see the man).
--
Christian Pélissier
Office National d'Études et de Recherches Aérospatiales
BP 72 92322 Chatillon
Tel: 33 1 46 73 44 19, Fax: 33 1 46 73 41 50
2010-02-17 by Christian Pélissier
It seems that libumem has no effect for me.
milter-greylist start with 17 Mb memory. 4 hours later milter-greylist
is 138 Mb and will probably reach 1 Gb tomorrow.
My greylist.db is 7 Gb.
Others milter have no leak problems :
- j-chkmail stable at 71M
- clamav-milter stable at 3,2M
- dkim-filter stable at 8,5M
===
OS Solaris 10 U1 + recommended cluster + sendmail 8.13.8 Sun
Milter order :
O InputMailFilters=j-chkmail, clmilter, greylist, dkim-milter
greylist first ?
Compiled with Sun gcc :
PATH=/usr/sfw/bin:/usr/ccs/bin:/usr/bin ; export PATH
LIBS="-lumem" ./configure --prefix=/usr/local --enable-dnsrbl
gmake clean
gmake
# ldd /usr/local/bin/milter-greylist
libumem.so.1 => /lib/libumem.so.1
libpthread.so.1 => /lib/libpthread.so.1
libsocket.so.1 => /lib/libsocket.so.1
libresolv.so.2 => /lib/libresolv.so.2
libnsl.so.1 => /lib/libnsl.so.1
libmilter.so.1 => /usr/lib/libmilter.so.1
libc.so.1 => /lib/libc.so.1
libmp.so.2 => /lib/libmp.so.2
libmd.so.1 => /lib/libmd.so.1
libscf.so.1 => /lib/libscf.so.1
libdoor.so.1 => /lib/libdoor.so.1
libuutil.so.1 => /lib/libuutil.so.1
libgen.so.1 => /lib/libgen.so.1
libm.so.2 => /lib/libm.so.2
Same leak problems (without umem) with Sun Studio cc2010-02-18 by Dietmar Rieder
On 02/16/2010 11:37 PM, Bob Friesenhahn wrote:
>
>
> On Tue, 16 Feb 2010, Dietmar Rieder wrote:
> >
> >> Regardless, if switching to libumem solves problems, there is surely a
> >> bug somewhere. Libumem supports some built in debugging features
> >> which might help to discover the location of the problem.
> >
> > ...I suspected that, but unfortunately I do not have the skills nor
> > resources for debugging this problem, but hopefully there is someone out
> > there who does...
>
> Consult 'man umem_debug'. This documents a UMEM_DEBUG environment
> variable and some other features which can be used to enable debugging
> and reporting of issues.
>
> It seems that
>
> UMEM_DEBUG=default
>
> is useful. With this enabled, the software may run a bit slower, but
> it should report any issues. Allocated data is non-zero so if the
> software somehow depended on this, then the problem should soon become
> evident.
Hi, thanks for the hint, I tried to use umem debugging and here is what
I get:
gcore 25869
mdb core.25869
> ::findleaks -dvf
findleaks: maximum buffers => 871328
findleaks: actual buffers => 870422
findleaks:
findleaks: potential pointers => 8295213
findleaks: dismissals => 2109500 (25.4%)
findleaks: misses => 4738170 (57.1%)
findleaks: dups => 578557 ( 6.9%)
findleaks: follows => 868986 (10.4%)
findleaks:
findleaks: elapsed wall time => 5 seconds
findleaks:
CACHE LEAKED BUFCTL CALLER
080ee710 1436 09f79648 libresolv.so.2`__res_vinit+0x118
----------------------------------------------------------------------
Total 1436 buffers, 1286656 bytes
umem_alloc_896 leak: 1436 buffers, 896 bytes each, 1286656 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
9f79648 9f7ab80 2dcdb5b1a541 12
80ee710 80ceed8 0
libumem.so.1`umem_cache_alloc_debug+0x16c
libumem.so.1`umem_cache_alloc+0x15c
libumem.so.1`umem_alloc+0x3f
libumem.so.1`malloc+0x23
libresolv.so.2`__res_vinit+0x118
libresolv.so.2`res_ninit+0x1a
dnsrbl_check_source+0x14a
acl_filter+0x281
real_envrcpt+0x201
mlfi_envrcpt+0x19
st_rcpt+0x66
mi_engine+0x2b5
mi_handle_session+0x36
mi_thread_handle_wrapper+0xe
libc.so.1`_thr_setup+0x4e
Unfortunately I'm no C programmer so I'm not sure what all of this
means, it seems to be something with lirbresolve.so.2...
Didi2010-02-18 by manu@netbsd.org
Dietmar Rieder <adrieder@...> wrote:
> libumem.so.1`malloc+0x23
> libresolv.so.2`__res_vinit+0x118
> libresolv.so.2`res_ninit+0x1a
> dnsrbl_check_source+0x14a
A quick look at NetBSD's res_ninit() sources (it's from BIND, and I
beleive most system have the same), indeed shows a malloc():
statp->_u._ext.ext = malloc(sizeof(*statp->_u._ext.ext));
This means that any successful call to res_ninit() must have a
corresponding res_ndestroy() call.
In dnsrbl_check_source, once res_ninit() is called, there is no way out
of the function without executing res_ndestroy(). But the trick is that
res_ninit() and res_ndestroy() may be macros in milter-greylist, so that
we can cope with older BIND resolvers without having #ifdef's everywhere
in our code.
In some situation, milter-greylist can define res_ndestroy() as
res_nclose(), and this is a problem, because at least in NetBSD's
resolver, res_ndestroy() does free statp->_u._ext.ext whereas
res_nclose() does not.
Hence, here is the offending code in dnsrbl.c
#if (defined(res_ninit) || (__RES >= 19991006) )
#define HAVE_RESN 1
#ifndef res_ndestroy
#define res_ndestroy(res) res_nclose(res) /* <--- XXXhereXXX */
#endif
#else
#define res_ninit(res) \
((_res.options & RES_INIT) == 0 && res_init())
#define res_nquery(res, req, class, type, ans, anslen) \
res_query(req, class, type, ans, anslen)
#define res_ndestroy(res)
#endif
The ifdef on res_ninit and res_ndestroy is an error, it will never be
evaluated as true. Please try this instead, and tell us if you get the
warning at compile time.
#if (__RES >= 19991006) )
#define HAVE_RESN 1
#else
#warn "no res_ninit!"
#define res_ninit(res) \
((_res.options & RES_INIT) == 0 && res_init())
#define res_nquery(res, req, class, type, ans, anslen) \
res_query(req, class, type, ans, anslen)
#define res_ndestroy(res)
#endif
Handling the BIND versions where res_nclose() exists but not
res_ndestroy() must be done by an autoconf test.
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...2010-02-18 by Dietmar Rieder
On 02/18/2010 11:40 AM, manu@... wrote:
>
>
> Dietmar Rieder <adrieder@...
> <mailto:adrieder%40sbox.tugraz.at>> wrote:
>
> > libumem.so.1`malloc+0x23
> > libresolv.so.2`__res_vinit+0x118
> > libresolv.so.2`res_ninit+0x1a
> > dnsrbl_check_source+0x14a
>
> A quick look at NetBSD's res_ninit() sources (it's from BIND, and I
> beleive most system have the same), indeed shows a malloc():
> statp->_u._ext.ext = malloc(sizeof(*statp->_u._ext.ext));
>
> This means that any successful call to res_ninit() must have a
> corresponding res_ndestroy() call.
>
> In dnsrbl_check_source, once res_ninit() is called, there is no way out
> of the function without executing res_ndestroy(). But the trick is that
> res_ninit() and res_ndestroy() may be macros in milter-greylist, so that
> we can cope with older BIND resolvers without having #ifdef's everywhere
> in our code.
>
> In some situation, milter-greylist can define res_ndestroy() as
> res_nclose(), and this is a problem, because at least in NetBSD's
> resolver, res_ndestroy() does free statp->_u._ext.ext whereas
> res_nclose() does not.
>
> Hence, here is the offending code in dnsrbl.c
>
> #if (defined(res_ninit) || (__RES >= 19991006) )
> #define HAVE_RESN 1
> #ifndef res_ndestroy
> #define res_ndestroy(res) res_nclose(res) /* <--- XXXhereXXX */
> #endif
> #else
> #define res_ninit(res) \
> ((_res.options & RES_INIT) == 0 && res_init())
> #define res_nquery(res, req, class, type, ans, anslen) \
> res_query(req, class, type, ans, anslen)
> #define res_ndestroy(res)
> #endif
>
> The ifdef on res_ninit and res_ndestroy is an error, it will never be
> evaluated as true. Please try this instead, and tell us if you get the
> warning at compile time.
>
> #if (__RES >= 19991006) )
> #define HAVE_RESN 1
> #else
> #warn "no res_ninit!"
> #define res_ninit(res) \
> ((_res.options & RES_INIT) == 0 && res_init())
> #define res_nquery(res, req, class, type, ans, anslen) \
> res_query(req, class, type, ans, anslen)
> #define res_ndestroy(res)
> #endif
>
> Handling the BIND versions where res_nclose() exists but not
> res_ndestroy() must be done by an autoconf test.
Hi thanks for the quick response and analysis. I changed the dnsrbl.c
file as indicated and recompiled the milter. I didn't get a warning at
compile time.
I genereated a core dump after some minute, and now it looks really
promising!!!!
gcore 6891
mdb core.6891
> ::findleaks -dvf
findleaks: maximum buffers => 873580
findleaks: actual buffers => 872852
findleaks:
findleaks: potential pointers => 8311361
findleaks: dismissals => 2091319 (25.1%)
findleaks: misses => 4770578 (57.3%)
findleaks: dups => 576612 ( 6.9%)
findleaks: follows => 872852 (10.5%)
findleaks:
findleaks: elapsed wall time => 5 seconds
findleaks:
CACHE LEAKED BUFCTL CALLER
----------------------------------------------------------------------
Total 0 buffers, 0 bytes
I'll observe it for a while and I'll let you know if there is any
leaking left.
THANK you so much!
Didi2010-02-18 by manu@netbsd.org
Dietmar Rieder <adrieder@...> wrote:
> THANK you so much!
Unfortunately I won't be able to release immediatly, as the machine that
host my CVS is dead. :-/
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2010-02-18 by manu@netbsd.org
Emmanuel Dreyfus <manu@...> wrote:
> #ifndef res_ndestroy
> #define res_ndestroy(res) res_nclose(res) /* <--- XXXhereXXX */
> #endif
Is there an autoconf hecker in the room?
Obviously we need autoconf to check for res_ninit, res_ndestroy,
res_nclose and devine HAVE_res_ndestroy so that we can use it. But I
have little success with it. Any hint?
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2010-02-18 by Bob Friesenhahn
On Thu, 18 Feb 2010, manu@... wrote:
> Emmanuel Dreyfus <manu@...> wrote:
>
>> #ifndef res_ndestroy
>> #define res_ndestroy(res) res_nclose(res) /* <--- XXXhereXXX */
>> #endif
>
> Is there an autoconf hecker in the room?
>
> Obviously we need autoconf to check for res_ninit, res_ndestroy,
> res_nclose and devine HAVE_res_ndestroy so that we can use it. But I
> have little success with it. Any hint?
Are these assured to be normal library functions? After you have
added any necessary resolver libraries to LIBS, you should be able to
use
AC_CHECK_FUNCS([res_ninit res_ndestroy res_nclose])
Bob
--
Bob Friesenhahn
bfriesen@..., http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
2010-02-18 by Johann E. Klasek
BTW, we had this strangeness on topic already back in Oct 2008, see
http://tech.groups.yahoo.com/group/milter-greylist/message/4805
The conclusion was that we need a autoconf rule that handles
res_ninit() existence in a proper way. My promise to provide
such a beast never came to reality ...
Well, I'll give them a 2nd try.
On Thu, Feb 18, 2010 at 11:40:07AM +0100, manu@... wrote:
[..]
> This means that any successful call to res_ninit() must have a
> corresponding res_ndestroy() call.
>
> In dnsrbl_check_source, once res_ninit() is called, there is no way out
> of the function without executing res_ndestroy(). But the trick is that
> res_ninit() and res_ndestroy() may be macros in milter-greylist, so that
> we can cope with older BIND resolvers without having #ifdef's everywhere
> in our code.
>
> In some situation, milter-greylist can define res_ndestroy() as
> res_nclose(), and this is a problem, because at least in NetBSD's
> resolver, res_ndestroy() does free statp->_u._ext.ext whereas
> res_nclose() does not.
>
> Hence, here is the offending code in dnsrbl.c
>
> #if (defined(res_ninit) || (__RES >= 19991006) )
> #define HAVE_RESN 1
> #ifndef res_ndestroy
> #define res_ndestroy(res) res_nclose(res) /* <--- XXXhereXXX */
> #endif
> #else
> #define res_ninit(res) \
> ((_res.options & RES_INIT) == 0 && res_init())
> #define res_nquery(res, req, class, type, ans, anslen) \
> res_query(req, class, type, ans, anslen)
> #define res_ndestroy(res)
> #endif
>
> The ifdef on res_ninit and res_ndestroy is an error, it will never be
> evaluated as true. Please try this instead, and tell us if you get the
> warning at compile time.
I'm not sure.
The origin in some 4.0 alpha version was
[..]
#ifdef res_ninit
#define HAVE_RESN 1
#ifndef res_ndestroy
#define res_ndestroy(res) res_nclose(res)
[..]
This tried to discover if a threaded version of res_ninit() do exist. But
this works only in environments which implement a macro layer for res_n* calls.
This is trues at least for Linux, NetBSD (AFAIK) but this will never hold on
systems which has real library functions for this (like Solaris).
The __RES condition was only added as a simply hack to this work for systems
with a newer Bind resolver (for which we can asume that the thread-safe
variants do exist).
>
> #if (__RES >= 19991006) )
> #define HAVE_RESN 1
> #else
> #warn "no res_ninit!"
> #define res_ninit(res) \
> ((_res.options & RES_INIT) == 0 && res_init())
> #define res_nquery(res, req, class, type, ans, anslen) \
> res_query(req, class, type, ans, anslen)
> #define res_ndestroy(res)
> #endif
>
> Handling the BIND versions where res_nclose() exists but not
> res_ndestroy() must be done by an autoconf test.
Maybe the check for __RES is sufficient (at least if we asume that
mostly Bind resolver are in use). But to get it really clean res_ninit()
should be also checked by autoconf ...
Johann E. K.
2010-02-18 by manu@netbsd.org
Bob Friesenhahn <bfriesen@...> wrote:
> AC_CHECK_FUNCS([res_ninit res_ndestroy res_nclose])
I tried that, but they are not found. The problem is that res_ninit is a
macro that expands to __res_ninit in the library.
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2010-02-18 by Johann E. Klasek
On Thu, Feb 18, 2010 at 09:19:48PM +0100, manu@... wrote:
> Bob Friesenhahn <bfriesen@...> wrote:
>
> > AC_CHECK_FUNCS([res_ninit res_ndestroy res_nclose])
>
> I tried that, but they are not found. The problem is that res_ninit is a
> macro that expands to __res_ninit in the library.
Not in general - this holds at least for Linux, NetBSD but not for Solaris.
Probably we must combine the check for a macro and existance of a function ...
#if (defined(HAVE_RES_NINIT) || defined(res_ninit))
#define HAVE_RESN 1
...
Johann E. K.
2010-02-19 by manu@netbsd.org
Johann E. Klasek <johann@...> wrote:
> Not in general - this holds at least for Linux, NetBSD but not for Solaris.
> Probably we must combine the check for a macro and existance of a function ...
>
> #if (defined(HAVE_RES_NINIT) || defined(res_ninit))
> #define HAVE_RESN 1
Ok, could people test this patch on Solaris and on other systems?
http://ftp.espci.fr/shadow/manu/resn.patch
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2010-02-19 by Dietmar Rieder
On 02/19/2010 05:36 AM, manu@... wrote:
>
>
> Johann E. Klasek <johann@... <mailto:johann%40klasek.at>> wrote:
>
> > Not in general - this holds at least for Linux, NetBSD but not for
> Solaris.
> > Probably we must combine the check for a macro and existance of a
> function ...
> >
> > #if (defined(HAVE_RES_NINIT) || defined(res_ninit))
> > #define HAVE_RESN 1
>
> Ok, could people test this patch on Solaris and on other systems?
> http://ftp.espci.fr/shadow/manu/resn.patch
> <http://ftp.espci.fr/shadow/manu/resn.patch>
is this patch against milter-greylist-4.2.3?
Here it doesn't apply cleanly:
# gpatch -p0 < resn.patch
patching file Makefile
Hunk #1 FAILED at 1.
Hunk #2 FAILED at 30.
Hunk #3 FAILED at 59.
3 out of 3 hunks FAILED -- saving rejects to file Makefile.rej
patching file config.h
Hunk #1 FAILED at 3.
1 out of 2 hunks FAILED -- saving rejects to file config.h.rej
patching file config.h.in
patching file configure
Hunk #1 succeeded at 726 (offset -2 lines).
Hunk #3 succeeded at 4389 (offset -2 lines).
Hunk #5 succeeded at 10098 (offset -104 lines).
patching file configure.ac
Hunk #3 succeeded at 745 with fuzz 1 (offset -33 lines).
patching file dnsrbl.c
patching file milter-greylist.spec
Hunk #1 FAILED at 1.
1 out of 1 hunk FAILED -- saving rejects to file milter-greylist.spec.rej
Didi
2010-02-19 by Hajimu UMEMOTO
Hi,
>>>>> On Fri, 19 Feb 2010 05:36:24 +0100
>>>>> manu@... said:
manu> Johann E. Klasek <johann@...> wrote:
> Not in general - this holds at least for Linux, NetBSD but not for Solaris.
> Probably we must combine the check for a macro and existance of a function ...
>
> #if (defined(HAVE_RES_NINIT) || defined(res_ninit))
> #define HAVE_RESN 1
I wrote the check. I didn't notice that there is an environment res_*
are not macro.
manu> Ok, could people test this patch on Solaris and on other systems?
manu> http://ftp.espci.fr/shadow/manu/resn.patch
It seems bit redundant to me. How about the attached patch? It is
against CVS version.
Sincerely,
2010-02-19 by manu@netbsd.org
Dietmar Rieder <adrieder@...> wrote:
> is this patch against milter-greylist-4.2.3?
> Here it doesn't apply cleanly:
>
> # gpatch -p0 < resn.patch
The important part applied. Just run autoheader; autoconf and rerun
configure
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2010-02-19 by manu@netbsd.org
Hajimu UMEMOTO <ume@...> wrote:
> It seems bit redundant to me. How about the attached patch? It is
> against CVS version.
Is it a right move to remove the test on (__RES >= 19991006)? That one
should be much more reliable than attempting to link a program.
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2010-02-20 by Johann E. Klasek
On Fri, Feb 19, 2010 at 11:03:37PM +0100, manu@... wrote:
> Hajimu UMEMOTO <ume@...> wrote:
>
> > It seems bit redundant to me. How about the attached patch? It is
> > against CVS version.
>
> Is it a right move to remove the test on (__RES >= 19991006)? That one
> should be much more reliable than attempting to link a program.
As mentioned before this condition was only a quick hack to
cope with environments, where res_n* functions are not based
on macros. It is not a general solution, it worked for Solaris (even there not
in all situations), maybe for other too, but will be never safe as
the autoconf check is.
Johann E. K.
2010-02-20 by manu@netbsd.org
Johann E. Klasek <johann@...> wrote:
> > Is it a right move to remove the test on (__RES >= 19991006)? That one
> > should be much more reliable than attempting to link a program.
>
> As mentioned before this condition was only a quick hack to
> cope with environments, where res_n* functions are not based
> on macros.
Is it? I thought it was to test for res_n* availability.
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...
2010-02-20 by Hajimu UMEMOTO
Hi,
(2010/02/20 07:03), manu@... wrote:
> Is it a right move to remove the test on (__RES>= 19991006)? That one
> should be much more reliable than attempting to link a program.
If detecting res_ninit() by configure fails, detecting
res_ndestroy() will fail, too. In such case, res_ninit() will
still be used by (__RES>= 19991006) test with regardless of
the existence of res_ndestroy(). Then, it will end up with
the memory leakage in question.
I think this situation is not desirable. Since, the user will
notice that detecting res_ninit() fail, it is better, IMHO.
--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@... ume@... ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/2010-02-21 by Johann E. Klasek
On Sat, Feb 20, 2010 at 09:24:42AM +0100, manu@... wrote:
> Johann E. Klasek <johann@...> wrote:
>
> > > Is it a right move to remove the test on (__RES >= 19991006)? That one
> > > should be much more reliable than attempting to link a program.
> >
> > As mentioned before this condition was only a quick hack to
> > cope with environments, where res_n* functions are not based
> > on macros.
>
> Is it? I thought it was to test for res_n* availability.
My answer was maybe too short: Sure, in case res_ninit is not a macro
the simplyfied asumption from __RES >= 19991006 is, that we have at
least a Bind resolver library and this version seem new enough (Bind 8
...) and that we can hope a threaded version is also available. This
seems to hold at least on Solaris but just for res_ninit(). It does not
help much if we want to know if res_ndestroy exists (Solaris 9 has
res_ndestroy, Solaris 8 has't, but both have the same __RES value ...).
So IMHO it is mandatory to use autoconf for optimal portability.
Johann E. K.
2010-02-22 by Christian Pélissier
The patch has a missing )
$ gpatch -p0 < resn.patch
$ autoheader
$ autoconf
$ ./configure --prefix=/usr/local --enable-dnsrbl
...
checking for ns_initparse in -lresolv... no
configure: WARNING: ns_initparse not found
checking for res_ninit... yes
checking for res_ndestroy... yes
checking for res_nclose... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: WARNING: 'Makefile.in' seems to ignore the --datarootdir
settingconfig.status: creating milter-greylist.spec
config.status: creating config.h
$ gmake
gcc -O -Wall -DUSE_DNSRBL -D_BSD_SOURCE -I. -I. -c -o
milter-greylist.o milter-greylist.c
gcc -O -Wall -DUSE_DNSRBL -D_BSD_SOURCE -I. -I. -c -o pending.o
pending.c
gcc -O -Wall -DUSE_DNSRBL -D_BSD_SOURCE -I. -I. -c -o sync.o sync.c
gcc -O -Wall -DUSE_DNSRBL -D_BSD_SOURCE -I. -I. -c -o dnsrbl.o
dnsrbl.c
dnsrbl.c:70:61: missing ')' in expression
gmake: *** [dnsrbl.o] Error 1
#if (defined(HAVE_RES_NINIT) || defined(res_ninit) || (__RES >=
19991006) )
#define HAVE_RESN 1
#if (!defined(res_ndestroy) && (!defined(HAVE_RES_NDESTROY))
#define res_ndestroy(res) res_nclose(res)
#endif
#else
#define res_ninit(res) \
((_res.options & RES_INIT) == 0 && res_init())
#define res_nquery(res, req, class, type, ans, anslen) \
res_query(req, class, type, ans, anslen)
#define res_ndestroy(res)
#endif
#if (!defined(res_ndestroy)) && (!defined(HAVE_RES_NDESTROY))
^^^missing )
Le ven. 19/02/2010 à 23:03, manu@... a écrit :
>
>
> Dietmar Rieder <adrieder@...> wrote:
>
> > is this patch against milter-greylist-4.2.3?
> > Here it doesn't apply cleanly:
> >
> > # gpatch -p0 < resn.patch
>
> The important part applied. Just run autoheader; autoconf and rerun
> configure
>
> --
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz
> manu@...
>
>
>
>
--
Christian Pélissier
Office National d'Études et de Recherches Aérospatiales
BP 72 92322 Chatillon
Tel: 33 1 46 73 44 19, Fax: 33 1 46 73 41 50