Milter-greylist

The consumption of virtual memory milter-greylist uses grows constantly 
(currently just under 2GB), however resident memory use is constant. 
This does not seem to be affecting performance. Any thoughts?

Here is a dump of /proc/status/13531

Name:   milter-greylist
State:  S (sleeping)
SleepAVG:       88%
Tgid:   13531
Pid:    13531
PPid:   1
TracerPid:      0
Uid:    15      15      15      15
Gid:    16000   16000   16000   16000
FDSize: 256
Groups: 11 16000
VmSize:  1967444 kB
VmLck:         0 kB
VmRSS:    133336 kB
VmData:  1964960 kB
VmStk:       772 kB
VmExe:        97 kB
VmLib:      1539 kB
StaBrk: 08086000 kB
Brk:    104d0000 kB
StaStk: bff72d30 kB
ExecLim:        08061000
Threads:        13
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000004003
SigIgn: 0000000000001000
SigCgt: 0000000180000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000


Thanks again,
Fredrik Nyberg

Fredrik Nyberg DC wrote:
> The consumption of virtual memory milter-greylist uses grows constantly
> (currently just under 2GB), however resident memory use is constant.
> This does not seem to be affecting performance. Any thoughts?


It sounds like your greylist database is getting huge. It could be you're
getting flooded with an absurd number of greylist entries, causing heavy memory
usage. Certainly the number of dictionary attacks that are prevalent these days
can cause this.

Things that will reduce the memory load would be:

1) Cut off the dictionary attacks on your system by using sendmail's bad
recipient throttling option (sendmail.mc syntax):

# after 5 invalid recipients, start slowing them down with
# 1 second sleeps. Legit mailservers will wait, but
# dictionary attackers will give up
define(`confBAD_RCPT_THROTTLE',5)

2) using a shorter timeout will cause "one-off" connections to be dumped faster.
This has the risk of causing slow-to-retry systems to never deliver mail.

3) using a shorter autowhite period will cause these entries to be dumped
faster. This will purge addresses that did retry but were only used once.
However, this means that legitimate addresses will get greylisted more often.

4) using the lazyaw option will also greatly reduce the number of entries, at
the expense of making it much easier to get past the greylist.

On Wed, Oct 12, 2005 at 11:55:31AM -0400, Matt Kettler wrote:
> 4) using the lazyaw option will also greatly reduce the number of entries, at
> the expense of making it much easier to get past the greylist.

5) buy some RAM :o)

Seriously, 2 GB seems really huge to me. I'd be curious to learn how many
message per day flow through this machine. The problem may be caused by
a memory leak (in milter-greylist or in a library)

-- 
Emmanuel Dreyfus
manu@...

On Wed, 12 Oct 2005, Emmanuel Dreyfus wrote:
> On Wed, Oct 12, 2005 at 11:55:31AM -0400, Matt Kettler wrote:
>> 4) using the lazyaw option will also greatly reduce the number of entries, at
>> the expense of making it much easier to get past the greylist.
> 5) buy some RAM :o)
> Seriously, 2 GB seems really huge to me. I'd be curious to learn how many
> message per day flow through this machine. The problem may be caused by
> a memory leak (in milter-greylist or in a library)

greylist still uses hundreds of megs of ram on busy machines (many 
millions of user-ip combinations). so really, need to move to external db.

-Dan

Emmanuel Dreyfus wrote:
> On Wed, Oct 12, 2005 at 11:55:31AM -0400, Matt Kettler wrote:
> 
>>4) using the lazyaw option will also greatly reduce the number of entries, at
>>the expense of making it much easier to get past the greylist.
> 
> 
> 5) buy some RAM :o)
> 
> Seriously, 2 GB seems really huge to me. I'd be curious to learn how many
> message per day flow through this machine.

I'd guess quite a lot of messages. abo.fi is a university and Fredrik appears to
be one of their IT staff. and "The University has nearly 8000 students".


> The problem may be caused by
> a memory leak (in milter-greylist or in a library)
> 


I don't know.. Really, given the rate of dictionary attacks these days it
doesn't seem that unreasonable if you have no mechanisms to protect against them.

How big is an entry? 2 email addresses and 4-byte IP address? 40 bytes or so?

Based on that you're talking about 50 million entries. If you've got a 5-day
timeout that's only 10 million attempts per day. For a site with 8,000 users
that doesn't seem unreasonable.

On Wed, Oct 12, 2005 at 04:09:40PM -0400, Matt Kettler wrote:
> > Seriously, 2 GB seems really huge to me. I'd be curious to learn how many
> > message per day flow through this machine.
> 
> I'd guess quite a lot of messages. abo.fi is a university and Fredrik appears to
> be one of their IT staff. and "The University has nearly 8000 students".

Well, if mail with inexistant recipient is not accepted, the amount of 
messages seen by milter-greylist should be reasonnable. 

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:
> On Wed, Oct 12, 2005 at 04:09:40PM -0400, Matt Kettler wrote:
> 
>>>Seriously, 2 GB seems really huge to me. I'd be curious to learn how many
>>>message per day flow through this machine.
>>
>>I'd guess quite a lot of messages. abo.fi is a university and Fredrik appears to
>>be one of their IT staff. and "The University has nearly 8000 students".
> 
> 
> Well, if mail with inexistant recipient is not accepted, the amount of 
> messages seen by milter-greylist should be reasonnable. 

Yes, but that's not how milter-greylist works.. at least, not on my system.

For me, the greylist check happens before the recipient is validated, thus I
constantly see greylisting going on for nonexistent users:

For example:

Oct 12 17:00:52 xanadu milter-greylist: j9CL0p9O020818: addr 203.42.55.196 from
<> to <loiterer@...> delayed for 00:01:00

loiterer is not a valid user here.. If the dictionary attacker tries the same
user again it will then be told the user is unknown.

On Wed, Oct 12, 2005 at 05:11:05PM -0400, Matt Kettler wrote:
> Yes, but that's not how milter-greylist works.. at least, not on my system.
> 
> For me, the greylist check happens before the recipient is validated, thus I
> constantly see greylisting going on for nonexistent users:

I assume you cannot configure sendmail to reject mail sent to nonexistent
usrers. Maybe milter-rcptfilter can fix your problem? You feed it with the
list of the valid addresses and it will reject everything else.

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:
> On Wed, Oct 12, 2005 at 05:11:05PM -0400, Matt Kettler wrote:
> 
>>Yes, but that's not how milter-greylist works.. at least, not on my system.
>>
>>For me, the greylist check happens before the recipient is validated, thus I
>>constantly see greylisting going on for nonexistent users:
> 
> 
> I assume you cannot configure sendmail to reject mail sent to nonexistent
> usrers. Maybe milter-rcptfilter can fix your problem? You feed it with the
> list of the valid addresses and it will reject everything else.
> 

My sendmail *does* reject nonexistent senders, and always has. re-read my
message. The rejection happens *AFTER* the greylist runs.

Matt Kettler wrote:

> 
> 
> My sendmail *does* reject nonexistent senders, and always has. re-read my
> message. The rejection happens *AFTER* the greylist runs.

I realized that's not all that clear.

Let me clarify.

My mailserver will never accept mail to a nonexistent user. I'm not describing
post-delivery bounces, it will will never allow a SMTP data phase to start.

However, when a remote server connects and tries to deliver mail,
milter-greylist declares the message greylisted. This happens _before_ sendmail
checks to see if the user exists.

The second time they connect (if they do) milter-greylist will allow the message
past, and then sendmail will 550 the message with an unknown user error.

All of this happens before the data phase in both cases.

If milter-greylist was not installed they'd have gotten a 550 the first time,
but it appears that on my server the order of operations is such that
milter-greylist runs before sendmail does any checks to see if the recipient exists.

Well I was going to say that on my system the unknown user rejection comes before greylisting. However, I remembered that the 
greylisting is on the hubs that have no local users. The unknown users are rejected via a milter that sits in front of the greylist 
milter.

I also believe that anything in the access.db will be rejected prior to the greylist milter, so if there's a bunch of unknown users 
you see over and over you can load them in the access file to reject them before greylisting. I know that's really a pain, and on 
the dictionary attacks I have experienced there are only a few emails from and/or to the same email address then the attack moves on 
to another, the server IP also changes every 10-25 emails as well so it's pretty much impossible to block by hand. I have several 
domains that have been under this type of attach for almost a year. Every day the same domains get the dictionary run against them.

One thing I did on the worst domains is; since I have several hubs setup as backup MX receivers, I created one (the lowest level MX) 
that only captures the email and doesn't pass it on to end user email server. It has a fairly long hello delay also. Only a very few 
real emails go to this machine but most of the dictionary attacks do. Most of the dictionary attacks don't wait for the really long 
hello delay, so very little email actually gets captured. Every so often I would review what was captured and send on anything that 
looked important. So little email was real that I have stopped doing that and have cron job that deletes any email older than a 
certain number of days old (just in case someone says their missing some email, I can look before it's deleted). No complains so 
far. It took a good load off the main real working hubs.

I know your pain.

Dennis

Matt Kettler wrote:

> Emmanuel Dreyfus wrote:
> 
>>On Wed, Oct 12, 2005 at 05:11:05PM -0400, Matt Kettler wrote:
>>
>>
>>>Yes, but that's not how milter-greylist works.. at least, not on my system.
>>>
>>>For me, the greylist check happens before the recipient is validated, thus I
>>>constantly see greylisting going on for nonexistent users:
>>
>>
>>I assume you cannot configure sendmail to reject mail sent to nonexistent
>>usrers. Maybe milter-rcptfilter can fix your problem? You feed it with the
>>list of the valid addresses and it will reject everything else.
>>
> 
> 
> My sendmail *does* reject nonexistent senders, and always has. re-read my
> message. The rejection happens *AFTER* the greylist runs.
> 
> 
> 
> 
> 
>  
> Yahoo! Groups Links
> 
> 
> 
>  
> 
> 
>

On Wed, Oct 12, 2005 at 06:26:54PM -0400, Matt Kettler wrote:
> > I assume you cannot configure sendmail to reject mail sent to nonexistent
> > usrers. Maybe milter-rcptfilter can fix your problem? You feed it with the
> > list of the valid addresses and it will reject everything else.
> My sendmail *does* reject nonexistent senders, and always has. re-read my
> message. The rejection happens *AFTER* the greylist runs.

Ok, let me rephrase:

I assume you cannot configure sendmail to reject mail sent to nonexistent
users before milter-greylist runs. Maybe milter-rcptfilter can fix your 
problem? You feed it with the list of the valid addresses and it will 
reject everything else.

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:
> On Wed, Oct 12, 2005 at 06:26:54PM -0400, Matt Kettler wrote:
> 
>>>I assume you cannot configure sendmail to reject mail sent to nonexistent
>>>usrers. Maybe milter-rcptfilter can fix your problem? You feed it with the
>>>list of the valid addresses and it will reject everything else.
>>
>>My sendmail *does* reject nonexistent senders, and always has. re-read my
>>message. The rejection happens *AFTER* the greylist runs.
> 
> 
> Ok, let me rephrase:
> 
> I assume you cannot configure sendmail to reject mail sent to nonexistent
> users before milter-greylist runs. Maybe milter-rcptfilter can fix your 
> problem? You feed it with the list of the valid addresses and it will 
> reject everything else.

Call me crazy, but it seems rather odd to add a milter to re-implement
functionality that's already built into sendmail. Actually, IMHO it's not just
odd, it's downright stupid.

milter-rcptfilter is for people who don't have user accounts on the local
mailserver and are relaying their mail. I DO have accounts on the server. It's
just that sendmail calls milter-greylist before doing the recipient checks.

However, when it all boils down, this is not a problem for me. I mitigate
dictionary attacks, so they don't flood my greylist.

This is really only a problem for "normal" sendmail users. ie: those who don't
relay, have local accounts, and don't mitigate dictionary attacks.

Matt Kettler wrote:
> Based on that you're talking about 50 million entries. If you've got a 5-day
> timeout that's only 10 million attempts per day. For a site with 8,000 users
> that doesn't seem unreasonable.

We have around 1 million entries in our greylist database. What really 
bugs me is that the current 2.3 GB of VmData that milter-greylist is 
using does not show up in physical memory or swap... it will be 
interresting to see what happens when the process hits 3 GB, since this 
is the amount of memory + swap that one of the servers has.

Could perhaps the kernel (Linux 2.6.9-11.ELsmp, CentOS 4.1) be reporting 
the memory usage wrong? This problem seems to have started since I began 
using the mxsync function.

Time for me to "Use the Source, Luke".

Cheers,
Fredrik Nyberg

Le jeu 13/10/2005 à 09:12, Fredrik Nyberg DC a écrit :
> Matt Kettler wrote:
> > Based on that you're talking about 50 million entries. If you've got a 5-day
> > timeout that's only 10 million attempts per day. For a site with 8,000 users
> > that doesn't seem unreasonable.
> 
> We have around 1 million entries in our greylist database. What really 
> bugs me is that the current 2.3 GB of VmData that milter-greylist is 
> using does not show up in physical memory or swap... it will be 
> interresting to see what happens when the process hits 3 GB, since this 
> is the amount of memory + swap that one of the servers has.
> 
> Could perhaps the kernel (Linux 2.6.9-11.ELsmp, CentOS 4.1) be reporting 
> the memory usage wrong? This problem seems to have started since I began 
> using the mxsync function.
> 
> Time for me to "Use the Source, Luke".
> 
> Cheers,
> Fredrik Nyberg

Storing the md5 of the tuple instead of the tuple itself could be a
solution for a large site and an a new option for milter-greylist.

md5 is only 16 byte long (against  ~ 64) 

md5(Sender IP:Sender e-mail:Recipient e-mail) Time

-- 
Christian Pélissier
Office National d'Études et de Recherches Aérospatiales
BP 72 92322 Chatillon
Tel: 33 1 46 73 44 19, Fax: 33 1 46 73 41 50

On Thu, Oct 13, 2005 at 02:13:44AM -0400, Matt Kettler wrote:
> Call me crazy, but it seems rather odd to add a milter to re-implement
> functionality that's already built into sendmail. Actually, IMHO it's not just
> odd, it's downright stupid.

Well, the idea is to get the thing working without loosing too much time.
For me it was even faster to develop milter-rcptfilter than to fix the
sendmail setup. I never said that tool was an universal solution, I just
suggested it could help in some situations.

-- 
Emmanuel Dreyfus
manu@...

Christian PELISSIER wrote:
> Storing the md5 of the tuple instead of the tuple itself could be a
> solution for a large site and an a new option for milter-greylist.
> 
> md5 is only 16 byte long (against  ~ 64) 
> 
> md5(Sender IP:Sender e-mail:Recipient e-mail) Time

Not a bad idea, but I think that it would break the subnet matching, 
since IPs on the same subnet would have different hashes... unless they 
were stored with wildcards. But then you would be unable to change the 
subnet mask later?

Cheers,
Fredrik Nyberg

Emmanuel Dreyfus wrote:
> On Thu, Oct 13, 2005 at 02:13:44AM -0400, Matt Kettler wrote:
> 
>>Call me crazy, but it seems rather odd to add a milter to re-implement
>>functionality that's already built into sendmail. Actually, IMHO it's not just
>>odd, it's downright stupid.
> 
> 
> Well, the idea is to get the thing working without loosing too much time.
> For me it was even faster to develop milter-rcptfilter than to fix the
> sendmail setup. I never said that tool was an universal solution, I just
> suggested it could help in some situations.
> 

True, and I think it's a *great* tool for people doing relay mailservers that
don't have any other list of users.

However, for me it would be a maintenance nightmare to have to edit
milter-rcptfilter every time I add a new user to the system. Eventually I'd
forget the second stem for someone, and they'd not realize it since their pop3
login would work...

On Thu, 13 Oct 2005, Christian PELISSIER wrote:
>> We have around 1 million entries in our greylist database. What really
>> bugs me is that the current 2.3 GB of VmData that milter-greylist is
>> using does not show up in physical memory or swap... it will be
>> interresting to see what happens when the process hits 3 GB, since this
>> is the amount of memory + swap that one of the servers has.
>> Could perhaps the kernel (Linux 2.6.9-11.ELsmp, CentOS 4.1) be reporting
>> the memory usage wrong? This problem seems to have started since I began
>> using the mxsync function.
>> Time for me to "Use the Source, Luke".
> Storing the md5 of the tuple instead of the tuple itself could be a
> solution for a large site and an a new option for milter-greylist.
> md5 is only 16 byte long (against  ~ 64)
> md5(Sender IP:Sender e-mail:Recipient e-mail) Time

no need, just use on-disk database rather than in-memory one.
someone really needs to integrate sqlite backend for tuple access.

-Dan

On Thu, Oct 13, 2005 at 02:13:44AM -0400, Matt Kettler wrote:
> This is really only a problem for "normal" sendmail users. ie: those who don't
> relay, have local accounts, and don't mitigate dictionary attacks.

Like me. The easy way out of this (which I implemented) is to
enable greylisting only for existing users (acl greylist rcpt ...)
and have a default whitelist clause (acl whitelist default).
Non-existant recipients will be whitelisted immediately
by milter-greylist, and sendmail will reject them at once.

The "acl greylist rcpt ..." lines for greylist.conf could be
generated automatically, but in fact it's not a problem if
you do this only from time to time, as for new users it 
tends to take some time until they collect big amounts of spam.

mp.

Hi,

>>>>> On Thu, 13 Oct 2005 10:12:33 +0300
>>>>> Fredrik Nyberg DC <nba@...> said:

nba> Matt Kettler wrote:
> Based on that you're talking about 50 million entries. If you've got a 5-day
> timeout that's only 10 million attempts per day. For a site with 8,000 users
> that doesn't seem unreasonable.

nba> We have around 1 million entries in our greylist database. What really 
nba> bugs me is that the current 2.3 GB of VmData that milter-greylist is 
nba> using does not show up in physical memory or swap... it will be 
nba> interresting to see what happens when the process hits 3 GB, since this 
nba> is the amount of memory + swap that one of the servers has.

I found that a pending entry has a redundant p_addr field which holds
string form of an IP address.  Since a pending entry has a p_sa field,
I think it is elective.  Actually, an autowhite entry doesn't have
such field.
The following patch should reduce about 20 bytes per each pending
entry.
This patch also fixes one minor problem.  It shouldn't happen, but the
pending_get() may return NULL on error.  The pending_check() doesn't
check it, and simply passes it to peer_create().

Index: pending.c
diff -u -p pending.c.orig pending.c
--- pending.c.orig	Thu May 12 01:25:43 2005
+++ pending.c	Fri Oct 14 21:54:49 2005
@@ -103,11 +103,15 @@ pending_get(sa, salen, from, rcpt, date)
 	char *rcpt;
 	time_t date;
 {
-	struct pending *pending;
+	struct pending *pending = NULL;
 	struct timeval tv;
 	int delay = conf.c_delay;
 	char addr[IPADDRSTRLEN];
 
+	/* make sure having only valid IP address */
+	if (!iptostring(sa, salen, addr, sizeof(addr)))
+		goto out;
+
 	if ((pending = malloc(sizeof(*pending))) == NULL)
 		goto out;
 
@@ -127,15 +131,7 @@ pending_get(sa, salen, from, rcpt, date)
 	}
 	memcpy(pending->p_sa, sa, salen);
 	pending->p_salen = salen;
-	if (!iptostring(sa, salen, addr, sizeof(addr)) ||
-	    (pending->p_addr = strdup(addr)) == NULL) {
-		free(pending->p_sa);
-		free(pending);
-		pending = NULL;
-		goto out;
-	}
 	if ((pending->p_from = strdup(from)) == NULL) {
-		free(pending->p_addr);
 		free(pending->p_sa);
 		free(pending);
 		pending = NULL;
@@ -143,7 +139,6 @@ pending_get(sa, salen, from, rcpt, date)
 	}
 	if ((pending->p_rcpt = strdup(rcpt)) == NULL) {
 		free(pending->p_from);
-		free(pending->p_addr);
 		free(pending->p_sa);
 		free(pending);
 		pending = NULL;
@@ -160,7 +155,7 @@ pending_get(sa, salen, from, rcpt, date)
 
 	if (conf.c_debug) {
 		syslog(LOG_DEBUG, "created: %s from %s to %s delayed for %lds",
-		    pending->p_addr, pending->p_from, pending->p_rcpt, 
+		    addr, pending->p_from, pending->p_rcpt,
 		    pending->p_tv.tv_sec - tv.tv_sec);
 	}
 out:
@@ -171,9 +166,13 @@ void
 pending_put(pending) /* pending list should be write-locked */
 	struct pending *pending;
 {
+	char addr[IPADDRSTRLEN];
+
 	if (conf.c_debug) {
+		iptostring(pending->p_sa, pending->p_salen, addr,
+		    sizeof(addr));
 		syslog(LOG_DEBUG, "removed: %s from %s to %s",
-		    pending->p_addr, pending->p_from, pending->p_rcpt);
+		    addr, pending->p_from, pending->p_rcpt);
 	}
 
 	TAILQ_REMOVE(&pending_head, pending, p_list);	
@@ -198,8 +197,6 @@ pending_del(sa, salen, from, rcpt, time)
 	struct timeval tv;
 
 	gettimeofday(&tv, NULL);
-	if (!iptostring(sa, salen, addr, sizeof(addr)))
-		return;
 
 	PENDING_WRLOCK;	/* XXX take it as read and upgrade it */
 	for (pending = TAILQ_FIRST(&pending_head); pending; pending = next) {
@@ -208,7 +205,7 @@ pending_del(sa, salen, from, rcpt, time)
 		/*
 		 * Look for our entry.
 		 */
-		if ((strncmp(addr, pending->p_addr, sizeof(addr)) == 0) &&
+		if (ip_equal(sa, pending->p_sa) &&
 		    (strcmp(from, pending->p_from) == 0) &&
 		    (strcmp(rcpt, pending->p_rcpt) == 0) &&
 		    (pending->p_tv.tv_sec == time)) {
@@ -221,10 +218,11 @@ pending_del(sa, salen, from, rcpt, time)
 		 */
 		if (tv.tv_sec - pending->p_tv.tv_sec > conf.c_timeout) {
 			if (conf.c_debug) {
+				iptostring(pending->p_sa, pending->p_salen,
+				    addr, sizeof(addr));
 				syslog(LOG_DEBUG,
 				    "del: %s from %s to %s timed out",
-				    pending->p_addr, pending->p_from,
-				    pending->p_rcpt);
+				    addr, pending->p_from, pending->p_rcpt);
 			}
 
 			pending_put(pending);
@@ -273,10 +271,11 @@ pending_check(sa, salen, from, rcpt, rem
 		 */
 		if (now - accepted > conf.c_timeout) {
 			if (conf.c_debug) {
+				iptostring(pending->p_sa, pending->p_salen,
+				    addr, sizeof(addr));
 				syslog(LOG_DEBUG,
 				    "check: %s from %s to %s timed out",
-				    pending->p_addr, pending->p_from,
-				    pending->p_rcpt);
+				    addr, pending->p_from, pending->p_rcpt);
 			}
 
 			pending_put(pending);
@@ -319,8 +318,8 @@ pending_check(sa, salen, from, rcpt, rem
 	 * It was not found. Create it and propagagte it to peers.
 	 * Error handling is useless here, we will tempfail anyway
 	 */
-	pending = pending_get(sa, salen, from, rcpt, (time_t)0);
-	peer_create(pending);
+	if ((pending = pending_get(sa, salen, from, rcpt, (time_t)0)) != NULL)
+		peer_create(pending);
 	rest = delay;
 	dirty = 1;
 
@@ -349,6 +348,7 @@ pending_textdump(stream)
 	struct pending *pending;
 	int done = 0;
 	char textdate[DATELEN + 1];
+	char textaddr[IPADDRSTRLEN];
 	struct tm tm;
 
 	fprintf(stream, "\n\n#\n# greylisted tuples\n#\n");
@@ -360,10 +360,12 @@ pending_textdump(stream)
 		localtime_r((time_t *)&pending->p_tv.tv_sec, &tm);
 		strftime(textdate, DATELEN, "%Y-%m-%d %T", &tm);
 
-		fprintf(stream, "%s\t%s\t%s\t%ld # %s\n", 
-		    pending->p_addr, pending->p_from, 
-		    pending->p_rcpt, (long)pending->p_tv.tv_sec, textdate);
-		
+		iptostring(pending->p_sa, pending->p_salen, textaddr,
+		    sizeof(textaddr));
+		fprintf(stream, "%s\t%s\t%s\t%ld # %s\n",
+		    textaddr, pending->p_from, pending->p_rcpt,
+		    (long)pending->p_tv.tv_sec, textdate);
+
 		done++;
 	}
 	PENDING_UNLOCK;
@@ -393,7 +395,6 @@ pending_free(pending)
 	}
 	UNLOCK(refcnt_lock);
 	free(pending->p_sa);
-	free(pending->p_addr);
 	free(pending->p_from);
 	free(pending->p_rcpt);
 	free(pending);
Index: pending.h
diff -u pending.h.orig pending.h
--- pending.h.orig	Tue Sep 14 03:41:55 2004
+++ pending.h	Fri Oct 14 19:16:17 2005
@@ -62,7 +62,6 @@
 TAILQ_HEAD(pendinglist, pending);
 
 struct pending {
-	char *p_addr;
 	struct sockaddr *p_sa;
 	socklen_t p_salen;
 	char *p_from;
Index: sync.c
diff -u -p sync.c.orig sync.c
--- sync.c.orig	Sun Jan 23 02:23:17 2005
+++ sync.c	Fri Oct 14 21:49:56 2005
@@ -236,6 +236,7 @@ sync_send(peer, type, pending)	/* peer l
 	char *replystr;
 	int replycode;
 	char line[LINELEN + 1];
+	char addr[IPADDRSTRLEN];
 	char *cookie = NULL;
 
 	if ((peer->p_stream == NULL) && (peer_connect(peer) != 0))
@@ -246,9 +247,10 @@ sync_send(peer, type, pending)	/* peer l
 	else
 		fprintf(peer->p_stream, "del ");
 
+	iptostring(pending->p_sa, pending->p_salen, addr, sizeof(addr));
 	fprintf(peer->p_stream, "addr %s from %s rcpt %s date %ld\r\n", 
-	    pending->p_addr, pending->p_from, 
-	    pending->p_rcpt, (long)pending->p_tv.tv_sec);
+	    addr, pending->p_from, pending->p_rcpt,
+	    (long)pending->p_tv.tv_sec);
 	fflush(peer->p_stream);
 
 	/* 


Sincerely,

--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@...  ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/

On Wed, Oct 12, 2005 at 06:26:54PM -0400, Matt Kettler wrote:
> My sendmail *does* reject nonexistent senders, and always has. re-read my
> message. The rejection happens *AFTER* the greylist runs.

You can change that behavior via "/etc/mail/access". Addresses which
are blacklister there e.g. like this ...

To:foobar@...			error:5.1.1:550 User unknown

... get refused immediately. My system automatically creates an
access list based on the local accounts and alias and merges that
into "/etc/mail/access".

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

Matthias Scheler wrote:
> On Wed, Oct 12, 2005 at 06:26:54PM -0400, Matt Kettler wrote:
> 
>>My sendmail *does* reject nonexistent senders, and always has. re-read my
>>message. The rejection happens *AFTER* the greylist runs.
> 
> 
> You can change that behavior via "/etc/mail/access". Addresses which
> are blacklister there e.g. like this ...
> 
> To:foobar@...			error:5.1.1:550 User unknown
> 
> ... get refused immediately. My system automatically creates an
> access list based on the local accounts and alias and merges that
> into "/etc/mail/access".
> 
> 	Kind regards
> 

Matthias, how does that help against a dictionary attack?

You can't put all possible combinations of username into /etc/mail/access. The
access list would be infinitely large.

On Mon, Oct 17, 2005 at 10:45:35AM -0400, Matt Kettler wrote:
> Matthias, how does that help against a dictionary attack?

Very well.

> You can't put all possible combinations of username into /etc/mail/access.
> The access list would be infinitely large.

You whitelist valid e-mail addresses and blacklist everything else:
From the automatically generated part of my "/etc/mail/access":

To:tron@...			RELAY
To:postmaster@...			RELAY
[...]
To:colwyn.zhadum.de				error:5.1.1:550 User unknown

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

Matthias Scheler wrote:
> On Mon, Oct 17, 2005 at 10:45:35AM -0400, Matt Kettler wrote:
> 
>>Matthias, how does that help against a dictionary attack?
> 
> 
> Very well.
> 
> 
>>You can't put all possible combinations of username into /etc/mail/access.
>>The access list would be infinitely large.
> 
> 
> You whitelist valid e-mail addresses and blacklist everything else:
>>From the automatically generated part of my "/etc/mail/access":
> 
> To:tron@...			RELAY
> To:postmaster@...			RELAY
> [...]
> To:colwyn.zhadum.de				error:5.1.1:550 User unknown

Fair enough. That's workable, although cumbersome.

Do you have any suggestions that would work without having to maintain two
separate userlists? Adding this to /etc/mail/access is more-or-less the same as
using milter-rcptfilter as Emmanuel suggested.

Really, I'd be more interested in suggestions for how to change when
milter-greylist gets called by sendmail, so it happens after recipients are
validated.

On Mon, Oct 17, 2005 at 11:09:16AM -0400, Matt Kettler wrote:
> Do you have any suggestions that would work without having to maintain two
> separate userlists? Adding this to /etc/mail/access is more-or-less the same as
> using milter-rcptfilter as Emmanuel suggested.

You can build the files from a common source...

-- 
Emmanuel Dreyfus
manu@...

On Mon, Oct 17, 2005 at 11:09:16AM -0400, Matt Kettler wrote:
> Do you have any suggestions that would work without having to maintain two
> separate userlists?

I don't maintain the above entries. They get auto generated from the
user and alias database.

> Adding this to /etc/mail/access is more-or-less the same as
> using milter-rcptfilter as Emmanuel suggested.

It causes less overhead e.g. because you don't have to run an additional
milter process.

> Really, I'd be more interested in suggestions for how to change when
> milter-greylist gets called by sendmail, so it happens after recipients are
> validated.

I'm not sure if that's possible without modifying sendmail.

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

Emmanuel Dreyfus wrote:
> On Wed, Oct 12, 2005 at 11:55:31AM -0400, Matt Kettler wrote:
> 
>>4) using the lazyaw option will also greatly reduce the number of entries, at
>>the expense of making it much easier to get past the greylist.
> 
> 
> 5) buy some RAM :o)
> 
> Seriously, 2 GB seems really huge to me. I'd be curious to learn how many
> message per day flow through this machine. The problem may be caused by
> a memory leak (in milter-greylist or in a library)

We have just under 1 000 000 rows in the database.

The problems seem to begin with mxsyncing enabled. The amount of virtual
memory used seems to grow if milter-greylist is _recieveing_ mxsyncs. I 
have one of the servers configured to recieve syncs only and there is no 
milter-traffic to this server (and thus it generates no entries on its 
own). The instance of milter-greylist recieveing syncs allocates more 
and more VmData while the one actually doing the greylisting and sending 
the syncs does not.

Valgrind does not seem to find any leaks.

Please note that I am speaking of VmData, not resident memory. My system 
does not swap, so it seems that the memory is only reserved, and never used.

Any ideas?

Cheers,
Fredrik Nyberg