Milter-greylist

1) This first idea was privately e-mailed to Emmanuel Dreyfus in the last week, but he did 
not seem to see its merit. I feel it might at least be a global option option that defaults to 
off, and those like myself, would be able to turn it on.

The idea that any SMTP transaction that fails to call EOM (which probably means calls 
abort), cause milter close call to flush all records in either the gray-list or the auto-white-
list for that relay IP address.

This can occur for several reasons: another previously called milter returned a temporary 
or permanent reject, Sendmail (sorry for the 5% Postfix users, although it may be identical) 
rejected the e-mail or the TCP/IP connection failed. With modern broadband, TCP/IP 
failure is _very_ rare; so basically a spam and/or virus milter rejected the e-mail, or 
Sendmail rejected it most likely because of a bogus account. I feel, and have good 
evidence to support, that this heuristic works well and reduces the size of the lists, 
especially the greylist.

Both Emmanuel and myself have this patch and I think having more people trying it out 
would be good, since it is a heuristic and may well cause bad effects over time - only 
experimentation will prove or disprove its merit. It is not a patch that will destabilize the 
milter; rather it may increase or not delays and/or may decrease greylist and auto whitelist 
sizes.

2) The second proposal, which I have not shown to Emmanuel yet, is an extension on his 
idea for pruning envelop user names for mailing lists. I have found machine-generated 
user names, often with many more digits than letters, occur when certain automated e-
mail's are sent by non-mailing list daemons. These can clutter the auto-white-list and 
never will match another e-mail.

My proposal, which is still in the works, is to identify these envelop sender user names and 
completely replace them by generic text, e.g. AUTO_GENERATED_USER. This results in a 
whitelist entry that _may_ match and is not something that would let span through. I am 
still trying to refine my heuristic here. I have nearly ever non-spam e-mail sent to me for 
the last 8 years and I need to do data reduction to refine the heuristic. As I said before, a 
long name with lots of digits versus letters seems to be the key.


I hope others might be interested in these ideas and I welcome constructive (whether 
negative or positive) feedback.

Tom Hedges <tom@...> wrote:

> The idea that any SMTP transaction that fails to call EOM (which probably
> means calls abort), cause milter close call to flush all records in either
> the gray-list or the auto-white- list for that relay IP address.

My problem with that is that if sendmail tempfails because it got
overloaded, the message will not be in the greylist. When the message
will get back later, it will have to get greylisted again.

> 2) The second proposal, which I have not shown to Emmanuel yet, is an
> extension on his idea for pruning envelop user names for mailing lists. I
> have found machine-generated user names, often with many more digits than
> letters, occur when certain automated e- mail's are sent by non-mailing
> list daemons. These can clutter the auto-white-list and never will match
> another e-mail.
> 
> My proposal, which is still in the works, is to identify these envelop
> sender user names and completely replace them by generic text, e.g.
> AUTO_GENERATED_USER. This results in a whitelist entry that _may_ match
> and is not something that would let span through. I am still trying to
> refine my heuristic here. I have nearly ever non-spam e-mail sent to me
> for the last 8 years and I need to do data reduction to refine the
> heuristic. As I said before, a long name with lots of digits versus
> letters seems to be the key.

Perhaps we can thing about a more generic scheme. For instance a way to
override a typle before it gets inserted in the greylist or whitelist
database.

acl greylist default from /@mailfarm\.example\.net/  \
        greytuple "%i","AUTO_GENERATED_USER","%r"

And there would be a awtuple clause to override the autowhite tuple

That way it's possible to implement your idea, and many others tricks.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

--- In milter-greylist@yahoogroups.com, manu@... wrote:
> My problem with that is that if sendmail tempfails because it got
> overloaded, the message will not be in the greylist. When the message
> will get back later, it will have to get greylisted again.

My understanding of Sendmail is that the main process, running as root, vforks a child that 
setuid's to "smmtp" for each incoming SMTP connection, there is a limit on how many 
children are forked and an overload condition results in the incoming port 25 connection 
failing. Nothing in this chain of events results in a child Sendmail process failing with an 
"overload" abort. This might only happen if the configuration files do not match the network, 
CPU and memory resources of the mail server. I would not think a milter author should worry 
about this lack of knowledge by the person configuering the mail server.

... There is an option define(`confREFUSE_LA , `18 )dnl in the mc config sendmail which will reject Mails if the load is higher than 18 ... M.Menge

--- In milter-greylist@yahoogroups.com, Michael Menge <Michael.Menge@...> wrote:
> There is an option
> define(`confREFUSE_LA', `18')dnl
> in the mc config sendmail which will reject Mails if the
> load is higher than 18

You are correct, in addition the maximum number of user smmsp child processes may be 
limited as well as maximum number of vfork'ed child processes per second may be limited.

In other words, the "Sendmail overload" possibility has been well dealt with in Sendmail 
8.13.x, now may we consider my proposal on its merits, be they bad or good!

Tom Hedges <tom@...> wrote:

> In other words, the "Sendmail overload" possibility has been well dealt
> with in Sendmail 8.13.x, now may we consider my proposal on its merits, be
> they bad or good!

I'm not really hostile, but I'd like comments from other users. Any
input, anyone?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

wrote on Thu, 1 Feb 2007 18:59:19 +0100:

> I'm not really hostile, but I'd like comments from other users. Any
> input, anyone?

I think you can't find out the merits if you don't implement it. Off by 
default. And identify it in logging, so one can actually see when it 
happens.

Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

wrote on Thu, 1 Feb 2007 08:27:40 +0100:

> Perhaps we can thing about a more generic scheme. For instance a way to
> override a typle before it gets inserted in the greylist or whitelist
> database.

If the idea behind this is to reduce the database size why not go to 
2-tuple saving only if lazyaw is set? Someone who wants to use this 
AUTO_GENERATED_USER approach might just want to use lazyaw (as I do on all 
machines).

Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

--- In milter-greylist@yahoogroups.com, Kai Schaetzl <maillists@...> wrote:
> If the idea behind this is to reduce the database size why not go to 
> 2-tuple saving only if lazyaw is set? Someone who wants to use this 
> AUTO_GENERATED_USER approach might just want to use lazyaw (as I do on all 
> machines).

I find only 1% to 2% of greylist and whitelist entries match these criteria, thus subjecting the 
other 98% to it is a major algorithm change; although I never propose rejecting an idea 
without trying it "in the wild".

For many spammers, the enveloper sender distinguishes different machines using a common 
relay. Although most spam-bots use SMTP directly to the "victim", quite a few ISP's block port 
25 outbound and a thus there is benefit in matching the envelop sender.

--- In milter-greylist@yahoogroups.com, manu@... wrote:
> I'm not really hostile, but I'd like comments from other users. Any
> input, anyone?

I'd like to pursue this a little more. From what you and another has posted
a reasonable implementation would be as follows:

An optional global (non-acl) conf argument, that defaults to off, i.e. current
behavior. Optionally, purge whitelists and perhaps -- as a parameter --
greylists.

To go over the rational again for this behavior; if, for example, spamassassin
is running via another milter (or clamAV), then the greylist delay may be the
only reason spamassassin "hit" the e-mail and bounced due to very high spam
scores. If auto-whitelisted, the same spammer may well get through with
later spams. Purging the entries enforces a new delay.

The objection that the mailer may temp fail under overload is possible, but current 
Sendmail has several throttling parameters to prevent this, thus only a small minority of 
sites are at risk of this; obviously, they should leave it this option disabled. Further, since
milter-greylist keeps everything in memory, high volume SMTP sites already have an issue 
with excessive memory use by the milter.