Milter-greylist

I've heard users describe instances where mail is delayed for about a
day.  This time, I had the time to dig up (and scrub) the logs.

I'm running milter-greylist 3.0 and sendmail 8.13.8 on Debian.

The important part of the log to note is:

Mar 6 16:17:37 milter-greylist from sender to userb delays 00:17:12
Mar 6 16:17:37 sm-mta from sender via relay 123.45.67.89
Mar 7 20:42:18 milter-greylist addr 123.45.67.89 from sender to userb
               autowhitelisted for 504:00:00

sender uses only one mail relay in all transactions.
sender has no problems sending the same message to usera.
userb got sender's message 27 hours after usera got it.
... why?

Thanks,
Adam

Hi, the sending mailserver decides when it will do the next delivery attempt. You have no influence if or when it will retry. The cause may be high load, long

> > I've heard users describe instances where mail is delayed for about a
> > day.

> the sending mailserver decides when it will do the next delivery
> attempt. You have no influence if or when it will retry.
>
> The cause may be high load, long mailqueue, bad configuration or many
> other things

okay, so that seems to allege that the problem isn't with
milter-greylist's configuration ... it is still a problem, and caused by
greylisting itself.  I've had to add a very large number of problematic
mail relays to the whitelisting section of my greylist.conf file.

Example:  gmail doesn't play nice with greylisting (as noted at
http://tinyurl.com/27ey2t ) so the fix is to traverse their SPF and
whitelist every named server (noted at http://tinyurl.com/2znyqh )  ...
other companies aren't as easily marked (lack of SPF plus the fact that I
have to do it on a case-by-case basis).

Is there a public exchange of servers that don't play nice with
greylisting that is larger than the default?

This is a large enough problem that I may stop using the milter, or
perhaps enable it only for hits in DNSRBLs.

Adam Katz wrote:
> I've heard users describe instances where mail is delayed for about a
> day.  This time, I had the time to dig up (and scrub) the logs.
> 
> I'm running milter-greylist 3.0 and sendmail 8.13.8 on Debian.
> 
> The important part of the log to note is:
> 
> Mar 6 16:17:37 milter-greylist from sender to userb delays 00:17:12
> Mar 6 16:17:37 sm-mta from sender via relay 123.45.67.89
> Mar 7 20:42:18 milter-greylist addr 123.45.67.89 from sender to userb
>                autowhitelisted for 504:00:00
> 
> sender uses only one mail relay in all transactions.
> sender has no problems sending the same message to usera.
> userb got sender's message 27 hours after usera got it.
> ... why?
> 

If you're greylisted, Milter-Greylist will accept the email on the next retry 
after the greylist expires, but exactly how long that takes depends on how long 
it takes for the sending server to retry. Milter-greylist can't control how 
often a sending server retries delivering mail.

The RFCs suggest retrying every 15 minutes, but an overloaded server with a lot 
of mail in its queue can take a very long time to retry. Also, most MTAs allow 
you to adjust the retry interval, and some admins reduce the retry frequency in 
order to limit the load and log output from retries..

Check your logs to see if the message delivery was retried between 16:17 and 
20:42.. If it wasn't retried, there's nothing MG can do about it. It can't call 
up and say "deliver your message now", it has to wait for the sending to present 
the message again.

This is actually a fundamental drawback of greylisting. Greylisting deals poorly 
with broken servers that don't retry, which is desirable and why it works. 
However it also excessively delays mail from servers that don't retry promptly 
like they should, which is desirable in some cases, and undesirable in others.

On 03/11/2008 09:45 AM the voices made Matt Kettler write:

> If you're greylisted, Milter-Greylist will accept the email on the next retry 
> after the greylist expires, but exactly how long that takes depends on how long 
> it takes for the sending server to retry. Milter-greylist can't control how 
> often a sending server retries delivering mail.
> 
> The RFCs suggest retrying every 15 minutes, but an overloaded server with a lot 
> of mail in its queue can take a very long time to retry. Also, most MTAs allow 
> you to adjust the retry interval, and some admins reduce the retry frequency in 
> order to limit the load and log output from retries..
> 
> Check your logs to see if the message delivery was retried between 16:17 and 
> 20:42.. If it wasn't retried, there's nothing MG can do about it. It can't call 
> up and say "deliver your message now", it has to wait for the sending to present 
> the message again.
> 
> This is actually a fundamental drawback of greylisting. Greylisting deals poorly 
> with broken servers that don't retry, which is desirable and why it works. 
> However it also excessively delays mail from servers that don't retry promptly 
> like they should, which is desirable in some cases, and undesirable in others.
> 

My experience has been that 15 minutes greylisting is needlessly too
long. Many high volume mailers were resending in as little as 30
seconds. If the mail was again tempfailed by greylisting the sending MTA
moved the mail to a slow queue and it might be 2-4 hours before it was
retried again.

I have my greylisting set for 10 seconds and autowhitelisting for 30
days. The spammers who never try - well, they never retry. It doesn't
matter what the greylisting time is set for. Why needlessly delay the
MTAs that _are_ going to retry?

> I have my greylisting set for 10 seconds and autowhitelisting for 30
> days. The spammers who never try - well, they never retry. It doesn't
> matter what the greylisting time is set for. Why needlessly delay the
> MTAs that _are_ going to retry?

What I'd really like to do is simplify the triplet ...
so instead of "from@... + smtp.sender.com + user@..."
we could have "[from] + smtp.sender.com"
where "from" is only needed if sender.com has 4+ MX records
(or some other simple check for a big company/service).

I don't care about which of my users get the mail because my users are all
aware of each other; the only case in which user@... is of use
is for a large-scale deployment like an ISP or university ... and even
then, I might argue it's not worth the minimal gain, especially given
other enterprise spam-fighting tools.

Another potential feature would be to utilize SPF (though not blindly as
the milter-greylist option does) in the following manner:  if
smtp.sender.com is listed in SPF, add all servers listed by SPF in the
autowhitelisting.  This would solve the issue created by redundant smtp
servers (whose extreme case is exhibited by gmail as noted earlier).

I unfortunately lack the time to help write such a modification :-(

-Adam

> I have my greylisting set for 10 seconds and autowhitelisting for 30
> days. The spammers who never try - well, they never retry. It doesn't
> matter what the greylisting time is set for. Why needlessly delay the
> MTAs that _are_ going to retry?

I completely forgot to reply to the text I cited ...

There are two merits to greylisting:  the required retry and the delay.
You are correct in that a ten second delay is enough to deter those
servers that never re-send.

However, the secondary benefit of greylisting is lost.  A delay ensures
that other servers without the delay have a chance to report the spam to
various databases:  your local bayes database, spamcop and other relay/URL
DNSRBLs, DCC, etc.  After this delay, spamassassin or whatever other tool
you use is more likely to trap spam.  This mostly works because servers
using greylisting are still a minority.

If the delay is only ten seconds, and the sending mail server re-sends in
thirty seconds, the spam is less of a known quantity and therefore less
likely to get flagged as such.


I suppose I could add another milter between greylisting and spamassassin
that merely delays greylisted mail for an extra 20 minutes, or perhaps
get even more complicated and delay spam scoring 3+ points meeting that
criteria for a second scan 20 minutes later ... hmm ...

racl whitelist spf pass
would do what you mentioned.
But the other feature you mentioned - i.e. greylisting to from+smtp_IP -
i.e. the some "light" version of the lazyaw option we have already would
be really cute.

Maybe already on Emmanuel's roadmap? :-)

I agree that even 10sec greylisting delay might be useful, but
definitely NOT with the lazyaw option. With the lazyaw option there is a
risk that the spammer who tries to send junk in a big amount, trying
systematically every recipient in your company will get eventually
autowhitelisted.

I certainly do now want to see autowhitelisted junk mails in my inbox.

Ondrej

Adam Katz wrote:

>
> > I have my greylisting set for 10 seconds and autowhitelisting for 30
> > days. The spammers who never try - well, they never retry. It doesn't
> > matter what the greylisting time is set for. Why needlessly delay the
> > MTAs that _are_ going to retry?
>
> What I'd really like to do is simplify the triplet ...
> so instead of "from@... <mailto:from%40sender.com> +
> smtp.sender.com + user@... <mailto:user%40greylisting.com>"
> we could have "[from] + smtp.sender.com"
> where "from" is only needed if sender.com has 4+ MX records
> (or some other simple check for a big company/service).
>
> I don't care about which of my users get the mail because my users are all
> aware of each other; the only case in which user@...
> <mailto:user%40greylisting.com> is of use
> is for a large-scale deployment like an ISP or university ... and even
> then, I might argue it's not worth the minimal gain, especially given
> other enterprise spam-fighting tools.
>
> Another potential feature would be to utilize SPF (though not blindly as
> the milter-greylist option does) in the following manner: if
> smtp.sender.com is listed in SPF, add all servers listed by SPF in the
> autowhitelisting. This would solve the issue created by redundant smtp
> servers (whose extreme case is exhibited by gmail as noted earlier).
>
> I unfortunately lack the time to help write such a modification :-(
>
> -Adam
>
>

On 03/11/2008 11:04 AM the voices made Ondrej Valousek write:
> racl whitelist spf pass
> would do what you mentioned.
> But the other feature you mentioned - i.e. greylisting to from+smtp_IP -
> i.e. the some "light" version of the lazyaw option we have already would
> be really cute.
> 
> Maybe already on Emmanuel's roadmap? :-)
> 
> I agree that even 10sec greylisting delay might be useful, but
> definitely NOT with the lazyaw option. With the lazyaw option there is a
> risk that the spammer who tries to send junk in a big amount, trying
> systematically every recipient in your company will get eventually
> autowhitelisted.
> 
> I certainly do now want to see autowhitelisted junk mails in my inbox.
> 

Yes, you would not want to use the lazyaw option with a 10 second delay.

Many spammers do retry. They get autowhitelisted. Whitelist from
greylisting, though, not whitelisted from the next layer of spam
control. Greylist alone will not take care of your spam, but it will
knock a whole lot of it out. Especially trojaned windows machines,
viruses and, oddly enough here anyway, almost all of the pornography
spams. Ten seconds works great, but not using the lazyaw option.

Adam Katz a \ufffdcrit :
>>> I've heard users describe instances where mail is delayed for about a
>>> day.
> 
>> the sending mailserver decides when it will do the next delivery
>> attempt. You have no influence if or when it will retry.
>>
>> The cause may be high load, long mailqueue, bad configuration or many
>> other things
> 
> okay, so that seems to allege that the problem isn't with
> milter-greylist's configuration ... it is still a problem, and caused by
> greylisting itself.  I've had to add a very large number of problematic
> mail relays to the whitelisting section of my greylist.conf file.
> 
> Example:  gmail doesn't play nice with greylisting (as noted at
> http://tinyurl.com/27ey2t ) so the fix is to traverse their SPF and
> whitelist every named server (noted at http://tinyurl.com/2znyqh )  ...
> other companies aren't as easily marked (lack of SPF plus the fact that I
> have to do it on a case-by-case basis).
> 

All the major ISP and email providers (such as gmail) use "mail farms" 
to handle outbound SMTP connexions: mails in the output queue may be 
processed (and retried) by whichever server in the farm.

This is *NOT* compatible with greylisting, which expects mails to be 
retried from the same IP address they were initially submitted; symptoms 
are random long greylisting delays.
So workarounds *must* be taken in greylist.conf:
- define "lazyaw" and "subnetmatch" to be more tolerant about incoming 
submissions (sounds quite dirty to me);
- locally whitelist by ACL (IP or domain match) all known server farms 
(a pain to maintain, but quickly responsive in case of problem discovery);
- make use of centralized DNS whitelists such as list.dnswl.org by a 
DNSRBL ACL clause;
- whitelist on behalf of sender's SPF record.


-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.

Adam Katz a \ufffdcrit :
> Another potential feature would be to utilize SPF (though not blindly as
> the milter-greylist option does) in the following manner:  if
> smtp.sender.com is listed in SPF, add all servers listed by SPF in the
> autowhitelisting.  This would solve the issue created by redundant smtp
> servers (whose extreme case is exhibited by gmail as noted earlier).

It's a great idea.
That's exactly what I manually do from time to time, when I discover an 
ISP has changed his config and his new servers are not yet listed in 
dnswl: see if it has an SPF record, and if so take the IP listed and put 
it in my local whitelist.
Of course the better way would be that such ISPs implement SRS, so their 
SPF would always validate, even when forwarding...
And of course it's preferable to whitelist only "trusted" SPFs, ie not 
those who include "+all" and the like: this is what the "spf self" 
clause is meant for.

-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.

> All the major ISP and email providers (such as gmail) use "mail farms"
> to handle outbound SMTP connexions: mails in the output queue may be
> processed (and retried) by whichever server in the farm.

gmail is the only such farm I've encountered to both rotate servers on
sending attempts AND exceed the number of servers to cause a problem.
Now that I've been alerted to lazyaw, this is less of an issue (since
popular sites send mail to/from lots of people, and other "mail farms"
won't be as extensive as gmail, which I've already whitelisted).

Scary "fun fact" - gmail lists almost 140,000 IPs in their SPF record.


> So workarounds *must* be taken in greylist.conf:
> - define "lazyaw" and "subnetmatch" to be more tolerant about incoming
> submissions (sounds quite dirty to me);

I didn't know about lazyaw.  That almost solves my issue, though it breaks
with free mail, universities, ISPs, and other giants (which is of course
why I suggested using the sender address if the domain has 4+ MX records).

To formalize: I propose an optional argument to lazyaw specifying the
number of MX records a domain should have in order to make use of the
sender address (so my earlier proposal would be implemented with "lazyaw
4").  Not specifying a number would make the default very high, depending
on what Emmanuel et al think is a good default high threshold (maybe 5?);
hotmail has only four (each of which maps to three IPs), and other big
players seem to have 5+.  255 should be enough to disable the threshold.

"subnetmatch" would end up whitelisting compromised DHCP subnets (which is
likely common thanks to spammer viruses and thus very "dirty" indeed).
My SPF suggestion should alleviate the need for subnetmatch on domains
that properly use SPF.  I'll call this feature proposal "awbyspf" and
define it as "autowhitelist all of a domain's SPF-specified IPs when one
passes."  See also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445841

> - locally whitelist by ACL (IP or domain match) all known server farms
> (a pain to maintain, but quickly responsive in case of problem discovery);

This is in place, but I again agree that this is a pain to maintain and
not an acceptable long-term strategy.

> - make use of centralized DNS whitelists such as list.dnswl.org by a
> DNSRBL ACL clause;

hrm.  that's an idea (since i've already compiled dnsrbl support in).
http://www.dnswl.org/tech doesn't list configuration help for using dnswl
with milter-greylist.  it appears nontrivial due to their different
response codes and milter-greylist's apparent need to know the response
ahead of time.  (Or perhaps I don't understand the option.)  Advice?

> - whitelist on behalf of sender's SPF record.

I can't see using milter-greylist without the "nospf" option as
responsible.  If you're suggesting manual insertion of known problematic
domains based on the mail servers listed in their SPF records, yes, I do
that manually.  If there were an option for this, that would be nice (it
would be a trivial add-in if my "light lazy" feature is added).


Ondrej Valousek wrote:
>> racl whitelist spf pass

It took me a while to figure out what "racl" was (I'm using
milter-greylist 3.0 since 4.x isn't in debian unstable yet).  It turns out
that "racl" is a synonym for "acl," so your line appears to simply undo
the "nospf" option that I consider a must-have.

My "awbyspf" proposal autowhitelists all SPF records for a domain once one
of them becomes autowhitelisted.  milter-greylisting's built-in SPF
capabilities only implement the first half of that (all SPF servers for
all domains are always whitelisted).

Ondrej Valousek <webserv@...> wrote:

> Maybe already on Emmanuel's roadmap? :-)

Well, I'm not very motivated for working on new features, as I receive
very few spams these days :-)

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

> -----Original Message-----
> > All the major ISP and email providers (such as gmail) use "mail farms"
> > to handle outbound SMTP connexions: mails in the output queue may be
> > processed (and retried) by whichever server in the farm.
> 
> gmail is the only such farm I've encountered to both rotate servers on
> sending attempts AND exceed the number of servers to cause a problem.
> Now that I've been alerted to lazyaw, this is less of an issue (since
> popular sites send mail to/from lots of people, and other "mail farms"
> won't be as extensive as gmail, which I've already whitelisted).
> 
> Scary "fun fact" - gmail lists almost 140,000 IPs in their SPF record.

  2nd "fun fact" - gmail's servers also backscatter, which causes them
to be blacklisted a lot of places.  Just because they're big, and they're
"Google", doesn't mean they should be given a free pass.

Adam Katz <yegsa-yahoo@...> wrote:

> I can't see using milter-greylist without the "nospf" option as
> responsible.  If you're suggesting manual insertion of known problematic
> domains based on the mail servers listed in their SPF records, yes, I do
> that manually.  If there were an option for this, that would be nice (it
> would be a trivial add-in if my "light lazy" feature is added).

You can match the DNSRBL against 127.0.0.0/8, that should do it.

> My "awbyspf" proposal autowhitelists all SPF records for a domain once one
> of them becomes autowhitelisted.  milter-greylisting's built-in SPF
> capabilities only implement the first half of that (all SPF servers for
> all domains are always whitelisted).

"awbyspf" or the shorter "awspf"?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

I am using the "racl whitelist spf pass" and I have not seen any junk
mail whitelisted by SPF so far - so I consider this as a safe enough option.
That's also why I do not agree with the proposed "awbyspf" option.

Big mail farms tend to fight with greylisting, yes, but they also
usually have SPF defined so the clause above will take a care of them
without my intervention.

Ondrej

Adam Katz wrote:

>
> > All the major ISP and email providers (such as gmail) use "mail farms"
> > to handle outbound SMTP connexions: mails in the output queue may be
> > processed (and retried) by whichever server in the farm.
>
> gmail is the only such farm I've encountered to both rotate servers on
> sending attempts AND exceed the number of servers to cause a problem.
> Now that I've been alerted to lazyaw, this is less of an issue (since
> popular sites send mail to/from lots of people, and other "mail farms"
> won't be as extensive as gmail, which I've already whitelisted).
>
> Scary "fun fact" - gmail lists almost 140,000 IPs in their SPF record.
>
> > So workarounds *must* be taken in greylist.conf:
> > - define "lazyaw" and "subnetmatch" to be more tolerant about incoming
> > submissions (sounds quite dirty to me);
>
> I didn't know about lazyaw. That almost solves my issue, though it breaks
> with free mail, universities, ISPs, and other giants (which is of course
> why I suggested using the sender address if the domain has 4+ MX records).
>
> To formalize: I propose an optional argument to lazyaw specifying the
> number of MX records a domain should have in order to make use of the
> sender address (so my earlier proposal would be implemented with "lazyaw
> 4"). Not specifying a number would make the default very high, depending
> on what Emmanuel et al think is a good default high threshold (maybe 5?);
> hotmail has only four (each of which maps to three IPs), and other big
> players seem to have 5+. 255 should be enough to disable the threshold.
>
> "subnetmatch" would end up whitelisting compromised DHCP subnets (which is
> likely common thanks to spammer viruses and thus very "dirty" indeed).
> My SPF suggestion should alleviate the need for subnetmatch on domains
> that properly use SPF. I'll call this feature proposal "awbyspf" and
> define it as "autowhitelist all of a domain's SPF-specified IPs when one
> passes." See also
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445841
> <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445841>
>
> > - locally whitelist by ACL (IP or domain match) all known server farms
> > (a pain to maintain, but quickly responsive in case of problem
> discovery);
>
> This is in place, but I again agree that this is a pain to maintain and
> not an acceptable long-term strategy.
>
> > - make use of centralized DNS whitelists such as list.dnswl.org by a
> > DNSRBL ACL clause;
>
> hrm. that's an idea (since i've already compiled dnsrbl support in).
> http://www.dnswl.org/tech <http://www.dnswl.org/tech> doesn't list
> configuration help for using dnswl
> with milter-greylist. it appears nontrivial due to their different
> response codes and milter-greylist's apparent need to know the response
> ahead of time. (Or perhaps I don't understand the option.) Advice?
>
> > - whitelist on behalf of sender's SPF record.
>
> I can't see using milter-greylist without the "nospf" option as
> responsible. If you're suggesting manual insertion of known problematic
> domains based on the mail servers listed in their SPF records, yes, I do
> that manually. If there were an option for this, that would be nice (it
> would be a trivial add-in if my "light lazy" feature is added).
>
> Ondrej Valousek wrote:
> >> racl whitelist spf pass
>
> It took me a while to figure out what "racl" was (I'm using
> milter-greylist 3.0 since 4.x isn't in debian unstable yet). It turns out
> that "racl" is a synonym for "acl," so your line appears to simply undo
> the "nospf" option that I consider a must-have.
>
> My "awbyspf" proposal autowhitelists all SPF records for a domain once one
> of them becomes autowhitelisted. milter-greylisting's built-in SPF
> capabilities only implement the first half of that (all SPF servers for
> all domains are always whitelisted).
>
>

:-)
Maybe it is not needed anyway - lazyaw is "sexy" enough as it is now.
And if I needed something too much, I might try to implement it on my
own - just a pity the sources are not well commented.
Ondrej

manu@... wrote:

>
> Ondrej Valousek <webserv@... <mailto:webserv%40s3group.cz>> wrote:
>
> > Maybe already on Emmanuel's roadmap? :-)
>
> Well, I'm not very motivated for working on new features, as I receive
> very few spams these days :-)
>
> -- 
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz <http://hcpnet.free.fr/pubz>
> manu@... <mailto:manu%40netbsd.org>
>
>

Hi Ondrej,

> I am using the "racl whitelist spf pass" and I have not seen any junk

I'd like to do the same and implement this, the less admin work I have to do
the better. 

What is required to make this work, just the command above? or do I need some
sort of plugin or module somewhere?

Thanks.

Michael.

> mail whitelisted by SPF so far - so I consider this as a safe enough 
> option. That's also why I do not agree with the proposed "awbyspf" option.
> 
> Big mail farms tend to fight with greylisting, yes, but they also
> usually have SPF defined so the clause above will take a care of them
> without my intervention.
> 
> Ondrej
> 
> Adam Katz wrote:
> >
> > > All the major ISP and email providers (such as gmail) use "mail farms"
> > > to handle outbound SMTP connexions: mails in the output queue may be
> > > processed (and retried) by whichever server in the farm.
> >
> > gmail is the only such farm I've encountered to both rotate servers on
> > sending attempts AND exceed the number of servers to cause a problem.
> > Now that I've been alerted to lazyaw, this is less of an issue (since
> > popular sites send mail to/from lots of people, and other "mail farms"
> > won't be as extensive as gmail, which I've already whitelisted).
> >
> > Scary "fun fact" - gmail lists almost 140,000 IPs in their SPF record.
> >
> > > So workarounds *must* be taken in greylist.conf:
> > > - define "lazyaw" and "subnetmatch" to be more tolerant about incoming
> > > submissions (sounds quite dirty to me);
> >
> > I didn't know about lazyaw. That almost solves my issue, though it breaks
> > with free mail, universities, ISPs, and other giants (which is of course
> > why I suggested using the sender address if the domain has 4+ MX records).
> >
> > To formalize: I propose an optional argument to lazyaw specifying the
> > number of MX records a domain should have in order to make use of the
> > sender address (so my earlier proposal would be implemented with "lazyaw
> > 4"). Not specifying a number would make the default very high, depending
> > on what Emmanuel et al think is a good default high threshold (maybe 5?);
> > hotmail has only four (each of which maps to three IPs), and other big
> > players seem to have 5+. 255 should be enough to disable the threshold.
> >
> > "subnetmatch" would end up whitelisting compromised DHCP subnets (which is
> > likely common thanks to spammer viruses and thus very "dirty" indeed).
> > My SPF suggestion should alleviate the need for subnetmatch on domains
> > that properly use SPF. I'll call this feature proposal "awbyspf" and
> > define it as "autowhitelist all of a domain's SPF-specified IPs when one
> > passes." See also
> > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445841
> > <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445841>
> >
> > > - locally whitelist by ACL (IP or domain match) all known server farms
> > > (a pain to maintain, but quickly responsive in case of problem
> > discovery);
> >
> > This is in place, but I again agree that this is a pain to maintain and
> > not an acceptable long-term strategy.
> >
> > > - make use of centralized DNS whitelists such as list.dnswl.org by a
> > > DNSRBL ACL clause;
> >
> > hrm. that's an idea (since i've already compiled dnsrbl support in).
> > http://www.dnswl.org/tech <http://www.dnswl.org/tech> doesn't list
> > configuration help for using dnswl
> > with milter-greylist. it appears nontrivial due to their different
> > response codes and milter-greylist's apparent need to know the response
> > ahead of time. (Or perhaps I don't understand the option.) Advice?
> >
> > > - whitelist on behalf of sender's SPF record.
> >
> > I can't see using milter-greylist without the "nospf" option as
> > responsible. If you're suggesting manual insertion of known problematic
> > domains based on the mail servers listed in their SPF records, yes, I do
> > that manually. If there were an option for this, that would be nice (it
> > would be a trivial add-in if my "light lazy" feature is added).
> >
> > Ondrej Valousek wrote:
> > >> racl whitelist spf pass
> >
> > It took me a while to figure out what "racl" was (I'm using
> > milter-greylist 3.0 since 4.x isn't in debian unstable yet). It turns out
> > that "racl" is a synonym for "acl," so your line appears to simply undo
> > the "nospf" option that I consider a must-have.
> >
> > My "awbyspf" proposal autowhitelists all SPF records for a domain once one
> > of them becomes autowhitelisted. milter-greylisting's built-in SPF
> > capabilities only implement the first half of that (all SPF servers for
> > all domains are always whitelisted).

> :-)
> Maybe it is not needed anyway - lazyaw is "sexy" enough as it is now.
> And if I needed something too much, I might try to implement it on my
> own - just a pity the sources are not well commented.
> Ondrej

Yes, it is "sexy" but I would like to have even more "sexy".
The lazyaw option is way too global and not generic enough semantically.
My users like to have more flexibility.
I would like to have a dynamic clause for that for example:

racl greylist awoption-match sender,domain rcpt <rcpt0@my.domain>
racl greylist awoption-match domain        rcpt <rcpt1@my.domain>
racl greylist awoption-match sender        rcpt <rcpt2@my.domain>
racl greylist awoption-match none          rcpt <rcpt3@my.domain>

The first line gives for rcpt0 like no lazyaw on for him.
The last line gives for rcpt3 like lazyaw is off for him.
The lines in the middle may give options not possible to do today's version.
(May be the use of one of them is suggested earlier in this thread?)
With this change the existing lazyaw option would give the default behaviour
if no awoption-match clause is defined on an racl line.
This would maintain the backward compatibility of the old config file.
I am not proposing the exact syntax, rather the concept.

Bests,
Attila

Ondrej Valousek wrote:
> I am using the "racl whitelist spf pass" and I have not seen any junk
> mail whitelisted by SPF so far - so I consider this as a safe enough option.
> That's also why I do not agree with the proposed "awbyspf" option.

Milter-greylist's default behavior is to whitelist any server that
passes SPF.  It can be disabled with the "nospf" command.  What does
"racl whitelist spf pass" do that isn't done by default?

As to junk mail that passes SPF, its volume is ever-increasing.  SPF
only protects you against spammers who fraudulently claim to use mail
servers you might recognize.  I receive spam from hotmail servers
daily; all of that passes SPF but is still spam.

Recall that there are two benefits to greylisting:  some spammers
don't re-send, and the delay grants time for the blocklists to receive
reports (so by the time spamassassin processes the mail, it's more
likely to be trapped by URIBL/DNSRBL/DCC and the like).  SPF-passes
are almost guaranteed to also re-send, but the delay is key in
trapping it with blocklists.

"awbyspf" simply clusters mail by a domain's legitimate servers, so as
to circumvent issues like gmail's farm and greylisting the second mail
between two users.

> Big mail farms tend to fight with greylisting, yes, but they also
> usually have SPF defined so the clause above will take a care of them
> without my intervention.

This is the opposite of what has been said so far.  Big mail farms
tend to NOT cater to greylisting.  I believe Yahoo greylists known
spammers, but otherwise I don't know of any big mail providers that
use the technology -- there seem to be more big mail farms that are
incompatible with greylisting than there are that use greylisting.

> Milter-greylist's default behavior is to whitelist any server that
> passes SPF. It can be disabled with the "nospf" command. What does
> "racl whitelist spf pass" do that isn't done by default?
>







"racl whitelist spf pass" is syntax as of version > 4.1 of milter-greylist.
The advantage of this version is:
"racl greylist spf self delay 1h" which handles those "+all" spammers -
that was not possible to do before.

I agree the volume of junk that passes SPF is increasing but it was you
who correctly said, that those who passes the SPF test would likely to
resend anyway -> so in either case, it is no use to greylist them. Same
is also true for those who are willing to use STARTTLS encryption -
those senders use a proper MTA so why should I bother and greylist them?
Ok if you use dnsrbl it make a sense - as you get some time, but
otherwise it is useless

Moreover I still thing the vast majority of spam is from infected
computers - and here has the greylisting something to say.

As of the mail farms - I said the they tend to fight with recipients
that use greylisting - they do not use greylisting themself.

Ondrej

>
>

<attila.bruncsak@...> wrote:

> racl greylist awoption-match sender,domain rcpt <rcpt0@...>
> racl greylist awoption-match domain        rcpt <rcpt1@...>
> racl greylist awoption-match sender        rcpt <rcpt2@...>
> racl greylist awoption-match none          rcpt <rcpt3@...>
> 
> The first line gives for rcpt0 like no lazyaw on for him.
> The last line gives for rcpt3 like lazyaw is off for him.
> The lines in the middle may give options not possible to do today's version.
> (May be the use of one of them is suggested earlier in this thread?)
> With this change the existing lazyaw option would give the default behaviour
> if no awoption-match clause is defined on an racl line.
> This would maintain the backward compatibility of the old config file.
> I am not proposing the exact syntax, rather the concept.

It's true we could improve how autowhitelisting works. The global knob
for lazyaw is not satisfying. 

But as usual, finding the right config syntax is difficult.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Hi,

> > Milter-greylist's default behavior is to whitelist any server that
> > passes SPF. It can be disabled with the "nospf" command. What does
> > "racl whitelist spf pass" do that isn't done by default?
> >
> 
> "racl whitelist spf pass" is syntax as of version > 4.1 of milter-greylist.
> The advantage of this version is:
> "racl greylist spf self delay 1h" which handles those "+all" 
> spammers - that was not possible to do before.

Hmm.. interesting. When will 4.1 be made available?

Michael.

Hi,

> "awbyspf" simply clusters mail by a domain's legitimate servers, so 
> as to circumvent issues like gmail's farm and greylisting the second 
> mail between two users.

Where is this awbyspf option located? I can't find it in the milter-greylist
4.0 man pages?

Thanks.

Michael.

Michael Mansour <mic@...> wrote:

> Where is this awbyspf option located? I can't find it in the milter-greylist
> 4.0 man pages?

It's a suggestion for future developement.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Michael Mansour <mic@...> wrote:

> > "racl whitelist spf pass" is syntax as of version > 4.1 of milter-greylist.
> > The advantage of this version is:
> > "racl greylist spf self delay 1h" which handles those "+all" 
> > spammers - that was not possible to do before.
> 
> Hmm.. interesting. When will 4.1 be made available?

spf clauses in ACL has been available since 3.1.4. You have in in 4.0.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...