Milter-greylist

Here are the next planned features for the CVS version. Feel free to
comment:

1) SMTP DATA stage ACL

We now have (in CVS) dacl rules, for ACL evaluated after the DATA stage.
Former acl rules have been renamed racl (for RCPT stage ACL). The acl
keyword still works for backward compatibility. 

DATA stage ACL cannot perform the greylist action, nor they can filter
on recipient.

Handling a rcpt clause in DATA stage ACL is not easy, because there can
be multiple recipients. And if we tempfail or reject at that stage, it's
done for all recipients at once: no way to reject for one recipient and
accept for another one.

How should such a rcpt clause work?  
dacl blacklist rcpt foo@...     
I guess it should match if foo@... is one of the recipients.

Is there any need for specifying a set of recipients? And how should
that be interpreted? Match if this is the exact set of recipient? Match
if it is a subset?

Performing the greylist action at DATA stage would mean inserting as
many tuples in the greylist database as we have recipients. I am not
convinced it would be a useful feature at all (feedback welcome here)


2) content filtering in DATA stage ACL
I think about a body clause:
dacl blacklist body "Subject: whatever" msg "I don't like your whatever"
As usual it would also accept regexps.

We could also check lengths: number of lines (lines keyword), number of
bytes (length keyword):
dacl blacklist lines 4000- msg "no more than 4000 lines"

Proposed syntax for lines and length values:
2000            exactly 2000 
2000-4000       between 2000 and 4000
4000-2000       Undefined behavior
2000-           More than 2000
-4000           Less than 4000


3) Time based filtering
We could specify time intervals and use them in ACL. I think about a
crontab(5)-like format specification: 
time "office" * 8-18 * 1,2,3,4,5

racl greylist time "office" delay 15m
racl greylist delay 1h

The time specification parsing will not be easy :-)


4) SPF, AUTH and STARTTLS integration in ACL
Example will tell more than a login explanation
racl whitelist spf
racl whitelist auth
racl whitelist starttls

I wonder if this is of some use, or if it's overkill: 
racl whitelist auth "login"
racl whitelist tls "cn=login,dc=example,dc=net"
 

5) delayedreject keyword in RCPT stage ACL
That would tell milter-greylist to await the DATA stage before
rejecting, just like we do for mail from <>
racl greylist rcpt "foo" delayedreject

Problem with that feature: what about multiple recipients? If one
recipient is whitelisted, and a second one is greylisted with
delayedreject, what should happen?

  
And now, the call for volunteers: Writing good docs takes time. Help
would really be appreciated here. As we add flexibility to the ACL, the
man page got more and more confusing. Anyone would like to work on
greylist.conf(5) and README?
 
-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

> 4) SPF, AUTH and STARTTLS integration in ACL
> Example will tell more than a login explanation
> racl whitelist spf

I'm not sure if this has already been dealt with, but the mechanisms
in 3.0 for dealing with mail farms by whitelisting them, be it based
on IP or SPF, seems a misfeature to me. There's no guarantee that just
because mail comes from a farm that it is not spam, and in fact
there's a good chance that an SPF-compliant source is not well-behaved...

http://www.theregister.co.uk/2004/09/03/email_authentication_spam/

I'm already seeing spam come through the milter because it's SPF
compliant or because it's from a whitelisted mail farm IP.

Anyway, the right solution, I think, is something closer to the CIDR
mask idea; equivalence classes of IPs and a change in the code that
looks up IPs in the greylist so that the comparison is on this equivalence
instead of identity.

All that's needed is an extra bit of syntax in the config file like

equivalent { ... }

where "..." is a list of IPs with optional masks. For each such list
the conf loader generates a unique ID to stand for all the addresses in
the list, and the ACL code can do its stuff based on this ID instead of
the IP.

It'd be easy to use a global assocative array for this with the address
being the key and the class ID being the value. You can then sanity check
the equivalence classes by making sure no address is duplicated
(masked addresses might need to be enumerated).

Has something like this been done or planned?

Cheers,

	- Joel

> > 4) SPF, AUTH and STARTTLS integration in ACL
> > Example will tell more than a login explanation
> > racl whitelist spf
> 
> I'm not sure if this has already been dealt with, but the mechanisms
> in 3.0 for dealing with mail farms by whitelisting them, be it based
> on IP or SPF, seems a misfeature to me.

Sorry, I should have been making this point only with respect to SPF,
so forget about the conf file syntax stuff I mentioned.

I forgot about this obviously correct observation in the README:

  Another workaround is simply to whitelist the netblocks allocated to
  mail farms. As any machine in theses IP address ranges are real SMTP
  servers that will always resend their messages, there is no point in
  greylisting them.

Anyway, I still think SPF needs to be used to define an equivalence class.

Cheers,

	- Joel

Joel Reicher <joel@...> wrote:

> > racl whitelist spf
> 
> I'm not sure if this has already been dealt with, but the mechanisms
> in 3.0 for dealing with mail farms by whitelisting them, be it based
> on IP or SPF, seems a misfeature to me. There's no guarantee that just
> because mail comes from a farm that it is not spam, and in fact
> there's a good chance that an SPF-compliant source is not well-behaved...

The point about having SPF as an ACL clause is that you can use it with
other clauses. You could decide to trust SPF for mail coming from a set
of IP netblocks, for instance.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

> Joel Reicher <joel@...> wrote:
> 
> > > racl whitelist spf
> > 
> > I'm not sure if this has already been dealt with, but the mechanisms
> > in 3.0 for dealing with mail farms by whitelisting them, be it based
> > on IP or SPF, seems a misfeature to me. There's no guarantee that just
> > because mail comes from a farm that it is not spam, and in fact
> > there's a good chance that an SPF-compliant source is not well-behaved...
> 
> The point about having SPF as an ACL clause is that you can use it with
> other clauses. You could decide to trust SPF for mail coming from a set
> of IP netblocks, for instance.

How would that be different from the current situation? I want to use SPF
to deal with mail farms so that I don't have to enter them all by hand, but
I don't want just to whitelist SPF-compliant senders because some of them
are spammers.

In other words I still want to greylist SPF-compliant senders, but use
the SPF record to deal automatically with potential mail farms.

As far as I can tell, using the SPF record as if it were a "subnetmatch"
for just those IPs is the only correct solution.

Thanks,

	- Joel

Joel Reicher <joel@...> wrote:

> > The point about having SPF as an ACL clause is that you can use it with
> > other clauses. You could decide to trust SPF for mail coming from a set
> > of IP netblocks, for instance.
> 
> How would that be different from the current situation?

Today, SPF whitelisting is global and apply to any message. If we
include it in the ACL framework, you can use it in conjunction with
other criterions, such as IP netblocks.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

> Joel Reicher <joel@...> wrote:
> 
> > > The point about having SPF as an ACL clause is that you can use it with
> > > other clauses. You could decide to trust SPF for mail coming from a set
> > > of IP netblocks, for instance.
> > 
> > How would that be different from the current situation?
> 
> Today, SPF whitelisting is global and apply to any message. If we
> include it in the ACL framework, you can use it in conjunction with
> other criterions, such as IP netblocks.

Sorry, I must be missing something, because I really don't see the point
of that. Why use SPF if you're going to enter the IP netblock by hand
anyway?

To my mind the strength of SPF is that it *can* be applied globally, for
some purposes. One of those purposes is mail farms, since the SPF record
might define the required IP block. If this block is then *greylisted* in an
appropriate way (equivalence class), you don't have to whitelist it by hand,
and we can get out of this situation of trying to find every mail farm out
there.

Cheers,

	- Joel

Joel Reicher <joel@...> wrote:

> One of those purposes is mail farms, since the SPF record
> might define the required IP block. 

So you'd want this, to whitelist example.net's mail farm:

acl whitelist from /@example\.net$/ spf

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

> Joel Reicher <joel@...> wrote:
> 
> > One of those purposes is mail farms, since the SPF record
> > might define the required IP block. 
> 
> So you'd want this, to whitelist example.net's mail farm:
> 
> acl whitelist from /@example\.net$/ spf

OK, at least that addresses my concern for known mail farms, but the
larger part of my concern is for *unknown* mail farms. When mail first
arrives from them, it can be greylisted a very, very long time if the
maillog and greylist.db trawling isn't done often enough. I was hoping
making the greylisting facility SPF-aware would solve this.

Thanks,

	- Joel

Joel Reicher <joel@...> wrote:

> OK, at least that addresses my concern for known mail farms, but the
> larger part of my concern is for *unknown* mail farms. When mail first
> arrives from them, it can be greylisted a very, very long time if the
> maillog and greylist.db trawling isn't done often enough. I was hoping
> making the greylisting facility SPF-aware would solve this.

If I understand correctly your idea, you want to do this:

if (spf) 
        greylist (*, from, rcpt)
else
        greylist (addr, from, rcpt)

What happens if a spammert sends from a botnet with from addresses in a
domain that has a ?all SPF record (ie: any host may send mail from the
domain)? 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

According to manu@...:
> 
> We could also check lengths: number of lines (lines keyword), number of
> bytes (length keyword):
> dacl blacklist lines 4000- msg "no more than 4000 lines"
> 
> Proposed syntax for lines and length values:
> 2000            exactly 2000 
> 2000-4000       between 2000 and 4000
> 4000-2000       Undefined behavior

issue a warning (syslog) at compile time then swap and use like
the previous one.

> 2000-           More than 2000
> -4000           Less than 4000

not sure that this "lines" command is useful though as people often use
confMAX_MESSAGE_SIZE directly in sendmail.mc when the goal is to reject
huge emails. When it comes to spam, it's not that big (the biggest being those
with animated GIFs, maybe 60k top) and impossible to differenciate from
legit emails.

BTW, in case some wonders, I use fuzzyocr for those spam pics:
http://fuzzyocr.own-hero.net/

/Fabien

> Joel Reicher <joel@...> wrote:
> 
> > OK, at least that addresses my concern for known mail farms, but the
> > larger part of my concern is for *unknown* mail farms. When mail first
> > arrives from them, it can be greylisted a very, very long time if the
> > maillog and greylist.db trawling isn't done often enough. I was hoping
> > making the greylisting facility SPF-aware would solve this.
> 
> If I understand correctly your idea, you want to do this:
> 
> if (spf) 
>         greylist (*, from, rcpt)
> else
>         greylist (addr, from, rcpt)

No, not quite. What I have in mind is the effect of the following (but
I wouldn't actually implement it this way)

if (spf)
	foreach (i in spfRecord)
		if(i is pass-qualified)
			greylist (i, from, rcpt)
else
	greylist (addr, from, rcpt)

The intent is to treat the SPF record as a "subnetmatch".

> What happens if a spammert sends from a botnet with from addresses in a
> domain that has a ?all SPF record (ie: any host may send mail from the
> domain)? 

Firstly, I think any administrator that took SPF seriously and used it
for a mail farm wouldn't qualify the farm machines as neutral.

But you could ask the same question about +all in the record, and I see
two ways of going on that. One is to treat it as a special case and
never greylist it.

On the other hand, what would be the harm in treating it the same
way as any other part of the record, and let the other two fields of the
greylisted tuple do their job? I see no problem with opening the system up
to a spammer who might resend from a different IP later, since that's no
more likely then resending from the same IP, in which case the spam gets
past greylisting anyway.

Cheers,

	- Joel

Fabien Tassin <fta+miltergreylist@...> wrote:

> > 2000-           More than 2000
> > -4000           Less than 4000
> 
> not sure that this "lines" command is useful though as people often use
> confMAX_MESSAGE_SIZE directly in sendmail.mc when the goal is to reject
> huge emails.

It can be interesting to mix that with other conditions in the ACL:
accept big mail from some senders or to some addresses.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Fabien Tassin <fta+miltergreylist@...> wrote:

> BTW, in case some wonders, I use fuzzyocr for those spam pics:
> http://fuzzyocr.own-hero.net/

I'd like to use an OCR tool, but I'm not tempted by spamassassin. My
idea would be to have a stand-alone milter that scans the GIF and attach
the scanned text as an additionnal MIME section. Then grep in it using
milter-regex, or directly with milter-greylist once I'll have
implemented content searching (It's on the way).

The big part is the milter that perform the OCR. Anyone is aware of such
an existing tool, or will I have to write it on my own?
-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

manu@... wrote:
> Here are the next planned features for the CVS version. Feel free to
> comment:
>
> 1) SMTP DATA stage ACL
>
>   
One of the most powerfull features of milter-greylist is that it
processes the mail before the data stage, saving a ton of bandwidth and
resources. Remember that in some areas, all the ISPs charge by the MB.

If you decide to do this, at least have a ./configure --nodata

On the server I take care of, I need to have a resorse light version
because its allready 400Megs deep in swap and I can't add more without
slowing it down even more.(1) And I'me sure i'me not the only one that
wants this, imaged the ones that processed a million mails per week/day.
They would have to add more servers just to handle the data stage filter.

There are plenty of data stage milters out there. Like milter-clamav,
milter-spamassassan, milter-mimedefang, and a search would get even more
data stage filters.

Perhaps a sujustion of seperating this into a milter-datafilter? There
can be more then on milter, I have three on my server.

> 4) SPF, AUTH and STARTTLS integration in ACL
> Example will tell more than a login explanation
> racl whitelist spf
> racl whitelist auth
> racl whitelist starttls
>
> I wonder if this is of some use, or if it's overkill: 
> racl whitelist auth "login"
> racl whitelist tls "cn=login,dc=example,dc=net"
>  
>
>   
I like to have spf move from global to acl rules. spl for spl pass,
splfail for spl fail and splnone for server that don't have spl setup.

An example

dnsrbl "spamhaus" zen.spamhaus.org 127.0.0.2
acl greylist dnsrbl "spamhaus" delay 24h autowhite 3d
acl greylist splpass delay 5m
acl greylist default delay 30m

The above would allow the use of spl to lower or whitelist spf server but the spammers that use it will get greylisted or blocked by the rbl acl before the acl spl check.


Sorry for the spelling, thunderbird spellcheck sucks and won't let me
compose a repley in plain text.

--Techwolf

1) Plans are in the works to tranfer to a larger and better server, but
the migration is going to take quite a while to upgrade it all from
Centos 3.x to later version or gentoo, still desiding and is going to
test things first to see what is better.

Techwolf <techwolf@...> wrote:
  
> One of the most powerfull features of milter-greylist is that it
> processes the mail before the data stage, saving a ton of bandwidth and
> resources. Remember that in some areas, all the ISPs charge by the MB.
> 
> If you decide to do this, at least have a ./configure --nodata

I'm aware of that. There are two safeguards:

1) If you don't use header or body clauses in ACL, neither postmsg in an
urlcheck definition, milter-greylist will not register header and body
callbacks, and will just not see message content (just as it did
before).

2) The maxpeek option can be used to tell milter-greylist to limit how
much of the message milter-greylist will look at. e.g.: You can tell it
to ignore anything beyond the 2048 bytes. 
   
> I like to have spf move from global to acl rules. spl for spl pass,
> splfail for spl fail and splnone for server that don't have spl setup.

Yes, we can do that.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

--- In milter-greylist@yahoogroups.com, manu@... wrote:
>
> Joel Reicher <joel@...> wrote:
> 
> > OK, at least that addresses my concern for known mail farms, but the
> > larger part of my concern is for *unknown* mail farms. When mail first
> > arrives from them, it can be greylisted a very, very long time if the
> > maillog and greylist.db trawling isn't done often enough. I was hoping
> > making the greylisting facility SPF-aware would solve this.
> 
> If I understand correctly your idea, you want to do this:
> 
> if (spf) 
>         greylist (*, from, rcpt)
> else
>         greylist (addr, from, rcpt)
> 
> What happens if a spammert sends from a botnet with from addresses in a
> domain that has a ?all SPF record (ie: any host may send mail from the
> domain)? 
> 
> 

I beleive the answer is in two parts.

When a mail is recieved from a mail farm that has spl set up, instead of
using just that one ip number for the tuples in the db, use the spl ip
number list in the tuples so that when the same mail is resend from the
different ip but from the same mailfarm, greylist will treat it as the
same mail.

What about the spammers that use 0.0.0.0/0 ccdr(?), use a acl dnsrpl
check to get them before the acl spf check.

--Techwolf