Milter-greylist

Hi

I user per-recipient settings, where each user can choose the level of
tiltering: nothing, greylisting for 15 minutes, using DNSRBL, and so on.

For now, users ask for various filtering levels, and the milter-greylist
configuration file is modified to satisfy their requests. 

It would be better to pull the per-recipient info from a centralized 
database that users could modify on their own: it would save admin
time.

LDAP seems a good candidate for the database.

Are there some experienced LDAP users here? How would you fit the
filtering level attribute in the database? How do you imagine the 
config file syntax to specify the use of the LDAP directory?

I imagine something like this:
ldap "ldap1" "ldaps://ldap.example.net" "mail=%s,dc=example,dc=net" \
    "cn=user,dc=example,dc=net" "greylist_level" "level 1" \
    "/etc/mail/greylist.crt" "/etc/mail/greylist.key"

acl greylist rcpt_ldap "ldap1" greylist 15m
acl whitelist default

With ldap statement arguments being:
	"ldap1" name of the config referenced later in the file
	"ldaps://ldap.example.net" URI of the server
	"mail=%s,dc=example,dc=net" filter, %s is replaced by rcpt e-mail.
	"cn=user,dc=example,dc=net" name used for connecting to the server
	"greylist_level" the attribute to look up
	"level 1" the value we look for a match
	"/etc/mail/greylist.crt" certificate (optional)
	"/etc/mail/greylist.key" key (optionnal)

But that looks a bit overkill (well, I assume it can't be otherwise with
LDAP)...

Or are there other good ways of handling the issue?

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus wrote:
> I user per-recipient settings, where each user can choose the level of
> tiltering: nothing, greylisting for 15 minutes, using DNSRBL, and so on.

(...)

> LDAP seems a good candidate for the database.

> I imagine something like this:

(...)

It would be much "easier" then to be able to be able to specify the
whole filter and suffix, with "%r" "%d" and such variables to extract
domain, realm etc...

Rene Luria <operator@...> wrote:

> It would be much "easier" then to be able to be able to specify the
> whole filter and suffix, with "%r" "%d" and such variables to extract
> domain, realm etc...

Can you describe how you immagine the syntax in more details?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Le mar 21/11/2006 à 17:50, Emmanuel Dreyfus a écrit :
> Hi
> 
> I user per-recipient settings, where each user can choose the level of
> tiltering: nothing, greylisting for 15 minutes, using DNSRBL, and so
> on.
> 
> For now, users ask for various filtering levels, and the
> milter-greylist
> configuration file is modified to satisfy their requests. 
> 
> It would be better to pull the per-recipient info from a centralized 
> database that users could modify on their own: it would save admin
> time.
> 
> LDAP seems a good candidate for the database.
> 
> Are there some experienced LDAP users here? How would you fit the
> filtering level attribute in the database? How do you imagine the 
> config file syntax to specify the use of the LDAP directory?
> 
> I imagine something like this:
> ldap "ldap1" "ldaps://ldap.example.net" "mail=%s,dc=example,dc=net" \
> "cn=user,dc=example,dc=net" "greylist_level" "level 1" \
> "/etc/mail/greylist.crt" "/etc/mail/greylist.key"
> 
> acl greylist rcpt_ldap "ldap1" greylist 15m
> acl whitelist default
> 
> With ldap statement arguments being:
> "ldap1" name of the config referenced later in the file
> "ldaps://ldap.example.net" URI of the server
> "mail=%s,dc=example,dc=net" filter, %s is replaced by rcpt e-mail.
> "cn=user,dc=example,dc=net" name used for connecting to the server
> "greylist_level" the attribute to look up
> "level 1" the value we look for a match
> "/etc/mail/greylist.crt" certificate (optional)
> "/etc/mail/greylist.key" key (optionnal)
> 
> But that looks a bit overkill (well, I assume it can't be otherwise
> with
> LDAP)...

If you choose LDAP you have to write your own LDAP schema adding milter
greylist required fields. Then you have to write a GUI allowing LDAP
users to modify the milter-greylist fields. You must define LDAP ACLs
that allow that. It's probably a good solution for a big site having an
LDAP server but it needs users to have a GUI.


> 
> Or are there other good ways of handling the issue?
>
The simple way to allow users to interact with some milter-greylist
parameters is  a web interface (form/cgi or php) running on the mail
server to add/delete their own ACL in a separate conf file (eg
greyusracls.conf). You also have to authentify them and the best for
that is to have an LDAP server .... So LDAP seems to be the right choice
but it's heavy for admin and milter-greylist.

> 
> -- 
> Emmanuel Dreyfus
> manu@...
> 
> 
> 
>  
-- 
Christian Pélissier
Office National d'Études et de Recherches Aérospatiales
BP 72 92322 Chatillon
Tel: 33 1 46 73 44 19, Fax: 33 1 46 73 41 50

manu@... wrote:
>> It would be much "easier" then to be able to be able to specify the
>> whole filter and suffix, with "%r" "%d" and such variables to extract
>> domain, realm etc...

> Can you describe how you immagine the syntax in more details?

This is what I have in some other imap software:

# Filter for user lookup. Some variables can be used:
#   %u - username
#   %n - user part in user@domain, same as %u if there's no domain
#   %d - domain part in user@domain, empty if user there's no domain

then something like:

user_filter = (whatever filter you want, replacing search attributes by
one of those "variables")

> If you choose LDAP you have to write your own LDAP schema adding milter
> greylist required fields. Then you have to write a GUI allowing LDAP
> users to modify the milter-greylist fields. You must define LDAP ACLs
> that allow that. It's probably a good solution for a big site having an
> LDAP server but it needs users to have a GUI.
> 
> 
> > 
> > Or are there other good ways of handling the issue?
> >
> The simple way to allow users to interact with some milter-greylist
> parameters is  a web interface (form/cgi or php) running on the mail
> server to add/delete their own ACL in a separate conf file (eg
> greyusracls.conf). You also have to authentify them and the best for
> that is to have an LDAP server .... So LDAP seems to be the right choice
> but it's heavy for admin and milter-greylist.

Having per-user conf file is an error prone architecture. In general,
file I/O is evil. In particular, milter-greylist need to parse conf
files of all users everytime just an user changes his/her preference.

For a LDAP-running site it will be easy to introduce a greylisting
preference in its existing LDAP database. For a SASL or UNIX passwd
site a monolithic file like /etc/passwd may be more attractive.

AIDA Shinra <shinra@...> wrote:

> For a LDAP-running site it will be easy to introduce a greylisting
> preference in its existing LDAP database. For a SASL or UNIX passwd
> site a monolithic file like /etc/passwd may be more attractive.

Where will you store the information in /etc/passwd?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

At Wed, 22 Nov 2006 19:17:02 +0100,
manu@... wrote:
> > For a LDAP-running site it will be easy to introduce a greylisting
> > preference in its existing LDAP database. For a SASL or UNIX passwd
> > site a monolithic file like /etc/passwd may be more attractive.
> Where will you store the information in /etc/passwd?

Sorry, I meant a monolithic file which is similar to /etc/passwd
(or /etc/aliases).

* It can be parsed by only fgets() and strtok().
* It can be edited by a machine without breaking comment lines.
* No macro nor ifdef nor wildcard.

AIDA Shinra <shinra@...> wrote:

> Sorry, I meant a monolithic file which is similar to /etc/passwd
> (or /etc/aliases).
> 
> * It can be parsed by only fgets() and strtok().
> * It can be edited by a machine without breaking comment lines.
> * No macro nor ifdef nor wildcard.

Well, that does half of the work, since we still lack a way of having
users modifying it. The point with using a directory is that there are
tools already available for that. Moreover, using a local file means
sync issues if you have multiple MX.

An alternative: assume the site already has some web machinery as a
front end for its user database, and that a CGI can give the
information. That would give something like that:

rcpturl "level 1" "http://www.foo.net/gr.php?rcpt=%r" "level1"
rcpturl "level 2" "http://www.foo.net/gr.php?rcpt=%r" "level2"

acl greylist rcpturl "level 1" greylist 15m
acl greylist rcpturl "level 2" greylist 1h 

So when we have a mail for bar@..., we query
http://www.foo.net/gr.php?rcpt=bar@... and depending on the string
we get back, we can apply different ACL. 

That's much more simple than the LDAP stuff, but one must have a web
middleware to handle it. I'm not sure that's a better path.

We can also immagine querying the DNS, a la Hesiod.

I'm not sure of the best way to go.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

AIDA Shinra wrote:
 > manu@... wrote:
 > > Where will you store the information in /etc/passwd?
 > 
 > Sorry, I meant a monolithic file which is similar to /etc/passwd
 > (or /etc/aliases).
 > 
 > * It can be parsed by only fgets() and strtok().
 > * It can be edited by a machine without breaking comment lines.
 > * No macro nor ifdef nor wildcard.

Such a plain file is terribly inefficient and has bad
performance for any significant number of users.

I would recommend to use a real database (e.g. PostgreSQL).
It has several advantages:
 - Good performance (with proper indices).
 - No problem syncing between MXs, they simply access
   the same database (or a replicated one: most databases
   already support replication themselves).
 - There already exist solutions for users to edit their
   own data in SQL tables.

Best regards
   Oliver


-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

$ dd if=/dev/urandom of=test.pl count=1
$ file test.pl
test.pl: perl script text executable

On Thu, 2006-11-23 at 09:13 +0100, manu@... wrote:
> 
*snip*
> 
> An alternative: assume the site already has some web machinery as a
> front end for its user database, and that a CGI can give the
> information. That would give something like that:
> 
> rcpturl "level 1" "http://www.foo.net/gr.php?rcpt=%r" "level1"
> rcpturl "level 2" "http://www.foo.net/gr.php?rcpt=%r" "level2"
> 
> acl greylist rcpturl "level 1" greylist 15m
> acl greylist rcpturl "level 2" greylist 1h 
> 
> So when we have a mail for bar@..., we query
> http://www.foo.net/gr.php?rcpt=bar@... and depending on the string
> we get back, we can apply different ACL. 
> 
> That's much more simple than the LDAP stuff, but one must have a web
> middleware to handle it. I'm not sure that's a better path.
> 

I'd vote for HTTP approach as it gives you a single interface to
virtually unlimited types of data sources behind it.

I'd also include in rcpturl the "%s" for the sender part, so as to allow
the user to specify e-mail addresses or wildcards for which it does not
want to have the greylisting in place.

R.

On Thu, Nov 23, 2006 at 09:13:36AM +0100, manu@... wrote:
> rcpturl "level 1" "http://www.foo.net/gr.php?rcpt=%r" "level1"
> rcpturl "level 2" "http://www.foo.net/gr.php?rcpt=%r" "level2"
> 
> acl greylist rcpturl "level 1" greylist 15m
> acl greylist rcpturl "level 2" greylist 1h 

While I think about it: delegating the decision to a web middleware
does not have to be done only on the recipient address. We could immagine
adding various information to the quesry such as sender IP, sender e-mail
and so on. 

If we introduce racl (evaluated at RCPT stage) and dacl 
(evaluated at DATA stage), we can even imagine sending the message body 
using a POST request.

Such a feature seems to allow much more flexibility than just a recipient
lookup in a LDAP directory, a SQL database, or whatever. 

The syntax would look like that:
urlfilter "level 1" "http://www.foo.net/gr.php?rcpt=%r+ipaddr=%a" "level1"
urlfilter "level 2" "http://www.foo.net/gr.php?rcpt=%r+ipaddr=%a" "level2"

acl greylist urlfilter "level 1" greylist 15m
acl greylist urlfilter "level 2" greylist 1h

In the urlfilter statement, we could use:
%r recipient e-mail
%f sender e-mail
%a sender IP
%d sender DNS name
%mfoo sendmail macro foo's value

Opinions?

-- 
Emmanuel Dreyfus
manu@...

On Thu, 2006-11-23 at 10:01 +0000, Emmanuel Dreyfus wrote:
> On Thu, Nov 23, 2006 at 09:13:36AM +0100, manu@... wrote:
> > rcpturl "level 1" "http://www.foo.net/gr.php?rcpt=%r" "level1"
> > rcpturl "level 2" "http://www.foo.net/gr.php?rcpt=%r" "level2"
> > 
> > acl greylist rcpturl "level 1" greylist 15m
> > acl greylist rcpturl "level 2" greylist 1h 
> 
> While I think about it: delegating the decision to a web middleware
> does not have to be done only on the recipient address. We could immagine
> adding various information to the quesry such as sender IP, sender e-mail
> and so on. 
> 
> If we introduce racl (evaluated at RCPT stage) and dacl 
> (evaluated at DATA stage), we can even imagine sending the message body 
> using a POST request.
> 
> Such a feature seems to allow much more flexibility than just a recipient
> lookup in a LDAP directory, a SQL database, or whatever. 
> 
> The syntax would look like that:
> urlfilter "level 1" "http://www.foo.net/gr.php?rcpt=%r+ipaddr=%a" "level1"
> urlfilter "level 2" "http://www.foo.net/gr.php?rcpt=%r+ipaddr=%a" "level2"
> 
> acl greylist urlfilter "level 1" greylist 15m
> acl greylist urlfilter "level 2" greylist 1h
> 
> In the urlfilter statement, we could use:
> %r recipient e-mail
> %f sender e-mail
> %a sender IP
> %d sender DNS name
> %mfoo sendmail macro foo's value
> 
> Opinions?

I'm 100% with you on this, however I would do the config in a different
manner here.

First define a pair of URL references:
urlquery-rcpt <url>
urlquery-data <url>

i.e.:
urlquery-rcpt "http://www.foo.net/rq.php?rcpt=%r+ipaddr=%a+from=%f+ipdns=%s"
urlquery-data "http://www.foo.net/dq.php?rcpt=%r+ipaddr=%a+from=%f+ipdns=%s"

Define query timeouts:
urlquery-timeout <seconds>

And - define the possible responses as follows:
urlquery-response (<custom_response>|DEFAULT) <well_defined_action>

Where:
        <custom_response> - is a string value to be mapped to a well
        defined
        		action or the special case DEFAULT to be used in
        		case of the comm failure with the HTTP server or an
        		undecipherable response
        
        <well_defined_action> - is one of the following:
                greylist <time>	 -      final decision, cached in
                pending,
                			distributed via mxsync
                
                whitelist <time> -      final decision, cached in awl,
                			distributed via mxsync
                
                reject		 - 	final decision
                
                next		 -      allows milter to use other means like
                			SPF and DNSBL to make it's decision
                
                ...

I.e:
urlquery-response "grey1" greylist 15m
urlquery-response "grey2" greylist 30m
urlquery-response "white" whitelist 5d
urlquery-response "no opinion" next
urlquery-response DEFAULT next


The internal search order for the decision within the milter-greylist
would be:

     1. awl cache
     2. pending cache
     3. http query (if defined)
     4. the "old" behavior if the "next" is the action

What do you think?

R.

Ranko Zivojnovic <ranko@...> wrote:

> What do you think?

Well, I'd prefer the thing to be an ACL clause, as it would allow mixing
it with other clauses.

Other opinions from other people?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

At Thu, 23 Nov 2006 09:35:01 +0100 (CET),
Oliver Fromme wrote:
> > > Where will you store the information in /etc/passwd?
> >
> > Sorry, I meant a monolithic file which is similar to /etc/passwd
> > (or /etc/aliases)
> 
> .
> >
> > * It can be parsed by only fgets() and strtok().
> > * It can be edited by a machine without breaking comment lines.
> > * No macro nor ifdef nor wildcard.
> Such a plain file is terribly inefficient and has bad
> performance for any significant number of users.
> I would recommend to use a real database (e.g. PostgreSQL).
> It has several advantages:
> - Good performance (with proper indices).
> - No problem syncing between MXs, they simply access
> the same database (or a replicated one: most databases
> already support replication themselves).
> - There already exist solutions for users to edit their
> own data in SQL tables.

Plain file is simple solution for small sites. Imagine a site which
has 20 users and every user has a shell account. In this case LDAP
will be overkilling unless the site is already running LDAP for
authentication purpose. Such a site can provide a setuid perl script
to let users modify greylisting preferences. MXes will fetch the
preference file from the master server every 5 minutes. Simple enough.

Shared SQL server is not a good choice because if the SQL server
failed then all MXes stop working. Replication is necessary unless
multiple MXes are managed in the same network for load balancing
purpose.

In contrast RDBMS is worth considering as an alternative of LDAP.

RDBMS advantages:
* Some sites are already using SQL servers as their backends.
* Better supported in scripting languages.

RDBMS disadvantages and LDAP advantages:
* LDAP is designed as a standard mechanism to keep per-user
information.
* I guess more sites are running LDAP than RDBMS.
* There exist many SQL implementations and their APIs are different
each other. Do we have to implement many bindings for the same
purpose?

At Thu, 23 Nov 2006 10:01:56 +0000,
Emmanuel Dreyfus wrote:
> 
> > rcpturl "level 1" "[[http://www.foo.net/gr.php?rcpt=%r]]" "level1"
> > rcpturl "level 2" "[[http://www.foo.net/gr.php?rcpt=%r]]" "level2"
> >
> > acl greylist rcpturl "level 1" greylist 15m
> > acl greylist rcpturl "level 2" greylist 1h
> While I think about it: delegating the decision to a web middleware
> does not have to be done only on the recipient address. We could immagine
> adding various information to the quesry such as sender IP, sender e-mail
> and so on.
> If we introduce racl (evaluated at RCPT stage) and dacl
> (evaluated at DATA stage), we can even imagine sending the message body
> using a POST request.
> Such a feature seems to allow much more flexibility than just a recipient
> lookup in a LDAP directory, a SQL database, or whatever.
> The syntax would look like that:
> urlfilter "level 1" "[[http://www.foo.net/gr.php?rcpt=%r+ipaddr=%a]]" "level1"
> urlfilter "level 2" "[[http://www.foo.net/gr.php?rcpt=%r+ipaddr=%a]]" "level2"
> acl greylist urlfilter "level 1" greylist 15m
> acl greylist urlfilter "level 2" greylist 1h
> In the urlfilter statement, we could use:
> %r recipient e-mail
> %f sender e-mail
> %a sender IP
> %d sender DNS name
> %mfoo sendmail macro foo's value
> Opinions?

Programmers will prefer hacking milter-greylist itself to implementing
such a glue mechanism. Though PHP programming will be easier than C,
programmers still need to take much care of security and error
handling.

The more declarative, the easier. Plugin mechanism is not for simple
programs such as milter-greylist. LDAP filter is enough flexible for
most cases.