Milter-greylist

Last try:
  I posted a message about this in the past.  I'm sorry for the repetition.
  My support of the sendmail systems on my current network is ending and 
I am most likely not going to have an occasion to update the patches for 
poprelayd support that I've been producing for my systems.  Poprelayd 
performs smtp authentication via pop/imap.  It is a perl script that 
parses logs and populates a berkeleydb.  It's a bit of a hack, but it is 
what we have always used on our systems and because it's perl, it is 
incredibly flexible.

  I've included the patch for the current release of milter-greylist 
which loads a berkeley db and adds an "authenticated" flag to any 
message who's IP is listed in the defined database.  The patch includes 
the appropriate configure file edit which check for a "--with-poprelayd" 
and "--with-berkeleydb" and unless these are called the patched code is 
excluded via "#ifdef" commands.
  The database can only be loaded via a command line argument to 
milter-greylist (-b <db-path>).  I was hoping to add a conf file 
interface for the database, but never did.
  Unfortunately this patch is really geared towards poprelayd in that it 
only reads from the database, but it is a single berkeleydb call which 
would open the db for writing and creation.  The data included in the IP 
is not parsed by my patch, but it is available, so it would be possible 
to call the function elsewhere to implement a berkeleydb backend for the 
greylist.db file.  That may speed things up and improve memory usage.
  If I find the time this is something I would like to do in the future, 
but if someone else wants to beat me to the punch then by all means have 
at it.  this patch is a start at any rate.

  I know the department that I'm currently in would really appreciate 
having this patch included in the milter-greylist source as I will no 
longer be here to implement the new versions.  I don't know how widely 
poprelayd is used any more, so I'm not sure it will be worth it, but my 
patch could have other uses.  It is a really easy way to whitelist IPs 
in milter greylist from an external process.
  Thanks for everything.  I know I will be implementing milter-greylist 
on future sendmail systems, so I'll be back...
  -Lawren Quigley-Jones

Hi,

>>>>> On Fri, 16 Dec 2005 16:08:38 -0500
>>>>> Lawren Quigley-Jones <lawrenqj@...> said:

lawrenqj> +	key.data = (char *)ipAddr;      /* key.data is the string we want to search for */
lawrenqj> +	key.size = strlen(ipAddr);      /* in this case size is the number of character in the string */
lawrenqj> +
lawrenqj> +	if ( key.size > 20 )
lawrenqj> +	{
lawrenqj> +		syslog(LOG_DEBUG, "The IP requested for was %d character long...  Possible stack attack!!!", key.size);
lawrenqj> +		key.size = 20;
lawrenqj> +	}

It seems you allow not only an IPv4 address but also an IPv6 as an
argument to popIPCheck().  So, key.size larger than 20 is still valid
length for an IP address.
I don't know if poprelayd supports an IPv6.  If poprelayd doesn't
support an IPv6, you should prevent popIPCheck() from getting an IPv6
address like DRAC support did.

Sincerely,

--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@...  ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/

Hajimu UMEMOTO <ume@...> wrote:

> It seems you allow not only an IPv4 address but also an IPv6 as an
> argument to popIPCheck().  So, key.size larger than 20 is still valid
> length for an IP address.
> I don't know if poprelayd supports an IPv6.  If poprelayd doesn't
> support an IPv6, you should prevent popIPCheck() from getting an IPv6
> address like DRAC support did.

I saw no updated patch addressing that problem. Did I missed it?

-- 
Emmanuel Dreyfus
Un bouquin en français sur BSD:
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...

Lets try that again... ... Lets try that again... On 1/8/06, Lawren Quigley-Jones  wrote: Thanks for the reminder. I just moved so that

Hi,

>>>>> On Sun, 8 Jan 2006 13:12:22 +0000
>>>>> Lawren Quigley-Jones <lawrenqj@...> said:

lawrenqj>     I couldn't find anything in the poprelayd documentation about ipv6, but
lawrenqj>     there didn't seem to be anything in the perl code that would prohibit it. 
lawrenqj>     I don't see the harm in asking the db for ipv6 values regardless.

Yes, perhaps it is not vulnerability issue.  But, your previous code
didn't do right thing against an IPv6 address, and issued wrong
warning message.

lawrenqj>     I'm not that familiar with ipv6.  I gathered from a quick google search
lawrenqj>     that the max string length is 240, so I set my keylenth check to 255.  is
lawrenqj>     this safe?

The max. length of an IPv6 address is 46 bytes (including trailing \0)
without regarding of a string form of a scope-id.

In anyway, while you think to need just checking a length of an IP
address without regarding an address family, the following check is
useless, and it should be go away, IMHO.

	if ( key.size > 255 )
	{
		syslog(LOG_DEBUG, "The IP requested for was %d character long...  Possible stack attack!!!", key.size);
		key.size = 255;
	}

Sincerely,

--
Hajimu UMEMOTO @ Internet Mutual Aid Society Yokohama, Japan
ume@...  ume@{,jp.}FreeBSD.org
http://www.imasy.org/~ume/

key length check is now gone... Thanks for the help...

Lawren Quigley-Jones <lawrenqj@...> wrote:

> key length check is now gone...

Lasts bits before I include your code:
1) style. I'm not a style maniac, but mixing different styles makes the
code less readable. Here are a few rules in milter-greylist sources:
- no lines longer thant 80 chars
- multiline comments are like this:
/*
 * foo bar buz
 * foo bar buz
 */
- K&R style function declarations with the function name at the
beginning of a line (helpful for grep ^function_name)
int 
popIPCheck(ipAddr)
char *ipAddr 
{

2) command line options: [-b popipdb]
There are already too may options. Could you add a config file directive
instead? If you can't I'll do it?

3) Documentation
We need documentation for the new feature. Of course the documentation
differs if we add a config line option instead of a command line flag...

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

I made the format changes. If you could do the lex and yak changes I would appreciate it.  I figured out one of those when I first wrote the patch for 1.6 but

On Wed, Jan 11, 2006 at 10:56:30AM -0500, Lawren Quigley-Jones wrote:
> You should decide the config line.  I was thinking
>        acl newdb <filename>
> or    acl bdb <filename>
> 
> It's not actually handled like an acl in milter-greylist, but it acts as one
> during implementation.

I haven't looked at the code. Can we write something like this?
acl bdb "/foo/bar" rcpt manu@... 

-- 
Emmanuel Dreyfus
manu@...

I haven t given it a whole lot of thought, but the problem with the rcpt list is that poprelayd whitelists the entire IP not just for a given username.  I

Lawren Quigley-Jones <lawrenqj@...> wrote:

> I haven't given it a whole lot of thought, but the problem with the rcpt
> list is that poprelayd whitelists the entire IP not just for a given
> username.  I think it would get complicated. 

Well the idea to merge poprelayd with ACL is to tell milter-greylist to
check poprelayd only for some users (or IP, or whatever the ACL allows)

For now we have an addr keyword to match an IP against the ACL argument:
acl whitelist addr 10.0.0.8/24 rcpt toto@...

We could have lines like this:
acl whitelist addr bdb "/foo/bar.db" rcpt toto@toto.com 

This would tell milter-greylist to whitelist if the sender IP address is
in DB file "/foo/bar" and recipient is toto@...

While we are there, we could also have flat files:
acl whitelist addr file "/foo/bar.txt" rcpt toto@...

And we could do it for other keywords:
acl whitelist rcpt file "recipients.txt"

And we could also have support for checking against a DNSRBL:
acl greylist addr dns "dnsrbl.example.net"

Of course in order to get decent performances, we should build a list of
external config sources (bdb, file...) when parsing the ACL, and check
addr, domain, rcpt and from against all the sources before evaluating
the ACL.

That's a rather intrusive change. I'm ok to work on it, but I'd like
feedback of other users for the design.

-- 
Emmanuel Dreyfus
Un bouquin en français sur BSD:
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@netbsd.org

This is interesting, but is it really doable?

My understanding (limited as it is), is that greylisting works so  
well, is because it keeps the whitelist in memory. Once you introduce  
db's or external files, that memory footprint could explode.

Then what about when those files get updated? or someone introduces a  
concept of 'on the fly' whitelisting (outside of greylist)... ie  
poprelayd or non-standard smtp-auth.

It just feels like these changes could impede performance.

Bill

On Jan 11, 2006, at 12:55 PM, manu@... wrote:

> Lawren Quigley-Jones <lawrenqj@...> wrote:
>
>> I haven't given it a whole lot of thought, but the problem with  
>> the rcpt
>> list is that poprelayd whitelists the entire IP not just for a given
>> username.  I think it would get complicated.
>
> Well the idea to merge poprelayd with ACL is to tell milter- 
> greylist to
> check poprelayd only for some users (or IP, or whatever the ACL  
> allows)
>
> This would tell milter-greylist to whitelist if the sender IP  
> address is
> in DB file "/foo/bar" and recipient is toto@...
>
> While we are there, we could also have flat files:
> acl whitelist addr file "/foo/bar.txt" rcpt toto@...
>
> And we could do it for other keywords:
> acl whitelist rcpt file "recipients.txt"
>
> And we could also have support for checking against a DNSRBL:
> acl greylist addr dns "dnsrbl.example.net"
>
> Of course in order to get decent performances, we should build a  
> list of
> external config sources (bdb, file...) when parsing the ACL, and check
> addr, domain, rcpt and from against all the sources before evaluating
> the ACL.
>
> That's a rather intrusive change. I'm ok to work on it, but I'd like
> feedback of other users for the design.
>
> -- 
> Emmanuel Dreyfus
> Un bouquin en français sur BSD:
> http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
> manu@...
>
>
>
> Yahoo! Groups Links
>
>
>
>
>
>
>

On Wed, Jan 11, 2006 at 01:18:49PM -0800, Bill Levering wrote:
> This is interesting, but is it really doable?
> 
> My understanding (limited as it is), is that greylisting works so  
> well, is because it keeps the whitelist in memory. Once you introduce  
> db's or external files, that memory footprint could explode.

My proposition was only for the access list, not for the greylisting
or autowhitelisting database.

> Then what about when those files get updated? or someone introduces a  
> concept of 'on the fly' whitelisting (outside of greylist)... ie  
> poprelayd or non-standard smtp-auth.
> 
> It just feels like these changes could impede performance.

Of course they would, but you do not have to use the feature. If you
don't use it you don't have a performance tradeoff.

-- 
Emmanuel Dreyfus
manu@...

Well the db is read each time milter-greylist is run.  This is so that poprelayd can update it real time.  Any feature we add to this is going to increase the

On Wed, 11 Jan 2006, manu@... wrote:

> Well the idea to merge poprelayd with ACL is to tell milter-greylist to
> check poprelayd only for some users (or IP, or whatever the ACL allows)
>
> For now we have an addr keyword to match an IP against the ACL argument:
> acl whitelist addr 10.0.0.8/24 rcpt toto@...
>
> We could have lines like this:
> acl whitelist addr bdb "/foo/bar.db" rcpt toto@...
>
> This would tell milter-greylist to whitelist if the sender IP address is
> in DB file "/foo/bar" and recipient is toto@...
>
> While we are there, we could also have flat files:
> acl whitelist addr file "/foo/bar.txt" rcpt toto@...
>
> And we could do it for other keywords:
> acl whitelist rcpt file "recipients.txt"
>
> And we could also have support for checking against a DNSRBL:
> acl greylist addr dns "dnsrbl.example.net"

Good idea. It's a first step to:
acl whitelist addr sql ... :-)

Suggestion: macro (or something else) to make this kind of constructions 
possible:
acl whitelist from bdb "${rcpt}.db"
so, we can check in only one line, if someone from address is in whitelist 
database for specific recipient.

> Of course in order to get decent performances, we should build a list of
> external config sources (bdb, file...) when parsing the ACL, and check
> addr, domain, rcpt and from against all the sources before evaluating
> the ACL.

hmmm...
This will make all things slow (in my opinion). I think that better will 
be fast partial evaluation of acl (is it already done?) and runtime 
checking.
I mean:
acl whitelist addr file "/foo/bar.txt" rcpt toto@...
This will check addr in file and if match occurs, then check recipient.
But for speed-up of evaluation of acl we can write:
acl whitelist rcpt toto@... addr file "/foo/bar.txt"
So, for addresses other than toto@... we don't have to check 
this external file.

And to avoid multiple query to one external database, there should be 
implemented some kind of cache...


This is my 3 cents...
I'm using milter-greylist for about half a year in production server and 
generating config for every client with his own whitelist isn't what I
like...
But, this despite of this - very good job!


Regards

Marek Wodzinski
-- 
"If you want something done...do yourself!"
Jean-Baptiste Emmanuel Zorg

Marek Wodzinski <majek@...> wrote:

> > Of course in order to get decent performances, we should build a list of
> > external config sources (bdb, file...) when parsing the ACL, and check
> > addr, domain, rcpt and from against all the sources before evaluating
> > the ACL.
> 
> hmmm...
> This will make all things slow (in my opinion). I think that better will
> be fast partial evaluation of acl (is it already done?) and runtime 
> checking.
> I mean:
> acl whitelist addr file "/foo/bar.txt" rcpt toto@...
> This will check addr in file and if match occurs, then check recipient.
> But for speed-up of evaluation of acl we can write:
> acl whitelist rcpt toto@toto.com addr file "/foo/bar.txt"
> So, for addresses other than toto@toto.com we don't have to check 
> this external file.

Well, that makes the logic more complicated. And indeed it requires a
cache so that the same external database is not queried again until the
end of ACL evaluation.

-- 
Emmanuel Dreyfus
Un bouquin en français sur BSD:
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...