Milter-greylist

First, I know this is a beta, so I understand if there's a bug in this 
release. However I would like to at least make sure I'm not doing something 
wrong or foolish.

Scenario: I'm in a situation where I can't blanket greylist all email in 
the network, because of the delay. This is a business, and we generaly work 
better when our email runs reasonably fast. I greylist my own account, and 
I've had some sites take several hours to retry despite my using a 1 minute 
greylist delay.

I'm also hesitant to use much in the way of RBLs, due to the occasional 
false listing.

So, what I'd like to do is use milter-greylist as a form of "soft" RBL, 
where only mail from certain domain names gets subjected to greylisting.

I wrote some regexes to try to only greylist end-user nodes of certain 
networks. For example this one:

acl greylist domain /pcp.*\...\.comcast\.net/

This targets the naming convention comcast uses for cablemodem subscriber 
PC's which are along the format of:
	pcp00########pcs.howard01.md.comcast.net

I use all this with a "acl whitelist default" at the end of my 
greylist.conf, and I've made sure that I have no old-syntax statements in 
my greylist.conf.

Here's a quick grep for lines that do not contain "acl", have at least one 
character on them, and removing those that are comments:
	$grep -v "acl" greylist.conf |grep . |grep -v ^\#
	greylist 1m
	autowhite 5d
	nospf

So there's no old pre-acl syntax in the file.

Unfortunately, when I add the domain ACL to my greylist, ALL email starts 
getting greylisted unless previously whitelisted. Even hosts that have no 
resemblance to the regex like:

Mar  9 15:52:08 xanadu milter-greylist: j29Kq82X00xxx: addr 64.4.56.40 from 
<xxxxxx@...> to <xxxxxxxx@...> delayed for 00:01:00

	$host 64.4.56.40
	40.56.4.64.in-addr.arpa domain name pointer bay101-f30.bay101.hotmail.com.


Disabling the line and restarting milter-greylist clears the problem back up.

Mar  9 15:53:29 xanadu milter-greylist: j29KrT2X022xxx: skipping greylist 
because this is the default action, 
(from=<xxxxx@...>,rcpt=<xxxxx@...>, addr=64.4.56.40)


What's going on? Is there some bug where domain lookups aren't working right?

Matt Kettler <mkettler@...> wrote:

> What's going on? Is there some bug where domain lookups aren't working right?

milter-greylist doesn't do any DNS lookup (except for SPF). The resolved
peer name is handed by sendmail.

Let's debug that. In the beginning of acl.c:acl_filter(), add a 
syslog(LOG_DEBUG, "hostname = \"%s\"", hostname);

Then in the block of if (acl->a_domain_re != NULL), you can add syslog
statements to check the code path: does the regex matches or not?  

Tell me the result.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Matt Kettler <mkettler@...> wrote:

> Unfortunately, when I add the domain ACL to my greylist, ALL email starts
> getting greylisted unless previously whitelisted.

You tried running in ACL debug mode? It's the -l flag
-- 
Emmanuel Dreyfus
Le cahier de l'admin BSD 2eme ed. est dans toutes les bonnes librairies
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...

At 04:56 PM 3/9/2005, manu@... wrote:

>Matt Kettler <mkettler@...> wrote:
>
> > What's going on? Is there some bug where domain lookups aren't working 
> right?
>
>milter-greylist doesn't do any DNS lookup (except for SPF). The resolved
>peer name is handed by sendmail.
>
>Let's debug that. In the beginning of acl.c:acl_filter(), add a
>syslog(LOG_DEBUG, "hostname = \"%s\"", hostname);
>
>Then in the block of if (acl->a_domain_re != NULL), you can add syslog
>statements to check the code path: does the regex matches or not?

This is really ODD.. I added a debug to every case in the switch...

Mar  9 17:22:13 xanadu milter-greylist: acl_filter hostname = 
"[222.62.149.123]"
Mar  9 17:22:13 xanadu milter-greylist: j29MM62X026578: addr 222.62.149.123 
from <xxxxx@...> to <xxxx@...> delayed for
00:01:00

None of the rules are matching, but it's being greylisted...

Here's my diff for acl.c so you can check what I added:

$ diff -u acl.c.orig acl.c
--- acl.c.orig  2005-03-09 17:11:32.000000000 -0500
+++ acl.c       2005-03-09 17:21:09.000000000 -0500
@@ -430,6 +430,8 @@

         ACL_RDLOCK;

+       syslog(LOG_DEBUG, "acl_filter hostname = \"%s\"", hostname);
+
         match = 0;
         retval = 0;
         TAILQ_FOREACH(acl, &acl_head, a_list) {
@@ -438,7 +440,10 @@

                 if (acl->a_addr != NULL) {
                         if (ip_match(sa, acl->a_addr, acl->a_mask))
+                       {
+                               syslog(LOG_DEBUG, "acl_filter match by 
ip_match");
                                 retval |= EXF_ADDR;
+                       }
                         else  {
                                 match = 0;
                                 continue;
@@ -447,7 +452,10 @@
                 if (acl->a_domain != NULL) {
                         /* Use emailcmp even if it's not an e-mail */
                         if (emailcmp(hostname, acl->a_domain) == 0)
+                       {
+                               syslog(LOG_DEBUG, "acl_filter match by 
hostname emailcmp");
                                 retval |= EXF_DOMAIN;
+                       }
                         else {
                                 match = 0;
                                 continue;
@@ -456,7 +464,10 @@
                 if (acl->a_domain_re != NULL) {
                         if (regexec(acl->a_domain_re,
                             hostname, 0, NULL, 0) == 0)
+                           {
+                               syslog(LOG_DEBUG, "acl_filter match by 
hostname regex");
                                 retval |= EXF_DOMAIN;
+                           }
                         else {
                                 match = 0;
                                 continue;
@@ -465,6 +476,7 @@
                 if (acl->a_from != NULL) {
                         if (emailcmp(from, acl->a_from) == 0) {
                                 retval |= EXF_FROM;
+                               syslog(LOG_DEBUG, "acl_filter match by from 
address emailcmp");
                         } else {
                                 match = 0;
                                 continue;
@@ -472,6 +484,7 @@
                 }
                 if (acl->a_from_re != NULL) {
                         if (regexec(acl->a_from_re, from, 0, NULL, 0) == 0) {
+                               syslog(LOG_DEBUG, "acl_filter match by from 
address regex");
                                 retval |= EXF_FROM;
                         } else {
                                 match = 0;
@@ -480,6 +493,7 @@
                 }
                 if (acl->a_rcpt != NULL) {
                         if (emailcmp(rcpt, acl->a_rcpt) == 0) {
+                               syslog(LOG_DEBUG, "acl_filter match by rcpt 
address emailcmp");
                                 retval |= EXF_RCPT;
                         } else {
                                 match = 0;
@@ -488,6 +502,7 @@
                 }
                 if (acl->a_rcpt_re != NULL) {
                         if (regexec(acl->a_rcpt_re, rcpt, 0, NULL, 0) == 0) {
+                               syslog(LOG_DEBUG, "acl_filter match by 
rcmpt address regex");
                                 retval |= EXF_RCPT;
                         } else {
                                 match = 0;

rMatt Kettler <mkettler@...> wrote:

> This is really ODD.. I added a debug to every case in the switch...
> 
> Mar  9 17:22:13 xanadu milter-greylist: acl_filter hostname = 
> "[222.62.149.123]"
> Mar  9 17:22:13 xanadu milter-greylist: j29MM62X026578: addr 222.62.149.123
> from <xxxxx@blswe.com> to <xxxx@...> delayed for
> 00:01:00
> 
> None of the rules are matching,

That's no surprise, as the hostname given by sendmail is not resolved.

> but it's being greylisted...

This means the default is broken. Any info with -l?

-- 
Emmanuel Dreyfus
Publicité subliminale: achetez ce livre!
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...

At 05:39 PM 3/9/2005, manu@... wrote:
> > This is really ODD.. I added a debug to every case in the switch...
> >
> > Mar  9 17:22:13 xanadu milter-greylist: acl_filter hostname =
> > "[222.62.149.123]"
> > Mar  9 17:22:13 xanadu milter-greylist: j29MM62X026578: addr 222.62.149.123
> > from <xxxxx@...> to <xxxx@...> delayed for
> > 00:01:00
> >
> > None of the rules are matching,
>
>That's no surprise, as the hostname given by sendmail is not resolved.

Perhaps I gave a bad example with that one... That example was a host with 
no ptr record, so there was nothing for sendmail to resolve.

Mar  9 17:43:27 xanadu milter-greylist: acl_filter hostname = 
"ns1.hagenhosting.com"
Mar  9 17:43:27 xanadu milter-greylist: j29MhR2X030xxx: addr 63.97.115.194 
from <> to <xxxxxx@...> delayed for 00:01:00

So the names are being resolved by sendmail where possible, but they are 
still being greylisted with no viable greylist rules matching.

> > but it's being greylisted...
>
>This means the default is broken. Any info with -l?

Erm, -l isn't a useful option, as it winds up generating printf's that go 
nowhere on my system.

It's weird that the default works normally.

I've also observed that the default sometimes works if I only add one 
domain ACL, but adding 4 always breaks it.

The above test was run with these enabled:
acl greylist domain /pcp.*\...\.comcast\.net/
acl greylist domain /.*\.client\.comcast\.net/
acl greylist domain /.*\.client2\.attbi\.com/
acl grelyist domain /.*\.client\.attbi\.com/

At 05:39 PM 3/9/2005, manu@... wrote:
>This means the default is broken. Any info with -l?

OK, I did some more checking, the default is definitely broken with the 
regex ACL rules in place..

I commented out the acl debug requirement for the syslog message at the 
bottom of acl_filter, and added another debug for the "default" case.

Without:
Mar  9 17:58:58 xanadu milter-greylist: acl_filter hostname = 
"postfixprim.ofir.com"
Mar  9 17:58:58 xanadu milter-greylist: acl_filter loop bottom with 
check==0 && match
Mar  9 17:58:58 xanadu milter-greylist: Mail from=<>, 
rcpt=<xxxxxx@...>, addr=193.0.243.235 is matched by entry acl 
whitelist default

With:
Mar  9 17:57:33 xanadu milter-greylist: acl_filter hostname = 
"209-232-74-254.kingschools.com"
Mar  9 17:57:33 xanadu milter-greylist: acl_filter using default. 
EXF_DEFAULT 4 retval 5 testmode 0
Mar  9 17:57:33 xanadu milter-greylist: j29MvW2X032628: addr 209.232.74.254 
from <> to <xxxxxxxxx@...> delayed for 00:01:00

So it would appear that the regex domain acl's are clobbering the acl 
whitelist default somehow and it's falling into the default case.

Matt Kettler <mkettler@...> wrote:
 
> OK, I did some more checking, the default is definitely broken with the
> regex ACL rules in place..

Can you send me a config file that exhibit the problem?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

At 06:29 PM 3/9/2005, manu@... wrote:
> > OK, I did some more checking, the default is definitely broken with the
> > regex ACL rules in place..
>
>Can you send me a config file that exhibit the problem?

I sent you an off-list message with it.

Also of note:
adding non-regex domain ACLs works fine...
adding regex based rcpt rules works fine...
adding address based rules works fine...

Pretty much the only thing that causes problems so far is regex based 
domain acls, and you seem to need 2 of em before it will stomp the "acl 
whitelist default" and everything will default to being greylisted.

At 06:29 PM 3/9/2005, manu@... wrote:
>Matt Kettler <mkettler@...> wrote:
>
> > OK, I did some more checking, the default is definitely broken with the
> > regex ACL rules in place..
>
>Can you send me a config file that exhibit the problem?

Ugh! I found the problem.. Unfortunately, the message I needed was being 
logged to /var/log/messages instead of /var/log/maillog.

I've got a typo in the word "greylist" on the second domain acl...

Sorry for the false alarm...

Matt Kettler <mkettler@...> wrote:

> Ugh! I found the problem.. Unfortunately, the message I needed was being
> logged to /var/log/messages instead of /var/log/maillog.
> 
> I've got a typo in the word "greylist" on the second domain acl...

What is the message?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

At 02:00 AM 3/10/2005, manu@... wrote:
> > Ugh! I found the problem.. Unfortunately, the message I needed was being
> > logged to /var/log/messages instead of /var/log/maillog.
> >
> > I've got a typo in the word "greylist" on the second domain acl...
>
>What is the message?

Mar  9 19:19:07 xanadu milter-greylist: config error at line 281: parse error

For some reason, those, along with the startup and shutdown messages are 
being sent to messages instead of maillog.

Matt Kettler <mkettler@...> wrote:

> Mar  9 19:19:07 xanadu milter-greylist: config error at line 281: parse error
> 
> For some reason, those, along with the startup and shutdown messages are
> being sent to messages instead of maillog.

Oh yes, it's emitted through yyerror instead of syslog. 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...