Milter-greylist

A few weeks ago I sent here a couple emails saying that since I upgraded 
from version 3.1.1 to 4.0a1 I noticed that from time to time
a lot of error messages are logged by sendmail about the greylist socket.

Now I fond the real problem: a few times a day milter-greylist crashes!

I have a little script that checks that greylist is running and 
eventually re-launches it, so I didn't noticed those crashes...

The OS is OpenBSD 4.1, both i386 and amd64.
No (unusual) error message is logged before the crash.
No coredump is generated.

What else I can look at?


Thanks.

-- 
___________________________________________________
     __
    |-                      giannici@...
    |ederico Giannici      http://www.neomedia.it
___________________________________________________

> I have a little script that checks that greylist is running and 
> eventually re-launches it, so I didn't noticed those crashes...
> 

Hello,

I strongly recommend to everybody to include notification in case
of automatic re-lunch of milter-greylist.

Here is my /usr/local/etc/check_milter_greylist which is scheduled to run
every five minutes from crontab on SuSE LINUX:

#! /bin/bash
PIDLIST="$(ps axo ppid,pid,comm | awk '{ if ($1 == 1 && $3 == "milter-greylist") print $2}')"
if [[ x"$PIDLIST" == x ]]
then
  /etc/init.d/milter-greylist start > /dev/null 2>&1
  recipient=postmaster
  mailx -s "Milter-greylist restarted on $(hostname)" "$recipient" < /dev/null
else
 :
#  printf 'RUNNING: %s\n' "$PIDLIST"
fi

You may just use as it is replacing the recipient variable better fitting to your site.
It must be trivial to port to other platforms.

Bests,
Attila

Federico Giannici <giannici@...> wrote:

> The OS is OpenBSD 4.1, both i386 and amd64.
> No (unusual) error message is logged before the crash.
> No coredump is generated.
> What else I can look at?

You can run the milter within gdb, with the -D flag. When it crashes,
issue the bt command to get a backtrace. It may help.

Indeed 4.0 alpha series have a bug somewhere. Nobody have been able to
find it yet. Probably a race condition.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

> Here is my /usr/local/etc/check_milter_greylist which is scheduled to run
> every five minutes from crontab on SuSE LINUX:
> 
> #! /bin/bash
> PIDLIST="$(ps axo ppid,pid,comm | awk '{ if ($1 == 1 && $3 == "milter-greylis
> t") print $2}')"
> if [[ x"$PIDLIST" == x ]]
> then
>   /etc/init.d/milter-greylist start > /dev/null 2>&1
>   recipient=postmaster
>   mailx -s "Milter-greylist restarted on $(hostname)" "$recipient" < /dev/nul
> l
> else
>  :
> #  printf 'RUNNING: %s\n' "$PIDLIST"
> fi

Why do something so complex? You could instead run milter-greylist in
the foreground inside an infinite loop, putting whatever else you like
in the loop for notification and other cleanup, e.g.

#!/bin/sh

while true; do
        ...some stuff...
        /usr/local/bin/milter-greylist -D
        ...some stuff...
done &

If an infinite loop makes you nervous, you can create some safety that
will exit the loop in the event of something like 5 relaunches in the
space of 5 seconds.

Better than periodically checking the process table, IMHO.

Cheers,

	- Joel

On Sun, May 27, 2007 at 02:34:06PM +1000, Joel Reicher wrote:
> Why do something so complex? You could instead run milter-greylist in
> the foreground inside an infinite loop, putting whatever else you like
> in the loop for notification and other cleanup, e.g.

This has the advantage that you can even run it within gdb and collect a
backtrace after a crash...

-- 
Emmanuel Dreyfus
manu@...

> Why do something so complex? You could instead run milter-greylist in
> the foreground inside an infinite loop, putting whatever else you like
> in the loop for notification and other cleanup, e.g.
> 
> #!/bin/sh
> 
> while true; do
>         ...some stuff...
>         /usr/local/bin/milter-greylist -D
>         ...some stuff...
> done &
> 
> If an infinite loop makes you nervous, you can create some safety that
> will exit the loop in the event of something like 5 relaunches in the
> space of 5 seconds.
> 
> Better than periodically checking the process table, IMHO.
> 
> Cheers,
> 
> 	- Joel

Sure, you can do this way too...
The main point is to have notification about the re-lunch of milter-greylist,
since it is a major event which should not go unnoticed.

Bests,
Attila

manu@... wrote:
> 
> Federico Giannici <giannici@... <mailto:giannici%40neomedia.it>> 
> wrote:
> 
>  > The OS is OpenBSD 4.1, both i386 and amd64.
>  > No (unusual) error message is logged before the crash.
>  > No coredump is generated.
>  > What else I can look at?
> 
> You can run the milter within gdb, with the -D flag. When it crashes,
> issue the bt command to get a backtrace. It may help.
> 
> Indeed 4.0 alpha series have a bug somewhere. Nobody have been able to
> find it yet. Probably a race condition.

In case it could be useful, I have restricted the range of time the bug 
has been introduced: the problem IS present in version 3.1.3 and IS NOT 
present in 3.1.1.


Bye.

-- 
___________________________________________________
     __
    |-                      giannici@...
    |ederico Giannici      http://www.neomedia.it
___________________________________________________

Tere.

Postfix includes good helo check options 
(http://www.postfix.org/uce.html#smtpd_helo_restrictions):

smtpd_helo_restrictions = reject_non_fqdn_hostname, 
reject_invalid_hostname, reject_unknown_hostname

And sendmail does not, which is sad as this is quite efficient, 
especially when unknown_hostname_reject_code is set to 550.

One way to implement it somehow into sendmail is using milter-regex 
(http://www.benzedrine.cx/milter-regex.html), but as this is again 
another milter and as milter-greylist also includes regular expressions 
support, then maybe is possible to to add invalid helo strings check 
directly into milter-greylist?

-- 
Mart

Mart Pirita <sysadmin@...> wrote:

> One way to implement it somehow into sendmail is using milter-regex 
> (http://www.benzedrine.cx/milter-regex.html), but as this is again 
> another milter and as milter-greylist also includes regular expressions
> support, then maybe is possible to to add invalid helo strings check 
> directly into milter-greylist?

My concern is always limitting the config file mess.

We can already set an ACL clause on HELO against a substring or a regex.
How do you imagine the syntax for performing a DNS check?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Tere.
>
> My concern is always limitting the config file mess.
>   
:).
> We can already set an ACL clause on HELO against a substring or a regex.
>   
Hmm, didn't know, any example how to do it?

> How do you imagine the syntax for performing a DNS check?
>
>   
Like dnsbl maybe?

-- 
Mart

Mart Pirita <sysadmin@...> wrote:

> > We can already set an ACL clause on HELO against a substring or a regex.
> Hmm, didn't know, any example how to do it?

In 4.0 beta:
racl blacklist helo /domain\.com/ msg "go away!" 

> > How do you imagine the syntax for performing a DNS check?
> Like dnsbl maybe?

An example.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Mart Pirita wrote:
> Tere.
> 
> Postfix includes good helo check options 
> (http://www.postfix.org/uce.html#smtpd_helo_restrictions):

I find it rather funny that the above page has this statement in it:

"Reject the request when the hostname in the client HELO (EHLO) command is not
in fully-qualified domain form, as required by the RFC."

Which is funny, because RFC 1123 section 5.2.5 also explicitly prohibits
refusing mail based on HELO failing checks.

"However, the receiver MUST NOT refuse to accept a message, even if the sender's
HELO command fails verification."

I guess postfix figures "an eye for an eye" is reasonable when it comes to RFC
violations...

Mart Pirita <sysadmin@...> wrote:

> > How do you imagine the syntax for performing a DNS check?
> Like dnsbl maybe?

An example?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Tere.
>
> In 4.0 beta:
> racl blacklist helo /domain\.com/ msg "go away!" 
>
>
>   
But isn't it manually adjustable blacklist? And fighting manually 
against spammers seems mission impossible?
>
> An example.
>
>   
Dnsbl also checks senders ip against something, and so checking helo= 
resolving seems easy for me, but as You did ask an example, then I'm 
afraid, it's not so easy:(.

-- 
Mart

Mart Pirita <sysadmin@...> wrote:

> > racl blacklist helo /domain\.com/ msg "go away!" 
> But isn't it manually adjustable blacklist? And fighting manually 
> against spammers seems mission impossible?

You can't fight on the HELO string, as the spammer can send whatever he
wants. HELO filtering is only going to help in dealing with very special
situations IMO.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Wed, Jun 20, 2007 at 06:50:12AM +0200, Emmanuel Dreyfus wrote:
> > > racl blacklist helo /domain\.com/ msg "go away!" 
> > But isn't it manually adjustable blacklist? And fighting manually 
> > against spammers seems mission impossible?
> 
> You can't fight on the HELO string, as the spammer can send whatever he
> wants. HELO filtering is only going to help in dealing with very special
> situations IMO.

But it works. I use "milter-regex" for this purpose and it rejected
831 SMTP connections (on a low traffic mail server) which used
"HELO localhost" in the last 24 hours.

	Kind regards

-- 
Matthias Scheler                                  http://zhadum.org.uk/

Matthias Scheler <tron@...> wrote:

> > You can't fight on the HELO string, as the spammer can send whatever he
> > wants. HELO filtering is only going to help in dealing with very special
> > situations IMO.
> 
> But it works. I use "milter-regex" for this purpose and it rejected
> 831 SMTP connections (on a low traffic mail server) which used
> "HELO localhost" in the last 24 hours.

Yes, I added HELO filtering in milter-greylist for this kind of
problems. But I don't call it fighting against spammers. What can be
using HELO localhost, except a broken MTA configured as an open relay?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Wed, Jun 20, 2007 at 02:24:30PM +0200, Emmanuel Dreyfus wrote:
> What can be using HELO localhost, except a broken MTA configured as
> an open relay?

Apparently Bot Net Spam software. I can find 289 unique IP address
which got blocked because of this. And from a quick look they look
like dynamic broadband IP addresses.

	Kind regards

P.S. Blocking "HELO localhost" not the only "HELO" filter I'm using.

-- 
Matthias Scheler                                  http://zhadum.org.uk/

Matthias Scheler wrote:
 > Emmanuel Dreyfus wrote:
 > > You can't fight on the HELO string, as the spammer can send whatever he
 > > wants. HELO filtering is only going to help in dealing with very special
 > > situations IMO.
 > 
 > But it works. I use "milter-regex" for this purpose and it rejected
 > 831 SMTP connections (on a low traffic mail server) which used
 > "HELO localhost" in the last 24 hours.

How many of those were spam?  And how many of those were
legitimate mails?  Can you even tell for sure?

It should be noted that using greylisting (which is still
the main purpose of milter-greylist) is perfectly RFC-
compliant.  But rejecting mails based on the HELO string
is not.

Therefore, if there's any mentioning of such HELO-based
filtering in the manpage or other docs, it should be
accompanied by a fat warning, IMHO.  So at least those
who intentionally make their servers violate the RFCs
are aware of it.

I expect that many people who use milter-greylist have
never read the appropriate RFCs.  The internet is becoming
more and more of a tough place because people don't read
standards or use "trial and error" methods.  I think that
milter-greylist should try not to make it worse, and at
least document the most critical things.  Milters often
operate at the very edge of RFC-compliance ...

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"Whatever happened to the days when hacking started
at the cerebral cortex, and not at the keyboard?"
  --  Sid on userfriendly.org by Illiad, 2007-06-20

Oliver Fromme <olli@...> wrote:

> Therefore, if there's any mentioning of such HELO-based
> filtering in the manpage or other docs, it should be
> accompanied by a fat warning, IMHO.  So at least those
> who intentionally make their servers violate the RFCs
> are aware of it.

Well, you can always enjoy the power of a nifty ACL without breaking any
RFC:

racl greylist helo "localhost" delay 12h msg "Think about fixing HELO"
racl greylist default delay 30m

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

manu@... wrote:
 > Oliver Fromme wrote:
 > > Therefore, if there's any mentioning of such HELO-based
 > > filtering in the manpage or other docs, it should be
 > > accompanied by a fat warning, IMHO.  So at least those
 > > who intentionally make their servers violate the RFCs
 > > are aware of it.
 > 
 > Well, you can always enjoy the power of a nifty ACL without breaking
 > any RFC:
 > 
 > racl greylist helo "localhost" delay 12h msg "Think about fixing HELO"
 > racl greylist default delay 30m

Yes, that's right.  :-)
Although I doubt that it really helps.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"Clear perl code is better than unclear awk code; but NOTHING
comes close to unclear perl code"  (taken from comp.lang.awk FAQ)