Milter-greylist

Hi,

First of all, thank you very much for creating milter-
greylist.  It really helps a lot.

Now I've got a question.  My MX server gets a lot of
probes from spammers for mail accounts that don't exist.
That is, I have thousands of these in my logs:

Jan 20 00:01:19 [...] <guirre@mydomain>... User unknown
Jan 20 00:01:19 [...] <gulfier@mydomain>... User unknown
Jan 20 00:01:20 [...] <gullit@mydomain>... User unknown
Jan 20 00:01:21 [...] <gummo@mydomain>... User unknown

After installing milter-greylist, all of those are
greylisted:

Jan 20 13:17:44 [...] to <arioch@mydomain> delayed for 00:03:00
Jan 20 13:17:56 [...] to <arjani@mydomain> delayed for 00:03:00
Jan 20 13:18:33 [...] to <arkeen@mydomain> delayed for 00:03:00
Jan 20 13:18:34 [...] to <arkosic@mydomain> delayed for 00:03:00

Those just take up space in the greylist for no reason.
In fact, 99% of the contents of my dump file is from
such useless probes.  And the address harvester might
even come back and probe the same account again.

Is it possible _not_ to greylist those, but reject the
mails permanently (and immediately)?  In other words,
a permanent "User unknow" should take precedence over
temporary greylisting.

Best regards
   Oliver

-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

Python is executable pseudocode.  Perl is executable line noise.

On Fri, Jan 20, 2006 at 02:43:00PM +0100, Oliver Fromme wrote:
> Now I've got a question.  My MX server gets a lot of
> probes from spammers for mail accounts that don't exist.
> That is, I have thousands of these in my logs:
> 
> Jan 20 00:01:19 [...] <guirre@mydomain>... User unknown

I use milter-rcptfilter to kick out mail to unexistant users.

-- 
Emmanuel Dreyfus
manu@...

On Fri, Jan 20, 2006 at 02:43:00PM +0100, Oliver Fromme wrote:
> 
> Is it possible _not_ to greylist those, but reject the
> mails permanently (and immediately)?  In other words,
> a permanent "User unknow" should take precedence over
> temporary greylisting.

While not to Oliver's degree I have noticed this as well on my recent
implementation of milter-greylist.  I suspect it should just be some
precidence issue insine the sendmail config, but I'm not conversant
enough with config details to ferret it out.

-- 
Till Later,
Jake <karrde@...>
http://www.viluppo.net/

Oliver Fromme wrote:
> Hi,
> 
> First of all, thank you very much for creating milter-
> greylist.  It really helps a lot.
> 
> Now I've got a question.  My MX server gets a lot of
> probes from spammers for mail accounts that don't exist.
> That is, I have thousands of these in my logs:
> 
> Jan 20 00:01:19 [...] <guirre@mydomain>... User unknown
> Jan 20 00:01:19 [...] <gulfier@mydomain>... User unknown
> Jan 20 00:01:20 [...] <gullit@mydomain>... User unknown
> Jan 20 00:01:21 [...] <gummo@mydomain>... User unknown
> 
> After installing milter-greylist, all of those are
> greylisted:
> 
> Jan 20 13:17:44 [...] to <arioch@mydomain> delayed for 00:03:00
> Jan 20 13:17:56 [...] to <arjani@mydomain> delayed for 00:03:00
> Jan 20 13:18:33 [...] to <arkeen@mydomain> delayed for 00:03:00
> Jan 20 13:18:34 [...] to <arkosic@mydomain> delayed for 00:03:00
> 
> Those just take up space in the greylist for no reason.
> In fact, 99% of the contents of my dump file is from
> such useless probes.  And the address harvester might
> even come back and probe the same account again.
> 
> Is it possible _not_ to greylist those, but reject the
> mails permanently (and immediately)?  In other words,
> a permanent "User unknow" should take precedence over
> temporary greylisting.

I asked this a long time ago. Unfortunately, with the Sendmail Milter API this
isn't possible without changing to post-data-phase greylisting, which has a lot
of problems with excessive bandwidth consumption.


You can add on another milter to run ahead of milter-greylist. This is a little
silly, but unfortunately it's the only way to get user-validation before
greylisting.

On Fri, 20 Jan 2006, Oliver Fromme wrote:

OF> 
OF> Is it possible _not_ to greylist those, but reject the
OF> mails permanently (and immediately)?  In other words,
OF> a permanent "User unknow" should take precedence over
OF> temporary greylisting.
OF> 

I have a wild card entry in my sendmail's virtual user file (note I have 
added  .example a few times in this post):

@...            error:550 "Spam problems. Try 
alan+newcontactone@..."

which means I have to have separate entries for the names I actually use, 
like:

alan@...   alan
lists@...            alan

milter-greylist doesn't leave any trace in the maillog for false addresses 
except, for example, when I sent one myself from my gmail account when it 
noted that the ip was in the exception list.  But there is no entry in the 
greylist.db for the address I used.

Something is not 100% right though, as the failure message delivered to my 
gamail account had error 553 in it rather than 550 but it appears to work 
anyway.

Technical details of permanent failure:
PERM_FAILURE: SMTP Error (state 9): 553 5.3.0 
<falseaddress@...>... Spam problems. Try 
alan+newcontactone@...

Looking at greylist.db, all the @... entries have a known local 
part.

-- 
Alan


( Please do not email me AS WELL as replying to the list.  Please 
  address personal email to alan+1@ as lists@ is not read. A
  password autoresponder may be invoked if this email is very old. )

Only a short note on this. When you only accept mails for existing users 
and return a error for all
non existing users, it's easy for a spammer to checkout which email 
addresses actually exist. If you
have milter-greylist in the first line of defense (under the condition 
the spammer only makes one try
per email address) he wouldn't get the info if the user exist or not....

Alan Clifford schrieb:

> On Fri, 20 Jan 2006, Oliver Fromme wrote:
>
> OF> 
> OF> Is it possible _not_ to greylist those, but reject the
> OF> mails permanently (and immediately)?  In other words,
> OF> a permanent "User unknow" should take precedence over
> OF> temporary greylisting.
> OF> 
>
> I have a wild card entry in my sendmail's virtual user file (note I have 
> added  .example a few times in this post):
>
> @...            error:550 "Spam problems. Try 
> alan+newcontactone@..."
>
> which means I have to have separate entries for the names I actually use, 
> like:
>
> alan@...   alan
> lists@...            alan
>
> milter-greylist doesn't leave any trace in the maillog for false addresses 
> except, for example, when I sent one myself from my gmail account when it 
> noted that the ip was in the exception list.  But there is no entry in the 
> greylist.db for the address I used.
>
> Something is not 100% right though, as the failure message delivered to my 
> gamail account had error 553 in it rather than 550 but it appears to work 
> anyway.
>
> Technical details of permanent failure:
> PERM_FAILURE: SMTP Error (state 9): 553 5.3.0 
> <falseaddress@...>... Spam problems. Try 
> alan+newcontactone@...
>
> Looking at greylist.db, all the @... entries have a known local 
> part.
>
>

Andreas Unterkircher <unki@...> wrote:

> Only a short note on this. When you only accept mails for existing users
> and return a error for all
> non existing users, it's easy for a spammer to checkout which email 
> addresses actually exist. 

Do they really collect addresses this way? It sounds rather unefficient
(compared to parsing web sites)

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

I don't know if spammers really actually doing this but in my last job 
where we send out newsletter mails (for
a local amazon-like webshop, user agreed to receive this letters...) to 
around 300tsd customers we used the
return code (over a longer periode) to disable the newsletter for this 
addresses.

Also when I'm searching through the maillog of an qmail server which 
doesn't accept mails for non-existing users
I can see that a stupid DSL-account spammer only retries the really 
existing email addresses (different spams)
over a longer periode. This selfmixed firstname.lastnames addresses 
which are not existing don't appear here twice
while the really existing email addresses appear in every spam run...

manu@... schrieb:

> Andreas Unterkircher <unki@...> wrote:
>
>   
>> Only a short note on this. When you only accept mails for existing users
>> and return a error for all
>> non existing users, it's easy for a spammer to checkout which email 
>> addresses actually exist. 
>>     
>
> Do they really collect addresses this way? It sounds rather unefficient
> (compared to parsing web sites)
>
>

On Sat, 21 Jan 2006 manu@... wrote:

> Andreas Unterkircher <unki@...> wrote:
>
>> Only a short note on this. When you only accept mails for existing users
>> and return a error for all
>> non existing users, it's easy for a spammer to checkout which email
>> addresses actually exist.
>
> Do they really collect addresses this way? It sounds rather unefficient
> (compared to parsing web sites)
>

I seem to get most stuff to more well known mail addresses, particulary 
lists@ (lists address) and alan@ (probably most visible in the ports list, 
silly me).  As these are accepted, maybe there is something in the case 
that these stick around in the spam lists whilst rejected ones don't.

I get some sent to nonsensical names like <4a5c07ce@...>  - I 
just can't see the point of these.

Maybe it would be better to accept and dump mail to non-existing users? 
If they did actually retry after the greylist reject, it would keep the 
spam databases full of rubbish addresses and every spam sent to my 
non-exists would be one less sent to real people.  However, considering 
the success of greylisting, I'm not too sure how much notice is taken of 
the rejects.

On the other hand, I get stuff to scatology@(usenet address) which is 
highly visible but rejected.

On the other hand, the address on my website is not popular but that maybe 
because it has a + in it.

-- 
Alan


( Please do not email me AS WELL as replying to the list.  Please
   address personal email to alan+1@ as lists@ is not read. A
   password autoresponder may be invoked if this email is very old. )

On Fri, Jan 20, 2006 at 02:43:00PM +0100, Oliver Fromme wrote:
> After installing milter-greylist, all of those are
> greylisted:
> 
> Jan 20 13:17:44 [...] to <arioch@mydomain> delayed for 00:03:00
> Jan 20 13:17:56 [...] to <arjani@mydomain> delayed for 00:03:00
> Jan 20 13:18:33 [...] to <arkeen@mydomain> delayed for 00:03:00
> Jan 20 13:18:34 [...] to <arkosic@mydomain> delayed for 00:03:00

Use "/etc/mail/access" or "/etc/mail/virtusertable" to define a list
of valid e-mail addresses and the problem will go away.

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

Matthias Scheler wrote:
 > Oliver Fromme wrote:
 > > After installing milter-greylist, all of those are
 > > greylisted:
 > > 
 > > Jan 20 13:17:44 [...] to <arioch@mydomain> delayed for 00:03:00
 > > Jan 20 13:17:56 [...] to <arjani@mydomain> delayed for 00:03:00
 > > Jan 20 13:18:33 [...] to <arkeen@mydomain> delayed for 00:03:00
 > > Jan 20 13:18:34 [...] to <arkosic@mydomain> delayed for 00:03:00
 > 
 > Use "/etc/mail/access" or "/etc/mail/virtusertable" to define a list
 > of valid e-mail addresses and the problem will go away.

That doesn't help.  I _do_ have access and virtusertable,
and also userdb and alias file.  But milter-greylist comes
_before_ all of the valid user checking.

By the way, thanks to everybody else for explaining that
the problem cannot easily be solved.  So far the best
suggestion was given by Emmanuel, i.e. to use the milter-
rcpfilter.  However, the problem is that it has is own
configuration file of valid users, so I would have to
maintain users in multiple different places, which is
annoying and error-prone.

So, unfortunately, there doesn't seem to be a good
solution for the problem.

Best regards
   Oliver

-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

C++: "an octopus made by nailing extra legs onto a dog"
        -- Steve Taylor, 1998

Oliver Fromme <olli@...> wrote:

> By the way, thanks to everybody else for explaining that
> the problem cannot easily be solved.  So far the best
> suggestion was given by Emmanuel, i.e. to use the milter-
> rcpfilter.  However, the problem is that it has is own
> configuration file of valid users, so I would have to
> maintain users in multiple different places, which is
> annoying and error-prone.

I generate the config files for milter-greylist and milter-rcptfilter
from a single source, using a shell script. It works quite well...

-- 
Emmanuel Dreyfus
Un bouquin en français sur BSD:
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...

On Sun, Jan 22, 2006 at 11:02:41AM +0100, Oliver Fromme wrote:
>  > Use "/etc/mail/access" or "/etc/mail/virtusertable" to define a list
>  > of valid e-mail addresses and the problem will go away.
> 
> That doesn't help.  I _do_ have access and virtusertable,
> and also userdb and alias file.  But milter-greylist comes
> _before_ all of the valid user checking.

That's not true. Try to connect to "mail.zhadum.de" and see for yourself:

220 colwyn.zhadum.org.uk ESMTP Sendmail 8.13.3/8.13.3; Sun, 22 Jan 2006 22:25:09 GMT
EHLO xxx
250-colwyn.zhadum.org.uk Hello xxx@xxx [xxx.xxx.xxx.xxx], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE 20971520
250-DSN
250-ETRN
250-STARTTLS
250-DELIVERBY
250 HELP
MAIL FROM:<foo@...>
250 2.1.0 <foo@...>... Sender ok
RCPT TO:<foobar@...>
553 5.3.0 <foobar@...>... User unknown

And that server is using "milter-greylist". Here is the logfile entry:

Jan 22 22:25:38 colwyn sm-mta[28295]: k0MMP9kR028295: <foobar@...>... User unknown
Jan 22 22:25:40 colwyn sm-mta[28295]: k0MMP9kR028295: from=<foo@...>, size=0, class=0, nrcpts=0, proto=ESMTP, daemon=MTA, relay=xxx [xxx.xxx.xxx.xxx]

As you can see "milter-greylist" was not involved.

> So, unfortunately, there doesn't seem to be a good
> solution for the problem.

Using "/etc/mail/access" and "/etc/mail/virtusertable" works fine if they
are configured accordingly. My "/etc/mail/access" lists all valid users
like this ...

To:tron@...                        RELAY

... and ends with a rule which blocks all unknown mail addresses:

To:colwyn.zhadum.de                             error:5.1.1:550 User unknown

The list is automatically created from the list of local users and aliases.
And each domain in "virtusertable" has a rule to reject unknown addresses:

@...                              error:nouser "User unknown"

With such a setup "sendmail" will reject unknown users immediately even
if "milter-greylist" is used.

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

> By the way, thanks to everybody else for explaining that
> the problem cannot easily be solved.  So far the best
> suggestion was given by Emmanuel, i.e. to use the milter-
> rcpfilter.  However, the problem is that it has is own
> configuration file of valid users, so I would have to
> maintain users in multiple different places, which is
> annoying and error-prone.

I've solved the problem by listing all valid email addresses
that should be protected by greylisting in greylist.conf,
and changing the default action to whitelist, like:

  acl greylist rcpt user1@...
  acl greylist rcpt user2@...
  acl whitelist default

Like this you don't need an extra milter. You still have to
maintain the list of valid e-mail addresses in greylist.conf.
If accounts aren't added/deleted frequently, that's not a big
issue. But even if so, you could generate the greylist.conf
entries (semi-)automatically with a script. It's not a problem
if a new account isn't listed immediately, as the default is
whitelist, and new accounts take some time to catch big amounts
of spam anyway.

mp.
-- 
                         Martin Paul | Systems Administrator
   Institute of Scientific Computing | martin@...
 Nordbergstrasse 15/C/3, A-1090 Wien | Tel: 01 4277 39403
        http://www.par.univie.ac.at/ | Fax: 01 4277 9394

Alan Clifford wrote on Sat, 21 Jan 2006 15:22:51 +0000 (GMT):

> I get some sent to nonsensical names like <4a5c07ce@...>  - I 
> just can't see the point of these.

Clifford, these are probably grabbed Message-IDs since most of the mail 
worms can't distinguish between email address and Message-ID, they just 
grab everything with an @ in-between.

Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com