Milter-greylist

Hi,

I seem to remember that I've heard Emmanuel talk about implementing 
greytrapping, but I've not seen it discussed here. I've tried to 
implement that in mimedefang, but there are some drawbacks that could be 
  avoided by using milter-greylist for that.

There are at least two ways to trap spammer IP's:

o messages sent to e-mail addresses that never existed (and will never 
exist) in the destination domain.
  An amazingly common case is spam received to e-mail addresses that are 
in fact message-id of usenet posts where XEmacs/GNUS was used, you get 
messages addressed to things like  "wzizo1at5ti.fsf@...". The IP of 
this sender can sent to an acl to be  blocked for an extended delay...

o messages sent to the lowest priority MX of a site when a higher 
priority MX is alive. One trick to detect that is to assign 2 IP 
addresses to you main MX and to declare the second one as another MX for 
your domain(s) with a very low priority. Now when a t-uple arrives to 
this IP and has not been seen by milter-greylist before (using the 
lazywaw or the subnetmatch feature if needed), it's for sure something 
that violates the RFC and deserves to wait longer (or get rejected 
directly).

I think those 2 kinds of trap are pretty easy to implement, and I'm 
willing to try to give them a shot, unless someone else beats me or 
finds this really stupid and useless.

-- 
Matthieu Herrb

Matthieu Herrb escribi\ufffd:
> Hi,
>
> I seem to remember that I've heard Emmanuel talk about implementing 
> greytrapping, but I've not seen it discussed here. I've tried to 
> implement that in mimedefang, but there are some drawbacks that could 
> be  avoided by using milter-greylist for that.
>
> There are at least two ways to trap spammer IP's:
>
> o messages sent to e-mail addresses that never existed (and will never 
> exist) in the destination domain.
>  An amazingly common case is spam received to e-mail addresses that 
> are in fact message-id of usenet posts where XEmacs/GNUS was used, you 
> get messages addressed to things like  "wzizo1at5ti.fsf@...". The 
> IP of this sender can sent to an acl to be  blocked for an extended 
> delay...
>
> o messages sent to the lowest priority MX of a site when a higher 
> priority MX is alive. One trick to detect that is to assign 2 IP 
> addresses to you main MX and to declare the second one as another MX 
> for your domain(s) with a very low priority. Now when a t-uple arrives 
> to this IP and has not been seen by milter-greylist before (using the 
> lazywaw or the subnetmatch feature if needed), it's for sure something 
> that violates the RFC and deserves to wait longer (or get rejected 
> directly).
>
> I think those 2 kinds of trap are pretty easy to implement, and I'm 
> willing to try to give them a shot, unless someone else beats me or 
> finds this really stupid and useless.
>
For the first case i thought a different solution. I'm using MScanner 
and to do some statistics i use Vispan. Vispan 3.0.0 includes a 
"heuristic" engine that identifies IP's that send spam (watching the 
maillog) to avoid false-positives it count the amount of spam that an ip 
send. All this IP's are stored on a txt file generaly /tmp/rblfile.txt.

If you use milter-greylist you can compile with --enable-dnsrbl to 
enable acls with rbl verification.

The next step is to set up rbldnsd locally to read rblfile.txt. When the 
rbl is ok, you can set an ACL to delay more time the email if the ip of 
the sender is in your blacklist.

Its possible to extend this, i have 4 servers collecting ips to a single 
rbl server (while processing real trafic) and i have 44000 ips in 
blacklist. The problem is that i cannot enable dnsrbl on greylisting.

Did someone enable it on the new beta?

Regards.

Matthieu Herrb <matthieu.herrb@...> wrote:

> I seem to remember that I've heard Emmanuel talk about implementing 
> greytrapping, but I've not seen it discussed here. I've tried to 
> implement that in mimedefang, but there are some drawbacks that could be
>   avoided by using milter-greylist for that.

How are you going to store the blacklist? IMO, a reasonnable approach
would be to match the messages using milter-greylist ACL, then feed a
DNSRBL. 

You'd need to add a DNS update action clause to milter-greylist ACL,
something such as:

dnsrblupdate "MYRBL"  ns.example.net  127.0.0.10
racl blacklist rcpt wzizo1at5ti.fsf@... dnsrblupdate "MYRBL"

You'll also need to think about entries expiration. What do you think?


Your other proposal (matching messages that goes to secondary MX before
primary) seems a bit more difficult. MX sync may help: the messages you
want to match arrive at secondary MX without being already greylisted.
It seems you need to add an information to the greylisting database:
where the message was presented last time (IP of MX for instance).

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Edux <edux@...> wrote:

> Its possible to extend this, i have 4 servers collecting ips to a single
> rbl server (while processing real trafic) and i have 44000 ips in 
> blacklist. The problem is that i cannot enable dnsrbl on greylisting.
> 
> Did someone enable it on the new beta?

I have been using it in production for more than 10 months now. The
usual problem is linking with a thread-unsafe DNS resolver, which cause
instabbility. Linking with BIND9's libbind fixes the problem.
-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

manu@... wrote:
> Matthieu Herrb <matthieu.herrb@...> wrote:
> 
>> I seem to remember that I've heard Emmanuel talk about implementing 
>> greytrapping, but I've not seen it discussed here. I've tried to 
>> implement that in mimedefang, but there are some drawbacks that could be
>>   avoided by using milter-greylist for that.
> 
> How are you going to store the blacklist? IMO, a reasonnable approach
> would be to match the messages using milter-greylist ACL, then feed a
> DNSRBL. 
> 
> You'd need to add a DNS update action clause to milter-greylist ACL,
> something such as:
> 
> dnsrblupdate "MYRBL"  ns.example.net  127.0.0.10
> racl blacklist rcpt wzizo1at5ti.fsf@... dnsrblupdate "MYRBL"
> 
> You'll also need to think about entries expiration. What do you think?

I was thinking about using the greylist database itself with a special 
attribute to store those dynamically blacklisted entries. The from and 
to fields are not meaningful in that case, but they can be kept for 
debugging purposes.
This database already have all mechanisms to manage the expiration of 
the entries.

> 
> 
> Your other proposal (matching messages that goes to secondary MX before
> primary) seems a bit more difficult. MX sync may help: the messages you
> want to match arrive at secondary MX without being already greylisted.
> It seems you need to add an information to the greylisting database:
> where the message was presented last time (IP of MX for instance).
> 

The trick is to have one unique milter-greylist instance that will 
handle the messages sent to 2 different IP addresses. The $if_addr 
sendmail macro can be used to distinguish them and treat the incoming 
connection as a spam attempt if the relay,from,to tuple doesn't already 
exist in the greylist database.
If it's present (whitelisted or greylisted) the message can be handled 
normally. Otherwise the relay IP will be auto-blacklisted.
-- 
Matthieu Herrb

wrote on Thu, 23 Aug 2007 05:29:27 +0200:

> Your other proposal (matching messages that goes to secondary MX before
> primary) seems a bit more difficult. MX sync may help: the messages you
> want to match arrive at secondary MX without being already greylisted.
> It seems you need to add an information to the greylisting database:
> where the message was presented last time (IP of MX for instance).

For clarification: he wants to run that second MX on the same machine, 
it's just a different IP. By the time you replied I had already forgotten 
that as well, but Matthieu's new message reminds me of that.
So it will go thru the same sendmail and milter, no matter where it was 
sent. And if it comes in via the second MX IP and the tuple is not already 
known to the greylist db it must be a first attempt which is supposed to 
come from a spammer.
I think this method can only be used if you don't have any real backup 
MXes. But with MX syncing it might work even for real backup MX, but in 
that case you also need some testing that verifies that the primary is up 
and accepting messages.

Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Matthieu Herrb <matthieu.herrb@...> wrote:

> The trick is to have one unique milter-greylist instance that will 
> handle the messages sent to 2 different IP addresses. The $if_addr 
> sendmail macro can be used to distinguish them and treat the incoming
> connection as a spam attempt if the relay,from,to tuple doesn't already
> exist in the greylist database.
> If it's present (whitelisted or greylisted) the message can be handled
> normally. Otherwise the relay IP will be auto-blacklisted.

So you already have the thing working? Is there anything remaining to
implement?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Thu, Aug 23, 2007 at 01:31:38PM +0200, Kai Schaetzl wrote:
> And if it comes in via the second MX IP and the tuple is not already 
> known to the greylist db it must be a first attempt which is supposed to 
> come from a spammer.

People tried that scheme in the past and didn't work very will. Problems
like intermittend routing problem cause to many false positives.

	Kind regards

-- 
Matthias Scheler                                  http://zhadum.org.uk/

Matthias Scheler wrote:
> On Thu, Aug 23, 2007 at 01:31:38PM +0200, Kai Schaetzl wrote:
>> And if it comes in via the second MX IP and the tuple is not already 
>> known to the greylist db it must be a first attempt which is supposed to 
>> come from a spammer.
> 
> People tried that scheme in the past and didn't work very will. Problems
> like intermittend routing problem cause to many false positives.

The idea is that if the 2 addresses are on the same physical interface, 
on the same subnet of the same machine, it should not be affected by 
intermittend routing problems (or you are probably in greater troubles 
than just false positives).

Moreover, integrating this in milter-greylist makes it possible to use 
its database to let messages matching the (auto-)whitelist to get 
through even if they arrive on the address used as a spam trap, reducing 
even more the risk of false positives.

And greytrapping can be very dynamic. Blacklisting a bad IP for a small 
period (one hour or two) at a time is enough to be efficient. After this 
period, either you don't see it again, or it continues to attempt to 
deliver spam and gets blocked again. or it comes back with a "normal" 
activity -- so you had a false positive, but only lost the connection 
for a small period (too bad, but it's better than false positive in 
external DNSBLs that takes days to be removed),
-- 
Matthieu Herrb

manu@... wrote:
> Matthieu Herrb <matthieu.herrb@...> wrote:
> 
>> The trick is to have one unique milter-greylist instance that will 
>> handle the messages sent to 2 different IP addresses. The $if_addr 
>> sendmail macro can be used to distinguish them and treat the incoming
>> connection as a spam attempt if the relay,from,to tuple doesn't already
>> exist in the greylist database.
>> If it's present (whitelisted or greylisted) the message can be handled
>> normally. Otherwise the relay IP will be auto-blacklisted.
> 
> So you already have the thing working? Is there anything remaining to
> implement?

I've just started to implement it on milter-greylist. For now I've only 
implemented the mx based greytrapping and have not done any 
configuration for it: it has my IP configuration hard-coded.

I'm going to be busy in the next weeks so don't wait on this to release 4.0.
-- 
Matthieu Herrb

On Fri, Aug 24, 2007 at 04:13:55PM +0200, Matthieu Herrb wrote:
> >People tried that scheme in the past and didn't work very will. Problems
> >like intermittend routing problem cause to many false positives.
> 
> The idea is that if the 2 addresses are on the same physical interface,

Yes, I understood that part.
 
> on the same subnet of the same machine, it should not be affected by 
> intermittend routing problems ...

People tried that, it didn't work very well.

	Kind regards

-- 
Matthias Scheler                                  http://zhadum.org.uk/

Matthieu Herrb wrote:
 > Matthias Scheler wrote:
 > > Kai Schaetzl wrote:
 > > > And if it comes in via the second MX IP and the tuple is not already
 > > > known to the greylist db it must be a first attempt which is supposed to
 > > > come from a spammer.
 > >
 > > People tried that scheme in the past and didn't work very will. Problems
 > > like intermittend routing problem cause to many false positives.
 >
 > The idea is that if the 2 addresses are on the same physical interface,
 > on the same subnet of the same machine, it should not be affected by
 > intermittend routing problems (or you are probably in greater troubles
 > than just false positives).

It's not that simple.

Suppose someone somewhere wants to send mail, and his MTA
selects your highest-priority MX.  While trying to send the
mail, there is a problem somewhere.  It could be _anywhere_
on the way:  At his end, at your end, or somewhere in the
middle.  It doesn't matter where, and it certainly doesn't
have to be on the receiver side.  It could be his cat
tripping on the modem cable or whatever.

In any case, the result is that sending the mail fails, and
the MTA choses to try another MX with lower priority.
That's perfectly legal behaviour.  If you assume such mails
are spam, you'll certainly get false positives.  I wouldn't
even penalize such mails with a longer greylist delay,
because they could be completely innocent.  YMMV, of course.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"C++ is to C as Lung Cancer is to Lung."
        -- Thomas Funke