Milter-greylist

Hi,

any best practice how to autowhitelist recipients from outgoing smtp 
relay on the incoming relay? On the incoming relay one could use 
urlcheck (any script available?), but how catch recipients from the 
outgoing relay using milter-greylist and make it available for the 
milter-greylist on the incoming server?

Ciao
Marcus

10 \u0441\u0435\u043d\u0442\u044f\u0431\u0440\u044f 2016�\u0433. 11:50:50 CEST, "Marcus Schopen lists-yahoogroups@... [milter-greylist]" <milter-greylist@yahoogroups.com> \u043f\u0438\u0448\u0435\u0442:
>Hi,
>
>any best practice how to autowhitelist recipients from outgoing smtp 
>relay on the incoming relay? On the incoming relay one could use 
>urlcheck (any script available?), but how catch recipients from the 
>outgoing relay using milter-greylist and make it available for the 
>milter-greylist on the incoming server?
>
>Ciao
>Marcus

Hi,

As I mentioned, I was looking for something like this (lazily) and did not find. But it was a few years back, and it is a nice to have (really nice) but not life-critical feature so we've managed without and users got used to the current behavior (that initial comms with a remote domain may lag). Our MGL rules use a scoring system, so for reasonably well-managed domains the greylisting delay is either 2 or 25 minutes which is often (alas not always) negligible; suspicious domains (mostly bad matching of DNS entries or consumerish PTR names, poor SPF, workstation OSes according to p0f, etc.) go into hours range so DNSRBLs have chance to speak up. Very few are blacklisted outright by MGL from the first connection, so whatever trickle passes the initial filter goes to spamassassin (which is cheaper than bayesing the whole big incoming stream) that marks suspicious content and the mailer puts messages into spam folders. Little is lost, few spams get into inbox unmarked, users ar!
 e happy.

Yet regarding the feature to pre-whitelist recipients as we send them our emails, I wanted to do something inside MGL code, such as to use the replication protocol to query and feed the grey/autowhite database. For some reason this did not work for me, maybe a daemon instance can not talk to itself this way (I don't think I tried with two daemons, a second one running just for this purpose and replicating back asynchronously, which might work, now that I think of it). Alternative solution inside MGL would be to somehow explicitly use the grey/white/black-listing keywords with formatting strings (probably this also requires hack the code to make it usable like this).

As a solution outside, that sounds like an easy workaround, you could try to whip up a CGI script to tickle with urlcheck, that would detect if email is incoming or outgoing, and based on that either store the expected reverse direction credentials, or find that those already exist and autowl them (for mails coming back and perhaps sender-verification schemes). Maybe it might also hop on the replication protocol to feed the entries into MGL database directly.

Good luck, and hope this helps (and I hope you're luckier and more persistent than myself to find or implement a solution and share back),
Jim Klimov
--
Typos courtesy of K-9 Mail on my Samsung Android

10 \u0441\u0435\u043d\u0442\u044f\u0431\u0440\u044f 2016�\u0433. 11:50:50 CEST, "Marcus Schopen lists-yahoogroups@... [milter-greylist]" <milter-greylist@yahoogroups.com> \u043f\u0438\u0448\u0435\u0442:
>Hi,
>
>any best practice how to autowhitelist recipients from outgoing smtp 
>relay on the incoming relay? On the incoming relay one could use 
>urlcheck (any script available?), but how catch recipients from the 
>outgoing relay using milter-greylist and make it available for the 
>milter-greylist on the incoming server?
>
>Ciao
>Marcus

Hi,

As I mentioned, I was looking for something like this (lazily) and did not find. But it was a few years back, and it is a nice to have (really nice) but not life-critical feature so we've managed without and users got used to the current behavior (that initial comms with a remote domain may lag). Our MGL rules use a scoring system, so for reasonably well-managed domains the greylisting delay is either 2 or 25 minutes which is often (alas not always) negligible; suspicious domains (mostly bad matching of DNS entries or consumerish PTR names, poor SPF, workstation OSes according to p0f, etc.) go into hours range so DNSRBLs have chance to speak up. Very few are blacklisted outright by MGL from the first connection, so whatever trickle passes the initial filter goes to spamassassin (which is cheaper than bayesing the whole big incoming stream) that marks suspicious content and the mailer puts messages into spam folders. Little is lost, few spams get into inbox unmarked, users ar!
 e happy.

Yet regarding the feature to pre-whitelist recipients as we send them our emails, I wanted to do something inside MGL code, such as to use the replication protocol to query and feed the grey/autowhite database. For some reason this did not work for me, maybe a daemon instance can not talk to itself this way (I don't think I tried with two daemons, a second one running just for this purpose and replicating back asynchronously, which might work, now that I think of it). Alternative solution inside MGL would be to somehow explicitly use the grey/white/black-listing keywords with formatting strings (probably this also requires hack the code to make it usable like this).

As a solution outside, that sounds like an easy workaround, you could try to whip up a CGI script to tickle with urlcheck, that would detect if email is incoming or outgoing, and based on that either store the expected reverse direction credentials, or find that those already exist and autowl them (for mails coming back and perhaps sender-verification schemes). Maybe it might also hop on the replication protocol to feed the entries into MGL database directly.

You also asked "how to learn of those outgoing mails"? That's easy - run MGL on the outgoing relay (for smaller orgs it may be same as incoming) and have the MTA (sendmail) milter the outgoing/authenticated stream too. Usually there is an ACL early in MGL rules to skip processing (static whitelist) for email that originated locally - but you can also add a few rules on messages that match this ACL (e.g. to call urlcheck or the new keyword in MGL) first, and then whitelist them.

Good luck, and hope this helps (and I hope you're luckier and more persistent than myself to find or implement a solution and share back),
Jim Klimov
--
Typos courtesy of K-9 Mail on my Samsung Android
--
Typos courtesy of K-9 Mail on my Samsung Android

Hi Jim,

On 2016-09-12 06:59, Jim Klimov jimklimov@... [milter-greylist] 
wrote:
[...]
> Hi,
> 
> As I mentioned, I was looking for something like this (lazily) and did
> not find. But it was a few years back, and it is a nice to have
> (really nice) but not life-critical feature so we've managed without
> and users got used to the current behavior (that initial comms with a
> remote domain may lag). Our MGL rules use a scoring system, so for
> reasonably well-managed domains the greylisting delay is either 2 or
> 25 minutes which is often (alas not always) negligible; suspicious
> domains (mostly bad matching of DNS entries or consumerish PTR names,
> poor SPF, workstation OSes according to p0f, etc.) go into hours range
> so DNSRBLs have chance to speak up. Very few are blacklisted outright
> by MGL from the first connection, so whatever trickle passes the
> initial filter goes to spamassassin (which is cheaper than bayesing
> the whole big incoming stream) that marks suspicious content and the
> mailer puts messages into spam folders. Little is lost, few spams get
> into inbox unmarked, users ar!
>  e happy.
> 
> Yet regarding the feature to pre-whitelist recipients as we send them
> our emails, I wanted to do something inside MGL code, such as to use
> the replication protocol to query and feed the grey/autowhite
> database. For some reason this did not work for me, maybe a daemon
> instance can not talk to itself this way (I don't think I tried with
> two daemons, a second one running just for this purpose and
> replicating back asynchronously, which might work, now that I think of
> it). Alternative solution inside MGL would be to somehow explicitly
> use the grey/white/black-listing keywords with formatting strings
> (probably this also requires hack the code to make it usable like
> this).
> 
> As a solution outside, that sounds like an easy workaround, you could
> try to whip up a CGI script to tickle with urlcheck, that would detect
> if email is incoming or outgoing, and based on that either store the
> expected reverse direction credentials, or find that those already
> exist and autowl them (for mails coming back and perhaps
> sender-verification schemes). Maybe it might also hop on the
> replication protocol to feed the entries into MGL database directly.
> 
> Good luck, and hope this helps (and I hope you're luckier and more
> persistent than myself to find or implement a solution and share
> back),
> Jim Klimov

Thanks for your time to respond that considerably. I was rethinking my 
milter-greylist setup in these days, searched the mailinglist to get 
some ideas how to better fight spam using milter-greylist. A 
pre-whitelisting would really nice, yes. I will try to figure out a 
solution without code modification.

Beside that, I'd like to ask in the round, what concepts for fighting 
spam are you using, especially what kind of milter-greylist ACLs are 
helpful?

This is basically what I do:

1. sendmail rejects unknown users using virtusertable or access or 
mimedefang's md_check_against_smtp_server feature

2. milter-greylist ACLs in order:

2.1. individual whitelisting lists on from, rcpt, relay

2.2. individual blacklist lists on from, rcpt, relay

2.3. blacklisting if a host is listed on _two_ of the following RBLs

    zen.spamhaus.org
    ix.dnsbl.manitu.net
    hostkarma.junkemailfilter.com
    db.wpbl.info
    bl.mailspike.net

    The Blacklist Monitor on 
http://www.intra2net.com/de/support/antispam/ is good help to check 
accuracy and overlapping of different RBLs.

    Any other good RBLs?

2.3. longer greylisting for 3 hours / 1 day autowhite if a host passes 
barrier 2.3. and is listed on only _one_ of the above RLBs

2.4. greylisting hosts without or not matching rDNS for 3 hours / 1 day 
autowhite

2.5. some moderate geoip greylisting

2.6. last barrier is standard greylisting for 15 minutes / 60 days 
autowhite

2.7. whitelist default

3. Content filtering with mimedefang (spamassassin/clamav)



Ciao
Marcus

Marcus Schopen lists-yahoogroups@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

> any best practice how to autowhitelist recipients from outgoing smtp 
> relay on the incoming relay? 

Looking at the sources, it seems we have an undocumented mx clause that
matches if sender IP is MX for sender address. You need to configure
with --enable-mx and I do not know if the code works or even builds.

If someone explores that hidden code and get it working, please
contribute an update to greylist.conf(5) man page.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...