Milter-greylist

I wonder if it makes sense to test "spf self" with 127.0.0.1 for
example, in case neither {if_addr} nor "localaddr" are set, instead
of bailing out completely? After all, we are checking that the SPF
is so loose it allows unexpected addresses?..

//Jim Klimov

Jim Klimov <jimklimov@...> wrote:

> I wonder if it makes sense to test "spf self" with 127.0.0.1 for
> example, in case neither {if_addr} nor "localaddr" are set, instead
> of bailing out completely? After all, we are checking that the SPF
> is so loose it allows unexpected addresses?..

The spammer may use a domain that passes 127.0.0.1. It is less easy to
prepare a domain that will mathc the MX of all the target recipient.
-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On 2013-08-12 05:20, manu@... wrote:
> Jim Klimov <jimklimov@... <mailto:jimklimov%40cos.ru>> wrote:
>
>  > I wonder if it makes sense to test "spf self" with 127.0.0.1 for
>  > example, in case neither {if_addr} nor "localaddr" are set, instead
>  > of bailing out completely? After all, we are checking that the SPF
>  > is so loose it allows unexpected addresses?..
>
> The spammer may use a domain that passes 127.0.0.1. It is less easy to
> prepare a domain that will mathc the MX of all the target recipient.

But isn't the point of the "spf self" test to detect that a spammer
owns or abuses a domain whose SPF record allows too much - such as
our server's address or loopback net, or "+all" completely - so that
we would not trust a "pass" result from this domain?

BTW, how is it supposed to work, given that currently there are no
"if" or "goto" structures in milter-greylist config language? :)
Should some "and-or-xor" evaluate both results in one rule?

The more I think about it, the less I am certain of my understanding.
It seems that now the SPF checks should be near the end of config,
just before the "greylist default" line - and after all our explicit
white and black lists, or custom-length greylists like those I posted
based on regex matching of "dialup-like hostnames"; so that we blacklist
"fail"ures, greylist-long "softfail" and "spf self" hits, whitelist
"pass"es and default the others?

Is there a way to permit "spf pass" EXCEPT "spf self" hits early
in the rules, and have those "spf self" hits fall through and be
subjected to all the other tests like regex - not plain greylisted?

On a side note, the libspf2 headers quote the SPF standard that the
mailers "SHOULD" add a header about processed SPF - verdict and
details (more details at http://www.openspf.org/SPF_Received_Header).

I've tried "addheader" on an spf rule, but milter-greylist crashed
for some reason when this line was in the config; the header text
was rather short (shouldn't exceed 2048 bytes by a long shot),
though it had some format strings; I did not debug any further.

On another note, I'd like to log the remote host's IP, HELO, DNS PTR
and FROM/RCPT addresses into X-Greylist headers of each processed
message. Is it possible to add via config - without hacking into
the "report all" source code?

As I've asked earlier - are there any ways to unconditionally inject
SMTP responses (like "msg") and/or headers without dependency on a
particular ACL rule hit and its msg/addheader parameters?

Thanks,
//Jim

Jim Klimov <jimklimov@...> wrote:

> As I've asked earlier - are there any ways to unconditionally inject
> SMTP responses (like "msg") and/or headers without dependency on a
> particular ACL rule hit and its msg/addheader parameters?

racl continue msg "stuff" ?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On 2013-08-12 08:27, Emmanuel Dreyfus wrote:
> Jim Klimov <jimklimov@...> wrote:
>
>> As I've asked earlier - are there any ways to unconditionally inject
>> SMTP responses (like "msg") and/or headers without dependency on a
>> particular ACL rule hit and its msg/addheader parameters?
>
> racl continue msg "stuff" ?

Something like that, yes :)

There is nothing like it here:
http://hcpnet.free.fr/milter-greylist/greylist.conf.5.txt
...though now that you said it, I see it in the source code
(though not in the source manpage either).

Thanks, I'll give it a shot,
//Jim Klimov

Jim Klimov <jimklimov@...> wrote:

> Is there a way to permit "spf pass" EXCEPT "spf self" hits early
> in the rules, and have those "spf self" hits fall through and be
> subjected to all the other tests like regex - not plain greylisted?

Have you tried this?
rcpt whitelist not spf self spf pass

> On another note, I'd like to log the remote host's IP, HELO, DNS PTR
> and FROM/RCPT addresses into X-Greylist headers of each processed
> message. Is it possible to add via config - without hacking into
> the "report all" source code?

I beleive you can with format strings:
%i sender IP address
%h HELLO string
%d sender DNS address
%f FROM
%r RCPT (beware of multi-recipient scenarios) 

> As I've asked earlier - are there any ways to unconditionally inject
> SMTP responses (like "msg") and/or headers without dependency on a
> particular ACL rule hit and its msg/addheader parameters?

racl continue msg "stuff"

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On 2013-08-13 05:54, manu@... wrote:
> Jim Klimov <jimklimov@... <mailto:jimklimov%40cos.ru>> wrote:
>
>  > Is there a way to permit "spf pass" EXCEPT "spf self" hits early
>  > in the rules, and have those "spf self" hits fall through and be
>  > subjected to all the other tests like regex - not plain greylisted?
>
> Have you tried this?
> rcpt whitelist not spf self spf pass

Thanks, not yet, I'll give it a shot. Now I'd have to also forge
a very permissive SPF domain (which is not pre-whitelisted as the
server's "$self"), I guess ;)


>  > On another note, I'd like to log the remote host's IP, HELO, DNS PTR
>  > and FROM/RCPT addresses into X-Greylist headers of each processed
>  > message. Is it possible to add via config - without hacking into
>  > the "report all" source code?
>
> I beleive you can with format strings:
> %i sender IP address
> %h HELLO string
> %d sender DNS address
> %f FROM
> %r RCPT (beware of multi-recipient scenarios)
>
>  > As I've asked earlier - are there any ways to unconditionally inject
>  > SMTP responses (like "msg") and/or headers without dependency on a
>  > particular ACL rule hit and its msg/addheader parameters?
>
> racl continue msg "stuff"

Ok, I'll also try and see how it gos :)

Is it currently possible to cache the SPF test result and refer to it
with some format string (I asked earlier if there is such caching or
if DNS lookups and full libspf processing are done for each instance
of the SPF keyword - with the same result for given relay, domain and
from)?

Thnaks,
//Jim

On 2013-08-12 09:27, Jim Klimov wrote:
> On 2013-08-12 08:27, Emmanuel Dreyfus wrote:
>  > racl continue msg "stuff" ?
>
> Something like that, yes :)
>
> There is nothing like it here:
> http://hcpnet.free.fr/milter-greylist/greylist.conf.5.txt
> ...though now that you said it, I see it in the source code
> (though not in the source manpage either).

Sorry for the noise, my tools have failed me. The keyword is
described in the source manpage. For some reason, the Midnight
Commander viewer did not find it there, though some other text
searches in the same manpage did work :\

//Jim

On 2013-08-13 10:38, Jim Klimov wrote:
>  > racl continue msg "stuff"


For some reason, this gives me an error with whatever I try,
from trivial
   racl continue msg "Test"
to
   racl continue addheader "X-Greylist: inspected by %V for IP:'%i' 
DOMAIN:'%d' HELO:'%h' FROM:'%f' RCPT:'%r'"

Startup of the milter just aborts with:

   config error at line 240: syntax error

Commenting away the line lets it load normally.

//Jim

On 2013-08-12 06:51, Jim Klimov wrote:
> On a side note, the libspf2 headers quote the SPF standard that the
> mailers "SHOULD" add a header about processed SPF - verdict and
> details (more details at http://www.openspf.org/SPF_Received_Header).
>
> I've tried "addheader" on an spf rule, but milter-greylist crashed
> for some reason when this line was in the config; the header text
> was rather short (shouldn't exceed 2048 bytes by a long shot),
> though it had some format strings; I did not debug any further.

I've traced the crash to the recipient domain - format string macros
'%r' (empty in spf addheader) and '%sr' (causes the crash). This does
happen with both racl and dacl rules, and the syslog debug entry for
the rule-hit contains the sender address among strings added by the
milter-greylist itself. Interestingly, the crash occurs (every time)
a couple of seconds after the message was fully received and processed
and logged. The message itself is not handed off back to the Messaging
Server and is not delivered into the mailbox.

I have used %r in other rules and it worked as expected...

So there a couple of things to do:
1) revise that domain/username format substrings do not crash with
empty input;
2) understand why the recipient is empty in this context?..

HTH,
//Jim Klimov

On Tue, Aug 13, 2013 at 01:02:33PM +0200, Jim Klimov wrote:
> from trivial
>   racl continue msg "Test"
> to
>   racl continue addheader "X-Greylist: inspected by %V for IP:'%i'
> DOMAIN:'%d' HELO:'%h' FROM:'%f' RCPT:'%r'"
> 
>   config error at line 240: syntax error

I discover that command clause are not accepted if there are no filter
clauses.  This will work:
racl continue from /.*/ addheader "X-Greylist: inspected ..."

Feel free to track that bug down and submit a patch fixing it.

-- 
Emmanuel Dreyfus
manu@...

On 2013-08-13 05:54, manu@... wrote:
> Jim Klimov <jimklimov@... <mailto:jimklimov%40cos.ru>> wrote:
>
>  > Is there a way to permit "spf pass" EXCEPT "spf self" hits early
>  > in the rules, and have those "spf self" hits fall through and be
>  > subjected to all the other tests like regex - not plain greylisted?
>
> Have you tried this?
> rcpt whitelist not spf self spf pass


I've seen a number of strange entries like the one below after enabling
the rule as suggested above.

The domain's SPF does include this IP address as a trusted source,
so this ACL rule's trust is well earned (and the message is delivered).

But why is this logged: "skipping greylist because sender is not
SPF-compliant"? Where is it *not* compliant? The sender domain has
an SPF rule and the sender host matches it as a direct "pass".

Details follow...

Aug 13 16:31:02 ucs milter-greylist: [ID 471652 mail.debug] Incoming 
connection from host '[46.165.235.131]'
Aug 13 16:31:02 ucs milter-greylist: [ID 308029 mail.debug] Got an 
unresolved host name [46.165.235.131], will try to resolve
Aug 13 16:31:02 ucs milter-greylist: [ID 682236 mail.debug] Requesting 
PTR entry for 131.235.165.46.in-addr.arpa.
Aug 13 16:31:02 ucs milter-greylist: [ID 356585 mail.debug] res_nquery 
failed: Error 0
Aug 13 16:31:02 ucs milter-greylist: [ID 283800 mail.debug] Retaining 
name '[46.165.235.131]'
Aug 13 16:31:02 ucs milter-greylist: [ID 703198 mail.debug] 
0MRG0017UYROPJ00: addr = [46.165.235.131][46.165.235.131], from = 
<admin@...>, rcpt = <taisia@...>
Aug 13 16:31:02 ucs milter-greylist: [ID 503426 mail.debug] SPF return 
code 4 (pass, result 0)
Aug 13 16:31:02 ucs milter-greylist: [ID 859003 mail.debug] SPF lookup 
performed in 0.183734s
Aug 13 16:31:02 ucs milter-greylist: [ID 503426 mail.debug] SPF return 
code 2 (pass, result 1)
Aug 13 16:31:02 ucs milter-greylist: [ID 859003 mail.debug] SPF lookup 
performed in 0.010663s
Aug 13 16:31:02 ucs milter-greylist: [ID 561399 mail.debug] Mail 
from=<admin@...>, rcpt=<taisia@...>, 
addr=[46.165.235.131][46.165.235.131] is matched by entry racl 1477 
whitelist not spf self spf pass [addheader "Received-SPF: whitelist not 
spf self spf pass"] [maxpeek -1]
Aug 13 16:31:02 ucs milter-greylist: [ID 902575 mail.info] 
0MRG0017UYROPJ00: skipping greylist because sender is not SPF-compliant, 
(from=<admin@...>, rcpt=<taisia@...>, 
addr=[46.165.235.131][46.165.235.131]) ACL 1477



# nslookup -q=txt xbosnomer.net 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
xbosnomer.net   text = "v=spf1 ip4:46.165.235.131 a mx ~all"

# nslookup -q=a xbosnomer.net 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   xbosnomer.net
Address: 46.165.235.131

# nslookup 46.165.235.131 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

** server can't find 131.235.165.46.in-addr.arpa.: NXDOMAIN

On 2013-08-13 14:34, Emmanuel Dreyfus wrote:
> On Tue, Aug 13, 2013 at 01:52:44PM +0200, Jim Klimov wrote:
>> So there a couple of things to do:
>> 1) revise that domain/username format substrings do not crash with
>> empty input;
>> 2) understand why the recipient is empty in this context?..
>
> Does that mean milter-greylist-4.4.3-p0f-noCrash.patch is not
> finalized yet?

No, that is a different crash in a different location for different
reasons. Unlike p0f, it left no clues in the log, so it is my guess
from experimentation (trying different addheader strings) that the
culprit is with an empty '%r' in this context, and so on as detailed
in that letter.

> On Tue, Aug 13, 2013 at 12:07:33PM +0200, Jim Klimov wrote:
>> The main question is: if the "local address" is wrong in p0f.c:380,
>> should we really abort the whole program, or just not do further p0f
>> processing?
>
> Please send code snippet? I do not see what part you talk about.

I believe that since you have already seen and/or reviewed the
milter-greylist-4.4.3-p0f-noCrash.patch, you don't need that
snippet (it is the default case, where I changed exit to return).

>
>> Aug 13 10:07:05 ucs milter-greylist: [ID 447650 mail.error] unexpected AF
>
> milter-greylist was built without Ipv6 support?

I think not, should have IPv6. It just seems to happen that for
the Messaging Server's queue reprocessing, there is no address
set at all, since this is not happening in the context of a live
SMTP dialog. Maybe this is a bug on their behalf, but unlikely
to change soon anyway.

In general, maybe localaddr should have been used here too as a
fallback in a situation like this, but at least for p0f - an MTA's
offline reprocessing stage is anyway too late to try any (passive)
IP fingerprinting of the remote original host.

//Jim Klimov

Jim Klimov <jimklimov@...> wrote:

> I wonder if it makes sense to test "spf self" with 127.0.0.1 for
> example, in case neither {if_addr} nor "localaddr" are set, instead
> of bailing out completely? After all, we are checking that the SPF
> is so loose it allows unexpected addresses?..

Well, I am not sure we need to add black magic in order to catch the
situation where the admin forgot to set an option. 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On 2013-08-15 06:52, manu@... wrote:
> Jim Klimov <jimklimov@... <mailto:jimklimov%40cos.ru>> wrote:
>
>  > I wonder if it makes sense to test "spf self" with 127.0.0.1 for
>  > example, in case neither {if_addr} nor "localaddr" are set, instead
>  > of bailing out completely? After all, we are checking that the SPF
>  > is so loose it allows unexpected addresses?..
>
> Well, I am not sure we need to add black magic in order to catch the
> situation where the admin forgot to set an option.

Possibly not... at least, as long as we remind the admin to set a more
correct value for the option than we can guess and hardcode beforehand ;)

//Jim

Jim Klimov <jimklimov@...> wrote:

> The headers are added after milter-greylist processing (standard
> reports) and by the spf rules, so there may be some problem with the
> continue handling in particular. Any fast ideas before I have time
> to look at the code? :)

No idea, look at the code.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...