Milter-greylist

A while ago I was suggested to use 'helo not "%d"' constructs
to verify if DNS PTR and HELO names of a remote relay agree.
Unfortunately, either I do something wrong, or it misfires
too often. With a config snippet like this:

racl continue \
     not helo "%d" \
     set $dnsscore+=10 \
     log "(%i SP:+10) %P{conninfo}: Malformed HELO (HELO '%h' does not 
match reverse DNS '%d')" \
     set $msgDNS="Malformed HELO (HELO '%h' does not match reverse DNS 
'%d')"


... I often see logs like these, which (seem to) mention the
same string in both HELO and DNSPTR parts:

Aug 29 01:06:14 dao.virt.cos.ru milter-greylist: [ID 536703 mail.info] 
(216.227.218.80 SP:+10) IP:'216.227.218.80' DOMAIN:'joy.lunarbreeze.com' 
HELO:'joy.lunarbreeze.com' FROM:'gVIwyErHMkoOi@...' 
RCPT:'bugaev@...': Malformed HELO (HELO 'joy.lunarbreeze.com' does 
not match reverse DNS 'joy.lunarbreeze.com')

...and I can't find any mismatches in real DNS (perhaps dot-ended
names, etc.) Any ideas on debugging this issue?
NOTE: Okay, the example above is a spammer example, but I have
such issues with big-name providers as well which are unlikely to
be lax in their DNS setups. Their logs rolled off-screen at the
moment ;(


..These are possibly more reasonable "mismatches", as far as
case-sensitivity might be concerned (although not really
relevant for DNS):

Aug 29 04:57:51 dao sendmail[19426]: [ID 801593 mail.notice] 
s7O0vmQv019426: MGL-TEMPFAIL-DNS-120: DNScheck: likely spam-source; 
repost from registered SMTP server, or contact 
bypass-antispam@.... Malformed HELO (HELO 
'DUB004-OMC1S3.hotmail.com' does not match reverse DNS 
'dub004-omc1s3.hotmail.com') SPF pass


...And these are proper hits that the rule is made for:

Aug 29 00:38:09 relay-mta milter-greylist: [ID 354941 mail.info] 
(89.240.10.108 SP:+10) IP:'89.240.10.108' 
DOMAIN:'host-89-240-10-108.static.as13285.net' 
HELO:'mail.netcellsolutions.com' FROM:'gzivhsom@...' 
RCPT:'hel@...': Malformed HELO (HELO 'mail.netcellsolutions.com' does 
not match reverse DNS 'host-89-240-10-108.static.as13285.net')



Thanks,
Jim Klimov

Jim Klimov jimklimov@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

> ..These are possibly more reasonable "mismatches", as far as
> case-sensitivity might be concerned (although not really
> relevant for DNS):

I think you found a bug: acl_helo_strstr() uses strstr(). It is case
sensitive. That does not explain your first example, though. 

Moreover, it probably match things that should not be matched (substring
match can occur anywhere, and not only right hand side). But changing
this should be optional, as some may rely on today's behavior.


-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

EHLO all,

Better late than never, they say... got a bit of quiet time late
in the year and finally took a look at the problem with misfiring
`not helo "%d"` clauses. Well, the case-sensitivity was just a rare
tip of the iceberg - thanks to my other rules, this one was not
often evaluated, although enough to cause a raised eyebrow when
some legitimate-looking relay was greylisted due to a mismatch with
seemingly identical HELO and DNSPTR names.

Guess what?

The "%d" was tested (and mis-matched) literally.

Patch attached to call format string expansion (any defined format)
in the helo_strstr clause. And case-insensitivity while we are at it.

This is being tested for the last night on our relay, seems working
as expected now.

HTH,
Jim Klimov


2014-08-29 12:54, Jim Klimov jimklimov@... [milter-greylist] wrote:

> A while ago I was suggested to use 'helo not "%d"' constructs
> to verify if DNS PTR and HELO names of a remote relay agree.
> Unfortunately, either I do something wrong, or it misfires
> too often. With a config snippet like this:
>
> racl continue \
> not helo "%d" \
> set $dnsscore+=10 \
> log "(%i SP:+10) %P{conninfo}: Malformed HELO (HELO '%h' does not
> match reverse DNS '%d')" \
> set $msgDNS="Malformed HELO (HELO '%h' does not match reverse DNS
> '%d')"
>
> ... I often see logs like these, which (seem to) mention the
> same string in both HELO and DNSPTR parts:
>
> Aug 29 01:06:14 dao.virt.cos.ru milter-greylist: [ID 536703 mail.info]
> (216.227.218.80 SP:+10) IP:'216.227.218.80' DOMAIN:'joy.lunarbreeze.com'
> HELO:'joy.lunarbreeze.com' FROM:'gVIwyErHMkoOi@...'
> RCPT:'bugaev@...': Malformed HELO (HELO 'joy.lunarbreeze.com' does
> not match reverse DNS 'joy.lunarbreeze.com')
>
> ...and I can't find any mismatches in real DNS (perhaps dot-ended
> names, etc.) Any ideas on debugging this issue?
> NOTE: Okay, the example above is a spammer example, but I have
> such issues with big-name providers as well which are unlikely to
> be lax in their DNS setups. Their logs rolled off-screen at the
> moment ;(
>
> ..These are possibly more reasonable "mismatches", as far as
> case-sensitivity might be concerned (although not really
> relevant for DNS):
>
> Aug 29 04:57:51 dao sendmail[19426]: [ID 801593 mail.notice]
> s7O0vmQv019426: MGL-TEMPFAIL-DNS-120: DNScheck: likely spam-source;
> repost from registered SMTP server, or contact
> bypass-antispam@.... Malformed HELO (HELO
> 'DUB004-OMC1S3.hotmail.com' does not match reverse DNS
> 'dub004-omc1s3.hotmail.com') SPF pass
>
> ...And these are proper hits that the rule is made for:
>
> Aug 29 00:38:09 relay-mta milter-greylist: [ID 354941 mail.info]
> (89.240.10.108 SP:+10) IP:'89.240.10.108'
> DOMAIN:'host-89-240-10-108.static.as13285.net'
> HELO:'mail.netcellsolutions.com' FROM:'gzivhsom@...'
> RCPT:'hel@...': Malformed HELO (HELO 'mail.netcellsolutions.com' does
> not match reverse DNS 'host-89-240-10-108.static.as13285.net')
>
> Thanks,
> Jim Klimov

Sorry, my bad: posted a patch against my earlier commit, not the
bare upstream release. The one attached now should apply cleanly
to milter-greylist-4.5.12 ;)

Hacky Christmas,
Jim Klimov


2014-12-23 14:30, Jim Klimov \ufffd\ufffd\ufffd\ufffd\ufffd:
> EHLO all,
>
> Better late than never, they say... got a bit of quiet time late
> in the year and finally took a look at the problem with misfiring
> `not helo "%d"` clauses. Well, the case-sensitivity was just a rare
> tip of the iceberg - thanks to my other rules, this one was not
> often evaluated, although enough to cause a raised eyebrow when
> some legitimate-looking relay was greylisted due to a mismatch with
> seemingly identical HELO and DNSPTR names.
>
> Guess what?
>
> The "%d" was tested (and mis-matched) literally.
>
> Patch attached to call format string expansion (any defined format)
> in the helo_strstr clause. And case-insensitivity while we are at it.
>
> This is being tested for the last night on our relay, seems working
> as expected now.
>
> HTH,
> Jim Klimov
>
>
> 2014-08-29 12:54, Jim Klimov jimklimov@... [milter-greylist] wrote:
>> A while ago I was suggested to use 'helo not "%d"' constructs
>> to verify if DNS PTR and HELO names of a remote relay agree.
>> Unfortunately, either I do something wrong, or it misfires
>> too often. With a config snippet like this:
>>
>> racl continue \
>> not helo "%d" \
>> set $dnsscore+=10 \
>> log "(%i SP:+10) %P{conninfo}: Malformed HELO (HELO '%h' does not
>> match reverse DNS '%d')" \
>> set $msgDNS="Malformed HELO (HELO '%h' does not match reverse DNS
>> '%d')"
>>
>> ... I often see logs like these, which (seem to) mention the
>> same string in both HELO and DNSPTR parts:
>>
>> Aug 29 01:06:14 dao.virt.cos.ru milter-greylist: [ID 536703 mail.info]
>> (216.227.218.80 SP:+10) IP:'216.227.218.80' DOMAIN:'joy.lunarbreeze.com'
>> HELO:'joy.lunarbreeze.com' FROM:'gVIwyErHMkoOi@...'
>> RCPT:'bugaev@...': Malformed HELO (HELO 'joy.lunarbreeze.com' does
>> not match reverse DNS 'joy.lunarbreeze.com')
>>
>> ...and I can't find any mismatches in real DNS (perhaps dot-ended
>> names, etc.) Any ideas on debugging this issue?
>> NOTE: Okay, the example above is a spammer example, but I have
>> such issues with big-name providers as well which are unlikely to
>> be lax in their DNS setups. Their logs rolled off-screen at the
>> moment ;(
>>
>> ..These are possibly more reasonable "mismatches", as far as
>> case-sensitivity might be concerned (although not really
>> relevant for DNS):
>>
>> Aug 29 04:57:51 dao sendmail[19426]: [ID 801593 mail.notice]
>> s7O0vmQv019426: MGL-TEMPFAIL-DNS-120: DNScheck: likely spam-source;
>> repost from registered SMTP server, or contact
>> bypass-antispam@.... Malformed HELO (HELO
>> 'DUB004-OMC1S3.hotmail.com' does not match reverse DNS
>> 'dub004-omc1s3.hotmail.com') SPF pass
>>
>> ...And these are proper hits that the rule is made for:
>>
>> Aug 29 00:38:09 relay-mta milter-greylist: [ID 354941 mail.info]
>> (89.240.10.108 SP:+10) IP:'89.240.10.108'
>> DOMAIN:'host-89-240-10-108.static.as13285.net'
>> HELO:'mail.netcellsolutions.com' FROM:'gzivhsom@...'
>> RCPT:'hel@...': Malformed HELO (HELO 'mail.netcellsolutions.com' does
>> not match reverse DNS 'host-89-240-10-108.static.as13285.net')
>>
>> Thanks,
>> Jim Klimov


-- 


+============================================================+
|                                                            |
| \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd,                                 Jim Klimov |
| \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd                                   CTO |
| \ufffd\ufffd\ufffd "\ufffd\ufffd\ufffd \ufffd \ufffd\ufffd"                                  JSC COS&HT |
|                                                            |
| +7-903-7705859 (cellular)          mailto:jimklimov@... |
|                        CC:admin@...,jimklimov@... |
+============================================================+
| ()  ascii ribbon campaign - against html mail              |
| /\                        - against microsoft attachments  |
+============================================================+

On Tue, Dec 23, 2014 at 02:30:02PM +0300, Jim Klimov jimklimov@... [milter-greylist] wrote:
> Patch attached to call format string expansion (any defined format)
> in the helo_strstr clause. And case-insensitivity while we are at it.

Thanks, but could you remove the debug printf?

-- 
Emmanuel Dreyfus
manu@...

2014-12-23 16:27, Emmanuel Dreyfus manu@... [milter-greylist] \u043f\u0438\u0448\u0435\u0442:
> On Tue, Dec 23, 2014 at 02:30:02PM +0300, Jim Klimov jimklimov@...
> [milter-greylist] wrote:
>  > Patch attached to call format string expansion (any defined format)
>  > in the helo_strstr clause. And case-insensitivity while we are at it.
>
> Thanks, but could you remove the debug printf?

Yes, no problem. It was there when I barged in, so I left it there.
I had no idea who would use it - I didn't see this in the outputs -
so chose to leave it as it was.

By the way, am I correct that there is currently no configurable
level of debugging verbosity - just an on/off toggle?

Finally, regarding substring matches: I saw this today:

Dec 23 16:39:12 relay-mta milter-greylist: [ID 421884 mail.debug]
   acl_helo_strstr -> 'ximea.com'/'mail.ximea.com' (lowercase of
   'ximea.com'/'mail.ximea.com') - MISMATCH

Indeed, the longer test string is not part of the shorter one.

Would it break any assumptions possibly used in the field today,
if we extend the test to try it both ways, roughly speaking:
   strstr(helo,dns)||strstr(dns,helo)
this?

Thanks,
Jim Klimov

New revision attached, without printf and with two strstr substring
matches (A in B or B in A).

Merry H-Mas,
Jim Klimov


2014-12-23 17:02, Jim Klimov jimklimov@... [milter-greylist] \u043f\u0438\u0448\u0435\u0442:

> 2014-12-23 16:27, Emmanuel Dreyfus manu@... [milter-greylist] \u043f\u0438\u0448\u0435\u0442:
>  > On Tue, Dec 23, 2014 at 02:30:02PM +0300, Jim Klimov jimklimov@...
>  > [milter-greylist] wrote:
>  > > Patch attached to call format string expansion (any defined format)
>  > > in the helo_strstr clause. And case-insensitivity while we are at it.
>  >
>  > Thanks, but could you remove the debug printf?
>
> Yes, no problem. It was there when I barged in, so I left it there.
> I had no idea who would use it - I didn't see this in the outputs -
> so chose to leave it as it was.
>
> By the way, am I correct that there is currently no configurable
> level of debugging verbosity - just an on/off toggle?
>
> Finally, regarding substring matches: I saw this today:
>
> Dec 23 16:39:12 relay-mta milter-greylist: [ID 421884 mail.debug]
> acl_helo_strstr -> 'ximea.com'/'mail.ximea.com' (lowercase of
> 'ximea.com'/'mail.ximea.com') - MISMATCH
>
> Indeed, the longer test string is not part of the shorter one.
>
> Would it break any assumptions possibly used in the field today,
> if we extend the test to try it both ways, roughly speaking:
> strstr(helo,dns)||strstr(dns,helo)
> this?
>
> Thanks,
> Jim Klimov
>

Jim Klimov jimklimov@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

> New revision attached, without printf and with two strstr substring
> matches (A in B or B in A).

I refactorized a bit. Tell me if it works fine and I'll commit:
http://ftp.espci.fr/shadow/manu/helo.patch

- space and comment style
- 80 char/line
- use sizeof(variable) instead of copying its size
- use same size for lh_priv and priv->priv_helo
- don't reuse a variable fortwo different usages
- free allocated  format string

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

2014-12-24 4:55, manu@... [milter-greylist] \u043f\u0438\u0448\u0435\u0442:
> Jim Klimov jimklimov@... [milter-greylist]
> <milter-greylist@yahoogroups.com> wrote:
>
>  > New revision attached, without printf and with two strstr substring
>  > matches (A in B or B in A).
>
> I refactorized a bit. Tell me if it works fine and I'll commit:
> http://ftp.espci.fr/shadow/manu/helo.patch
>
> - space and comment style
> - 80 char/line
> - use sizeof(variable) instead of copying its size
> - use same size for lh_priv and priv->priv_helo
> - don't reuse a variable fortwo different usages
> - free allocated format string

Thanks for the revision, looks good - almost: the check in LOG_DEBUG
now got inverted, e.g.:

Dec 24 16:18:51 myrelay milter-greylist: [ID 238697 mail.debug]
  acl_helo_strstr: "smtp461.unisnd.com"/"smtp461.unisnd.com" (lowercase
  of "smtp461.unisnd.com"/"smtp461.unisnd.com") => MISMATCH

This is cosmetic, but still should be fixed.

I see that you've removed the explicit checks like "==0" or '\0'
which makes the result somewhat more system-dependent (i.e. things
like the definition of an end-of-string character or a NULL pointer
are not necessarily series of zero bits, and there are historic
precedents ages ago when some other numeric values were used - so
we were trained to be more explicit in coding).

Even in this case of inverted results, for C the zero return is
"false" while for shell scripts it is "success"... both ways feel
"natural" but I get confused when switching between languages.
So I'd rather write explicit result-checks ;)

With that inversion of the logged message in mind, seems good to go.

Thanks, and a merry X-Mas,
Jim Klimov

Also tested, by chance, that the lowercasing logic does work:

...MGL-TEMPFAIL-DNS-480: DNScheck: probable spam-source; repost from
  registered SMTP server, or contact bypass-antispam@....
  Malformed  HELO (HELO 'SDC-EXEDGE-01.dp.mosreg.ru' does not match
  reverse DNS '[213.85.255.248]') SPF pass

Dec 24 21:04:59 relay-mta milter-greylist: [ID 238697 mail.debug]
  acl_helo_strstr: "sdc-exedge-01.dp.mosreg.ru"/"[213.85.255.248]"
  (lowercase of "SDC-EXEDGE-01.dp.mosreg.ru"/"[213.85.255.248]") =>
  MISMATCH


Jim