Milter-greylist

Hello all,

   FYI: for milter-greylist users who (want to) use p0f fingerprinting
and run under a Solaris-based OS.

   Over the past year I've applied assorted changes to p0f-3.06b in
order to enable its builds on Solaris 10 and OpenIndiana, as well as
integration with Solaris Service Management Framework (for further
integration with our relays' milter-greylist based antispam solution).
I've also hit a bug with the API connections not being closed after
the p0f-client disconnection, causing the failure of the p0f daemon
and/or its clients (milter-greylist) upon exhaustion of some resource.

   My research and discussion of these problems has been tracked on
the milter-greylist mailing list, and some of the patches were also
posted here... partially due to my laziness ;)

   I've unsuccesfully tried to post the changes to the original project 
author, but got no replies to any posts during the last year), and
ultimately decided to publish my fixes in an orderly bunch by forking
the repository which I've found at GitHub: https://github.com/p0f/p0f.
So the Solaris users who want p0f-3.06b can use my fork for now:
   https://github.com/jimklimov/p0f

   Still, I'd love to see this upstreamed someday, so that the vanilla
p0f (say, 3.06c?) can just work under Solaris :)

Thanks,
//Jim Klimov

On 2014-02-09 18:24, Jim Klimov wrote:
> Hello all,
>
> FYI: for milter-greylist users who (want to) use p0f fingerprinting
> and run under a Solaris-based OS.

Also some fixes/improvements were desired on the milter-greylist side.

I attached here the following:
* in the debug logs, don't just state the p0f query results, but do
   also report the IP address they pertain to - helps debugging a lot ;)
* try to reconnect to p0f once if "write failed" or "read failed",
   instead of bailing out immediately so that no p0f processing is done
   for this connection (i.e. if p0f daemon restarted)
* if the remote host is missing in the p0f cache, do a few retries,
   perhaps its packets will soon be processed by the daemon and show up.
   NOTE that this blocks milter-greylist until the loop completes
   and in case of at least Sendmail, this delays even the display
   of the SMTP banner.
* the clumsy part I need help with: configuration of the delay and
   count for the retry loop above. For some reason, my build refused to
   process the p0f_miss_retry_count and p0f_miss_retry_delay variables
   in the config file :( Possibly a late-night silly typo somewhere?

Thanks,
//Jim Klimov

On 2014-02-09 18:24, Jim Klimov wrote:
> FYI: for milter-greylist users who (want to) use p0f fingerprinting

I also found that if p0fsock is enabled, the filter is called - even
if there are no p0f ACLs requiring that. Moreover, this filter is
called at the beginning of connection-processing by milter-greylist
instead of the position where it is referenced by config file.

Can anything be done about this (to ensure orderly processing of
rules)?

Thanks,
//Jim Klimov

Jim Klimov <jimklimov@...> wrote:

> * if the remote host is missing in the p0f cache, do a few retries,<br>
>    perhaps its packets will soon be processed by the daemon and show up.<br>
>    NOTE that this blocks milter-greylist until the loop completes<br>
>    and in case of at least Sendmail, this delays even the display<br>
>    of the SMTP banner.<br>

I understand that sleep(3) will put the whole process to sleep, which
makes the milter unable to process other requests from sendmail.
sched_yield(3) seems to be the right way. Could you investigate?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On 2014-02-10 06:12, manu@... wrote:
> Jim Klimov <jimklimov@...> wrote:
>
>  > * if the remote host is missing in the p0f cache, do a few retries,<br>
>  > perhaps its packets will soon be processed by the daemon and show up.<br>
>  > NOTE that this blocks milter-greylist until the loop completes<br>
>  > and in case of at least Sendmail, this delays even the display<br>
>  > of the SMTP banner.<br>
>
> I understand that sleep(3) will put the whole process to sleep, which
> makes the milter unable to process other requests from sendmail.
> sched_yield(3) seems to be the right way. Could you investigate?

Hmmm, thanks

I'll take a look (hopefully tonight)

--Boundary_(ID_z4F1RdDzhgNp1chwB0jsCg)
Content-type: text/plain; charset=utf-8
Content-transfer-encoding: base64

V2VsbCwgYWNjb3JkaW5nIHRvIG1hbnBhZ2VzLCBzY2hlZF95aWVsZCgpIGRvZXMgbm90IHByb3Zp
ZGUgZm9yIGEgZGVsYXkgdmFsdWUuIEFuZCBhY2NvcmRpbmcgdG8gbWFueSBzb3VyY2VzLCBzbGVl
cCgpIHNob3VsZCBvbmx5IGFjdCBvbiBpdHMgdGhyZWFkLCBub3QgdGhlIHdob2xlIHByb2Nlc3Ms
IGF0IGxlYXN0IGluIHVuaXgtbGlrZSBzeXN0ZW1zLgoKU28gdW5sZXNzIHByb3ZlbiBvdGhlcndp
c2UsIHRoaXMgcGFydCBvZiB0aGUgc29sdXRpb24gc2hvdWxkIHdvcmsgd2l0aG91dCBkZWxheXMg
Zm9yIHVucmVsYXRlZCBjb25uZWN0aW9ucy4gSSBwbGFuIHRvIGV4cGVyaW1lbnQgd2l0aCBhIGRl
cGxveWVkIHJlbGF5IHRvbmlnaHQgdGhvdWdoLgoKVGhlIHF1ZXN0aW9ucyBvbiBjb25maWcgdG9n
Z2xlcyBhbmQgb24gY2FsbGluZyBwMGYgb25seSB3aGVuIHJlcXVlc3RlZCB0byAoYXQgbGVhc3Qs
IGluIHRoZSBjb25maWcgcG9pbnQgd2hlcmUgcDBmc29jayBpcyBkZWZpbmVkKSwgbm90IHNvb25l
ciwgZG8gcmVtYWluIGluIHBsYWNlIDspCgpGb3IgZXhhbXBsZSwgbm93IGkgaGF2ZSBtaWx0ZXIt
Z3JleWxpc3QgaW5xdWlyaW5nIHAwZiBhYm91dCBMQU4gaG9zdHMgd2hpY2ggYXJlIGV4ZW1wdCBm
cm9tIHAwZCdzIGJwZiBmaWx0ZXIgZGVmaW5pdGlvbiBhbmQgY2F1c2UgdGhlIHJldHJ5IGxvb3Ag
Zm9yIGxvY2FsIHN1Ym1pc3Npb25zLCBkZXNwaXRlIHRoZSBmYWN0IHRoYXQgYW55dGhpbmcgYWJv
dXQgcDBmIGlzIG1lbnRpb25lZCB3YXkgYWZ0ZXIgbWFudWFsIHdoaXRlbGlzdHMgaW4gdGhlIGNv
bmZpZy4gVGhpcyBraW5kYSBzdWNrcyA7KAoKVHlwb3MgY291cnRlc3kgb2YgbXkgU2Ftc3VuZyBN
b2JpbGUKCi0tLS0tLS0tINCY0YHRhdC+0LTQvdC+0LUg0YHQvtC+0LHRidC10L3QuNC1IC0tLS0t
LS0tCtCe0YI6IG1hbnVAbmV0YnNkLm9yZyAK0JTQsNGC0LA6IDIwMTQuMDIuMTAgIDY6MTIgIChH
TVQrMDE6MDApIArQmtC+0LzRgzogbWlsdGVyLWdyZXlsaXN0QHlhaG9vZ3JvdXBzLmNvbSAK0KLQ
tdC80LA6IFJlOiBbbWlsdGVyLWdyZXlsaXN0XSBtaWx0ZXItZ3JleWxpc3QgYW5kIHAwZiAtIGlu
dGVncmF0aW9uIHdpdGggU29sYXJpcyAKIApKaW0gS2xpbW92IDxqaW1rbGltb3ZAY29zLnJ1PiB3
cm90ZToKCj4gKiBpZiB0aGUgcmVtb3RlIGhvc3QgaXMgbWlzc2luZyBpbiB0aGUgcDBmIGNhY2hl
LCBkbyBhIGZldyByZXRyaWVzLDxicj4KPiBwZXJoYXBzIGl0cyBwYWNrZXRzIHdpbGwgc29vbiBi
ZSBwcm9jZXNzZWQgYnkgdGhlIGRhZW1vbiBhbmQgc2hvdyB1cC48YnI+Cj4gTk9URSB0aGF0IHRo
aXMgYmxvY2tzIG1pbHRlci1ncmV5bGlzdCB1bnRpbCB0aGUgbG9vcCBjb21wbGV0ZXM8YnI+Cj4g
YW5kIGluIGNhc2Ugb2YgYXQgbGVhc3QgU2VuZG1haWwsIHRoaXMgZGVsYXlzIGV2ZW4gdGhlIGRp
c3BsYXk8YnI+Cj4gb2YgdGhlIFNNVFAgYmFubmVyLjxicj4KCkkgdW5kZXJzdGFuZCB0aGF0IHNs
ZWVwKDMpIHdpbGwgcHV0IHRoZSB3aG9sZSBwcm9jZXNzIHRvIHNsZWVwLCB3aGljaAptYWtlcyB0
aGUgbWlsdGVyIHVuYWJsZSB0byBwcm9jZXNzIG90aGVyIHJlcXVlc3RzIGZyb20gc2VuZG1haWwu
CnNjaGVkX3lpZWxkKDMpIHNlZW1zIHRvIGJlIHRoZSByaWdodCB3YXkuIENvdWxkIHlvdSBpbnZl
c3RpZ2F0ZT8KCi0tIApFbW1hbnVlbCBEcmV5ZnVzCmh0dHA6Ly9oY3BuZXQuZnJlZS5mci9wdWJ6
Cm1hbnVAbmV0YnNkLm9yZwo=


--Boundary_(ID_z4F1RdDzhgNp1chwB0jsCg)
Content-type: text/html; charset=utf-8
Content-transfer-encoding: base64

PGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0
L2h0bWw7IGNoYXJzZXQ9VVRGLTgiPjwvaGVhZD48Ym9keSA+PGRpdj5XZWxsLCBhY2NvcmRpbmcg
dG8gbWFucGFnZXMsIHNjaGVkX3lpZWxkKCkgZG9lcyBub3QgcHJvdmlkZSBmb3IgYSBkZWxheSB2
YWx1ZS4gQW5kIGFjY29yZGluZyB0byBtYW55IHNvdXJjZXMsIHNsZWVwKCkgc2hvdWxkIG9ubHkg
YWN0IG9uIGl0cyB0aHJlYWQsIG5vdCB0aGUgd2hvbGUgcHJvY2VzcywgYXQgbGVhc3QgaW4gdW5p
eC1saWtlIHN5c3RlbXMuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5TbyB1bmxlc3MgcHJvdmVu
IG90aGVyd2lzZSwgdGhpcyBwYXJ0IG9mIHRoZSBzb2x1dGlvbiBzaG91bGQgd29yayB3aXRob3V0
IGRlbGF5cyBmb3IgdW5yZWxhdGVkIGNvbm5lY3Rpb25zLiBJIHBsYW4gdG8gZXhwZXJpbWVudCB3
aXRoIGEgZGVwbG95ZWQgcmVsYXkgdG9uaWdodCB0aG91Z2guPC9kaXY+PGRpdj48YnI+PC9kaXY+
PGRpdj5UaGUgcXVlc3Rpb25zIG9uIGNvbmZpZyB0b2dnbGVzIGFuZCBvbiBjYWxsaW5nIHAwZiBv
bmx5IHdoZW4gcmVxdWVzdGVkIHRvIChhdCBsZWFzdCwgaW4gdGhlIGNvbmZpZyBwb2ludCB3aGVy
ZSBwMGZzb2NrIGlzIGRlZmluZWQpLCBub3Qgc29vbmVyLCBkbyByZW1haW4gaW4gcGxhY2UgOyk8
L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkZvciBleGFtcGxlLCBub3cgaSBoYXZlIG1pbHRlci1n
cmV5bGlzdCBpbnF1aXJpbmcgcDBmIGFib3V0IExBTiBob3N0cyB3aGljaCBhcmUgZXhlbXB0IGZy
b20gcDBkJ3MgYnBmIGZpbHRlciBkZWZpbml0aW9uIGFuZCBjYXVzZSB0aGUgcmV0cnkgbG9vcCBm
b3IgbG9jYWwgc3VibWlzc2lvbnMsIGRlc3BpdGUgdGhlIGZhY3QgdGhhdCBhbnl0aGluZyBhYm91
dCBwMGYgaXMgbWVudGlvbmVkIHdheSBhZnRlciBtYW51YWwgd2hpdGVsaXN0cyBpbiB0aGUgY29u
ZmlnLiBUaGlzIGtpbmRhIHN1Y2tzIDsoPC9kaXY+PGRpdj48YnI+PC9kaXY+VHlwb3MgY291cnRl
c3kgb2YgbXkgU2Ftc3VuZyBNb2JpbGU8YnI+PGJyPjxicj4tLS0tLS0tLSDQmNGB0YXQvtC00L3Q
vtC1INGB0L7QvtCx0YnQtdC90LjQtSAtLS0tLS0tLTxicj7QntGCOiBtYW51QG5ldGJzZC5vcmcg
PGJyPtCU0LDRgtCwOiAyMDE0LjAyLjEwICA2OjEyICAoR01UKzAxOjAwKSA8YnI+0JrQvtC80YM6
IG1pbHRlci1ncmV5bGlzdEB5YWhvb2dyb3Vwcy5jb20gPGJyPtCi0LXQvNCwOiBSZTogW21pbHRl
ci1ncmV5bGlzdF0gbWlsdGVyLWdyZXlsaXN0IGFuZCBwMGYgLSBpbnRlZ3JhdGlvbiB3aXRoIFNv
bGFyaXMgPGJyPiA8YnI+PGJyPgo8c3BhbiBzdHlsZT0iZGlzcGxheTpub25lIj4mbmJzcDs8L3Nw
YW4+CgoKCiAgICA8ZGl2IGlkPSJ5Z3JwLXRleHQiPgogICAgICAKICAgICAgCiAgICAgIDxwPkpp
bSBLbGltb3YgJmx0O2ppbWtsaW1vdkBjb3MucnUmZ3Q7IHdyb3RlOjxicj4KPGJyPgomZ3Q7ICog
aWYgdGhlIHJlbW90ZSBob3N0IGlzIG1pc3NpbmcgaW4gdGhlIHAwZiBjYWNoZSwgZG8gYSBmZXcg
cmV0cmllcywmbHQ7YnImZ3Q7PGJyPgomZ3Q7ICAgIHBlcmhhcHMgaXRzIHBhY2tldHMgd2lsbCBz
b29uIGJlIHByb2Nlc3NlZCBieSB0aGUgZGFlbW9uIGFuZCBzaG93IHVwLiZsdDticiZndDs8YnI+
CiZndDsgICAgTk9URSB0aGF0IHRoaXMgYmxvY2tzIG1pbHRlci1ncmV5bGlzdCB1bnRpbCB0aGUg
bG9vcCBjb21wbGV0ZXMmbHQ7YnImZ3Q7PGJyPgomZ3Q7ICAgIGFuZCBpbiBjYXNlIG9mIGF0IGxl
YXN0IFNlbmRtYWlsLCB0aGlzIGRlbGF5cyBldmVuIHRoZSBkaXNwbGF5Jmx0O2JyJmd0Ozxicj4K
Jmd0OyAgICBvZiB0aGUgU01UUCBiYW5uZXIuJmx0O2JyJmd0Ozxicj4KPGJyPgpJIHVuZGVyc3Rh
bmQgdGhhdCBzbGVlcCgzKSB3aWxsIHB1dCB0aGUgd2hvbGUgcHJvY2VzcyB0byBzbGVlcCwgd2hp
Y2g8YnI+Cm1ha2VzIHRoZSBtaWx0ZXIgdW5hYmxlIHRvIHByb2Nlc3Mgb3RoZXIgcmVxdWVzdHMg
ZnJvbSBzZW5kbWFpbC48YnI+CnNjaGVkX3lpZWxkKDMpIHNlZW1zIHRvIGJlIHRoZSByaWdodCB3
YXkuIENvdWxkIHlvdSBpbnZlc3RpZ2F0ZT88YnI+Cjxicj4KLS0gPGJyPgpFbW1hbnVlbCBEcmV5
ZnVzPGJyPgpodHRwOi8vaGNwbmV0LmZyZWUuZnIvcHViejxicj4KbWFudUBuZXRic2Qub3JnPGJy
Pgo8L3A+CgogICAgPC9kaXY+CiAgICAgCgogICAgCgoKCgoKPCEtLSBlbmQgZ3JvdXAgZW1haWwg
LS0+Cgo8L2JvZHk+


--Boundary_(ID_z4F1RdDzhgNp1chwB0jsCg)--

> I understand that sleep(3) will put the whole process to sleep, which
> makes the milter unable to process other requests from sendmail.
> sched_yield(3) seems to be the right way. Could you investigate?
> 

At least on Tru64 this is in the man page of sleep (3):

  In a multi-threaded environment, the sleep() function is redefined so that
  only the calling thread is suspended.

Best,
Attila