Milter-greylist

well, since we now have support for everything else (dnsbl, spf,  
dkim), why not add p0f support (os fingerprinting) to selectively  
greylist against.

Patrick Domack <patrickdk@...> wrote:

> well, since we now have support for everything else (dnsbl, spf,  
> dkim), why not add p0f support (os fingerprinting) to selectively  
> greylist against.

Heavily greylisting windows XP boxen could be a major benefit.
How coudl that work? You would like to send probes to the sending SMTP
server? It might be unreachable.

Or do you plan to gather information from the SMTP connexion itself? How
will you do that? Can you ask some local server about a given TCP
connexion, given source and destination port and IP?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Quoting manu@...:

> Patrick Domack <patrickdk@...> wrote:
>
>> well, since we now have support for everything else (dnsbl, spf,
>> dkim), why not add p0f support (os fingerprinting) to selectively
>> greylist against.
>
> Heavily greylisting windows XP boxen could be a major benefit.
> How coudl that work? You would like to send probes to the sending SMTP
> server? It might be unreachable.
>
> Or do you plan to gather information from the SMTP connexion itself? How
> will you do that? Can you ask some local server about a given TCP
> connexion, given source and destination port and IP?

hmm, that was why I said 'P0f' support, p0f does all that without  
doing any probes.

Patrick Domack <patrickdk@...> wrote:

> hmm, that was why I said 'P0f' support, p0f does all that without  
> doing any probes.

Nice, but could you investigate and tell us how that works?

milter-greylist can gather the current SMTP connexion source and
destination IP and ports, using Sendmail's {client_addr}, {client_port},
{daemon_addr}, {daemon_port} macros (how does Postfix cope with that?
Anyone can tell?).

Now what should be done with it? Is there a library with an API for
parsing p0f logs? 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

there are two ways people interface to p0f

Most use the script that comes with amavis

/usr/sbin/p0f -l 'dst port 25' 2>&1 | /usr/sbin/p0f-analyzer 2345 &

and you just query it on a udp port in the format:

SourceIP randomjunk

and it replies

SourceIP randomjunk Os-type verions .......

Or you can use the p0f internal method of using a socket, but I'm not  
sure how flexible that is, it wants source ip, port, and dest ip,port,  
and I dunno if it can do wildcards for any of them.

Quoting manu@...:

> Patrick Domack <patrickdk@...> wrote:
>
>> hmm, that was why I said 'P0f' support, p0f does all that without
>> doing any probes.
>
> Nice, but could you investigate and tell us how that works?
>
> milter-greylist can gather the current SMTP connexion source and
> destination IP and ports, using Sendmail's {client_addr}, {client_port},
> {daemon_addr}, {daemon_port} macros (how does Postfix cope with that?
> Anyone can tell?).
>
> Now what should be done with it? Is there a library with an API for
> parsing p0f logs?
>
> --
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz
> manu@...
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

Patrick Domack <patrickdk@...> wrote:

> Or you can use the p0f internal method of using a socket, but I'm not
> sure how flexible that is, it wants source ip, port, and dest ip,port,
> and I dunno if it can do wildcards for any of them.

We do not need that: we have source and destination IP and ports. 
Do you have an URL with detailed documentation about this internal
method?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Quoting manu@...:

> We do not need that: we have source and destination IP and ports.
> Do you have an URL with detailed documentation about this internal
> method?

Ok, the point of using udp was for spamass/amavis running on a  
different machine could get that same data (unless we place a header  
in the email with this data in it ourselfs so they don't need to fetch  
it), but from a pure greylisting standpoint, I don't see why the  
milter and p0f would ever be on seperate boxes.

All this code is taken directly from the p0f source package that  
includes samples in a few languages. The header file isn't moved into  
a normal include directoy for inclusion easily :(

#define QUERY_MAGIC             0x0defaced

#define NO_SCORE                -100

/* Masquerade detection flags: */
#define D_GENRE   0x0001
#define D_DETAIL  0x0002
#define D_LINK    0x0004
#define D_DIST    0x0008
#define D_NAT     0x0010
#define D_FW      0x0020
#define D_NAT2_1  0x0040
#define D_FW2_1   0x0080
#define D_NAT2_2  0x0100
#define D_FW2_2   0x0200
#define D_FAST    0x0400
#define D_TNEG    0x0800

#define D_TIME    0x4000
#define D_FAR     0x8000

#define QTYPE_FINGERPRINT       1
#define QTYPE_STATUS            2

struct p0f_query {
   _u32 magic;                   /* must be set to QUERY_MAGIC */
   _u8  type;                    /* QTYPE_* */
   _u32 id;                      /* Unique query ID */
   _u32 src_ad,dst_ad;           /* src address, local dst addr */
   _u16 src_port,dst_port;       /* src and dst ports */
};

#define RESP_OK         0       /* Response OK */
#define RESP_BADQUERY   1       /* Query malformed */
#define RESP_NOMATCH    2       /* No match for src-dst data */
#define RESP_STATUS     255     /* Status information */

struct p0f_response {
   _u32 magic;                   /* QUERY_MAGIC */
   _u32 id;                      /* Query ID (copied from p0f_query) */
   _u8  type;                    /* RESP_* */

   _u8  genre[20];               /* OS genre (empty if no match) */
   _u8  detail[40];              /* OS version (empty if no match) */
   _s8  dist;                    /* Distance (-1 if unknown ) */
   _u8  link[30];                /* Link type (empty if unknown) */
   _u8  tos[30];                 /* Traffic type (empty if unknown) */
   _u8  fw,nat;                  /* firewall and NAT flags flags */
   _u8  real;                    /* A real operating system? */
   _s16 score;                   /* Masquerade score (or NO_SCORE) */
   _u16 mflags;                  /* Masquerade flags (D_*) */
   _s32 uptime;                  /* Uptime in hours (-1 = unknown) */
};


struct p0f_status {
   _u32 magic;                   /* QUERY_MAGIC */
   _u32 id;                      /* Query ID (copied from p0f_query) */
   _u8  type;                    /* RESP_STATUS */

   _u8  version[16];             /* p0f version */
   _u8  mode;                    /* p0f mode (S - SYN; A - SYN+ACK, R  
- RST, O - stray) */
   _u32 fp_cksum;                /* Fingerprint file checksum */
   _u32 cache;                   /* p0f query cache size */
   _u32 packets;                 /* Total number of all packet received */
   _u32 matched;                 /* Total number of packets matched */
   _u32 queries;                 /* Total number of queries handled */
   _u32 cmisses;                 /* Total number of cache query misses */
   _u32 uptime;                  /* Process uptime in seconds */
};



   struct sockaddr_un x;
   struct p0f_query p;
   struct p0f_response r;
   _u32 s,d,sp,dp;
   _s32 sock;

   if (argc != 6) {
     debug("Usage: %s p0f_socket src_ip src_port dst_ip dst_port\n",
           argv[0]);
     exit(1);
   }

   s  = inet_addr(argv[2]);
   sp = atoi(argv[3]);
   d  = inet_addr(argv[4]);
   dp = atoi(argv[5]);

   if (!sp || !dp || s == INADDR_NONE || d == INADDR_NONE)
     fatal("Bad IP/port values.\n");

   sock = socket(PF_UNIX,SOCK_STREAM,0);
   if (sock < 0) pfatal("socket");

   memset(&x,0,sizeof(x));
   x.sun_family=AF_UNIX;
   strncpy(x.sun_path,argv[1],63);

   if (connect(sock,(struct sockaddr*)&x,sizeof(x)))  pfatal(argv[1]);

   p.magic    = QUERY_MAGIC;
   p.id       = 0x12345678;
   p.type     = QTYPE_FINGERPRINT;
   p.src_ad   = s;
   p.dst_ad   = d;
   p.src_port = sp;
   p.dst_port = dp;

   if (write(sock,&p,sizeof(p)) != sizeof(p))
     fatal("Socket write error (timeout?).\n");

   if (read(sock,&r,sizeof(r)) != sizeof(r))
     fatal("Response read error (timeout?).\n");

   if (r.magic != QUERY_MAGIC)
     fatal("Bad response magic.\n");

   if (r.type == RESP_BADQUERY)
     fatal("P0f did not honor our query.\n");

   if (r.type == RESP_NOMATCH) {
     printf("This connection is not (no longer?) in the cache.\n");
     exit(3);
   }

   if (!r.genre[0]) {
     printf("Genre and OS details not recognized.\n");
   } else {
     printf("Genre    : %s\n",r.genre);
     printf("Details  : %s\n",r.detail);
     if (r.dist != -1) printf("Distance : %d hops\n",r.dist);
   }

   if (r.link[0]) printf("Link     : %s\n",r.link);
   if (r.tos[0])  printf("Service  : %s\n",r.tos);

   if (r.uptime != -1)  printf("Uptime   : %d hrs\n",r.uptime);

   if (r.score != NO_SCORE)
     printf("M-Score  : %d%% (flags %x).\n",r.score,r.mflags);

   if (r.fw) printf("The host is behind a firewall.\n");
   if (r.nat) printf("The host is behind NAT or such.\n");

   shutdown(sock,2);
   close(sock);

manu@... wrote:
 > Patrick Domack <patrickdk@...> wrote:
 > 
 > > well, since we now have support for everything else (dnsbl, spf,  
 > > dkim), why not add p0f support (os fingerprinting) to selectively  
 > > greylist against.
 > 
 > Heavily greylisting windows XP boxen could be a major benefit.

The important question is:  How reliable is it?  How likely
is it to get false positives?  It should be pointed out
that tools like nmap (and similar) just take a good guess,
but are often wrong.  For example, it doesn't detect one of
my backup MX machines correctly.

Also note that some server admins intentionally change the
parameters of their TCP/IP stack so fingerprinting does not
guess their OS correctly (just like many admins change the
welcome message of their MTA so it confuses potential
attackers).

I don't want to put a huge greylist delay on machines based
on their OS if the OS detection isn't 100% reliable.

And I *certainly* don't want my own MTAs greylisted for a
long time just because some other braindead server is unable
to detect my OS correctly.  :-(

That's why I feel a little uneasy adding such a "feature"
to milter-greylist.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"The ITU has offered the IETF formal alignment with its
corresponding technology, Penguins, but that won't fly."
        -- RFC 2549

Well isn't that like anything. Nothing is ever going be reliable, in  
greylisting triplites aren't even very accurate cause of firewalls,  
server pools that can cause the source ip and source email addresses  
to change each time.

Now people just using default setups or using someone elses config  
blindly is an issue, and it's their issue.

Not giving people an option, just for the sake that some people will  
use that option is an annoying or stupid way, is a different issue.

I know I personally collect stats of all kinds of things into my  
database, every greylist triplite, from address spam levels, from ip  
spam levels, os id spam levels, ....

I then compile the results ever now and again as I notice spam  
changing, and modify my rules, add new ones, and blacklist/whitelist  
people.


Quoting Oliver Fromme <olli@...>:

>
> manu@... wrote:
>  > Patrick Domack <patrickdk@...> wrote:
>  >
>  > > well, since we now have support for everything else (dnsbl, spf,
>  > > dkim), why not add p0f support (os fingerprinting) to selectively
>  > > greylist against.
>  >
>  > Heavily greylisting windows XP boxen could be a major benefit.
>
> The important question is:  How reliable is it?  How likely
> is it to get false positives?  It should be pointed out
> that tools like nmap (and similar) just take a good guess,
> but are often wrong.  For example, it doesn't detect one of
> my backup MX machines correctly.
>
> Also note that some server admins intentionally change the
> parameters of their TCP/IP stack so fingerprinting does not
> guess their OS correctly (just like many admins change the
> welcome message of their MTA so it confuses potential
> attackers).
>
> I don't want to put a huge greylist delay on machines based
> on their OS if the OS detection isn't 100% reliable.
>
> And I *certainly* don't want my own MTAs greylisted for a
> long time just because some other braindead server is unable
> to detect my OS correctly.  :-(
>
> That's why I feel a little uneasy adding such a "feature"
> to milter-greylist.
>
> Best regards
>    Oliver
>
> --
> Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
> Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
> secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
> chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart
>
> FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd
>
> "The ITU has offered the IETF formal alignment with its
> corresponding technology, Penguins, but that won't fly."
>         -- RFC 2549
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

Patrick Domack wrote:
 > Oliver Fromme wrote:
 > > manu@... wrote:
 > > > Patrick Domack <patrickdk@...> wrote:
 > > > 
 > > > > well, since we now have support for everything else (dnsbl, spf,
 > > > > dkim), why not add p0f support (os fingerprinting) to selectively
 > > > > greylist against.
 > > > 
 > > > Heavily greylisting windows XP boxen could be a major benefit.
 > > 
 > > The important question is:  How reliable is it?  How likely
 > > is it to get false positives?  It should be pointed out
 > > that tools like nmap (and similar) just take a good guess,
 > > but are often wrong.  For example, it doesn't detect one of
 > > my backup MX machines correctly.
 > > 
 > > Also note that some server admins intentionally change the
 > > parameters of their TCP/IP stack so fingerprinting does not
 > > guess their OS correctly (just like many admins change the
 > > welcome message of their MTA so it confuses potential
 > > attackers).
 > > 
 > > I don't want to put a huge greylist delay on machines based
 > > on their OS if the OS detection isn't 100% reliable.
 > > 
 > > And I *certainly* don't want my own MTAs greylisted for a
 > > long time just because some other braindead server is unable
 > > to detect my OS correctly.  :-(
 > > 
 > > That's why I feel a little uneasy adding such a "feature"
 > > to milter-greylist.
 > 
 > Well isn't that like anything. Nothing is ever going be reliable, [...]

Right.  Sadly.

There was a time (before spam existed, and before anybody
would even consider running MTAs on Windows) when e-mail
delivery in the internet was reliable.  Sadly this isn't
the case anymore today.

Talking about filter features:  Some features are more
reliable than others, and some features are easier to
abuse or misuse than others (or to use in an inappropriate
or wrong way).

If an OS fingerprinting feature will be implemented in
milter-greylist, it should at least be accompanied by
a fat warning, and it should not be included in the
sample configuration file by default.

Of course, nothing helps against clueless mail server
admins.  I'm already pretty much fed up with such people,
having dealt with a lot of them [*].  Of course this is
not to blame on milter-greylist.  But the more features
milter-greylist grows that are too easily misconfigured,
the more often it *will* be misconfigured, and the result
is that internet email becomes more and more unreliable.

Best regards
   Oliver

PS: [*] The most recent example was someone who greylisted
for 1 minute and expired tuples after one hour.  Since
one of my mailservers resends every two hours (which is
perfectly RFC-compliant), it was unable to send anything
to him.  When I temporarily ran the queue in order to
reach his postmaster@ address to inform him about the
problem, it bounced with a "user unknown" error.
No need to say more.

Sorry for grumbling.  I would just like to make email
more reliable, not more unreliable ...

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

$ dd if=/dev/urandom of=test.pl count=1
$ file test.pl
test.pl: perl script text executable

Quoting Oliver Fromme <olli@...>:

>
> Patrick Domack wrote:
>  > Oliver Fromme wrote:
>  > > manu@... wrote:
>  > > > Patrick Domack <patrickdk@...> wrote:
>  > > >
>  > > > > well, since we now have support for everything else (dnsbl, spf,
>  > > > > dkim), why not add p0f support (os fingerprinting) to selectively
>  > > > > greylist against.
>  > > >
>  > > > Heavily greylisting windows XP boxen could be a major benefit.
>  > >
>  > > The important question is:  How reliable is it?  How likely
>  > > is it to get false positives?  It should be pointed out
>  > > that tools like nmap (and similar) just take a good guess,
>  > > but are often wrong.  For example, it doesn't detect one of
>  > > my backup MX machines correctly.
>  > >
>  > > Also note that some server admins intentionally change the
>  > > parameters of their TCP/IP stack so fingerprinting does not
>  > > guess their OS correctly (just like many admins change the
>  > > welcome message of their MTA so it confuses potential
>  > > attackers).
>  > >
>  > > I don't want to put a huge greylist delay on machines based
>  > > on their OS if the OS detection isn't 100% reliable.
>  > >
>  > > And I *certainly* don't want my own MTAs greylisted for a
>  > > long time just because some other braindead server is unable
>  > > to detect my OS correctly.  :-(
>  > >
>  > > That's why I feel a little uneasy adding such a "feature"
>  > > to milter-greylist.
>  >
>  > Well isn't that like anything. Nothing is ever going be reliable, [...]
>
> Right.  Sadly.
>
> There was a time (before spam existed, and before anybody
> would even consider running MTAs on Windows) when e-mail
> delivery in the internet was reliable.  Sadly this isn't
> the case anymore today.
>
> Talking about filter features:  Some features are more
> reliable than others, and some features are easier to
> abuse or misuse than others (or to use in an inappropriate
> or wrong way).
>
> If an OS fingerprinting feature will be implemented in
> milter-greylist, it should at least be accompanied by
> a fat warning, and it should not be included in the
> sample configuration file by default.
>
> Of course, nothing helps against clueless mail server
> admins.  I'm already pretty much fed up with such people,
> having dealt with a lot of them [*].  Of course this is
> not to blame on milter-greylist.  But the more features
> milter-greylist grows that are too easily misconfigured,
> the more often it *will* be misconfigured, and the result
> is that internet email becomes more and more unreliable.

I agree, but there has been a milter_p0f for awhile, and I would much  
perfer to just modify greylisting based of fingerprinting results,  
than blacklist based on it, and in order for that to happen it would  
have to be understood by the greylisting milter.

And the paranoid admin that is going mess his system up with p0f  
anyways, if it isn't an option in greylisting, very well might opt for  
using the p0f milter and blacklist os's based on it.

Oliver Fromme <olli@...> wrote:

> If an OS fingerprinting feature will be implemented in
> milter-greylist, it should at least be accompanied by
> a fat warning, and it should not be included in the
> sample configuration file by default.

Well, this would just be another ACL clause. One can experiment random
setups with it. The worst thing that can happen if this is used for
heavy greylisting is an extra delay.

But I think you are right: the README should start with more warnings
about bad setups. Do you want to contribute them? (thr warnings, not the
setups)

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...