Milter-greylist

Hello!

I am looking at milter-greylist to replace our current
greylistingsoftware, mainly due to the nice features that
milter-greylist apparently has.

I am, however, a bit curious about the syncing of the database between
peers: what happens if the peer is down for more than 1024 connections?
Can the databases be synced after the peer comes up, or are the
databases permanently out of sync?

Cheers,
Fredrik Nyberg

Fredrik Nyberg DC <nba@...> wrote:

> I am, however, a bit curious about the syncing of the database between
> peers: what happens if the peer is down for more than 1024 connections?
> Can the databases be synced after the peer comes up, or are the
> databases permanently out of sync?

You have missing entries on the host that was down, but that only cause
extra delivery delays. Here is a sample scenario:

0) MX2 is down
1) senders sends an e-mail, it tries MX1
2) MX1 adds the mail to the greylist, and tempfail
3) MX1 attemps to sync with MX2 but cannot and the queue has overflowed
4) MX2 comes back on-line
5) sender try a resend to MX2
6) MX2 adds the mail to the greylist, and tempfail
7) MX2 syncs with MX1
8) sender try a resend to MX2
9) MX2 accepts the mail   

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

manu@... wrote:
> 0) MX2 is down
> 1) senders sends an e-mail, it tries MX1
> 2) MX1 adds the mail to the greylist, and tempfail
> 3) MX1 attemps to sync with MX2 but cannot and the queue has overflowed
> 4) MX2 comes back on-line
> 5) sender try a resend to MX2
> 6) MX2 adds the mail to the greylist, and tempfail
> 7) MX2 syncs with MX1
> 8) sender try a resend to MX2
> 9) MX2 accepts the mail   

OK, this seems to work fairly well with two greylist-databases. If there
were four or perhaps ten, then it would be a problem. But I assume there
is no "super-sync" functionality where one database could be synced in
full to another peer?

Cheers,
Fredrik Nyberg

Fredrik Nyberg DC <nba@...> wrote:

> OK, this seems to work fairly well with two greylist-databases. If there
> were four or perhaps ten, then it would be a problem.

There is never any real problem, just longer delivery. Note that if you
have ten MX, the chance to retry on the machine that was down and does
not have an up to date greylist is 1/10, so longer delivery will not
happen often. 

> But I assume there
> is no "super-sync" functionality where one database could be synced in
> full to another peer?

Not yet, but you can write it. That would be simple: a command to ask a
peer to dump the whole database. 

-- 
Emmanuel Dreyfus
Le cahier de l'admin BSD 2eme ed. est dans toutes les bonnes librairies
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...

manu@... wrote:
>>But I assume there
>>is no "super-sync" functionality where one database could be synced in
>>full to another peer?
>
>
> Not yet, but you can write it. That would be simple: a command to ask a
> peer to dump the whole database.

OK, I figured as much, just wanted to check that I don't start writing
code that exists. I'll look into it.

Cheers,
Fredrik Nyberg

Fredrik Nyberg DC <nba@...> wrote:

> > Not yet, but you can write it. That would be simple: a command to ask a
> > peer to dump the whole database. 
> OK, I figured as much, just wanted to check that I don't start writing
> code that exists. I'll look into it.

See in sync.c. The current commands are
add, del, help, quit

You can add two commands:
dump greylist
dump whitelist

That would perform a text dump. The easier was is probably to cat the
dump file. You can also do a dump, but that would require adding locking
logic for that (see dump.c)

-- 
Emmanuel Dreyfus
Le cahier de l'admin BSD 2eme ed. est dans toutes les bonnes librairies
http://www.eyrolles.com/Informatique/Livre/9782212114638/livre-bsd.php
manu@...

manu@... wrote:
> Fredrik Nyberg DC <nba@...> wrote:
> 
> 
>>>Not yet, but you can write it. That would be simple: a command to ask a
>>>peer to dump the whole database. 
>>
>>OK, I figured as much, just wanted to check that I don't start writing
>>code that exists. I'll look into it.
> 
> 
> See in sync.c. The current commands are
> add, del, help, quit
> 
> You can add two commands:
> dump greylist
> dump whitelist
> 
> That would perform a text dump. The easier was is probably to cat the
> dump file. You can also do a dump, but that would require adding locking
> logic for that (see dump.c)
> 

I'm working on a prototype, I'm using 'sync' to dump the entire textfile
with proper locking. However, I'll have to think about race-conditions a
bit, since I was thinking of locking _all_ the peers for the sync to
ensure that the databases are not flushed during the update. The peer
that is out of sync would then have to have some algortihm to decide
which database from the other peers it wants (oldest entry, newest
entry, most entries, etc) since the databases are not guaranteed to be
in sync (imagine a powerout in the server-room). I'll probably need some
kind of timeout for the actual transfer, too.

Thinking-cap turn-on.

Cheers,
Fredrik Nyberg

Fredrik Nyberg DC <nba@...> wrote:

> I'm working on a prototype, I'm using 'sync' to dump the entire textfile
> with proper locking. However, I'll have to think about race-conditions a
> bit, since I was thinking of locking _all_ the peers for the sync to
> ensure that the databases are not flushed during the update. The peer
> that is out of sync would then have to have some algortihm to decide
> which database from the other peers it wants (oldest entry, newest
> entry, most entries, etc) since the databases are not guaranteed to be
> in sync (imagine a powerout in the server-room). I'll probably need some
> kind of timeout for the actual transfer, too.

Yes, there is the problem of choosing the best source. Looking for
latest entry sounds good. This can be done by another command to get the
latest database update date. A starting machine would ask all the peers
and would look its dump file, then choose the best way to reload the
greylist and whitelist.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...