Milter-greylist

>


>
> How scalability would be improved? Accessing a mysqldb or any db  
> stored on Hard Disk is slower than accesing memory directly (only  
> mysql memory tables (HEAP) are fast enough.
>
>

Eduardo is right at this point I also do not believe in performance  
impacts but for data manipulation and processing a db would be nice.
In our case we are using the data generated by milter-greylist for  
building white and blacklists and detecting other possible spam  
sources so a native db backend would be much more comfortable.


>
>
>
> In the only case i think it might help is in configuration data, but  
> the text file is not so complicated, and adding layers might add  
> problems.
>
>


It´s much easier to handle the config in a single db or directory.  
Normally we generate the configs out of db/directory data but in some  
cases just in time changes are usefull and so a single db/directory  
would be nice to have.



Mit freundlichen Gruessen || with kind regards

Sven Holz

----- "Sven Holz" <sven.holz@...> wrote: 
> 
> 
> 
> 

> 
> 
> > 
> > How scalability would be improved? Accessing a mysqldb or any db 
> > stored on Hard Disk is slower than accesing memory directly (only 
> > mysql memory tables (HEAP) are fast enough. 
> > 
> > 
> 
> Eduardo is right at this point I also do not believe in performance 
> impacts but for data manipulation and processing a db would be nice. 
> In our case we are using the data generated by milter-greylist for 
> building white and blacklists and detecting other possible spam 
> sources so a native db backend would be much more comfortable. 
> 




How do you user milter-greylist data to generate white/blacklist? If a server doesn't retry you blacklist it? Please it would be kind if you can give me an example of how you re-use the data :D 




thanks! 



> > 
> > 
> > 
> > In the only case i think it might help is in configuration data, but 
> > the text file is not so complicated, and adding layers might add 
> > problems. 
> > 
> > 
> 
> It´s much easier to handle the config in a single db or directory. 
> Normally we generate the configs out of db/directory data but in some 
> cases just in time changes are usefull and so a single db/directory 
> would be nice to have. 
> 
> Mit freundlichen Gruessen || with kind regards 
> 
> Sven Holz 
> 
>  

-- 
----- 
Eduardo Casarero 
Informatica Avanzada SRL 
Email: eduardo.casarero@... 
Tel: (054) 011-52353939 int 115

>
>
> How do you user milter-greylist data to generate white/blacklist? If  
> a server doesn't retry you blacklist it? Please it would be kind if  
> you can give me an example of how you re-use the data :D
>


Not directly the data but together with logfiles our bayesian filters  
and so on we use the touples to get a trend for which IPs could be  
access pools and so on for our blacklists. Or just simple to get  
trusted IPs for our static whitelists.
It´s just another source in a larger pool of sources with information  
about good and bad IPs.

So we can white and blacklist IPs and domains jut before milter- 
greylist is involved thats much less cpu intensive :-)


Mit freundlichen Gruessen || with kind regards

             i.A. Sven Holz


WOBCOM GmbH
Sitz in Wolfsburg, Amtsgericht Braunschweig HRB 100052
Geschäftsführer: Prof. Dr. Markus Karp

-- 
Sven Holz - IP-Services - WOBCOM GmbH	Phone:	+49.5361.89.11.122
Hesslinger Str. 1-5, D-38440 Wolfsburg		Fax:		+49.5361.89.11.129
Email: sven.holz@... 				Mobile:	+49.151.544.491.22
Jabber: sholz@...

Hi Eduardo,

I agree that adding a simple embedded database backend such as SQLite
does not add much to milter-greylist. The current code is more than
adequate for a single server.

The intention is to make milter-greylist more scalable. I am assuming
that the 20 mail servers you are running do not serve the same domain;
please confirm.

Once you get into the realm of 20 servers for a single domain, the
amount of peer-to-peer traffic to keep the servers synchronized
quickly grows out of control.

With a backend that has the ability to connect to a scalable database
architecture, the peer-to-peer communication issue disappears, and
milter-greylist becomes truly scalable to any number of servers for a
single domain.

I choose MySQL as an example, because it can run high-volume and
high-availability clusters.

Let me know if anyone feels this makes sense or not.

--- In milter-greylist@yahoogroups.com, Eduardo Casarero
<eduardo.casarero@...> wrote:
>
> 
> ----- "Sven Holz" <sven.holz@...> wrote: 
> > 
> > 
> > 
> > 
> 
> Hi all 
> > 
> > > For scalability, the next step up for Milter-greylist will be to 
> > > implement a database backend. MySQL is a good candidate. 
> > > 
> > > In my discussions with Emmanuel, we have identified the following 
> > > steps to address this challenge, without breaking any current
setups 
> > > in the mean time. The proposed steps are: 
> > 
> > if an db backend is included it would be nice to place config parts 
> > like whitelists, peers and so on also in the db... 
> > 
> > This would make it possible to code a central nice an sweet 
> > webfrontend for milter-greylist :-) 
> > 
> > Mit freundlichen Gruessen || with kind regards 
> > 
> > Sven Holz 
> > 
> > 
> 
> 
> 
> 
> 
> Is really so important to have a DB backend? i've more than 20
servers running milter-greylist and they dump the tuples only once a day. 
> 
> 
> 
> 
> 
> How scalability would be improved? Accessing a mysqldb or any db
stored on Hard Disk is slower than accesing memory directly (only
mysql memory tables (HEAP) are fast enough. 
> 
> 
> 
> 
> 
> In the only case i think it might help is in configuration data, but
the text file is not so complicated, and adding layers might add
problems. 
> 
> 
> 
> 
> I think that milter-greylisting should focus in performance and the
smtp transaction (not body neither headers). In my setup i only use it
with spf and some acl to stop DDoS, harvesting, etc. Behind
greylisting, i have mailscanner, spamassassin and everything. 

> 
> 
> 
> 
> Just my 2 cents 
> 
> 
> 
> 
> Eduardo. 
> 
> 
>  
> 
> -- 
> ----- 
> Eduardo Casarero 
> Informatica Avanzada SRL 
> Email: eduardo.casarero@... 
> Tel: (054) 011-52353939 int 115
>

>
>
> I would prefer to have a "dummy greylist daemon" with something like  
> this http://www.ispras.ru/~knizhnik/fastdb/readme.htm as DB. All the  
> 20 greylists have this dummy greylist configured as peer (only 1)  
> and they send to this dummy peer all the transaccions. As the  
> transactions gets into the dummy, it's replyes to the other 19  
> clients.
>
>
>
>
> It will work as central node, behaving as a normal greylisting  
> daemon (so no need to alter p2p code to much) but acting as  
> redistribution daemon. I see as benefits, lightweight code, no need  
> to involve a huge db daemon, it can be reused all code of p2p trafic.


I think it´s a great idea to have master for the data.

And from my experience parsing a large file is much faster then  
parsing a large database but I´m not a db guru so don´t tak it as  
ultimate truth :-). So maybe you don´t even need any kind of db on the  
master.

And isn´t it possible with the actual software that you use one as  
master with a peering to all milter-greylist hosts and on a simple  
milter-greylist host just an entry to the master?
Hope you get what I mean.

I think you have to have all data in the memory on every host because  
it really would not perform well to generate a request for every  
single mail.
So wouldn´t it work right out of the box now?

Mit freundlichen Gruessen || with kind regards

            i.A. Sven Holz


WOBCOM GmbH
Sitz in Wolfsburg, Amtsgericht Braunschweig HRB 100052
Geschäftsführer: Prof. Dr. Markus Karp

-- 
Sven Holz - IP-Services - WOBCOM GmbH	Phone:	+49.5361.89.11.122
Hesslinger Str. 1-5, D-38440 Wolfsburg		Fax:		+49.5361.89.11.129
Email: sven.holz@... 				Mobile:	+49.151.544.491.22
Jabber: sholz@...

>
>
> No, now a days a greylisting daemon only fordward 1 level of  
> information, so in a multinode cluster there is no fordwarding  
> storm. But changing the code should not be very complicated .
>



Uhh yeah you a right stupid question 8-)



Mit freundlichen Gruessen || with kind regards

Sven Holz

--- In milter-greylist@yahoogroups.com, Eduardo Casarero
<eduardo.casarero@...> wrote:
> I would prefer to have a "dummy greylist daemon" with something like
this http://www.ispras.ru/~knizhnik/fastdb/readme.htm as DB. All the
20 greylists have this dummy greylist configured as peer (only 1) and
they send to this dummy peer all the transaccions. As the transactions
gets into the dummy, it's replyes to the other 19 clients. 


This is a good idea, and will help milter-greylist to handle larger
server clusters. The dummy daemon basically acts as a home-grown DB
server. There are two problems with it, though:

1. There is a limit to the scalability. Increasing the number of mail
servers, the load on the dummy greylist daemon will eventually become
too high and the setup will break

2. There is a reliability issue. If the dummy daemon fails,
greylisting on the whole server cluster fails. It breaks the
redundancy inherent to multiple mx servers for a single domain.

Have a look at http://en.wikipedia.org/wiki/MySQL_Cluster. These
database guys know how to do high availability and performance
clustering. Why not take advantage of that?

--- In milter-greylist@yahoogroups.com, Eduardo Casarero
<eduardo.casarero@...> wrote:

> I think that involving DB's Clusters change the spirit of
milter-greylisting (its a milter) also you need DBA skills to manage
such a cluster.

The idea _is_ to change the spirit of Milter-greylisting -- towards a
scalable configuration. Note that you don't need to use a MySQL
cluster. A simple MySQL server will do for most medium sized
operations. The scalability of MySQL will create an upgrade path as
the installation grows. Also, the old backend will still be available.
I am curious as to why you feel the fact that it is a milter does not
jive with scalability?

> I just think that the cost of implementing a big database such as
mysql is a waste of resources and time for something so simple as
milter-greylist. 
>

If that is the consensus, I will gladly rest my case...

Eduardo Casarero <eduardo.casarero@...> wrote:

> I just think that the cost of implementing a big database such as mysql is
> a waste of resources and time for something so simple as
> milter-greylist.

The plan is to allow using various backends, including the current
in-memory storage. You will never have to switch to MySQL. At least I
would not want to. But people that need it will be able to use it.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Helo,

My point of view is :
	-the DB API must be stable, efficient, reliable, and open. 
	-the choice of DB is not important if the API is OPEN all DB should be implemented.
	-API must meet a specification that we should begin to define.
	-the specifications must be accurate and not talk about techniques, but features.

This is only my view.
sorry for the quality of my English.
-------------------------------------------
Emmanuel SCIEUR
ESSBC
Web : http://www.essbc.com
-------------------------------------------
Avant d'imprimer cet email, réfléchissez à l'impact sur l'environnement, merci! 

De : milter-greylist@yahoogroups.com [mailto:milter-greylist@yahoogroups.com] De la part de Eduardo Casarero
Envoyé : mardi 20 janvier 2009 00:42
À : milter-greylist@yahoogroups.com
Objet : Re: [~Disarmed~] Re: [milter-greylist] Implement MySQL backend in Milter-greylist


----- "reschauzier" <reschauzier@...> wrote: 
> 
> 
> 
> 
--- In milter-greylist@yahoogroups.com, Eduardo Casarero
> <eduardo.casarero@...> wrote:
> 
> > I think that involving DB's Clusters change the spirit of
> milter-greylisting (its a milter) also you need DBA skills to manage
> such a cluster.
> 
> The idea _is_ to change the spirit of Milter-greylisting -- towards a
> scalable configuration. Note that you don't need to use a MySQL
> cluster. A simple MySQL server will do for most medium sized
> operations. The scalability of MySQL will create an upgrade path as
> the installation grows. Also, the old backend will still be available.
> I am curious as to why you feel the fact that it is a milter does not
> jive with scalability?

IMHO a milter is a little piece in the antispam machine it runs on the MTA (sendmail, postfix, etc) doing a great work at MTA level. In my experience users (i have like 50k) screw things with whitelist/blacklist, so having a well tuned milter greylisting with p2p, dummy daemon or the method you want to sincronize tuples its ok. 

Milter-greylist its not a MTA, so i try to set the point that adding features to milter-greylist that already you can do with the MTA is a waste of resources (or you can maybe put it in the MTA wishlist)

However is my point of view, and in my experience in Mail Gateway things shoud stay as simple as posible.

PD: Maybe we can check barricade MX and try to add to milter-greylist some new ideas.

> 
> > I just think that the cost of implementing a big database such as
> mysql is a waste of resources and time for something so simple as
> milter-greylist. 
> >
> 
> If that is the consensus, I will gladly rest my case...
> 
> 


-- 
-----
Eduardo Casarero
Informatica Avanzada SRL
Email: eduardo.casarero@informaticaavanzada.com.ar
Tel: (054) 011-52353939 int 115 
 
 

__________ Information provenant d'ESET Smart Security, version de la base des signatures de virus 3779 (20090119) __________

Le message a été vérifié par ESET Smart Security.

http://www.eset.com

Hello,

Regarding a DB backend for storing greylist tuples ...

What is the average ratio of creating tuples vs. looking
up tuples in the list?  In other words:  How many write
operations and how many read operations?  That is a very
important question, because it affects the scalability
of a DB backend solution.

Someone mentioned a situation with 20 servers.  Currently
every newly created tuple has to be replicated to all the
other servers, so that's 19 packet exchanges.  On the other
hand, a look-up is extremely cheap:  Every server has a
complete copy of the list, so it can look up tuples locally
without any network access.

Now consider the same 20 servers using a database cluster
(lets say 2 DB servers for redundancy and load-balancing).
When a new packet is generated, only the DB servers need
to be contacted, so there are two packet exchanges (lets
ignore the fact that an SQL command is more complicated
than milter-greylist's MX sync packet).  On the other hand,
every look-up also requires a network packet exchange (SQL
select query) between an MX server and one of the database
servers.  This means more delay on the greylist servers,
and possibly increases thread contention.

So which one scales better?  If 10% of all operations are
new tuple creation, and 90% are look-ups, then I think the
current implementation without separate DB servers scales
better.  If the numbers are 50%/50%, then the answer might
be different, but it probably depends on a lot of things
(number of servers, performance of DB servers, network
bandwith, and so on).  You can't even benchmark it properly
without actually putting the whole setup in production,
I guess, because it's very difficult to generate realistic
mail load on such a large setup.

What I'm trying to say:  Implementing a DB backend will not
necessarily solve any scalability propblems.  You should
first get some statistics on the behaviour (number of
newly created tuples vs. number of tuple look-ups per
time unit) in order to decide whether it's even worth the
effort to implement a DB backend.

I think there are other -- maybe better -- ways to improve
scalability.  For example, when you've got 20 servers, then
don't use all of them for the same set of domains.  Instead
you should partition them into sets that create roughly
the same load ("divide and conquer"), e.g. use 5 sets of
4 servers for different domains.  That will cut the MX sync
traffic down by 75%.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"And believe me, as a C++ programmer, I don't hesitate to question
the decisions of language designers.  After a decent amount of C++
exposure, Python's flaws seem ridiculously small." -- Ville Vainio

--- In milter-greylist@yahoogroups.com, Oliver Fromme <olli@...> wrote:

> I think there are other -- maybe better -- ways to improve
> scalability.  For example, when you've got 20 servers, then
> don't use all of them for the same set of domains.  Instead
> you should partition them into sets that create roughly
> the same load ("divide and conquer"), e.g. use 5 sets of
> 4 servers for different domains.  That will cut the MX sync
> traffic down by 75%.

This only works when serving multiple domains. As long as a single
domain can be handled by one or only a handful of mx servers, the
current milter-greylist solution is fine.

The focus here is what if we need a significant (>50) number of
servers to handle a single domain? A database backend, even with its
inherent overhead, then suddenly starts to make sense.

reschauzier wrote:
 > Oliver Fromme wrote:
 > 
 > > I think there are other -- maybe better -- ways to improve
 > > scalability.  For example, when you've got 20 servers, then
 > > don't use all of them for the same set of domains.  Instead
 > > you should partition them into sets that create roughly
 > > the same load ("divide and conquer"), e.g. use 5 sets of
 > > 4 servers for different domains.  That will cut the MX sync
 > > traffic down by 75%.
 > 
 > This only works when serving multiple domains. As long as a single
 > domain can be handled by one or only a handful of mx servers, the
 > current milter-greylist solution is fine.
 > 
 > The focus here is what if we need a significant (>50) number of
 > servers to handle a single domain? A database backend, even with its
 > inherent overhead, then suddenly starts to make sense.

Are there any real-use cases for that many servers using
milter-greylist for a single domain?  I don't think that
something like that really exists.

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"anyone new to programming should be kept as far from C++ as
possible;  actually showing the stuff should be considered a
criminal offence" -- Jacek Generowicz

Oliver Fromme <olli@...> wrote:

> Are there any real-use cases for that many servers using
> milter-greylist for a single domain?  I don't think that
> something like that really exists.

As I understand, the idea is to make it scale for such a setup. I have
to confess I fear that it would make milter-greylist a valuable target
for spammers. 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Tue, Jan 20, 2009 at 09:28:20PM +0100, manu@... wrote:
> Oliver Fromme <olli@...> wrote:
> 
> > Are there any real-use cases for that many servers using
> > milter-greylist for a single domain?  I don't think that
> > something like that really exists.
> 
> (...) I have to confess I fear that it would make milter-greylist a
> valuable target for spammers. 

I really didn't get that one. Could you please explain?



   Petar Bogdanovic

Petar Bogdanovic <petar@...> wrote:

> > (...) I have to confess I fear that it would make milter-greylist a
> > valuable target for spammers. 
> I really didn't get that one. Could you please explain?

Spam is a healthy business. People invest in developping 
- malware to take over the control of millions of machines
- spam engines in order to work around spam filters.
- various smart things to defeat CAPTCHA

Spammers work on developements that will have a good return on
investement. Since no milter-greylist implementation scales to a very
large installations, working on defeating milter-greylist itself is not
financially attractive: it will mean reachine a small amount of new
mailboxes. This seems uninteresting when you can work on injecting spams
in massive mailing platforms. 

If milter-greylist scales to high, that may not be the case any longer.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

I've just been reading this thread for a while and don't really have an 
opinion. But I developed two thoughts in the meantime.

- Do we need that much information stored as we do now?
- Do we need to have this info shared in realtime?

I think the answer to both is no.

- amount of information
what for do we need sender and recipient email address? Greylisting is a 
technical way of blocking mail servers that do not retry. From and to 
doesn't matter. (From does matter if you whitelist by SPF, but you don't 
need to store it.) If a mail server retries then it will eventually 
deliver all its mail, no matter which sender or recipient. Doesn't make 
sense to artificially delay that.
Yes, there's lazyaw, but the whole tuple is still getting saved at the 
moment and it's only "active" for matching. 
If we need less information we can store and share less information and 
have to process less information, which would be desirable for many uses, 
including a DBMS backend and for scaling. Much less actually, an IP 
address is much shorter than two email addresses. I think this would also 
make a good default.

- realtime sharing
do all servers in the cloud need to know any "event" immediately? I don't 
think so. In general, you can assume that the "next" mail from the same 
source (be it a retry or a completely new one) will either follow in a 
time period where the tuple (or single IP, see above) would still be 
(grey- or whatever-)listed even for the one server model or in a time 
period where there was enough time to share any necessary information for 
acting upon it "gradually". If it is true that the info doesn't need to be 
shared "immediately", what about using a "revolver" scheme? Given you have 
A - Z servers. Each, say, 5 seconds they contact *one* peer for updating 
in one direction. e.g. A -> B, B -> C, ... , Z -> A. So, for an already 
big cluster of ten servers this would take about one minute. Isn't that 
enough of a time period for synchronisation? If it is done this way I 
assume it uses much less ressources (in CPU and bandwidth) and may thus 
also easier accomodate if a DBMS gets used as a backend (on each machine).

As for using a "central" DBMS for better scaling. I fear this will somehow 
bite itself in the tail. If you have a larger cloud of mail servers that 
means you also have a *lot* of mail traffic and you were required to 
cluster it to cope with that. Now going to a single DBMS backend means 
"re-introducing" a single point of possible slowdown and failure. So, I 
rather think that any managing improvements coming with a DBMS backend 
will mostly apply for *smaller* mail clusters.

Maybe I'm missing a few points of this thread but this is what comes to my 
mind about it at the moment.

As I'm only using milter-greylist for "normal" greylisting my arguments 
may not apply too well to other uses, like blacklisting or whatever else 
can be done with mitler-greylist nowadays.


Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Kai Schaetzl <maillists@...> wrote:

> what for do we need sender and recipient email address?

If your ennemy is a spamware operating on a botnet, retaining (IP, from,
rcpt) in your greylisting database cost ressources to the ennemy: it has
to keep track of the tuple if it wants to defeat greylisting. 

For the botnet operator, resources are free (as it is operating on
hacked machines), but they are not infinite. If it does not want to be
too annoying, which would get it eradicated by the machine owner, it
cannot consume too much resources. This is why we see spamwares keeping
track of tuples for 15 minutes, but not for 4 hours.

If you retain only the IP, the spamware just has to send messages with
random (from, rcpt) to your server, and after some time, they will get
through. No need to keep track of the tuple, you make its life easier.

> - realtime sharing
> do all servers in the cloud need to know any "event" immediately? I don't
> think so. In general, you can assume that the "next" mail from the same
> source (be it a retry or a completely new one) will either follow in a
> time period

Yes, you are right here. I never thought about it, but real-time sync is
of no use, since a real server will not retry immediatly (and even if it
does, you want to tempfail that)

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Wed, Jan 21, 2009 at 5:41 PM,  <manu@...> wrote:
> Yes, you are right here. I never thought about it, but real-time sync is
> of no use, since a real server will not retry immediatly (and even if it
> does, you want to tempfail that)

I find that Exchange servers do this very often, that is try every MX
with no delay. What is even worse is that it only tries one round and
then no more. :-( My most common whitelist is Exchange servers for
this reason and their operators are always clueless so it's up to me
to fix it if my client wants to receive mail from them without
problems.

-- 
/peter

Manu@... wrote on Wed, 21 Jan 2009 17:41:18 +0100:

> > what for do we need sender and recipient email address?
> 
> If your ennemy is a spamware operating on a botnet, retaining (IP, from,
> rcpt) in your greylisting database cost ressources to the ennemy:

and for me, too!

it has
> to keep track of the tuple if it wants to defeat greylisting. 
> 
> For the botnet operator, resources are free (as it is operating on
> hacked machines), but they are not infinite. If it does not want to be
> too annoying, which would get it eradicated by the machine owner, it
> cannot consume too much resources. This is why we see spamwares keeping
> track of tuples for 15 minutes, but not for 4 hours.

I don't have any experience with this. I block most dialup/dynamic stuff 
right-away with RBLs and so probably don't see much coming in from Bots, 
anyway.

> 
> If you retain only the IP, the spamware just has to send messages with
> random (from, rcpt) to your server, and after some time, they will get
> through.

If I understand correctly, this model is in play when you greylist based 
on RBLs. e.g. greylist for four hours if in RBL. I think in other cases it 
doesn't make much sense, e.g. "normal" greylisting will not be longer than 
10 or 15 minutes or so. So, it's only a subset of users that need it. Fair 
enough to keep it for them.
But it's not necessary to store this data if lazyaw is in use. And in that 
case would dramatically reduce necessary storage size and bandwidth for 
sharing.



Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Shuttlebox wrote on Wed, 21 Jan 2009 17:49:21 +0100:

> I find that Exchange servers do this very often, that is try every MX
> with no delay. What is even worse is that it only tries one round and
> then no more. :-( My most common whitelist is Exchange servers for
> this reason and their operators are always clueless so it's up to me
> to fix it if my client wants to receive mail from them without
> problems.

Yeah, but you have to do this, anyway.

Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

On Wed, Jan 21, 2009 at 6:37 PM, Kai Schaetzl <maillists@...> wrote:
> Shuttlebox wrote on Wed, 21 Jan 2009 17:49:21 +0100:
>
>> I find that Exchange servers do this very often, that is try every MX
>> with no delay. What is even worse is that it only tries one round and
>> then no more. :-( My most common whitelist is Exchange servers for
>> this reason and their operators are always clueless so it's up to me
>> to fix it if my client wants to receive mail from them without
>> problems.
>
> Yeah, but you have to do this, anyway.

A *lot* less since I started using DNSWL.

http://www.dnswl.org/

-- 
/peter

Kai Schaetzl <maillists@...> wrote:

> If I understand correctly, this model is in play when you greylist based
> on RBLs. e.g. greylist for four hours if in RBL. I think in other cases it
> doesn't make much sense, e.g. "normal" greylisting will not be longer than
> 10 or 15 minutes or so. 

Yes, 15 minutes for normal people and 4 hours for cable/xDSL pools works
great.


-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Wed, Jan 21, 2009 at 06:06:19AM +0100, manu@... wrote:
> Petar Bogdanovic <petar@...> wrote:
> 
> > > (...) I have to confess I fear that it would make milter-greylist a
> > > valuable target for spammers. 
> > I really didn't get that one. Could you please explain?
> 
> Spam is a healthy business. People invest in developping 
> - malware to take over the control of millions of machines
> - spam engines in order to work around spam filters.
> - various smart things to defeat CAPTCHA
> 
> Spammers work on developements that will have a good return on
> investement. Since no milter-greylist implementation scales to a very
> large installations, working on defeating milter-greylist itself is not
> financially attractive: it will mean reachine a small amount of new
> mailboxes. This seems uninteresting when you can work on injecting spams
> in massive mailing platforms. 
> 
> If milter-greylist scales to high, that may not be the case any longer.

SpamAssassin is pretty popular too, but still (after nearly a decade of
being freely available on SourceForge) very effective..



   Petar Bogdanovic

Petar Bogdanovic wrote:
>
> >
> > If milter-greylist scales to high, that may not be the case any longer.
>
> SpamAssassin is pretty popular too, but still (after nearly a decade of
> being freely available on SourceForge) very effective..
>
> Petar Bogdanovic
>
>  <!-- #ygrp-mkp{ border: 1px solid #d8d8d8; 
> font-family: Arial; margin: 14px 0px; padding: 0px 14px; } #ygrp-mkp 
> hr{ border: 1px solid #d8d8d8; } #ygrp-mkp #hd{ color: #628c2a; 
> font-size: 85%; font-weight: bold; line-height: 122%; margin: 10px 
> 0px; } #ygrp-mkp #ads{ margin-bottom: 10px; } #ygrp-mkp .ad{ padding: 
> 0 0; } #ygrp-mkp .ad a{ color: #0000ff; text-decoration: none; } --> 
> <!-- #ygrp-sponsor #ygrp-lc{ font-family: Arial; } #ygrp-sponsor 
> #ygrp-lc #hd{ margin: 10px 0px; font-weight: bold; font-size: 78%; 
> line-height: 122%; } #ygrp-sponsor #ygrp-lc .ad{ margin-bottom: 10px; 
> padding: 0 0; } --> <!-- #ygrp-mlmsg {font-size:13px; font-family: 
> arial,helvetica,clean,sans-serif;*font-size:small;*font:x-small;} 
> #ygrp-mlmsg table {font-size:inherit;font:100%;} #ygrp-mlmsg select, 
> input, textarea {font:99% arial,helvetica,clean,sans-serif;} 
> #ygrp-mlmsg pre, code {font:115% monospace;*font-size:100%;} 
> #ygrp-mlmsg * {line-height:1.22em;} #ygrp-text{ font-family: Georgia; 
> } #ygrp-text p{ margin: 0 0 1em 0; } #ygrp-tpmsgs{ font-family: Arial; 
> clear: both; } #ygrp-vitnav{ padding-top: 10px; font-family: Verdana; 
> font-size: 77%; margin: 0; } #ygrp-vitnav a{ padding: 0 1px; } 
> #ygrp-actbar{ clear: both; margin: 25px 0; white-space:nowrap; color: 
> #666; text-align: right; } #ygrp-actbar .left{ float: left; 
> white-space:nowrap; } .bld{font-weight:bold;} #ygrp-grft{ font-family: 
> Verdana; font-size: 77%; padding: 15px 0; } #ygrp-ft{ font-family: 
> verdana; font-size: 77%; border-top: 1px solid #666; padding: 5px 0; } 
> #ygrp-mlmsg #logo{ padding-bottom: 10px; } #ygrp-reco { margin-bottom: 
> 20px; padding: 0px; } #ygrp-reco #reco-head { font-weight: bold; 
> color: #ff7900; } #reco-grpname{ font-weight: bold; margin-top: 10px; 
> } #reco-category{ font-size: 77%; } #reco-desc{ font-size: 77%; } 
> #ygrp-vital{ background-color: #e0ecee; margin-bottom: 20px; padding: 
> 2px 0 8px 8px; } #ygrp-vital #vithd{ font-size: 77%; font-family: 
> Verdana; font-weight: bold; color: #333; text-transform: uppercase; } 
> #ygrp-vital ul{ padding: 0; margin: 2px 0; } #ygrp-vital ul li{ 
> list-style-type: none; clear: both; border: 1px solid #e0ecee; } 
> #ygrp-vital ul li .ct{ font-weight: bold; color: #ff7900; float: 
> right; width: 2em; text-align:right; padding-right: .5em; } 
> #ygrp-vital ul li .cat{ font-weight: bold; } #ygrp-vital a{ 
> text-decoration: none; } #ygrp-vital a:hover{ text-decoration: 
> underline; } #ygrp-sponsor #hd{ color: #999; font-size: 77%; } 
> #ygrp-sponsor #ov{ padding: 6px 13px; background-color: #e0ecee; 
> margin-bottom: 20px; } #ygrp-sponsor #ov ul{ padding: 0 0 0 8px; 
> margin: 0; } #ygrp-sponsorv li{ list-style-type: square; padding: 6px 
> 0; font-size: 77%; } #ygrp-sponsor #ov li a{ text-decoration: none; 
> font-size: 130%; } #ygrp-sponsor #nc{ background-color: #eee; 
> margin-bottom: 20px; padding: 0 8px; } #ygrp-sponsor .ad{ padding: 8px 
> 0; } #ygrp-sponsor .ad #hd1{ font-family: Arial; font-weight: bold; 
> color: #628c2a; font-size: 100%; line-height: 122%; } #ygrp-sponsor 
> .ad a{ text-decoration: none; } #ygrp-sponsor .ad a:hover{ 
> text-decoration: underline; } #ygrp-sponsor .ad p{ margin: 0; } 
> o{font-size: 0; } .MsoNormal{
It's fascinating that milter-greylist still works as well as it does.  
It seems like it would be trivial for a spammer to make multiple 
attempts, and many do.  Do a lot of ips end up on blacklists before they 
can try again?  It seems like a lot of blacklists are very slow to log 
new spammers.  For instance I have week old spam whose ips aren't on 
spamhaus yet. 

But then again I definitely don't understand the economics of spam.  I'm 
guessing that a lot of spammers are themselves victims of scams.  
Something like "pay us a hundred bucks, reach millions of customers" or 
"make millions with internet marketing."  At this point, no products 
actually get sold via spam, the product is the spam itself.  Just 
guessing.  If that's the case there may be an actual disincentive to 
bypass spam filters because you don't want your customers getting shut 
down very quickly.  Just let them think everything's going fine.

I guess malware is a different issue, but is anybody still stupid enough 
to click on a spam link?  I still believe the biggest spam activity 
targets the actual spammers themselves.

Hi,

> Petar Bogdanovic wrote:
> >
> > >
> > > If milter-greylist scales to high, that may not be the case any longer.
> >
> > SpamAssassin is pretty popular too, but still (after nearly a decade of
> > being freely available on SourceForge) very effective..
> >
> > Petar Bogdanovic
> >
> It's fascinating that milter-greylist still works as well as it 
> does.  It seems like it would be trivial for a spammer to make 
> multiple attempts, and many do.  Do a lot of ips end up on 
> blacklists before they can try again?  It seems like a lot of 
> blacklists are very slow to log new spammers.  For instance I have 
> week old spam whose ips aren't on spamhaus yet.
> 
> But then again I definitely don't understand the economics of spam.  
> I'm guessing that a lot of spammers are themselves victims of scams. 
>  Something like "pay us a hundred bucks, reach millions of 
> customers" or "make millions with internet marketing."  At this 
> point, no products actually get sold via spam, the product is the 
> spam itself.  Just guessing.  If that's the case there may be an 
> actual disincentive to bypass spam filters because you don't want 
> your customers getting shut down very quickly.  Just let them think 
> everything's going fine.
> 
> I guess malware is a different issue, but is anybody still stupid 
> enough to click on a spam link?  I still believe the biggest spam 
> activity targets the actual spammers themselves.

In Australia, Nigerian and other scams total over $32mill a year sent
overseas. I'd shudder to think how many get scammed in Europe and USA, but the
worldwide figure is over $2bill a year. Australia is such a small percentage
of that world-wide figure, but it's still substantial enough for spammers to
continue their operations.

So yes, people click on the spam links and get scammed, and reading an article
earlier last year where a mid-twenties spammer retired a millionaire and
(anonymously) went public, 99% of his earnings was generated from the
_vulnerable_ people in our societies. Once he knew what they were vulnerable
with, he targeted them endlessly.

Those people may seem "stupid" to you, but they could have mental health
problems, uncontrollable addictions, not be in their right mind, etc and
continue to get scammed.

As email providers, it's our responsibility to try and help protect such
people from themselves by stopping the illegal activity of the spam/scam
operators.

Regards,

Michael.

Petar Bogdanovic <petar@...> wrote:

> SpamAssassin is pretty popular too, but still (after nearly a decade of
> being freely available on SourceForge) very effective..

Spammers are actively trying to get around SpamAssassin. You have to
keep rules up to date if you want it to keep being effective.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

--- In milter-greylist@yahoogroups.com, Kai Schaetzl <maillists@...>
wrote:

A single database server will be able to handle a significant number
of single domain mx servers. If that is not enough, the next step up
will be a high performance, high availability database cluster. These
type of clusters have multiple points of entry. This is much like how
a http://www.google.com request is directed to one of many web
servers. From the standpoint of milter-greylist interfacing with the
database, there is very little difference between connecting to a
simple SQL server, or a large cluster. Hence the scalability of the db
model.

> As for using a "central" DBMS for better scaling. I fear this will
somehow 
> bite itself in the tail. If you have a larger cloud of mail servers
that 

> means you also have a *lot* of mail traffic and you were required to 
> cluster it to cope with that. Now going to a single DBMS backend means 
> "re-introducing" a single point of possible slowdown and failure. So,

Reschauzier wrote on Thu, 22 Jan 2009 14:26:33 -0000:

> A single database server will be able to handle a significant number
> of single domain mx servers.

There is no such thing as a "single domain mx server". I assume you mean a 
server cluster that handles mail for one company? And further assume that 
a single company has only a small number of domains to handle mail for?
Why do you think that needs less ressources than a "multi-domain" (dozens 
or hundreds of domains) server? And how do you know that a single MySQL 
server will handle this without throttling the processing rate on the 
multiple mail servers?

I'm not saying that you are wrong. I and others just caution that there 
*might* be problems performancewise which in the end might greatly reduce 
the deployment figures of such a solution. So much that it may be not 
worth it to add such an API.

If that is not enough, the next step up
> will be a high performance, high availability database cluster.

You can do that. But think about that. What do you gain with this? I think 
you gain very little but invest a lot of hardware and personell 
ressources. I think most vendors are unlikely going to do that.

Kai

-- 
Kai Sch\ufffdtzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

--- In milter-greylist@yahoogroups.com, Kai Schaetzl <maillists@...>
wrote:

> 
> There is no such thing as a "single domain mx server". I assume you
mean a 
> server cluster that handles mail for one company? And further assume
that 
> a single company has only a small number of domains to handle mail for?
> Why do you think that needs less ressources than a "multi-domain"
(dozens 
> or hundreds of domains) server?

With "single domain mx servers" I mean a pool, cluster or farm of mx
servers that all serve the same domain name.

This is the worst-case scenario for greylisting, since consequent
attempts to deliver can be made to any machine in the mx pool.
Therefore, all mx servers need to be aware of previous attempts to
deliver to any of the other machines.

And how do you know that a single MySQL 
> server will handle this without throttling the processing rate on the 
> multiple mail servers?

I don't. But if not, there is a scaling solution (db cluster), while
with the current setup there is not.

> 
> I'm not saying that you are wrong. I and others just caution that there 
> *might* be problems performancewise which in the end might greatly
reduce 
> the deployment figures of such a solution. So much that it may be not 
> worth it to add such an API.

As I said, if no one is interested (and that certainly seems to be the
case), I will rest my case. So far, I have not heard a good
alternative to extend greylisting to single domains with (extremely)
large volumes. The question may be largely academic, but I feel it can
be an important vector for future development of milter-greylist.

> 
> If that is not enough, the next step up
> > will be a high performance, high availability database cluster.
> 
> You can do that. But think about that. What do you gain with this? I
think 
> you gain very little but invest a lot of hardware and personell 
> ressources. I think most vendors are unlikely going to do that.

Well maybe. But remember these are large scale operations to begin
with (domains with 50 mx servers or more), so adding a database server
is not unthinkable.