Milter-greylist

Hello,

Last week, I installed milter-greylist for some email addresses.
Last night, the file /var/milter-greylist/greylist.db displays:

---------------------- Sample 1 -----------------------------------
[ .... ]
222.136.25.31             <virginia@...>    <xxxx.xxxxxxxx@...> 1085536020 # 2004-05-26 03:47:00
24.60.250.191                  <ssu@...>    <xxxx.xxxxxxxx@...> 1085536023 # 2004-05-26 03:47:03
24.98.118.46                  <derrek@...>    <xxxx.xxxxxxxx@...> 1085536025 # 2004-05-26 03:47:05
24.34.148.233                <sakti@...>    <xxxx.xxxxxxxx@...> 1085536029 # 2004-05-26 03:47:09
61.84.231.102                <gaoyuan@...>    <xxxx.xxxxxxxx@...> 1085536033 # 2004-05-26 03:47:13
61.249.203.23                   <icap@...>    <xxxx.xxxxxxxx@...> 1085536039 # 2004-05-26 03:47:19
4.7.50.95                  <randall@...>    <xxxx.xxxxxxxx@...> 1085536048 # 2004-05-26 03:47:28
65.24.194.20                <subhednu@...>    <xxxx.xxxxxxxx@...> 1085536050 # 2004-05-26 03:47:30
66.188.95.20                    <gene@...>    <xxxx.xxxxxxxx@...> 1085536076 # 2004-05-26 03:47:56
67.163.217.46               <dryden@...>    <xxxx.xxxxxxxx@...> 1085536077 # 2004-05-26 03:47:57
67.162.129.13               <somasama@...>    <xxxx.xxxxxxxx@...> 1085536079 # 2004-05-26 03:47:59
[ .... ]
---------------------- Sample 2 -----------------------------------
200.53.248.142              <libpcap@...>     <yyyyyy@...>        1085544283 # 2004-05-26 06:04:43
24.16.253.21           <tudor@...>     <yyyyyy@...>        1085544284 # 2004-05-26 06:04:44
24.157.153.40          <cmsg@...>     <yyyyyy@...>        1085544361 # 2004-05-26 06:06:01
24.159.241.11         <orlandini@...>     <yyyyyy@...>        1085544364 # 2004-05-26 06:06:04
200.82.47.94      <Soille@...>     <yyyyyy@...>        1085544372 # 2004-05-26 06:06:12
24.128.119.17                <Qobi@...>     <yyyyyy@...>        1085544378 # 2004-05-26 06:06:18
24.14.26.219       <jamesm@...>     <yyyyyy@...>        1085544379 # 2004-05-26 06:06:19
218.48.37.81      <abiword_bugs@...>     <yyyyyy@...>        1085544384 # 2004-05-26 06:06:24
24.130.151.178              <snowhare@...>     <yyyyyy@...>        1085544388 # 2004-05-26 06:06:28
   [ ... 600 lines deleted : from 06:04:43 to 07:10:23 ]

Here 600 is a big number, but VERY OFTEN I have 20-30 connections in 
2 minutes for a SINGLE destination, but from 20-30 differents IP
and differents From:.

My interpretation: a spammer wants to send something to <yyyyyy@...>,
it fails from 200.53.248.142 / <libpcap@...>, and so he retries
from another PC within a pool of "relays", and so on.

So there are 2 denies of service:
. large amount of SMTP connections in a short time (= fork with sendmail);
. large amount of data collected in the greylist database.
	
	Question: isn't that the perfect tool to destroy the idea of greylisting?

--
Jacques Beigbeder                    |  Jacques.Beigbeder@...
Service de Prestations Informatiques |     http://www.spi.ens.fr
Ecole normale sup\ufffdrieure             |
45 rue d'Ulm                         |Tel : (+33 1)1 44 32 37 96
F75230 Paris cedex 05                |Fax : (+33 1)1 44 32 20 75

Jacques Beigbeder <jacques.beigbeder@...> writes:

> Here 600 is a big number, but VERY OFTEN I have 20-30 connections in 
> 2 minutes for a SINGLE destination, but from 20-30 differents IP
> and differents From:.
>
> My interpretation: a spammer wants to send something to <yyyyyy@...>,
> it fails from 200.53.248.142 / <libpcap@...>, and so he retries
> from another PC within a pool of "relays", and so on.
>
> So there are 2 denies of service:
> . large amount of SMTP connections in a short time (= fork with sendmail);

This can be controlled by fixing the max number of childs and combined with
connection refuse on high LA values you can dramaticaly mitigate any risk
of sendmail DoS, except a constant flood of port knocking at 25/TCP.

> . large amount of data collected in the greylist database.

No precise idea about this but imho the real persistent data is the whitelist,
anything else can be wiped on a regular basis or even lost without any severe
impact milter-greylist behaviour.

> Question: isn't that the perfect tool to destroy the idea of greylisting?

I believe that under flood conditions a spmmer would be able to DoS your
box but how would his crap be received by our MTA then ?
Is that his real goal ?

On Wed, May 26, 2004 at 01:27:31PM +0200, Jacques Beigbeder wrote:
> Here 600 is a big number, but VERY OFTEN I have 20-30 connections in 
> 2 minutes for a SINGLE destination, but from 20-30 differents IP
> and differents From:.
(snip)
> So there are 2 denies of service:
> . large amount of SMTP connections in a short time (= fork with sendmail);
> . large amount of data collected in the greylist database.

Surely this is a DDOS, but it is not especially aimed at greylisting. 
With greylisting, it eats your memory, and if you use SpamAssassin, it will
compeltely crush your machine under CPU load (good point: once the system
resources are exhausted, you crash and the attack is stopped)

Even if you don't filter anything, it can fill your spool, so it's just some
malicious attack not really related to mail filtering. 

The day spammers will want to defeat greylisting, they'll just have to 
upgrade the spam relay on virus-compromished hosts so that they correctly
handle temporary failures by retrying later.

But I have the hope we can win the next battle by using distributed honeypot
addresses networks. Speaking about this, I've already wrote some code, but I'm 
looking for testers.

-- 
Emmanuel Dreyfus
manu@...

On Wed, May 26, 2004 at 01:50:08PM +0200, Cyril Guibourg wrote:
> I believe that under flood conditions a spmmer would be able to DoS your
> box but how would his crap be received by our MTA then ?
> Is that his real goal ?

That's why I think this is just some evil assault which is not related
to spam. I already saw floods of messages that knocked down my servers,
and obviously it was not aimed to be readen at all: mail with wrong sender
addresses in my dimain sent to random wrong addresses with real domains: I got
crushed by my own postmaster notifications caused by remote site's mailer 
daemons messages hitting invalid addresses in my domain. Evil, isn't it?

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus <manu@...> writes:

> But I have the hope we can win the next battle by using distributed
> honeypot addresses networks.

This is clearly an escalation process.

> Speaking about this, I've already wrote some code, but I'm looking for
> testers.

Once I am done with the milter-greylist FreeBSD port I will be glad to play
with it.

On Wed, May 26, 2004 at 02:01:17PM +0200, Cyril Guibourg wrote:
> > But I have the hope we can win the next battle by using distributed
> > honeypot addresses networks.
> This is clearly an escalation process.

That's fine for me provided I remain ahead of the spammers during that 
process :)

-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus <manu@...> writes:

> That's fine for me provided I remain ahead of the spammers during that 
> process :)

Run ! run !!!   ;^)

Jacques Beigbeder wrote:
> Hello,
> 
> Last week, I installed milter-greylist for some email addresses.
> Last night, the file /var/milter-greylist/greylist.db displays:
> 
> ---------------------- Sample 1 -----------------------------------
> [ .... ]
> 222.136.25.31             <virginia@...>    <xxxx.xxxxxxxx@...> 1085536020 # 2004-05-26 03:47:00
> 24.60.250.191                  <ssu@...>    <xxxx.xxxxxxxx@...> 1085536023 # 2004-05-26 03:47:03
> 24.98.118.46                  <derrek@...>    <xxxx.xxxxxxxx@...> 1085536025 # 2004-05-26 03:47:05
> 24.34.148.233                <sakti@...>    <xxxx.xxxxxxxx@...> 1085536029 # 2004-05-26 03:47:09
> 61.84.231.102                <gaoyuan@...>    <xxxx.xxxxxxxx@...> 1085536033 # 2004-05-26 03:47:13
> 61.249.203.23                   <icap@...>    <xxxx.xxxxxxxx@...> 1085536039 # 2004-05-26 03:47:19
> 4.7.50.95                  <randall@...>    <xxxx.xxxxxxxx@...> 1085536048 # 2004-05-26 03:47:28
> 65.24.194.20                <subhednu@...>    <xxxx.xxxxxxxx@...> 1085536050 # 2004-05-26 03:47:30
> 66.188.95.20                    <gene@...>    <xxxx.xxxxxxxx@...> 1085536076 # 2004-05-26 03:47:56
> 67.163.217.46               <dryden@...>    <xxxx.xxxxxxxx@...> 1085536077 # 2004-05-26 03:47:57
> 67.162.129.13               <somasama@...>    <xxxx.xxxxxxxx@...> 1085536079 # 2004-05-26 03:47:59
> [ .... ]
> ---------------------- Sample 2 -----------------------------------
> 200.53.248.142              <libpcap@...>     <yyyyyy@...>        1085544283 # 2004-05-26 06:04:43
> 24.16.253.21           <tudor@...>     <yyyyyy@...>        1085544284 # 2004-05-26 06:04:44
> 24.157.153.40          <cmsg@...>     <yyyyyy@...>        1085544361 # 2004-05-26 06:06:01
> 24.159.241.11         <orlandini@...>     <yyyyyy@...>        1085544364 # 2004-05-26 06:06:04
> 200.82.47.94      <Soille@...>     <yyyyyy@...>        1085544372 # 2004-05-26 06:06:12
> 24.128.119.17                <Qobi@...>     <yyyyyy@...>        1085544378 # 2004-05-26 06:06:18
> 24.14.26.219       <jamesm@...>     <yyyyyy@...>        1085544379 # 2004-05-26 06:06:19
> 218.48.37.81      <abiword_bugs@...>     <yyyyyy@...>        1085544384 # 2004-05-26 06:06:24
> 24.130.151.178              <snowhare@...>     <yyyyyy@...>        1085544388 # 2004-05-26 06:06:28
>    [ ... 600 lines deleted : from 06:04:43 to 07:10:23 ]
> 
> Here 600 is a big number, but VERY OFTEN I have 20-30 connections in 
> 2 minutes for a SINGLE destination, but from 20-30 differents IP
> and differents From:.
> 
> My interpretation: a spammer wants to send something to <yyyyyy@...>,
> it fails from 200.53.248.142 / <libpcap@...>, and so he retries
> from another PC within a pool of "relays", and so on.
> 
> So there are 2 denies of service:
> . large amount of SMTP connections in a short time (= fork with sendmail);
> . large amount of data collected in the greylist database.

I've seen that too. I've ended up with

dnl Max connexions par secondes
define(confCONNECTION_RATE_THROTTLE,`10')

in my sendmail.mc to limit the impact of such attacks/spam farms behaviour.

-- 
Matthieu Herrb

>> Jacques Beigbeder wrote:
>> >Hello,
>> >
>> >Last week, I installed milter-greylist for some email addresses.
>> >Last night, the file /var/milter-greylist/greylist.db displays:
>> ...
>> >So there are 2 denies of service:
>> >. large amount of SMTP connections in a short time (= fork with sendmail);
>> >. large amount of data collected in the greylist database.
>> 
>> I've seen that too. I've ended up with
>> 
>> dnl Max connexions par secondes
>> define(confCONNECTION_RATE_THROTTLE,`10')
>> 
>> in my sendmail.mc to limit the impact of such attacks/spam farms behaviour.

May be but...
If I look at my setting, I estimate there are 100 more SMTP connections
per user per day due to greylisting. For a site with 5000 users (my
case), this means that 200.000 SMTP connections per day will become
700.000... My mail server will have 10 connections per second all
along the day, and also 10 records written in database every second.
(any SMTP connection ends in database, as new or as refreshed).
The database will be many (100?) Mb...

--
Jacques Beigbeder                    |  Jacques.Beigbeder@...
Service de Prestations Informatiques |     http://www.spi.ens.fr
Ecole normale supérieure             |
45 rue d'Ulm                         |Tel : (+33 1)1 44 32 37 96
F75230 Paris cedex 05                |Fax : (+33 1)1 44 32 20 75

On Wed, May 26, 2004 at 05:43:16PM +0200, Jacques Beigbeder wrote:
> May be but...
> If I look at my setting, I estimate there are 100 more SMTP connections
> per user per day due to greylisting. For a site with 5000 users (my
> case), this means that 200.000 SMTP connections per day will become
> 700.000... 

When you count 200k connections per day, do you count everything, or
everything minus spam and viruses?

If it's everything, then most of your connections won't be retried because
of greylisting, since spam and viruses never retry. So you don't jump to
700k. 

How many connections are legitimate mail on your system? That's the key 
point. I'd say it's less than 10% on my servers.

> My mail server will have 10 connections per second all
> along the day, and also 10 records written in database every second.
> (any SMTP connection ends in database, as new or as refreshed).
> The database will be many (100?) Mb...

That's getting really big. 10 modifications per second are alright provided 
you don't dump on each change (you need milter-greylist-1.3.3 or a patch
to avoid that). The greylist is just a chained list in RAM, so that should
be quite fast.

FWIW, 215 users asked for greylisting on my site, and my greylist database
is 1MB big. Assuming your users have the same usage as mine, with 5000
users, you'll have a 25MB database. Sounds okay, if you dump it every 10
minutes.

-- 
Emmanuel Dreyfus
manu@...

>> > If I look at my setting, I estimate there are 100 more SMTP connections
>> > per user per day due to greylisting. For a site with 5000 users (my
>> > case), this means that 200.000 SMTP connections per day will become
>> > 700.000... 
>> 
>> When you count 200k connections per day, do you count everything, or
>> everything minus spam and viruses?

Everything, including spams and virus.
 
>> If it's everything, then most of your connections won't be retried because
>> of greylisting, since spam and viruses never retry. So you don't jump to
>> 700k. 

I don't agree. I estimated 100 MORE connections per day per user.
5000 users = 500.000 more SMTP connections.

For virus, I know they can retry:
May 19 19:28:18 nef milter-greylist: i4JHSIl0089807: addr 217.174.203.201 from <> to <Webmaster@...> delayed for 00:58:00
May 19 19:34:59 nef milter-greylist: i4JHYxuG093055: addr 217.174.203.201 from <> to <Webmaster@...> delayed for 00:51:19
May 19 19:54:59 nef milter-greylist: i4JHsxmT003327: addr 217.174.203.201 from <> to <Webmaster@...> delayed for 00:31:19
May 19 20:28:20 nef milter-greylist: i4JISIFi019762: addr 217.174.203.201 from <> rcpt <Webmaster@...>: autowhitelisted for 720:00:00
This is W32/Sober.g@MM.

--
Jacques Beigbeder                    |  Jacques.Beigbeder@...
Service de Prestations Informatiques |     http://www.spi.ens.fr
Ecole normale sup\ufffdrieure             |
45 rue d'Ulm                         |Tel : (+33 1)1 44 32 37 96
F75230 Paris cedex 05                |Fax : (+33 1)1 44 32 20 75

Quoting Matthieu Herrb <matthieu.herrb@...>:

> Jacques Beigbeder wrote:
> > Hello,
> > 
> > Last week, I installed milter-greylist for some email addresses.
> > Last night, the file /var/milter-greylist/greylist.db displays:
> > 
> > ---------------------- Sample 1 -----------------------------------
> > [ .... ]
> > 222.136.25.31             <virginia@...>    <xxxx.xxxxxxxx@...>
> 1085536020 # 2004-05-26 03:47:00
> > 24.60.250.191                  <ssu@...>    <xxxx.xxxxxxxx@...>
> 1085536023 # 2004-05-26 03:47:03
> > 24.98.118.46                  <derrek@...>    <xxxx.xxxxxxxx@...>
> 1085536025 # 2004-05-26 03:47:05
> > [ .... ]
> > ---------------------- Sample 2 -----------------------------------
> > 200.53.248.142              <libpcap@...>     <yyyyyy@...>       
> 1085544283 # 2004-05-26 06:04:43
> > 24.16.253.21           <tudor@...>     <yyyyyy@...>       
> 1085544284 # 2004-05-26 06:04:44
> > 24.157.153.40          <cmsg@...>     <yyyyyy@...>       
> 1085544361 # 2004-05-26 06:06:01
> > 24.159.241.11         <orlandini@...>     <yyyyyy@...>       
> 1085544364 # 2004-05-26 06:06:04
> > 200.82.47.94      <Soille@...>     <yyyyyy@...>       
> 1085544372 # 2004-05-26 06:06:12
> > 24.128.119.17                <Qobi@...>     <yyyyyy@...>       
> 1085544378 # 2004-05-26 06:06:18
> > 24.14.26.219       <jamesm@...>     <yyyyyy@...>       
> 1085544379 # 2004-05-26 06:06:19
> > 218.48.37.81      <abiword_bugs@...>     <yyyyyy@...>       
> 1085544384 # 2004-05-26 06:06:24
> > 24.130.151.178              <snowhare@...>     <yyyyyy@...>       
> 1085544388 # 2004-05-26 06:06:28
> >    [ ... 600 lines deleted : from 06:04:43 to 07:10:23 ]
> > 
> > Here 600 is a big number, but VERY OFTEN I have 20-30 connections in 
> > 2 minutes for a SINGLE destination, but from 20-30 differents IP
> > and differents From:.
> > 
> > My interpretation: a spammer wants to send something to <yyyyyy@...>,
> > it fails from 200.53.248.142 / <libpcap@...>, and so he retries
> > from another PC within a pool of "relays", and so on.
> > 
> > So there are 2 denies of service:
> > . large amount of SMTP connections in a short time (= fork with sendmail);
> > . large amount of data collected in the greylist database.
> 
> I've seen that too. I've ended up with
> 
> dnl Max connexions par secondes
> define(confCONNECTION_RATE_THROTTLE,`10')
> 
> in my sendmail.mc to limit the impact of such attacks/spam farms behaviour.
> 

This kind of DoS occurs not only with grey-listing but also with 
use of blacklists at the sendmail level. Now some spammers seems to 
do retry from other IPs when getting a 4xx or 5xx SMTP reject code
(they seem to have some distributed sending robots). 
They also often use a different FROM: email address each time. 

Connection rate control (globally or on a per IP/network basis - only 
in sendmail 8.13beta2) helps but It's getting more and more difficult
to fight them - I see more and more spam coming from some IPs not yet
listed in serious blacklists). 


Stephane
---
http://milter.free.fr/intro/

Jacques Beigbeder <jacques.beigbeder@...> writes:


> May 19 20:28:20 nef milter-greylist: i4JISIFi019762: addr 217.174.203.201
> from <> rcpt <Webmaster@...>: autowhitelisted for 720:00:00
> This is W32/Sober.g@MM.

How do you tell this *is* a virus and *not* some MTA trying to deliver some
infected message ?

$ telnet vserver-raccourci.nexen.net 25
Trying 217.174.203.201...
Connected to vserver-raccourci.nexen.net.
Escape character is '^]'.
220 mail.raccourci.fr ESMTP

Jacques Beigbeder <jacques.beigbeder@...> wrote:

> For virus, I know they can retry:
> May 19 19:28:18 nef milter-greylist: i4JHSIl0089807: addr 217.174.203.201
> May 19 from <> to <Webmaster@...> delayed for 00:58:00 19:34:59 nef
> May 19 milter-greylist: i4JHYxuG093055: addr 217.174.203.201 from <> to
> May 19 <Webmaster@...> delayed for 00:51:19 19:54:59 nef
> May 19 milter-greylist: i4JHsxmT003327: addr 217.174.203.201 from <> to
> May 19 <Webmaster@...> delayed for 00:31:19 20:28:20 nef
> May 19 milter-greylist: i4JISIFi019762: addr 217.174.203.201 from <> rcpt
> May 19 <Webmaster@...>: autowhitelisted for 720:00:00
> This is W32/Sober.g@MM.

I'm pretty convinced it's a real mail server that relayed the virus. I
suspect it just accepted the virus from its internal domain and
propagates it outside. 217.174.203.201 even responds on port 25.  

AFAIK real viruses do not handle retries yet.

But you should consider virus filtering before greylisting. I'm not
surprised your greylist database grew so big if you greylist viruses. 

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Dan Hollis <goemon@...> wrote:

[Honeypots] 
> In order for honeypot to be useful it needs to be easy to deploy, like
> milter-greylist. E.g. not these spaghetti mess of other honeypot software.
> 
> Hopefully it's something like a few .c file and a single config in /etc
> 
> And it doesn't run as root...

Yes, DST (Distributed Spam Traps) already does all of this. But there is
no doc written yet, and the config file probably needs some refining.
Check out the code here ftp://ftp.espci.fr/pub/dst

There is basically two items:
dstd is a daemon used to relay spam reports in a NNTP fashion. It has
the ability to just forward reports to other dstd, the ability to
maintain a Berkeley DB database of seen reports, and it can feed a
DNSRBL by performing DNS updates.

dstc is the spam trap itself. It eats messages on stdin, parse the
headers looking for the sender IP address and report it to a dstd. It
can sign the report using a RSA key, and dstd can verify the signature
before feeding a DNSRBL. dstc is ran from /etc/aliases or .forward.

The gaol was to have something efficient and highly interoperable, so
that it can be easily deployed. Everything is written in C, and it will
work with any MTA: dstc works with delivery to a program, and
blacklisting works by using a DNSRBL, something that all MTA support.

Dependencies: OpenSSL (for RSA signing/verifying), BIND9 with the DNS
resolver (for DNS updates), Berkeley DB, and a POSIX thread library.

Now the only issue is to deal with this scenario: spammers discover a
spam trap and start throwing spam at it through a big ISP mail server.
How can we avoid blacklisting the ISP mail server? I think we need some
karma scheme here: you count spam reports divided by total messages and
see if you still want to receive messages from that server. 

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

On Wed, 26 May 2004, Emmanuel Dreyfus wrote:
> But I have the hope we can win the next battle by using distributed honeypot
> addresses networks. Speaking about this, I've already wrote some code, but I'm 
> looking for testers.

In order for honeypot to be useful it needs to be easy to deploy, like 
milter-greylist. E.g. not these spaghetti mess of other honeypot software.

Hopefully it's something like a few .c file and a single config in /etc

And it doesn't run as root...

-Dan

On Wed, 26 May 2004, Jacques Beigbeder wrote:
> My interpretation: a spammer wants to send something to <yyyyyy@...>,
> it fails from 200.53.248.142 / <libpcap@...>, and so he retries
> from another PC within a pool of "relays", and so on.

This is really it. He's thinking that his IP has been blacklisted in some 
RBL so he's trying again from another IP hoping that not all of them are 
blacklisted.

It's just the usual desperate criminals.

> 	Question: isn't that the perfect tool to destroy the idea of greylisting?

Only if the greylist daemon is inefficient. Efficient one should be able 
to withstand attack of any size.

-Dan

Dan Hollis <goemon@...> wrote:

> Database still has problems: long email addresses are truncated.

Please show me a real-life situation where this triggers a real problem.
 
> Is the database storage system going to change? You said you had given up
> on berkeley db.

It depends on the feedbacks about version 1.3.3. If people are happy
with it or not.

But don't take the text file as what it's not: it's just a checkpoint to
resume operation after a crash. milter-greylist only reads it on
startup, then everything is done in main memory. milter-greylist never
ever reads from the text file once started. 

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

On Wed, 26 May 2004, Emmanuel Dreyfus wrote:
> FWIW, 215 users asked for greylisting on my site, and my greylist database
> is 1MB big. Assuming your users have the same usage as mine, with 5000
> users, you'll have a 25MB database. Sounds okay, if you dump it every 10
> minutes.

Database still has problems: long email addresses are truncated.

Is the database storage system going to change? You said you had given up 
on berkeley db.

-Dan

On Wed, 26 May 2004, Cyril Guibourg wrote:
> Jacques Beigbeder <jacques.beigbeder@...> writes:
> > May 19 20:28:20 nef milter-greylist: i4JISIFi019762: addr 217.174.203.201
> > from <> rcpt <Webmaster@...>: autowhitelisted for 720:00:00
> > This is W32/Sober.g@MM.
> How do you tell this *is* a virus and *not* some MTA trying to deliver some
> infected message ?
> $ telnet vserver-raccourci.nexen.net 25
> Trying 217.174.203.201...
> Connected to vserver-raccourci.nexen.net.
> Escape character is '^]'.
> 220 mail.raccourci.fr ESMTP

most viruses don't have a listener on 25/tcp.

-Dan

Matthieu Herrb <matthieu.herrb@...> wrote:

> > But you should consider virus filtering before greylisting. I'm not
> > surprised your greylist database grew so big if you greylist viruses.
> 
> I wonder how you'd do that. Given how milter and greylists works, it 
> looks impossible to me. Greylisting happens before the body of the 
> message has been received, so you can't run a virus scanner before 
> greylisting.

Mmm you are right, I'm greylisting viruses while I thought I wasn't.
That's bad, I'd like to fix it.
 
It could be fixed by sending the answer after the end of the DATA
command, but I don't see how this could work with multi-recipient
messages where some recipients use greylisting and some others don't.

There is another bad thing with answering after RCPT commands: sender
callbacks such as milter-sender's one get greylisted.

Any ideas?

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

manu@... wrote:

> But you should consider virus filtering before greylisting. I'm not
> surprised your greylist database grew so big if you greylist viruses. 

I wonder how you'd do that. Given how milter and greylists works, it 
looks impossible to me. Greylisting happens before the body of the 
message has been received, so you can't run a virus scanner before 
greylisting.

And you can't add an anti-virus gateway in front of you sendmail with 
milter-greylist without breaking the greylist idea.
-- 
					Matthieu

On Thu, May 27, 2004 at 07:50:10AM +0200, Matthieu Herrb wrote:
> >But you should consider virus filtering before greylisting. I'm not
> >surprised your greylist database grew so big if you greylist viruses. 
> I wonder how you'd do that. Given how milter and greylists works, it 
> looks impossible to me. Greylisting happens before the body of the 
> message has been received, so you can't run a virus scanner before 
> greylisting.

And it is a bad idea anyway. Virus checking causes a lot of load on
mail server. So it is good if milter greylist blocks virus even
if that makes the database bigger.

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

On Thu, May 27, 2004 at 07:21:26AM +0200, Emmanuel Dreyfus wrote:
> Mmm you are right, I'm greylisting viruses while I thought I wasn't.
> That's bad, I'd like to fix it.

No, it is *good*. Scanning for a virus is expensive. One of the reasons
I'm using milter greylist is that it keeps a lot of virus infected
e-mails from my mailbox.

	Kind regards

-- 
Matthias Scheler                                  http://scheler.de/~matthias/

On Thu, May 27, 2004 at 12:49:24PM +0200, Matthias Scheler wrote:
> No, it is *good*. Scanning for a virus is expensive. One of the reasons
> I'm using milter greylist is that it keeps a lot of virus infected
> e-mails from my mailbox.

At least this could be configurable. I filter viruses using milter-regex,
it costs CPU but not memory, so I'd prefer greylisting to take place after
virus filtering.

-- 
Emmanuel Dreyfus
manu@...

manu@n... writes:

>> Database still has problems: long email addresses are truncated.
>
> Please show me a real-life situation where this triggers a real
> problem.

It triggers a problem when using regexp's in the configuration
file. E.g.

| rcpt /spamtrap-prefix.*@.../
| rcpt /maillists+foobarnet.blablub.com=mydomain.com@.*/
                               ^

would *never* be matched as no tested rcpt will have the characters
after the marked place (^). That's a very confusing behavior and it is
documented nowhere (afaik).




Enrico

On Tue, Jun 01, 2004 at 01:58:05PM -0000, enscensc wrote:
[e-mail addresses are truncated to 32 chars]
> It triggers a problem when using regexp's in the configuration
> file. E.g.
> 
> | rcpt /spamtrap-prefix.*@.../
> | rcpt /maillists+foobarnet.blablub.com=mydomain.com@.*/
>                                ^
> would *never* be matched as no tested rcpt will have the characters
> after the marked place (^). That's a very confusing behavior and it is
> documented nowhere (afaik).

Hum, that's a real problem. 
I see several way of fixing the problem
1) raise the truncation to a higher value (64? 128?)
This impacts the memory usage.

2) ensure regex are matched on the address before it is truncated

3) Document it as a known problem

Opinions?
-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus <manu@...> writes:

> Hum, that's a real problem. 
> I see several way of fixing the problem
> 1) raise the truncation to a higher value (64? 128?)
> This impacts the memory usage.
> 2) ensure regex are matched on the address before it is truncated
>
> 3) Document it as a known problem
>
> Opinions?

What about storing an MD5 hash instead of the address ? 
 
The drawback I see is that there would be no way to easily interpret
the dump db but after all, syslogs can help when tracking white list 
candidates.

On Tue, Jun 01, 2004 at 04:34:50PM +0200, Cyril Guibourg wrote:
> What about storing an MD5 hash instead of the address ? 
>  
> The drawback I see is that there would be no way to easily interpret
> the dump db but after all, syslogs can help when tracking white list 
> candidates.

And how do you match a regex on a hashed address?

-- 
Emmanuel Dreyfus
manu@...

Cyril Guibourg wrote:
> Emmanuel Dreyfus <manu@...> writes:
> 
> 
>>Hum, that's a real problem. 
>>I see several way of fixing the problem
>>1) raise the truncation to a higher value (64? 128?)
>>This impacts the memory usage.
>>2) ensure regex are matched on the address before it is truncated
>>
>>3) Document it as a known problem
>>
>>Opinions?
> 
> 
> What about storing an MD5 hash instead of the address ? 

The MD5 doesn't help if you want to do regex matching, but I'm not sure 
if there are cases were you need to match an address in the database 
against a regex.

I'd raise the limit, and change the database format to use a single 
white space character as separator, so that it remains of a reasonable 
size on disk.

This means that the adress needs to be encoded in quoted-printable or 
such in the case it contains some white-space (or these incorrect 
addresses could be filtered out by rejecting the message entierly).

Anyways, a limit is needed (even if computing a MD5 hash) to protect 
milter-greylist from DoS attack by arbitrary large addresses. But here 
too addresses over the limit can be simply rejected instead of inserted 
in the grey list.

-- 
Matthieu Herrb

Emmanuel Dreyfus <manu@...> writes:

> And how do you match a regex on a hashed address?

I wasn't speaking about the regex but the [non]matching address. If memory
consumption is a problem, you can store in the db md5 hashes of addresses
that have been already checked.

For the regex, my understanding is that once compiled, it has to be left
as is.

Matthieu Herrb <matthieu.herrb@...> writes:

> The MD5 doesn't help if you want to do regex matching, but I'm not
> sure if there are cases were you need to match an address in the
> database against a regex.

I thought about MD5 when storing adresses that do not match the 
regex. Once the test is done, storing the hash instead of the full adress
itself would save memory.

> I'd raise the limit, and change the database format to use a single
> white space character as separator, so that it remains of a reasonable
> size on disk.

Why not.

> This means that the adress needs to be encoded in quoted-printable or
> such in the case it contains some white-space (or these incorrect
> addresses could be filtered out by rejecting the message entierly).

This put a limit on mail address length driven by conservative
memory usage concerns.

> Anyways, a limit is needed (even if computing a MD5 hash) to protect
> milter-greylist from DoS attack by arbitrary large addresses. But here
> too addresses over the limit can be simply rejected instead of
> inserted in the grey list.

Agree with that. Btw, with MD5 you know that whatever the adress length
is, the room needed to store the hash is 33 bytes. Even if evil people
try to DoS milter-greylist with very long address they will not hurt
much more then with small length ones.

Cyril Guibourg <cg+milter-greylist@...> writes:

> the room needed to store the hash is 33 bytes.

If you store it in ascii.

-- 
^x^i ~/.signature

On Tue, Jun 01, 2004 at 04:48:07PM +0200, Matthieu Herrb wrote:
> I'd raise the limit, and change the database format to use a single 
> white space character as separator, so that it remains of a reasonable 
> size on disk.

Okay, but to what value should we raise the limit?
You'll laways fid some ill-configured software that sends long addreses.

> This means that the adress needs to be encoded in quoted-printable or 
> such in the case it contains some white-space (or these incorrect 
> addresses could be filtered out by rejecting the message entierly).

That problem is already handled: whitespaces are replaced by underscores
before storage and comparisons.

> Anyways, a limit is needed (even if computing a MD5 hash) to protect 
> milter-greylist from DoS attack by arbitrary large addresses. But here 
> too addresses over the limit can be simply rejected instead of inserted 
> in the grey list.

That will generate false positives...


-- 
Emmanuel Dreyfus
manu@...

On Tue, Jun 01, 2004 at 05:19:05PM +0200, Cyril Guibourg wrote:
> Agree with that. Btw, with MD5 you know that whatever the adress length
> is, the room needed to store the hash is 33 bytes. Even if evil people
> try to DoS milter-greylist with very long address they will not hurt
> much more then with small length ones.

You didn't understand the problem of MD5 and regex
I send amail, rcpt <foobarbuz@...>
It gets through a MD5 hash: ad71c6bc1d51a4be31aaddd7ce5c7ac3

Then you want to match this against /.*@example\.com/ How do you immagine that?


-- 
Emmanuel Dreyfus
manu@...

Emmanuel Dreyfus <manu@...> writes:

> You didn't understand the problem of MD5 and regex I send amail, rcpt
> <foobarbuz@...> It gets through a MD5 hash:
> ad71c6bc1d51a4be31aaddd7ce5c7ac3
>
> Then you want to match this against /.*@example\.com/ How do you
> immagine that?

Why should I try to match it against a regex ? It would be already in the
list of "greyed" addresses because the match test did occur before.

Something like:

    You send a mail to <foobarbuz@...>


    hash = md5(address)

    greyed = lookup(list, hash)
    
    if (greyed != NULL)
       do things for already seen addresses
    else      
       ok = match(whitelist, address)
       if (ok == NULL)
          install(list, hash, STATE_GREY)
       else
           install(list, hash, STATE_WHITE)
           return accept_message
       endif
    endif

The only reason to match it again would be a change in the config file
that would end in clearing all already stored hashes.

Imho this is what causes a potential problem: can we afford to flush
the whole greyed adresses db upon conf reload because regex were amended ?

This is a trade off between conservative memory usage and greyed db
reconstruction time.

With the hope it clarifies.

manu@... (Emmanuel Dreyfus) writes:

> [e-mail addresses are truncated to 32 chars]
>> It triggers a problem when using regexp's in the configuration
>> file. E.g.
>> 
>> | rcpt /spamtrap-prefix.*@.../
>> | rcpt /maillists+foobarnet.blablub.com=mydomain.com@.*/
>>                                ^
>> would *never* be matched as no tested rcpt will have the characters
>> after the marked place (^). That's a very confusing behavior and it is
>> documented nowhere (afaik).
>
> Hum, that's a real problem. 
> I see several way of fixing the problem
> 1) raise the truncation to a higher value (64? 128?)

Longest 'FROM' addresses are around 90 characters (maillist bounce-addresses)
here. So 128 seems to be a good value.


> This impacts the memory usage.

* Is this really a problem with nowadays hardware (1GB+ RAM)?
* Why not allocate the needed memory dynamically?


> 2) ensure regex are matched on the address before it is truncated

Sounds good, and is probably the best choice. Doing this would allow to
store hashes in the database: the comparisions with settings in the
configuration file (rcpt/from) happens on the real addresses (pointers
given by the milter-interface), and the lookup in the database is done
with hashes.

This would make milter-greylist very efficiently:

* generate *one* hash over the entire triple, e.g.
  | sha1("%s#%s#%s", relay, from, rcpt)  -->  40 bytes ASCII resp. 20 bytes binary
  - I would recommend sha1 over md5 since it is more collision resistent
  - to stay at case insensitive comparisions, transform everything to
    lower- or uppercase
* order the pending- and auto-whitelist 
* do a binary search over these lists (insert is more expensive than
  the current linked-list implementation, but there should be far more
  'lookup' than 'insert' operations)


Advantages:
* reduced memory consumption (20 bytes + timestamp per entry)
* faster (O(log n) for lookup)
* enhanced privacy


> 3) Document it as a known problem

A last resort solution... ;)




Enrico

On Tue, Jun 01, 2004 at 06:30:00PM +0200, Cyril Guibourg wrote:
> Why should I try to match it against a regex ? It would be already in the
> list of "greyed" addresses because the match test did occur before.

In fact, the fix to the original problem is trivial: I just have to give
except_rcpt_filter() and except_sender_filter() the pointer to the 
original string, not the copy that was truncated to 32 chars. This will
fix the regex problem.

I wonder I should do that before 1.4 or after 1.4. It seems trivial,
but it might be a can of worms waiting to be open, you never know.

-- 
Emmanuel Dreyfus
manu@...

On Tue, Jun 01, 2004 at 06:32:49PM +0200, Enrico Scholz wrote:
> Longest 'FROM' addresses are around 90 characters (maillist bounce-addresses)
> here. So 128 seems to be a good value.

It would be best to refer to the RFCs here. RFC2821 says:

4.5.3.1 Size limits and minimums

   There are several objects that have required minimum/maximum sizes.
   Every implementation MUST be able to receive objects of at least
   these sizes.  Objects larger than these sizes SHOULD be avoided when
   possible.  However, some Internet mail constructs such as encoded
   X.400 addresses [16] will often require larger objects: clients MAY
   attempt to transmit these, but MUST be prepared for a server to
   reject them if they cannot be handled by it.  To the maximum extent
   possible, implementation techniques which impose no limits on the
   length of these objects should be used.

   local-part
      The maximum total length of a user name or other local-part is 64
      characters.

   domain
      The maximum total length of a domain name or number is 255
      characters.

On Tue, Jun 01, 2004 at 06:32:49PM +0200, Enrico Scholz wrote:
> > This impacts the memory usage.
> * Is this really a problem with nowadays hardware (1GB+ RAM)?

My setup has 256 MB, so yes, memory usage can be a problem for me :)

> > 2) ensure regex are matched on the address before it is truncated
> Sounds good, and is probably the best choice. Doing this would allow to
> store hashes in the database: the comparisions with settings in the
> configuration file (rcpt/from) happens on the real addresses (pointers
> given by the milter-interface), and the lookup in the database is done
> with hashes.
> 
> This would make milter-greylist very efficiently:

I have no idea how costly is cryptographic hash computing. Are you sure
it won't consume more CPU if we go that way?

> * generate *one* hash over the entire triple, e.g.
>   | sha1("%s#%s#%s", relay, from, rcpt)  -->  40 bytes ASCII resp. 20 bytes binary

You can't do that: you loose the ability to perform subnet matching. At most,
you can hash from and rcpt addresses, but not the IP. Or you store the IP
next to the hash, and each time the config file is changed, you renegerate
the hashes. That's more complicated (I've done it for Bekeley DB support, 
it was a pain to debug).

>   - I would recommend sha1 over md5 since it is more collision resistent

I was wondering what was the best hash to use here. I suspect we don't care
about the security usage of hashes (ie: it is difficult to create two identical
hashes with different data on purpose), but we need efficiency, and collision 
resistance.

> * do a binary search over these lists (insert is more expensive than
>   the current linked-list implementation, but there should be far more
>   'lookup' than 'insert' operations)

I thought about this too, but before going that path, we need to measure
how long a lookup is. It's useless to optimize something that is low 
compared to other problems. I suspect the database dump to be the biggest
CPU hog. Would you like to run some tests?

> Advantages:
> * reduced memory consumption (20 bytes + timestamp per entry)
> * faster (O(log n) for lookup)
> * enhanced privacy

I disagree with the last advantage: the log file is only readen by the
administrator, and the ability to easily debug things is probably more
useful than the ability to hide what was greylisted to the sysadmin.

If you follow the privacy path too far, you configure syslog to send
mail.* to /dev/null...


-- 
Emmanuel Dreyfus
manu@...

jeffw@... (Jeff Wasilko) writes:

> On Tue, Jun 01, 2004 at 06:32:49PM +0200, Enrico Scholz wrote:
>> Longest 'FROM' addresses are around 90 characters (maillist bounce-addresses)
>> here. So 128 seems to be a good value.
>
> It would be best to refer to the RFCs here. RFC2821 says:
>    ...
>    local-part
>       The maximum total length of a user name or other local-part is 64
>       characters.

securityfocus.com has bounce-addresses exceeding this; e.g.

| focus-virus-euro-return-294-maillists+security.focus-virus=MYDOMAIN@...

('MYDOMAIN' is 6 characters long in reality)


>    domain
>       The maximum total length of a domain name or number is 255
>       characters.

This means, that 320 byte must be reserved for an address. This is
probably too much...




Enrico

On Tue, Jun 01, 2004 at 08:07:43PM +0200, Enrico Scholz wrote:
> 
> This means, that 320 byte must be reserved for an address. This is

> probably too much...

I guess this is one tradeoff of using an on-disk db (like BDB)
vs. trying to keep it in memory.

As someone who's tried to do high-transaction rate stuff with
BDB, I understand your reluctance to use it. It's so easy to get
deadlocked with it...

-j

On Tue, 1 Jun 2004, Emmanuel Dreyfus wrote:
> On Tue, Jun 01, 2004 at 04:48:07PM +0200, Matthieu Herrb wrote:
> > I'd raise the limit, and change the database format to use a single 
> > white space character as separator, so that it remains of a reasonable 
> > size on disk.
> Okay, but to what value should we raise the limit?
> You'll laways fid some ill-configured software that sends long addreses.

The current situation is too short, I think i've pointed that out already 
:-)

Is it possible to decide at runtime with a config option? Those with big 
boxes can decide how much memory they want to dedicate to it.

-Dan

On Tue, 1 Jun 2004, Emmanuel Dreyfus wrote:
> I have no idea how costly is cryptographic hash computing. Are you sure
> it won't consume more CPU if we go that way?

sha1 isnt _that_ expensive. Figure about 4000 cycles for 64 bytes.
This means on typical 1ghz cpu you can do 250,000 sha1/sec.

> >   - I would recommend sha1 over md5 since it is more collision resistent
> I was wondering what was the best hash to use here. I suspect we don't care
> about the security usage of hashes (ie: it is difficult to create two identical
> hashes with different data on purpose), but we need efficiency, and collision 
> resistance.

sha1 is more resistant. there have been studies (and warnings) of 
collisions on md5.

-Dan

manu@... (Emmanuel Dreyfus) writes:

>> > 2) ensure regex are matched on the address before it is truncated
>> Sounds good, and is probably the best choice. Doing this would allow to
>> store hashes in the database: the comparisions with settings in the
>> configuration file (rcpt/from) happens on the real addresses (pointers
>> given by the milter-interface), and the lookup in the database is done
>> with hashes.
>> 
>> This would make milter-greylist very efficiently:
>
> I have no idea how costly is cryptographic hash computing. Are you sure
> it won't consume more CPU if we go that way?

My P4 2.6 calculates the md5sum of 1GiB data in 5 seconds and the sha1sum
in 15 seconds:

| $ dd if=/dev/zero bs=1024k count=1024 | time md5sum
| 4.62user 0.51system 0:05.24elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
|
| $ dd if=/dev/zero bs=1024k count=1024 | time sha1sum
| 14.56user 0.40system 0:15.18elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k


For milter-greylist, we will calculate the checksum of 50-200 bytes of
data per connection. Somewhere I read about 200k connections/day, which
is still far behind these 1GiB above, so the costs for md5/sha1 should
not matter.


>> * generate *one* hash over the entire triple, e.g.
>>   | sha1("%s#%s#%s", relay, from, rcpt)  -->  40 bytes ASCII resp. 20 bytes binary
>
> You can't do that: you loose the ability to perform subnet matching.

Mmh... perhaps I am missing here something, but why not do

| relay &= mask;

*before* calculating the checksum? E.g. '10.0.0.0' will be used in
hash-calculation for both 10.0.0.1 and 10.0.0.2 when using '-L 24'.


> At most, you can hash from and rcpt addresses, but not the IP. Or
> you store the IP next to the hash, and each time the config file is
> changed, you renegerate the hashes.

Yes, you will loss the auto-whitelist when changing the '-L' parameter, but
since such changes are an exception and greylisting is self-generating, it
should not be a problem.


>>   - I would recommend sha1 over md5 since it is more collision resistent
>
> I was wondering what was the best hash to use here. I suspect we don't care
> about the security usage of hashes (ie: it is difficult to create two identical
> hashes with different data on purpose),

With relatively high costs a spammer could generate some addresses
with the same hash. But indeed, I do not see how this can be used to
circumvent greylisting.



>> * do a binary search over these lists (insert is more expensive than
>>   the current linked-list implementation, but there should be far more
>>   'lookup' than 'insert' operations)
>
> I thought about this too, but before going that path, we need to measure
> how long a lookup is. It's useless to optimize something that is low 
> compared to other problems. I suspect the database dump to be the biggest
> CPU hog.

ACK; but the delay-option should reduce the impact.


> Would you like to run some tests?

I fear, that I can not present representative results since I have only
a low mailtraffic (50 people company, some maillists, most frequently
connecting relays are white-listed). For today, I get

(1) neither in pending nor auto-white: 2 lookup() & 1 insert() & 0 remove() --> 192 times
(2) in pending, but still in timeout:  1 lookup() & 0 insert() & 0 remove() -->  66 times
(3) in pending, reached timeout:       2 lookup() & 1 insert() & 1 remove() -->  66 times
(4) already in auto-whitelist:         2 lookup() & 0 insert() & 0 remove() --> 129 times

1511 entries in auto-whitelist, 230 in pending. I do not have numbers
about expiration of database entries.


Results were gathered with

| # grep 'please come back in' /var/log/maillog  | \
|      grep    <initial-timeout> | wc -l              ## --> (1)
| # grep 'please come back in' /var/log/maillog  | \
|      grep -v <initial-timeout> | wc -l              ## --> (2)
| # grep 'X-Greylist:' /var/log/maillog | wc -l       ## --> (3)
| # grep 'for more' /var/log/greylist | wc -l         ## --> (4)


>> Advantages:
>> * enhanced privacy
>
> I disagree with the last advantage: the log file is only readen by the
> administrator, and the ability to easily debug things is probably more
> useful than the ability to hide what was greylisted to the sysadmin.

I am not an expert in privacy law but some people could interpret
'greylist.db' as a list of profiles which exceeds common logging.



Enrico

Jeff Wasilko <jeffw@...> wrote:

> I guess this is one tradeoff of using an on-disk db (like BDB)
> vs. trying to keep it in memory.

Keeping everything in memory makes sense since you walk the whole
greylist on each new spam that gets in. 
 
> As someone who's tried to do high-transaction rate stuff with
> BDB, I understand your reluctance to use it. It's so easy to get
> deadlocked with it...

The biggest issue is that you have no data integrity garantee if the
power goes down. My biggest concern being reliability, I prefer a text
based dump because you can always fix things by hand if you need it.
 
-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Dan Hollis <goemon@...> wrote:

> The current situation is too short, I think i've pointed that out already
> :-)
> 
> Is it possible to decide at runtime with a config option? Those with big
> boxes can decide how much memory they want to dedicate to it.

Change ADDRLEN in dump.h to the value you like and rebuild.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Enrico Scholz <greylist-milter@...> wrote:
 
> I am not an expert in privacy law but some people could interpret
> 'greylist.db' as a list of profiles which exceeds common logging.

There is no information in greylist.db that you couldn't find in
maillog...

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

On Tue, 1 Jun 2004 manu@... wrote:
> Dan Hollis <goemon@...> wrote:
> > Is it possible to decide at runtime with a config option? Those with big
> > boxes can decide how much memory they want to dedicate to it.
> Change ADDRLEN in dump.h to the value you like and rebuild.

I think I said runtime? Yeah, I did :-)

-Dan

On Tue, 1 Jun 2004 manu@... wrote:
> Jeff Wasilko <jeffw@...> wrote:
> > I guess this is one tradeoff of using an on-disk db (like BDB)
> > vs. trying to keep it in memory.
> Keeping everything in memory makes sense since you walk the whole
> greylist on each new spam that gets in.

I wonder if this won't cause scalability problems in the future on very 
large systems (eg 10,000's of users). Multiply by the unique senders and 
unique IPs...

> > As someone who's tried to do high-transaction rate stuff with
> > BDB, I understand your reluctance to use it. It's so easy to get
> > deadlocked with it...
> The biggest issue is that you have no data integrity garantee if the
> power goes down. My biggest concern being reliability, I prefer a text
> based dump because you can always fix things by hand if you need it.

Doesnt bdb have transactions? That would seem to solve the data integrity 
issue.

Another alternative would be sql.

-Dan

Dan Hollis <goemon@...> wrote:

> > > Is it possible to decide at runtime with a config option? Those with big
> > > boxes can decide how much memory they want to dedicate to it.
> > Change ADDRLEN in dump.h to the value you like and rebuild. 
> I think I said runtime? Yeah, I did :-)

Ok, since you really ask for it, let me rephrase:

Add a config option in conf_lex.l and conf_yacc.y for this, change every
occurence of a string of size ADDRLEN to a malloc'ed string, handle
memory disposal without a leak, and send me a patch.

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Dan Hollis <goemon@...> wrote:

> I wonder if this won't cause scalability problems in the future on very
> large systems (eg 10,000's of users). Multiply by the unique senders and
> unique IPs...

If you have such a big system, then you won't object throwing a few more
GB of RAM to fix that problem :)

-- 
Emmanuel Dreyfus
Il y a 10 sortes de personnes dans le monde: ceux qui comprennent 
le binaire et ceux qui ne le comprennent pas.
manu@...

Emmanuel Dreyfus <manu@...> wrote:
> Dan Hollis <goemon@...> wrote:
> > I wonder if this won't cause scalability problems in the future on very
> > large systems (eg 10,000's of users). Multiply by the unique senders and
> > unique IPs...
> If you have such a big system, then you won't object throwing a few more
> GB of RAM to fix that problem :)

I don't mean ram, but rather that the serial db lookups could end up 
taking a lot of cpu.

On-disk database would alleviate both memory pressure and lookups (via 
keying).

-Dan