Milter-greylist

I just had this happen again over the weekend. A simple restart with no change
in parameters dropped the "size" (as reported by top) of milter-greylist from
over 1 gig to 11,872k.

I'm upgrading to 3.0 now, however, it concerns me that I seem to be suffering
from memory leaks and there's no mention of any memory leak problems being fixed
in the ChangeLog.


Is anyone else seeing explosively-huge milter-greylists recently?

This doesn't look like dictionary-attack overload. My MTA died on Sunday at
14:08, but grepping my logs for milter-greylist create and removed messages
there's nothing unusual around this timeframe.

Number created/removed by hour:
"Dec 10 08:"    268/263
"Dec 10 09:"    216/314
"Dec 10 10:"    265/303
"Dec 10 11:"    208/273
"Dec 10 12:"    253/282
"Dec 10 13:"     208/255
"Dec 10 14:"      41/22

Looking at MRTG, memory usage on the server stayed normal from 11/21 until
Thursday 12/7. Starting on 12/7, memory usage on the server crept up slowly and
steadily until the MTA died on 12/10.


Looking back at last Sunday, for comparison, the load was lower, but not by that
much:

"Dec  3 11:"	190/542
"Dec  3 12:"	217/199
"Dec  3 13:"	166/193


However, looking at what happened when I restarted my server at 10:56 today:
"Dec 11 10:56"    14497/8858

But between 12/10 15:00 and 12/11 10:50, pretty much no create or removes happened.

"Dec 10 2.:"	0/0
"Dec 11 0.:"	0/0

Does MG only do management when sendmail does a connection? That explains all
the removes.. but what about the 14k creates the same minute I restarted? Is
that it just reloading the DB?

I reloaded it again, an hour later..
"Dec 11 11:56"	10060/6

So, my DB isn't a whole lot smaller than it was at the 10:56 reload. The removes
come first, so I suppose at reload time my DB was about 23,355 entries, and now
it's about 10,000. But that's a factor of 2 DB size, but memory is a factor of
100 less...

Any ideas?


on 11/22/2006 Matt Kettler wrote:
> I know, 3.0 final is out and I should upgrade to it, it's on my todo.
> 
> However, this morning I came in to find my milter-greylist running with a vsize
> of 975MB. That's substantially larger than I've ever seen it grow to before.
> Upon reducing my timeout from 5d to 3d and restarting, it shrank to 8mb. Right
> now, 4 hours later, it's up to 17mb.
> 
> Looking at the "dumping records" in my maillog, the largest number of records
> all week is 40,445. That was at 2:30 this morning.
> 
> Right before reload it dumped 34007 records. After reloading due to the shorter
> timeout it reduced itself to 23,280 records.
> 
> Currently it's got 23,253 records. Which is almost half of what the peak was,
> but the memory usage is a factor of 65 smaller than it was earlier. This leads
> me to suspect the size problem is something more than just a lot of records.
> 
> 2:30am	40445 records, unknown size
> 11am	34007 records, 975m process size (according to top)
> 11:15am 23280 records 8m process size
> 3pm	23253 records  17m process size
> 
> Currently (3pm) ps reports a vsz of 44692 and a RSS of 5280
> 		top reports a size 17460, rss of 5264  and share of 3224
> I don't have ps info for the other points in time.
> 
> 
> Looking at the ChangeLog, I don't see anything that would suggest any leaking or
> memory overflows was fixed in 3.0.
> 
> 3.0
>         Fix crashes during dump reloads (AIDA Shinra)
>         Fix DoS in MX sync protocol (AIDA Shinra)
> 
> Neither of those sounds like they'd cause my excessive memory use problem.
> 
> Anyone else seeing similar problems?
> 
> FWIW I am using the new DNSBL code, and am using 14 DNSBLs, but those are
> actually boiled down to 3 unique lists. (so milter-greylist makes 14 queries,
> but 11 of them will be cached because the query is redundant. The difference
> being what result-code the test checks for.)
>