Milter-greylist

Emmanuel Dreyfus wrote:
 > I tracked down the problem of slow database reloading in 3.1.x series.
 > 
 > The problem was introduced with the pending_insert_to_queue() and
 > autowhite_insert_to_queue() functions, that sort the in-memory database
 > by date. For each tuple to be inserted, the whole database is searched, 
 > which results in a major slow down when thousands of tuples are reloaded,
 > at startup time.
 > 
 > Are these functions useful at all? I don't see the point of sorting 
 > the tuples, since the code in charge of retreiving tuples does not use
 > their sorted property to win a shortcut.
 > 
 > Attached is a patch that remove the sorting of tuples. It restores the
 > instant loading time we had before 3.1 series. Feedback is welcome: does
 > it harm in any way to nuke that code?

Just a thought:  Maybe it would make sense to sort them,
so you can use a simple binary search algorithm to retrieve
them, which should improve retrieval speed considerably
(it would scale much better with large lists).

Of course, the sorting should be done only once, after
the whole databas has been reloaded.  On the other hand,
the list would need to be kept sorted whenever a new
tuple is added during operation.  For that purposed, maybe
it would be useful to use a balanced tree of some sort.

(Unfortunately, my time is too limited to have a look at
implementing it myself.  :-(  But maybe someone else can
pick up the idea.)

Of course, all of that would be unnecessary if we had a
real SQL backend.  ;-)  (SCNR)

Best regards
   Oliver

-- 
Oliver Fromme,  secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing
Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd
Any opinions expressed in this message may be personal to the author
and may not necessarily reflect the opinions of secnetix in any way.

"... there are two ways of constructing a software design:  One way
is to make it so simple that there are _obviously_ no deficiencies and
the other way is to make it so complicated that there are no _obvious_
deficiencies."        -- C.A.R. Hoare, ACM Turing Award Lecture, 1980