Milter-greylist

Hello,

We're running recent 4.0+ milter-greylist (2008-03-19 snapshot), and 
make extensive use of "urlcheck" feature for custom tests.

Currently our URL server is an Apache 2.2 which serves small CGI script.

All is fine as long as our CGI scripts only returns the one-line 
response "milterGreylistStatus: Ok" if they pass, and nothing if they 
fail (after a "Content-type: text/plain\n\n" MIME header).

But if we try to return multi-line responses, like :

milterGreylistStatus: Ok
milterGreylistMsg: You are not allowed to connect

strange things occur. The first line (the answer itself) is generally 
honoured, but subsequent lines are ignored and generate errors in the 
log like :

milter-greylist: ignoring unepxected "nimilterGreylistMsg" => "You are 
not allowed to connect"

It looks like garbage characters seems to be inserted on beginning of 
each line.

Curiously enough, error disappear and all statements are honoured (in 
this case: reject message is used) if we insert blank lines between 
statements (ie separating them with \n\n instead of \n):

milterGreylistStatus: Ok

milterGreylistMsg: You are not allowed to connect


Sanity of the output of our scripts has been verified by testing the URL 
manually with a web browser.

So something seems wrong in milter-greylist. Bad string pointer ? memory 
corruption ?

Any idea ?




-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.

Benoit Branciard <benoit.branciard@...> wrote:

> Any idea ?

Could that be a CRLF vs LF problem? Looking at the code, it seems it
does not attempt to handle \r in any way. I'd call it a bug.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Benoit Branciard wrote:
 > We're running recent 4.0+ milter-greylist (2008-03-19 snapshot), and 
 > make extensive use of "urlcheck" feature for custom tests.
 > 
 > Currently our URL server is an Apache 2.2 which serves small CGI script.
 > 
 > All is fine as long as our CGI scripts only returns the one-line 
 > response "milterGreylistStatus: Ok" if they pass, and nothing if they 
 > fail (after a "Content-type: text/plain\n\n" MIME header).
 > 
 > But if we try to return multi-line responses, like :
 > 
 > milterGreylistStatus: Ok
 > milterGreylistMsg: You are not allowed to connect
 > 
 > strange things occur. The first line (the answer itself) is generally 
 > honoured, but subsequent lines are ignored and generate errors in the 
 > log like :

The HTTP protocol follows the telnet standard which
requires that lines are separated by CR+LF sequences,
i.e. "\r\n".

Many programs also accept LF only, but you shouldn't
rely on it.  It seems like milter-greylist is strict.

Best regards
Oliver

PS:  I wish more programmers kept the well-known
sentence from RFC 1122 in mind:  "Be liberal in what
you accept, and conservative in what you send."
That means that your script should output "\r\n"
(conservative), and milter-greylist should also
accept "\n" only (liberal).

-- 
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Gesch\ufffdftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M\ufffdn-
chen, HRB 125758,  Gesch\ufffdftsf\ufffdhrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"I learned Java 3 years before Python.  It was my language of
choice.  It took me two weekends with Python before I was more
productive with it than with Java." -- Anthony Roberts

manu@... a \ufffdcrit :
> Benoit Branciard <benoit.branciard@...> wrote:
> 
>> Any idea ?
> 
> Could that be a CRLF vs LF problem? Looking at the code, it seems it
> does not attempt to handle \r in any way. I'd call it a bug.
> 

My scripts output only \n's, and I just verified Apache didn't mangle 
them to something else.

Should they send \r\n's ?


-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.

Benoit Branciard <benoit.branciard@...> wrote:

> My scripts output only \n's, and I just verified Apache didn't mangle
> them to something else.
> 
> Should they send \r\n's ?

No, it was just an idea. 
Could it be that it's just the last char that get mangled? What happens
if you stick a single space at the end of the line, instead of adding a
blank line?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

manu@... a \ufffdcrit :
> Benoit Branciard <benoit.branciard@...> wrote:
> 
>> My scripts output only \n's, and I just verified Apache didn't mangle
>> them to something else.
>>
>> Should they send \r\n's ?
> 
> No, it was just an idea. 
> Could it be that it's just the last char that get mangled? What happens
> if you stick a single space at the end of the line, instead of adding a
> blank line?
> 


1)

-- urlcheck script:
echo "milterGreylistStatus: Ok "
echo "milterGreylistMsg: You are not allowed to connect "

-- log:
milter-greylist: ignoring unepxected "milterGreylistStatus" => "Ok "
milter-greylist: ignoring unepxected "" => "You are not allowed to connect"

-- action:
both statements are ignord, message passes.


2)

-- urlcheck script:
echo " milterGreylistStatus: Ok"
echo " milterGreylistMsg: You are not allowed to connect"


-- log:
milter-greylist: ignoring unepxected "" => "You are not allowed to connec"

-- action:
milterGreylistStatus is honoured, milterGreylistMsg is not


more leading spaces (I tested 2 and 10) give similar result, but error 
message in log vary slightly (first "" argument seems to contain random 
1- or 2-char stings)


Conclusions (if any):

- milter-greylist does not cope with trailing spaces, it causes 
milterGreylistStatus statement value to be ignored. Actually the code in 
  urlcheck.c seems to skip leadins spaces (at beginning of line and 
after ":"), but not at end of line.

- leading spaces do not solve problem

- I can't see a logic in the bug for the moment

-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.

Benoit Branciard <benoit.branciard@...> wrote:

> - I can't see a logic in the bug for the moment

What's even more surprising is that it works at mine. 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

manu@... a \ufffdcrit :
> Benoit Branciard <benoit.branciard@...> wrote:
> 
>> - I can't see a logic in the bug for the moment
> 
> What's even more surprising is that it works at mine. 
> 

May it be a problem with libcurl ? I use (and compiled with) libcurl3 
7.13.2 (Debian package).

Is it possible that libcurl mangles end-of-lines on the client side if 
they are not CR-LF ? Or that there is thread-safety issues (quite 
unprobable, errors would not be so systematic) ?

-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.

Benoit Branciard a \ufffdcrit :
> manu@... a \ufffdcrit :
>> Benoit Branciard <benoit.branciard@...> wrote:
>>
>>> My scripts output only \n's, and I just verified Apache didn't mangle
>>> them to something else.
>>>
>>> Should they send \r\n's ?
>> No, it was just an idea. 

I think end-of-lines are out of concern.

If I get my test script output \r\n's:

echo -e "milterGreylistStatus: Ok\r"
echo -e "milterGreylistMsg: You are not allowed to connect\r"

errors in mail.log become:

milter-greylist: ignoring unepxected "milterGreylistStatus" => "Ok^M"
milter-greylist: ignoring unepxected "ncmilterGreylistMsg" => "You are 
not allowed to connect"

Problem is still present, and \r are not stripped and interfer with 
parameter values parsing.

I can't find a logic error in answer_parse() in urlcheck.c, so I believe 
data corruption occurs *before* this fucntion is called, but why/where...
Manual wget or lynx of my script URL return correct contents.

-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.

Benoit Branciard <benoit.branciard@...> wrote:

> May it be a problem with libcurl ? I use (and compiled with) libcurl3
> 7.13.2 (Debian package).

I have curl-7.16.0

I use urlcheck extensively to query an LDAP directory, where per-user
filtering settings are stored.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

manu@... a \ufffdcrit :
> Benoit Branciard <benoit.branciard@...> wrote:
> 
>> May it be a problem with libcurl ? I use (and compiled with) libcurl3
>> 7.13.2 (Debian package).
> 
> I have curl-7.16.0
> 
> I use urlcheck extensively to query an LDAP directory, where per-user
> filtering settings are stored.
> 

I upgraded to libcurl3 7.15.5, recompiled milter-greylist with some more 
debug logging messages, and I still get some garbage chars in urlcheck 
responses.

Corruption occurs *before* answer_parse() : I added a mg_log() at 
beginning of this function to dump the response buffer, it already 
contains garbage.

I tested an LDAP urlcheck: it seems to pass uncorrupted, *except* the 
first line, which should contain "DN: ...": two garbage chars appear 
before the DN value.

This may explain why you don't have problems in your setup if you only 
do LDAP checks, since the DN: line is always ignored by milter-greylist.

By comparison, a command-line "curl" on the same URL (LDAP or HTTP), 
executed on the same machine as milter-greylist (same libs), always 
returns expected result, not corrupted.

Still investigating...

-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.

Benoit Branciard a \ufffdcrit :
> 
> Still investigating...
> 

... Found !

in curl_outlet(), if the output buffer length was null, two chars were 
pre-allocated for an trailing \r\0.

The problem was that on subsequent calls of this function, those chars 
were ignored in the buffer length calculation, so that data were written 
at the wrong offset, inserting two uninitialized chars in the middle of 
the buffer.

This happened more or less frequently depending on the number of calls 
to curl_outlet() libcurl did to fill the complete answer.

The attached patch contains a possible fix, as well as a a bit of code 
to handle \r\n responses, and a typo fix.


-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.

On Sat, Apr 12, 2008 at 12:43:30PM +0200, Benoit Branciard wrote:
> The attached patch contains a possible fix, as well as a a bit of code 
> to handle \r\n responses, and a typo fix.

Thank you for tracking this down. I integrated your patch, and here is
milter-greylist 4.1.1

http://ftp.espci.fr/pub/milter-greylist-4.1.1.tgz
MD5 (milter-greylist-4.1.1.tgz) = 525f60bd74c8269f658b99dc5405d5c1

Changes from 4.0:
        Fix corruption in urlcheck parsing (Benoit Branciard)
        Document what package contains libmilter
        Fix error about stat in greylist.conf (Bob Smith)
        Add configurable dumpfile permissions
        Avoid C99 specific syntax (Greg Troxel)
        Documentation fixes
        Add SPF status selection to ACL
        Add id string to ACL
        Fix typos in documentation (Constantine A. Murenin)

-- 
Emmanuel Dreyfus
manu@...

At 12:34 PM 4.12.2008 +0000, you wrote: 
>>>>
  
On Sat, Apr 12, 2008 at 12:43:30PM +0200, Benoit Branciard wrote:
> The attached patch contains a possible fix, as well as a a bit of code 
> to handle \r\n responses, and a typo fix.

Thank you for tracking this down. I integrated your patch, and here is
milter-greylist 4.1.1

<http://ftp.espci.fr/pub/milter-greylist-4.1.1.tgz>http://ftp.espci.fr/pub/m
ilter-greylist-4.1.1.tgz
MD5 (milter-greylist-4.1.1.tgz) = 525f60bd74c8269f658b99dc5405d5c1

Emanuel:

File not found...????

Best,
Jack

(^_^)
Happy trails,
Jack L. Stone

System Admin
Sage-american

On Sat, Apr 12, 2008 at 3:08 PM, Jack L. Stone <jacks@...> wrote:
>  File not found...????

http://ftp.espci.fr/pub/milter-greylist/milter-greylist-4.1.1.tgz

-- 
/peter

On Sat, Apr 12, 2008 at 08:08:27AM -0500, Jack L. Stone wrote:
> http://ftp.espci.fr/pub/milter-greylist-4.1.1.tgz
> File not found...????

Oh, right, I screwed it again...
http://ftp.espci.fr/pub/milter-greylist/milter-greylist-4.1.1.tgz

-- 
Emmanuel Dreyfus
manu@...