Yahoo Groups archive

Milter-greylist

Index last updated: 2026-04-28 23:32 UTC

Thread

Postfix queue ID

Postfix queue ID

2015-09-13 by Steven Hiscocks

Hi,

I've created a patch which allows logging of queue IDs when using
postfix; useful when analysing logs.

This requires the postfix option 'smtpd_delay_open_until_valid_rcpt' be
set to 'no' (default 'yes'). This means the queue ID is assigned post
the "envfrom" milter stage, and therefore available in the "envrcpt"
stage of the milter.

For this I've delayed the setting of the queue ID to the "envrcpt"
milter stage, and also the logging of global SPF whitelisting (appears
to be only use of queue ID pre-"envrcpt" stage).

When not using postfix (or not having the postfix option set) the
behaviour should be the same.

-- 
Steven Hiscocks

Re: [milter-greylist] Postfix queue ID [1 Attachment]

2015-09-14 by Emmanuel Dreyfus

On Sun, Sep 13, 2015 at 08:30:35PM +0100, Steven Hiscocks steven@... [milter-greylist] wrote:
> I've created a patch which allows logging of queue IDs when using
> postfix; useful when analysing logs.

Thank you for that contribution.

I am in the middle of a home computer mess and I cannot merge patches
right now. Please be patient and do not hesitate to repost in a week
if you do not hear from me.

-- 
Emmanuel Dreyfus
manu@...

Re: [milter-greylist] Postfix queue ID [1 Attachment]

2015-09-27 by manu@...

Steven Hiscocks steven@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

> When not using postfix (or not having the postfix option set) the
> behaviour should be the same.

Thanks for taht patch that plugs a long standing complain. I am a bit
scared about the "should" above, and need to test the patch on sendmail
before merging it. Anyone feels free to beat me at testing it.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] Postfix queue ID [1 Attachment]

2015-10-04 by manu@...

Steven Hiscocks steven@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

> I've created a patch which allows logging of queue IDs when using
> postfix; useful when analysing logs.

This one is checked in.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

dnsrbl for IPv6 [1 Attachment]

2015-12-23 by Kazuyuki YOSHIDA

Hi,

I've created a patch which allows dnsrbl for IPv6 MTAs.
There are few Blacklist for IPv6 MTAs, but some Whitelists exist.
MTAWL.org includes IPv6 MTAs.

This patch is for 4.5.11, but it can use for 4.5.16, I think.
--
Kazuyuki Yoshida
Oita University

Attachments

Re: [milter-greylist] dnsrbl for IPv6 [1 Attachment]

2015-12-23 by Emmanuel Dreyfus

On Wed, Dec 23, 2015 at 04:09:28PM +0900, Kazuyuki YOSHIDA yoshida@... [milter-greylist] wrote:
> I've created a patch which allows dnsrbl for IPv6 MTAs.
> There are few Blacklist for IPv6 MTAs, but some Whitelists exist.
> MTAWL.org includes IPv6 MTAs.

For next submission please observe
- 80 char/line max
- spaces around binary operators such as == or &&

My comments:
- if (sa->sa_family == AF_INET) ; deserves a switch/case 
- I have trouble to understand why ip6tostringfordns cannot be made
more straightforward. Could you please detail the desired output format?

-- 
Emmanuel Dreyfus
manu@...

Re: [milter-greylist] dnsrbl for IPv6

2015-12-23 by Kazuyuki YOSHIDA

Dear Emmanuel Dreyfus,

Thank you for your comments.
"ip6tostringfordns " changes IPv6 address to String for DNS like
IPv6 Address 2607:f8b0:400e:c00::248
"8.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.c.0.e.0.0.4.0.b.8.f.7.0.6.2"

It is concatenated with DNS RBL URL like ".list.dnswl.org"  at line 196 
of dnsrbl.c, then
"8.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.c.0.e.0.0.4.0.b.8.f.7.0.6.2.list.dnswl.org"

In ip6tostringfordns, IPv6 address is changed per Byte (2 Hexadecimal 
digits),
like one byte of IPv6 Address like 0x12 should change String "2.1".
So large transformation table is used.
--
Kazuyuki Yoshida

On 2015/12/23 19:32, Emmanuel Dreyfus manu@... [milter-greylist] 
wrote:
Show quoted textHide quoted text
>
> On Wed, Dec 23, 2015 at 04:09:28PM +0900, Kazuyuki YOSHIDA 
> yoshida@... [milter-greylist] wrote:
> > I've created a patch which allows dnsrbl for IPv6 MTAs.
> > There are few Blacklist for IPv6 MTAs, but some Whitelists exist.
> > MTAWL.org includes IPv6 MTAs.
>
> For next submission please observe
> - 80 char/line max
> - spaces around binary operators such as == or &&
>
> My comments:
> - if (sa->sa_family == AF_INET) ; deserves a switch/case
> - I have trouble to understand why ip6tostringfordns cannot be made
> more straightforward. Could you please detail the desired output format?
>
> -- 
> Emmanuel Dreyfus
> manu@...
>
>

Re: [milter-greylist] dnsrbl for IPv6

2015-12-23 by Hajimu UMEMOTO

Hi,

>>>>> On Wed, 23 Dec 2015 21:19:59 +0900
>>>>> "Kazuyuki YOSHIDA yoshida@... [milter-greylist]" <milter-greylist@yahoogroups.com> said:

yoshida> In ip6tostringfordns, IPv6 address is changed per Byte (2 Hexadecimal 
yoshida> digits),
yoshida> like one byte of IPv6 Address like 0x12 should change String "2.1".
yoshida> So large transformation table is used.

It seems somewhat redundant to me.  How about the attached patch?
Sorry but I don't test it, actually.

Sincerely,

Attachments

Re: [milter-greylist] dnsrbl for IPv6 [1 Attachment]

2015-12-23 by Hajimu UMEMOTO

Hi,

>>>>> On Thu, 24 Dec 2015 03:48:45 +0900
>>>>> "Hajimu UMEMOTO ume@... [milter-greylist]" <milter-greylist@yahoogroups.com> said:

yoshida> In ip6tostringfordns, IPv6 address is changed per Byte (2 Hexadecimal 
yoshida> digits),
yoshida> like one byte of IPv6 Address like 0x12 should change String "2.1".
yoshida> So large transformation table is used.

ume> It seems somewhat redundant to me.  How about the attached patch?
ume> Sorry but I don't test it, actually.

Oops, I sent wrong version.  I send the patch, again.
Sorry for my inconvenience.

Sincerely,

Attachments

Re: [milter-greylist] dnsrbl for IPv6 [1 Attachment]

2015-12-24 by Kazuyuki YOSHIDA

Hi,

Thank you for your refinement > Hajimu.
It's nice!

# two patches at 3:48(+9) & 4:08(+9) look same.
There is a extra '(' in patch for dnsrbl.h.
+char *iptorevstr((struct sockaddr *, socklen_t, char *, size_t);

Sincerely,
--
Kazuyuki Yoshida
Oita University
Show quoted textHide quoted text
On 2015/12/24 4:05, Hajimu UMEMOTO ume@... [milter-greylist] wrote:
> [Attachment(s) <#TopText> from Hajimu UMEMOTO included below]
>
> Hi,
>
> >>>>> On Thu, 24 Dec 2015 03:48:45 +0900
> >>>>> "Hajimu UMEMOTO ume@... [milter-greylist]" 
> <milter-greylist@yahoogroups.com> said:
>
> yoshida> In ip6tostringfordns, IPv6 address is changed per Byte (2 
> Hexadecimal
> yoshida> digits),
> yoshida> like one byte of IPv6 Address like 0x12 should change String 
> "2.1".
> yoshida> So large transformation table is used.
>
> ume> It seems somewhat redundant to me. How about the attached patch?
> ume> Sorry but I don't test it, actually.
>
> Oops, I sent wrong version. I send the patch, again.
> Sorry for my inconvenience.
>
> Sincerely,
>
> 
>
>
> --
> Hajimu UMEMOTO
> ume@...  ume@...
> http://www.mahoroba.org/~ume/

Re: [milter-greylist] dnsrbl for IPv6

2015-12-24 by Hajimu UMEMOTO

Hi,

>>>>> On Thu, 24 Dec 2015 10:30:54 +0900
>>>>> "Kazuyuki YOSHIDA yoshida@... [milter-greylist]" <milter-greylist@yahoogroups.com> said:

yosh> # two patches at 3:48(+9) & 4:08(+9) look same.

No, there was a difference in an index of the addr array. :)

yosh> There is a extra '(' in patch for dnsrbl.h.
yosh> +char *iptorevstr((struct sockaddr *, socklen_t, char *, size_t);

Oops, thanks.

I reworked for further simplify.  Now, istorevstr() does thing which
reverse_endian() did, as well.

Attachments

Re: [milter-greylist] dnsrbl for IPv6 [1 Attachment]

2015-12-24 by Emmanuel Dreyfus

> Oops, I sent wrong version.  I send the patch, again.
> Sorry for my inconvenience.

It looks fine, I would just like to hear usage feedback before
checking it in. 

-- 
Emmanuel Dreyfus
manu@...

Re: [milter-greylist] dnsrbl for IPv6

2015-12-28 by Kazuyuki YOSHIDA

Hi,

Usage of DNSRBL for IPv6 is same as usage of DNSRBL for IPv4.
Some DNSRBLs, such as dnswl.org, virbl.dnsbl.bit.nl, and spamhaus.org, 
may have entries
of IPv6 addresses.

For example, Followings are added to greylist.conf.
--------
dnsrbl "MTAWL" list.dnswl.org 127.0.0.0/16
dnsrbl "VIRBL" virbl.dnsbl.bit.nl 127.0.0.2

racl whitelist dnsrbl "MTAWL"
racl blacklist dnsrbl "VIRBL" msg "Sender IP caught in VIRBL blacklist"
--------

There are many entries of IPv6 addresses in whitelist (dnswl.org),
but few entries of IPv6 addresses in blacklist(virbl.dnsbl.bit.nl).
Following entry can be found in /var/log/maillog.  (Local parts of mail 
addresses are omitted)

Dec 27 05:35:10 kuju milter-greylist: tBQKZ87K014136: skipping greylist 
because address 2607:f8b0:4002:c07::22a is in DNSRBL, 
(from=<??????@...>, rcpt=<?????@...>, 
addr=mail-yk0-x22a.google.com[2607:f8b0:4002:c07::22a]) ACL 150

I try to add DNSRBL for IPv6 to greylist.conf.5
--
Kazuyuki Yoshida


On 2015/12/24 14:15, Emmanuel Dreyfus manu@... [milter-greylist] 
wrote:
Show quoted textHide quoted text
>
> > Oops, I sent wrong version. I send the patch, again.
> > Sorry for my inconvenience.
>
> It looks fine, I would just like to hear usage feedback before
> checking it in.
>
> -- 
> Emmanuel Dreyfus
> manu@...
>
>

Attachments

Re: [milter-greylist] dnsrbl for IPv6 [1 Attachment]

2015-12-28 by Emmanuel Dreyfus

On Mon, Dec 28, 2015 at 03:22:47PM +0900, Kazuyuki YOSHIDA yoshida@... [milter-greylist] wrote:
> > It looks fine, I would just like to hear usage feedback before
> > checking it in.

I meant: feedback that latest patch from Hajimu UMEMOTO works (the 
obvious syntax error being fixed, of course).

-- 
Emmanuel Dreyfus
manu@...

Re: [milter-greylist] dnsrbl for IPv6

2015-12-28 by Kazuyuki YOSHIDA

Milter-greylist with the Hajimu UMEMOTO's patch of 11:33AM(+9) 24 Dec. 
2015 is operating
normaly in these 4 days(24-28 Dec.) on our server.
--
Kazuyuki Yohida


On 2015/12/28 17:32, Emmanuel Dreyfus manu@... [milter-greylist] 
wrote:
Show quoted textHide quoted text
>
> On Mon, Dec 28, 2015 at 03:22:47PM +0900, Kazuyuki YOSHIDA 
> yoshida@... [milter-greylist] wrote:
> > > It looks fine, I would just like to hear usage feedback before
> > > checking it in.
>
> I meant: feedback that latest patch from Hajimu UMEMOTO works (the
> obvious syntax error being fixed, of course).
>
> -- 
> Emmanuel Dreyfus
> manu@...
>
>

Re: [milter-greylist] dnsrbl for IPv6 [1 Attachment]

2016-01-03 by manu@...

Hajimu UMEMOTO ume@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

> I reworked for further simplify.  Now, istorevstr() does thing which
> reverse_endian() did, as well.

I checked this patch in.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: Postfix queue ID

2016-02-02 by mauricio.teixeira@...

Hello,

I'm new to this list, so I'm sorry if this has been discussed/answered before.

I have recently started testing milter-greylist-4.5.16, and I modified my postfix as suggested in this post. However the log output is not consistent, so it will be very hard to debug greylisting issues when moving this server to production. What I mean is that only one or two entries in the log get the queue ID, while it's missing for everything else, so it would be hard to co-relate facts if I would be receiving many messages in parallel at the exact same time (see the log output below).

So the are two possible ways of fixing it:
a) Adding the queue ID to every single logged message; or
b) Adding the process ID to each one of them.

The ideal, obviously, would be doing both.

Any thoughts?

Feb 2 16:29:58 mx01 postfix/smtpd[9818]: connect from cli01[10.10.16.1]
Feb 2 16:29:58 mx01 milter-greylist: smfi_getsymval failed for {if_addr}, consider setting localaddr option
Feb 2 16:29:58 mx01 milter-greylist: smfi_getsymval failed for {daemon_port}, using default smtp port
Feb 2 16:29:58 mx01 milter-greylist: p0f_lookup: 10.10.16.1[54750] -> 0.0.0.0[6400]
Feb 2 16:29:58 mx01 milter-greylist: SPF return temperror (test code 6, result 0)
Feb 2 16:29:58 mx01 milter-greylist: SPF lookup performed in 0.006320s
Feb 2 16:29:58 mx01 postfix/smtpd[9818]: 7F85CC005910: client=cli01[10.10.16.1]
Feb 2 16:29:58 mx01 milter-greylist: 7F85CC005910: addr = cli01[10.10.16.1], from = <user1@example.com>, rcpt = <user2@example.com>
Feb 2 16:29:58 mx01 milter-greylist: Mail from=<user1@example.com>, rcpt=<user2@example.com>, addr=cli01[10.10.16.1] is matched by entry racl 108 continue from_re /.*/ [addheader "X-Greylist: inspected by %V for IP:'%i' DOMAIN:'%d' HELO:'%h' FROM:'%f' RCPT:'%r'"]
Feb 2 16:29:58 mx01 milter-greylist: DNSRBL lookup 1.16.10.10.sh-sbl.example.com performed in 0.002837s
Feb 2 16:29:58 mx01 milter-greylist: DNSRBL lookup 1.16.10.10.sh-xbl.example.com performed in 0.002941s
Feb 2 16:29:58 mx01 milter-greylist: DNSRBL lookup 1.16.10.10.sh-xbl.example.com performed in 0.001323s
Feb 2 16:29:58 mx01 milter-greylist: Mail from=<user1@example.com>, rcpt=<user2@example.com>, addr=cli01[10.10.16.1] is matched by entry racl 118 greylist [maxpeek -1] default
Feb 2 16:29:58 mx01 milter-greylist: (local): 10.10.16.1 from <user1@example.com> to <user2@example.com>: greylisted entry timed out
Feb 2 16:29:58 mx01 milter-greylist: created: 10.10.16.1 from <user1@example.com> to <user2@example.com> delayed for 00:05:00
Feb 2 16:29:58 mx01 milter-greylist: 7F85CC005910: addr cli01[10.10.16.1] from <user1@example.com> to <user2@example.com> delayed for 00:05:00 (ACL 118)
Feb 2 16:29:58 mx01 milter-greylist: sync_sender running
Feb 2 16:29:58 mx01 postfix/smtpd[9818]: 7F85CC005910: milter-reject: RCPT from cli01[10.10.16.1]: 451 4.7.1 Greylisting in action, please come back in 00:05:00; from=<user1@example.com> to=<user2@example.com> proto=ESMTP helo=<cli01>
Feb 2 16:29:58 mx01 postfix/smtpd[9818]: disconnect from cli01[10.10.16.1]

Re: [milter-greylist] Re: Postfix queue ID

2016-02-02 by manu@...

mauricio.teixeira@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

>  So the are two possible ways of fixing it: 
>  a) Adding the queue ID to every single logged message; or
>  b) Adding the process ID to each one of them.

Only solution a will work, as milter-greylist is a multithreaded
program: on reasonable systems, all threads share the same PID.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] Re: Postfix queue ID

2016-02-02 by Mauricio Teixeira

> So the are two possible ways of fixing it:
> a) Adding the queue ID to every single logged message; or
> b) Adding the process ID to each one of them.

Only solution a will work, as milter-greylist is a multithreaded
program: on reasonable systems, all threads share the same PID.


Great, that should be enough. So how do we get that implemented? I'm horrible at C. :)

--
Mauricio Teixeira
Sao Paulo/SP/BR
http://mteixeira.wordpress.com
mauricio.teixeira{at}gmail.com
(irc: netmask on freenode)

Re: [milter-greylist] Re: Postfix queue ID

2016-02-03 by manu@...

Mauricio Teixeira mauricio.teixeira@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

> Great, that should be enough. So how do we get that implemented? I'm
> horrible at C. :)

That seems a very good opportunity to learn.

The mg_log() function could be modified to take a struct mlfi_priv *priv
as second argument (possibly NULL), and display priv->priv_queueid at
the beginning of each log line if (priv && priv->priv_queueid).

Then each mg_log() invocation needs to be modified to add priv or NULL
is unavailable.

Open question: before or after 4.6 release?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Re: [milter-greylist] Re: Postfix queue ID

2016-02-03 by Mauricio Teixeira

Sorry, I've been looking around the source code and I don't even know where to start. I'm pretty sure there's a big possibility that I will screw this up...
Show quoted textHide quoted text
On Tue, Feb 2, 2016 at 10:17 PM, manu@... [milter-greylist] <milter-greylist@yahoogroups.com> wrote:

Mauricio Teixeira mauricio.teixeira@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

> Great, that should be enough. So how do we get that implemented? I'm
> horrible at C. :)

That seems a very good opportunity to learn.

The mg_log() function could be modified to take a struct mlfi_priv *priv
as second argument (possibly NULL), and display priv->priv_queueid at
the beginning of each log line if (priv && priv->priv_queueid).

Then each mg_log() invocation needs to be modified to add priv or NULL
is unavailable.

Open question: before or after 4.6 release?

--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...




--
Mauricio Teixeira
Sao Paulo/SP/BR
http://mteixeira.wordpress.com
mauricio.teixeira{at}gmail.com
(irc: netmask on freenode)

Re: [milter-greylist] Re: Postfix queue ID

2016-02-03 by manu@...

Mauricio Teixeira mauricio.teixeira@... [milter-greylist]
<milter-greylist@yahoogroups.com> wrote:

> Sorry, I've been looking around the source code and I don't even know where
> to start. I'm pretty sure there's a big possibility that I will screw this
> up...

First update the mg_log function to add the new priv parameter and use
it. Next, change every mg_log() invocation to add priv or NULL if there
is no priv available.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Move to quarantaine

This moves the raw source file on disk only. The archive index is not changed automatically, so you still need to run a manual refresh afterward.