Milter-greylist

Hi,

I'd like to configure sendmail and milter-greylist, so that
sendmail listens to two different IP-address, but have
milter-greylist act on only one address.

Is this a sendmail/milter configuration issue. or should
this be handled inside the milter?

The reason for this setup is so I can have two MX
records for one mailserver, and only bother mail
that is sent to the MX with the lowest priority.

Spammers seem to prefer secondary mailservers
over primary ones; using milter-greylist on a
secondary MX only catches a lot of spam, without
bothering any legitimate mail.

Thanks,
Pim