First off - thanks to both of you for really helpful replies. Let me see if I can clarify some things...

ASSP is basically a proxy server that filters incoming mail for spam and receives all outgoing so it can whitelist all addresses mails are being sent to. This has the big advantage that addresses users send mails to are automatically whitelisted for a year. As for the configuration - I'm aware that it is somewhat complicated, but this is the official configuration recommended by the ASSP documentation when ASSP and Exchange are running on the same machine, which they do in my case.

@dilberts_left_nut: I'm currently only worried about Mercury as a possible cause for an open relay. Yet I see your point that Mercury itself cannot actually be the problem.

@Thomas R. Stephenson: Thanks for the advice - I'll add the refuse entry. From your recommendation I take it that entries that are higher up supersede ones below, right?

Taken together, I take your two posts to say that Mercury probably is not the cause of my problem. That means I'll turn to the Exchange server and recheck that one...

Thank you so much!

Cheers,
Stefan.

Hi everybody,

I have an installation with an ASSP spam proxy and an Exchange mailserver. In order to register outgoing mails with the ASSP proxy, Exchange sends mails to the ASSP proxy, which then uses Mercury/32 v. 4.62 as a relay (basically like a smarthost). Everything has been running fine for months now. My problem is that my ISP has suddenly claimed that my server is an open relay and has put me on their blacklist. Since my external IT support tells me my Exchange configuration is impeccable, I believe the problem must be somewhere in the Mercury/32 config. So basically I'm hoping for your help in making sure my Mercuy/32 SMTP relays only mails coming from my Exchange.

Okay, this is what my configuration looks like (the numbers are the ports used):

Setup incoming: Internet -> 25 ASSP -> 12525 Exchange -> Outlook
Setup outgoing: Outlook -> Exchange -> 225 ASSP -> 325 Mercury32 -> Internet

Note that all three programs (ASSP, Exchange and Mercury/32) run on the same machine and that the firewall only allows traffic on port 25. My main MTA is Exchange, Mercury/32 serves only as a smarthost for relaying purposes (i.e. to deliver to its final recipient mail that has come out of Exchange and has passed through ASSP).

In Mercury/32, in the MercuryS SMTP server configuration, in the tab "Connection control" I have two entries allowing connections from 127.0.0.1 and from 10.0.0.1, the internal IP of the machine everything runs on. I have currently not checked any of the relaying control checkboxes in this dialog, but am leaning towards checking the "Do not permit SMTP relaying of non-local mail" box. Should I do that? Is there anything else I should do to ensure only mails from the Exchange server are relayed through Mercury/32? In this configuration, can/should I employ the SMTP authentification feature of Mercury/32?

Cheers,
Stefan.

> Setup incoming: Internet -> 25 ASSP -> 12525 Exchange -> Outlook
> Setup outgoing: Outlook -> Exchange -> 225 ASSP -> 325 Mercury32 -> Internet
>
> In Mercury/32, in the MercuryS SMTP server configuration, in the tab
> "Connection control" I have two entries allowing connections from
> 127.0.0.1 and from 10.0.0.1, the internal IP of the machine everything
> runs on. I have currently not checked any of the relaying control
> checkboxes in this dialog, but am leaning towards checking the "Do not
> permit SMTP relaying of non-local mail" box. Should I do that? Is
> there anything else I should do to ensure only mails from the Exchange
> server are relayed through Mercury/32? In this configuration,
> can/should I employ the SMTP authentification feature of Mercury/32?

You should put in a refuse of 0.0.0.1 - 255.255.255.255 after the two allow entries to ensure nothing other that these two IP addresses are allowed to send mail via MercuryS.  

It's still possible though if there is any infection of this system that the worm/zombie will be able to send through this setup using the SMTP mailer of the infection via port 325 since there is no authentication.  

I really prefer turning off all relaying in MercuryS and use ESMTP authentication to send the mail using CRAM-MD5.  I do not know if ASSP can do the required authentication though.

BTW, do you really need to be using ASSP on the outbound mail?

[quote user="spalan"]

Setup incoming: Internet -> 25 ASSP -> 12525 Exchange -> Outlook
Setup outgoing: Outlook -> Exchange -> 225 ASSP -> 325 Mercury32 -> Internet
[/quote]

If MercS is ONLY listening on port 325 it will never see any traffic from the net (on 25) so cannot relay it.

It will howerver relay ANYTHING passed to it by ASSP.

If your ASSP / Exchange is not rejecting external relay attempts, then your 'whole system' IS an open relay.

AUTH-ing in Merc WILL NOT prevent this as any 'bad relay' mail will then be coming from an authenticated source (and allowed).

If your system is really an open relay, then the problem should be solved in ASSP as the receiving SMTP server in the first instance. Exchange also would seem to be guilty of relaying non-local mail. Mercury could be set to refuse any outgoing mail from anyone@NOT-YOUR-DOMAIN but that won't stop spoofing from external sources that get past ASSP / Exchange.

Seems unnecessarily complex to me, 3 servers to do the job of one decent one [;)]

 EDIT:

[quote user="spalan"]

ASSP -> 12525 Exchange
Outlook -> Exchange
[/quote]

Is exchange listening on port 25? If so, have you restricted which interface it uses?