Just a confirmation, in case anyone else has this problem; the cause and the fix I've posted appear to work, the only downside being that access to my website requires using www in front of the domain name, previously it was optional. an alternative fix might be to set up a forwarding email account with my hosting provider but not create an MX record to point to it. The reason for no MX being that the spam that I refuse connections to will end up there and get forwarded to me.

I run Mercury 4.8 on my home ADSL line. I've recently changed ISPs and my new ASP doesn't allocate static IPs. So I've registered with duckdns.org and have a subdomain there that they point to my IP, updating if it changes (so far it hasn't).

Since it's dynamic IP, it's listed in spamhaus.zen, and there's nothing I can do about that, so my outgoing mail goes through a smarthost belonging to a hosting company I have an account with. That's working fine.

The problem I didn't expect is that some email from at least two Hotmail users is only getting to getting to me intermittently. The senders are getting errors of the form:

Reporting-MTA: dns;BLU004-OMC2S29.hotmail.com
Received-From-MTA: dns;BLU437-SMTP52
Arrival-Date: Sat, 5 Mar 2016 13:39:27 -0800
Final-Recipient: rfc822 xxxx@mydomain.co.uk
Action: failed
Status: 5.5.0
Diagnostic-Code: smtp;550-Please turn on SMTP Authentication in your mail client. 
550-blu004-omc2s29.hotmail.com [65.55.111.104]:50245 is not permitted to relay
550 through this server without authentication.

 The first time this happened, the senders second attempt was delivered, and he'd thoughtfully attached the error from the first try. It looked to me like the hotmail server bouncing it back to him, so I assumed it was temporary error and had been fixed. But it's happened again, with a completely different sender, and several times in a row with the original sender. It still looks like a hotmail error, and I can't find any trace of the event in my MercuryS logs (I've now enabled session logging but didn't then) but it didn't happen before I changed to a dynamic IP, and a bit of Google searching suggest that others have had this problem, and a suspicion that it's caused by an ambiguity in the receiving servers DNS records.

Since I have a dynamic IP, and my MX record points to my duckdns.org subdomain, I assume my DNS records may look incorrect; I don't think I can have a valid reverse DNS in this situation.

I wondered if Hotmail is trying to validate my address and failing, so bouncing the email as a public service, and since I don't think there's a proper code for "rejected because I've decided the final recipient won't want it" is quoting spurious grounds. It wouldn't be the first time I've had an Microsoft originated error message that didn't match the actual error.

I have both my real domain and my dynamic subdomain on the right hand side in mercury.ini, and my real domain in the canonical name (maybe I should use the dynamic subdomain?).

Suggestions gratefully received. Thank you.

I think I've found the problem. It seems that Hotmail doesn't follow the RFC, and first tries to send mail to the A record. It only drops back to the MX if port 25 is closed on the server in the A record. This would be unbelievable if it was anyone but Microsoft but a search using "Hotmail ignoring MX records" finds lots of evidence.

 In my case, I have web sites hosted by an external provider, and my A record points there, and MX hosted at home, where my MX record points. The external provider runs an email server on the same IP as their web server, but I'm not currently set up as a user, so when Hotmail follows my A record it quite justifiably refuses Hotmail's attempt to send my mail to it. 

There's zero chance of getting Hotmail to behave (others have tried, it seems) so the solution seems to be to remove the A record. For the moment I've replaced the A record for the domain with one with www. and removed the CNAME that pointed www. to it.  

I don't think that's broken anything else; I'll wait for it to propagate and test.