I am half a step ahead of you. In a funny sort of way. It has been determined that there has to be a diagnostic text in the rule.  Which I have not been including.  I have added the text "Bad HELO greeting".  This now shows in the SMTP log.  So we now know that the file is being processed.  However, the 'D' command has been ignored as in the session log it still shows as connection being made.  As a test I have just now changed the drop command to refuse and will see what happens from there. 

I am having an issue with transflt.mer compliance.  I am even going to go so far as to say that it is not even being processed by Mercury. 

Steps that I have completed to come to this conclusion:

1.  Capitalized TRANSFLT.MER in the Mercury directory (grasping at straws)

2.  Confirmed that the file is correctly listed in the Mercury.ini file

3.  Started a new file called Nuke.mer and add the lines into that file

4.  Confirmed that the file listed in Mercury.ini was listed as Nuke.mer

5.  Deleted the file TRANSFLT.MER and had Mercury recreate it

6.  Added only one line into the file H, "EHLO User*", D 

7.  Changed the file extension from mer to lis

It should be noted that after any changes that were made to the files, I closed Mercury and did a Service Mercury32 restart. 

The log(s) still show the connections that are suppose to be refused and in the last case the User connection.

I have an asterisks * at the first of all my rules 

H, "*EHLO USER*" RS, "554 message refused - error code 1328"

I put an error code as the response just in case it catches something it shouldn't and the person can then tell me, it bounces with an error code 1328 or which ever one it is and I can adjust the rule, drop it completely or create some kind of exception for just that one person.

That had been tried to no avail.  As it stands right now the Transflt.mer file is not being processed by Mercury at all. 

Could it be that Mercury for some reason hasn't the main Mercury directory as the current directory when running the program as a service? What Windows version is being used on the server?

It is a Win10 Pro server. I have included the c:\mercury directory in the path for the tranflt.mer file. Also when I deleted the tranflt.met file, Mercury happily created a fresh one. Step 5 in my trouble shooting.

I made a quick test with this under Windows 10 Pro, using this transaction filter rule:

H, "*", L, "Filter active"

That rule will add a log line for every connection to MercuryS when the HELO event is processed. In program mode it can be checked directly in the console window, and when in service mode the SMTP log file can be checked. The rule file had the default name without path.

In both program mode and service mode the rule triggered and a line was added to the log.

Could you make a similar test on your system and see what the results are? If there are any special environment factors that possibly could affect Mercury (such as running in a virtual machine, using a third party service enabler, running an anti-malware solution other than the built-in Windows Defender) please make a note of that.