Stop the server, remove everything from the queue directory (save it somewhere in case you think there can be relevant messages in there that you need to salvage), clear the scratch directories. Before starting Mercury again make sure these two lines in mercury.ini look like this:
Relay : 0
Strict_Relay : 1
If you again start getting lots of outgoing messages after starting Mercury please stop the server again, and check logs and the contents some of the .QDF files in the queue directory to find out exactly what is happening (a mail loop, a spam-bot in your LAN, some relaying that isn't stopped, for instance due to an AUTH password that has been compromised).