I have the same setup outlook 2010 and mercury.  It works fine for me.  Just did a session log which I provided below in case you can see something in it that helps.

It doesn't look like yours is logging in?  The peer certificate thing popped up for me as well.

I wonder if you try providing the username and password instead of saying use same as pop will fix it.  That is how mine is set. under More settings | Outgoing server i have "my server requires authentication" but I am providing the username and password.

00:26:25.244: --- 15 Dec 2017, 0:26:25.244 ---
00:26:25.244: Accepted connection from '24.224.215.188', timeout 900 seconds.
00:26:25.307: Connection from 24.224.215.188, Fri Dec 15 00:26:25 2017<lf>
00:26:25.307: << 220 mercurymailsystem.ca ESMTP server ready.<cr><lf>
00:26:25.322: >> EHLO JimmyB<cr><lf>
00:26:25.322: << 250-mercurymailsystem.ca Hello JimmyB; ESMTPs are:<cr><lf>250-TIME<cr><lf>
00:26:25.322: << 250-SIZE 107286400<cr><lf>
00:26:25.322: << 250-8BITMIME<cr><lf>
00:26:25.322: << 250-AUTH CRAM-MD5 LOGIN<cr><lf>
00:26:25.322: << 250-AUTH=LOGIN<cr><lf>
00:26:25.322: << 250-STARTTLS<cr><lf>
00:26:25.322: << 250 HELP<cr><lf>
00:26:25.400: >> STARTTLS<cr><lf>
00:26:25.416: << 220 OK, begin SSL/TLS negotiation now.<cr><lf>
00:26:25.588: [*] SSL/TLS session established
00:26:25.588: [*] AES256-GCM-SHA384, TLSv1.2, Kx=RSA, Au=RSA, Enc=AESGCM(256), Mac=AEAD<lf>
00:26:25.588: [*] No peer certificate presented.
00:26:25.588: >> EHLO JimmyB<cr><lf>
00:26:25.588: << 250-mercurymailsystem.ca Hello JimmyB; ESMTPs are:<cr><lf>250-TIME<cr><lf>
00:26:25.588: << 250-SIZE 107286400<cr><lf>
00:26:25.588: << 250-8BITMIME<cr><lf>
00:26:25.604: << 250-AUTH CRAM-MD5 LOGIN<cr><lf>
00:26:25.604: << 250-AUTH=LOGIN<cr><lf>
00:26:25.604: << 250 HELP<cr><lf>
00:26:25.666: >> AUTH LOGIN<cr><lf>
00:26:25.666: << 334 HDFSLKjdfskchnge<cr><lf>
00:26:25.682: >> sdflkjfchangea;<cr><lf>
00:26:25.682: << 334 dfsalkjYUYsdlkjchange<cr><lf>
00:26:25.682: >> dfsalkjdfschange<cr><lf>
00:26:25.697: << 235 Authentication successful.<cr><lf>
00:26:25.697: >> MAIL FROM: <jbanks@APM.ca><cr><lf>
00:26:25.713: << 250 Sender OK - send RCPTs.<cr><lf>
00:26:25.713: >> RCPT TO: <jameshaven@outlook.com><cr><lf>
00:26:25.713: << 250 Recipient OK - send RCPT or DATA.<cr><lf>
00:26:25.729: >> DATA<cr><lf>
00:26:25.729: << 354 OK, send data, end with CRLF.CRLF<cr><lf>
00:26:25.760: >> From: "Jim Banks" <jbanks@APM.ca><cr><lf>
00:26:25.760: >> To: <jameshaven@outlook.com><cr><lf>
00:26:25.760: >> Subject: test 2<cr><lf>

Running Outlook 2010 against Mercury 4.8 with TLS / SSL and authentication.
With SMTP there is TLS on port 587, SSL3 on port 465 (port 25 should work too - doesn't matter).

IMAP is no problem - works fine. SMTP does not (neither with TLS or SSL3).

Error in Outlook on sending: "Error Message: 0x800CCC80 - None of the authentication methods supported by this client are supported by your server"

No error with Pegasus or Thunderbird with same configuration.

"SMTP needs Authentication"  is enabled with same authentication data like incoming. SPA is disabled!

Sessionlog Outlook:

10:39:47.061: << 250-wiso.tu-dortmund.de Hello BIKER; ESMTPs are:<cr><lf>250-TIME<cr><lf>
10:39:47.061: << 250-SIZE 20000000<cr><lf>
10:39:47.061: << 250-STARTTLS<cr><lf>
10:39:47.061: << 250 HELP<cr><lf>
10:39:47.264: >> STARTTLS<cr><lf>
10:39:47.264: << 220 OK, begin SSL/TLS negotiation now.<cr><lf>
10:39:47.358: [*] SSL/TLS session established
10:39:47.358: [*] AES256-SHA, SSLv3, Kx=RSA, Au=RSA, Enc=AES(256), Mac=SHA1<lf>
10:39:47.358: [*] No peer certificate presented.
10:39:47.358: >> EHLO BIKER<cr><lf>
10:39:47.358: << 250-wiso.tu-dortmund.de Hello BIKER; ESMTPs are:<cr><lf>250-TIME<cr><lf>
10:39:47.358: << 250-SIZE 20000000<cr><lf>
10:39:47.358: << 250 HELP<cr><lf>
10:39:47.577: 26: SSL_read returned zero (socket was closed)
10:39:47.577: >>
10:39:47.593: [!] Warning: SSL connection improperly closed by remote host.
 

Sessionlog Thunderbird:

 10:37:27.546: << 220 wiso.tu-dortmund.de ESMTP server ready.<cr><lf>
10:37:27.593: >> EHLO [129.217.232.10]<cr><lf>
10:37:27.593: << 250-wiso.tu-dortmund.de Hello [129.217.232.10]; ESMTPs are:<cr><lf>250-TIME<cr><lf>
10:37:27.593: << 250-SIZE 20000000<cr><lf>
10:37:27.593: << 250-STARTTLS<cr><lf>
10:37:27.593: << 250 HELP<cr><lf>
10:37:27.811: >> STARTTLS<cr><lf>
10:37:27.811: << 220 OK, begin SSL/TLS negotiation now.<cr><lf>
10:37:27.905: [*] SSL/TLS session established
10:37:27.905: [*] AES128-SHA, SSLv3, Kx=RSA, Au=RSA, Enc=AES(128), Mac=SHA1<lf>
10:37:27.905: [*] No peer certificate presented.
10:37:27.905: >> EHLO [129.217.232.10]<cr><lf>
10:37:27.905: << 250-wiso.tu-dortmund.de Hello [129.217.232.10]; ESMTPs are:<cr><lf>250-TIME<cr><lf>
10:37:27.905: << 250-SIZE 20000000<cr><lf>
10:37:27.905: << 250 HELP<cr><lf>
10:37:28.124: >> MAIL FROM:
...
10:37:28.171: << 250 Data received OK.<cr><lf>
10:37:28.171: >> QUIT<cr><lf>
10:37:28.186: << 221 wiso.tu-dortmund.de Service closing channel.<cr><lf>
10:37:28.233: [!] Warning: SSL connection improperly closed by remote host.

 The main difference seems to be the line with encryption.

Any help?

    Olaf