ShieldMarc

How to fix a STARTTLS failure

By Nathan Jeffery8 min read

Fixing a STARTTLS failure starts with identifying which of three things broke: the certificate, the port, or the TLS version, not DMARC or SPF. This guide gives each cause, the numbered fix and exact record or setting, and how to confirm it worked. The mistake to avoid: assuming opportunistic TLS on port 25 ever proves a certificate is trusted.

Contents
  1. What does a STARTTLS failure actually mean?
  2. How to confirm it is a STARTTLS problem and not something else
  3. Cause 1: the certificate is expired, self-signed or wrong for the hostname
  4. Cause 2: the port and encryption mode do not match
  5. Cause 3: a firewall or cloud provider is blocking the port
  6. Cause 4: the server or client still offers an obsolete TLS version
  7. Verifying the fix has actually worked
  8. Preventing a STARTTLS failure from coming back
  9. Start today
  10. Frequently asked questions

What does a STARTTLS failure actually mean?

A STARTTLS failure means a connection could not upgrade from plain-text SMTP to TLS partway through the session. STARTTLS is a command defined in RFC 3207, not a separate encrypted port: it starts in plain text and switches to TLS only once the server accepts it.

You will usually see one of these forms:

  • STARTTLS failed, a generic client-side error.
  • 530 5.7.0 Must issue a STARTTLS command first, when the server refuses to proceed without encryption.
  • Could not negotiate TLS connection, a handshake that broke down.
  • A certificate warning during send, an encrypted but untrusted connection.

This is a transport problem, separate from DMARC, SPF and DKIM, which judge authentication later in the SMTP session or after the message is received; see Why is my DMARC failing if that is your real question. RFC 3207 says a publicly referenced server, meaning port 25 on a host listed in the domain's MX record (or its A record if there is no MX), must not require STARTTLS to accept mail for local delivery.

A server that is not publicly referenced, such as a submission server on port 587, may require it and should answer a command such as MAIL FROM with 530 until the client sends STARTTLS.

How to confirm it is a STARTTLS problem and not something else

Confirm a STARTTLS problem from the reply code behind the error text: a 220 reply to STARTTLS means the server accepted it, so an error after that points at the TLS handshake, 530 means the server refused to proceed without it, and a TLS alert points at a certificate or protocol mismatch.

Run openssl s_client -starttls smtp -connect <mail-host>:<port> -verify_hostname <mail-host> -servername <mail-host> from an unblocked network and read the Verify return code line (OpenSSL's s_client manual); anything other than 0 (ok) names the certificate problem directly.

Check the EHLO response for STARTTLS in its capability list. If it is missing, either the server is not offering STARTTLS or something in the path is stripping it, a different fix to a broken handshake.

Rule out DMARC, SPF and DKIM: those checks run after STARTTLS, so they cannot cause a STARTTLS error, and a DMARC report only covers mail that got far enough to be evaluated.

Fixing a STARTTLS failure

  1. Read the exact error

    Note the reply code and text: 530, STARTTLS failed, or a certificate error.

  2. Test the handshake directly

    Run openssl s_client -starttls smtp against the exact host and port.

  3. Match the cause to a fix

    Certificate, port mismatch, a blocked port, or an outdated TLS version.

  4. Verify the fix worked

    Re-run the test and confirm Verify return code: 0 (ok) appears.

  5. Monitor to stop it recurring

    Track certificate expiry and MTA-STS enforcement mode over time.

Cause 1: the certificate is expired, self-signed or wrong for the hostname

Cause 1 is a STARTTLS certificate error: the certificate presented during the handshake has expired, is self-signed, or omits the mail server's hostname from its Subject Alternative Name.

Fix it:

  1. Get a certificate from a trusted CA, covering the mail server's exact hostname.
  2. Install it in the server's TLS configuration: Postfix's smtpd_tls_cert_file (the server certificate first, then its intermediate CA certificates) and smtpd_tls_key_file, or, once the certificate is imported on an Exchange server, Enable-ExchangeCertificate -Thumbprint <thumbprint> -Services SMTP.
  3. Restart the SMTP service so it presents the new certificate.

A message can still arrive despite this: opportunistic TLS on SMTP port 25 keeps delivering even past an untrusted or wrong-named certificate (Postfix's TLS documentation), so delivery alone proves nothing.

Verify by re-running openssl s_client and checking that the new certificate appears under Certificate chain with Verify return code: 0 (ok), since a failed handshake with no certificate also prints 0 (ok); add -verify_return_error so the test stops on a bad certificate instead of continuing past it.

Cause 2: the port and encryption mode do not match

The client or server is using the wrong encryption mode for its port. Port 465 is implicit TLS from the first byte; ports 587 and 25 start in plain text and switch to TLS only after STARTTLS. RFC 8314 sets out the port 465 and 587 split for mail submission, RFC 3207 defines the STARTTLS used on port 25, and the wrong mode for a port fails every time, regardless of certificates.

Fix it:

  1. Use implicit TLS or SSL only on port 465, never STARTTLS.
  2. Use STARTTLS only on port 587 or 25, never implicit TLS.
  3. Check the application's own mail settings, not only DNS or the firewall.

Pro Tip: Test with openssl s_client -starttls smtp -connect host:587 -verify_hostname host -servername host rather than -connect host:465, since port 465 expects TLS immediately and never uses STARTTLS, so it cannot test the upgrade you are trying to debug.

Verify with a real test message: on port 587 or 25 the log should show a 220 reply to STARTTLS, then a 250 reply to the fresh EHLO sent over TLS, while on port 465 the TLS handshake comes first with no STARTTLS.

Cause 3: a firewall or cloud provider is blocking the port

A firewall, proxy or cloud provider is blocking the SMTP port, so the connection fails before STARTTLS can begin, and a vague client error can make that look like a STARTTLS failure.

Azure blocks outbound port 25 on every subscription type except Enterprise Agreement and MCA-E (Microsoft's Azure SMTP guidance), and Google Compute Engine blocks it to external destinations (Google Cloud's mail guidance); ports 587 and 465 are not blocked on either.

Fix it:

  1. On a cloud platform, relay outbound mail through port 587 or 465 via the provider's supported smart host.
  2. On premise, open the specific port in both directions to the mail server's IP, and confirm no gateway is stripping the STARTTLS advertisement.

Verify with openssl s_client on the exact port, from the sending server itself for an outbound block and from outside the network for an inbound one, not from a machine that may be exempted.

Cause 4: the server or client still offers an obsolete TLS version

The server or client still offers only TLS 1.0 or TLS 1.1, both deprecated under RFC 8996, and more counterparts now refuse to negotiate them.

Fix it:

  1. Enable TLS 1.2 and TLS 1.3 in the mail server's configuration.
  2. Disable TLS 1.0 and TLS 1.1 entirely; RFC 8997 sets TLS 1.2 as the minimum for submission and access clients, and NCSC advises TLS 1.3, TLS 1.2 or both, with government systems upgraded to support TLS 1.3.

The NCSC recommends that government systems do not support deprecated versions of the protocol, and instead are upgraded to support TLS version 1.3. Source: NCSC: Using TLS to Protect Data

Watch for a trap: OpenSSL 3 disables SSL 3, TLS 1.0 and TLS 1.1 by default, so adding only -tls1_1 to the openssl s_client command fails on your machine regardless of the server. Add -tls1_1 -cipher 'DEFAULT:@SECLEVEL=0' instead to test whether the server still allows TLS 1.1 (OpenSSL's security level documentation).

Verify the s_client output names a cipher rather than Cipher is (NONE) and reads Protocol: TLSv1.2 or TLSv1.3, because a failed handshake can still print Protocol: TLSv1.3.

Verifying the fix has actually worked

Verifying the fix means the same test now passes on every count together. Re-run openssl s_client -starttls smtp -connect <host>:<port> -verify_hostname <host> -servername <host> and confirm it prints no Didn't find STARTTLS in server response warning, the handshake completes, and Verify return code: 0 (ok) appears.

Send a real end-to-end test message too, and check the receiving server's own log, not only yours, since some failures are one-directional.

If the certificate is shared with a website, ShieldMarc's free SSL checker confirms its issuer, SANs and expiry on port 443 only, without opening a mail-port STARTTLS session itself; its answer can be up to about an hour old, so it may still show the certificate from before the fix.

If MTA-STS is published, ShieldMarc's free MTA-STS checker reads your _mta-sts and MX records and fetches the policy file, but it never contacts the mail server, so it cannot confirm the TLS fix itself.

Preventing a STARTTLS failure from coming back

Preventing a repeat means putting certificate and policy checks on a schedule, not trusting that nothing changes. ShieldMarc's SSL monitor checks port 443 daily and alerts within 7 days of expiry when it watches a hostname sharing the mail server's certificate, though it does not watch the mail port itself.

Publish a TLS-RPT record so other servers report their own STARTTLS failures to you; ShieldMarc ingests and displays these reports but sends no email alert about them, so check the dashboard directly.

Roll out MTA-STS in testing mode with a TLS-RPT address first, watch for a month, then move to MTA-STS enforce mode, the order NCSC recommends. See MTA-STS and TLS-RPT explained for the full rollout. Record which port and mode each relay uses, so any future change can be checked against it; SSL certificate monitoring covers a fuller renewal cadence.

Delivery succeeding is not proof STARTTLS is healthy, since opportunistic TLS on port 25 delivers past a bad certificate anyway.

Start today

Run two free checks against the domain: ShieldMarc's MTA-STS checker and TLS-RPT checker confirm both records validate, with no account needed.

Run the free Security Grade scan too, for a wider view of the domain's DMARC, SPF, SSL and DNS posture.

ShieldMarc does not run a live SMTP STARTTLS handshake test against a mail server, so use the openssl s_client command from this guide for that check.

For ongoing monitoring, ShieldMarc's Starter plan covers one domain and one SSL monitor free, permanently; MTA-STS and TLS-RPT monitoring sit on the paid plans, with a 30-day trial and no card required.

Frequently asked questions

Why did STARTTLS suddenly start failing when nothing changed on my side?

A common trigger is a certificate that renewed badly, or the provider raising its minimum TLS version. A new firewall or mail gateway blocking the STARTTLS advertisement is another. Test from a different network: failing everywhere points to the receiving server.

Can I just disable STARTTLS or certificate verification to make the error go away?

Disabling certificate verification can hide a certificate error, but the connection then no longer proves it reached the right server, and disabling STARTTLS removes encryption altogether, so neither should stay in place. Opportunistic TLS on port 25 already delivers past a bad certificate, so this only hides the warning. Fix the certificate or configuration instead, then leave verification on.

Does a STARTTLS failure mean my DMARC or SPF is broken too?

No: STARTTLS is transport encryption, negotiated before the message is evaluated, while DMARC, SPF and DKIM judge authentication later in the SMTP session or after the message is received. They can look related in one investigation but have separate causes and fixes. See Why is my DMARC failing for alignment causes.

Why does the connection work in a browser or telnet but STARTTLS still fails for mail?

A browser tests the web service on port 443, which may present a different certificate, and telnet can show whether the mail server advertises STARTTLS but cannot perform the TLS handshake, so neither proves the handshake works. Typing STARTTLS in telnet only reaches the 220 Ready to start TLS reply, since telnet cannot switch to an encrypted stream, so it looks like it hangs. Use openssl s_client -starttls smtp -connect <mail-host>:<port> -verify_hostname <mail-host> instead, which performs the real handshake.

Sources