TLSx509certificatecachainintermediate

Certificates and the Chain of Trust

A certificate binds a hostname to a public key with a signature from an issuer the client already trusts; the client walks leaf → intermediate → root, checks names, dates and signatures, and every classic TLS outage is one of those checks failing.

Conceptual
▶ InteractiveInterview question
Progress

The problem

Key agreement gives two parties a shared secret, but not knowledge of who the other party is. The browser has never seen api.example.com’s public key before. How can it decide, offline and in microseconds, that a key presented by a stranger really belongs to that name — and how does that decision get revoked when the key is stolen?

What a certificate says

An X.509 certificate is a signed statement: "issuer *I* asserts that public key *K* belongs to subject *S*, valid from *t₁* to *t₂*". The parts that matter for TLS are the Subject Alternative Name extension listing the DNS names (and IPs) the certificate is valid for, the public key, the validity period, the issuer name, key usage flags, and the issuer’s signature over all of it. The Common Name field in the subject is historical; browsers have ignored it for hostname matching since around 2017 and only SANs count.

The signature is what makes the statement checkable: anyone with the issuer’s public key can verify it, and nobody without the issuer’s private key can forge it. So the question "is this key really api.example.com’s?" becomes "do I trust the issuer, and did the issuer really sign this?" — which is the same question one level up.

The fields that matter, from `openssl x509 -in leaf.pem -noout -text` (trimmed)
Certificate:
    Data:
        Serial Number: 04:a1:...:9c
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = US, O = Let's Encrypt, CN = E6
        Validity
            Not Before: Jun 12 09:14:33 2026 GMT
            Not After : Sep 10 09:14:32 2026 GMT
        Subject: CN = api.example.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey  (P-256)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:api.example.com, DNS:www.example.com
            X509v3 Key Usage: critical  Digital Signature
            X509v3 Extended Key Usage:  TLS Web Server Authentication
            Authority Information Access:
                OCSP - URI:http://e6.o.lencr.org
                CA Issuers - URI:http://e6.i.lencr.org/
            CT Precertificate SCTs: ...

The chain: leaf → intermediate → root

Conceptual

The recursion terminates at a root certificate that is self-signed and trusted not because of any signature but because it ships in the client’s trust store — the OS store (Windows, macOS, Android), Mozilla’s NSS bundle (Firefox, and the source of ca-certificates on most Linux distributions), or a language-runtime bundle (Java’s cacerts, Python’s certifi, Node’s compiled-in Mozilla list). Roots are kept offline and precious; they sign a handful of intermediate CAs, and intermediates sign the millions of leaf certificates servers present.

During the The TLS Handshake the server sends the leaf and the intermediates — not the root, which the client must already have. The client builds a path: leaf’s issuer must be the intermediate’s subject, the intermediate’s signature over the leaf must verify, the intermediate’s issuer must be a root in the store, and so on. Each certificate’s validity window, key usage and name constraints are checked along the way. A missing intermediate is the single most common chain misconfiguration, and it fails inconsistently: browsers often fetch the missing certificate via the AIA URL or have it cached, while curl, Java and mobile apps do not.

Hostname validation is the last step: the name the client connected to must match a SAN entry. A wildcard *.example.com matches exactly one label — api.example.com but not example.com and not a.b.example.com. IP addresses must appear as IP: SANs; connecting to a server by IP with a DNS-only certificate fails even when everything else is fine.

Verification walks upward until it hits something already trusted
  1. Leaf: CN=api.example.com, SAN: api.example.comSent by the server; signed by the intermediate; must match the hostname and be within validity
  2. Intermediate: Let’s Encrypt E6Sent by the server; signed by the root; missing it is the classic "works in Chrome, fails in curl"
  3. Root: ISRG Root X1Not sent; must already be in the client trust store; self-signed
  4. Trust storeOS, Mozilla/NSS, Java cacerts, certifi — different clients consult different stores

Revocation, and why it is weak

A certificate is valid until Not After regardless of what happens to the key. If the private key leaks, the issuer needs a way to say "no longer trust this one". Two mechanisms exist. A CRL is a signed list of revoked serial numbers published by the CA — simple, but large and slow to propagate, and a client must download it. OCSP lets the client ask the CA about one serial in real time, which leaks browsing history to the CA and adds a network round trip to every handshake.

Both mechanisms fail open in practice. If the OCSP responder is unreachable, browsers soft-fail and proceed, because a hard failure would let anyone who can block one URL take down every site using that CA. An attacker with a stolen key and on-path position can block the OCSP check, which is exactly the situation revocation was meant to cover. OCSP stapling helps: the server fetches a signed OCSP response periodically and sends it in the handshake, removing the client round trip and the privacy leak; Must-Staple in the certificate turns a missing staple into a hard failure, but it is rarely used.

The industry’s practical answer has been short lifetimes: Let’s Encrypt certificates last 90 days (with 6-day certificates being introduced), and the browser-CA forum has agreed to reduce the maximum public certificate lifetime to 47 days by 2029. A key that is only valid for days limits the damage of a leak without depending on a check that fails open. Browsers meanwhile push aggregated revocation data (Chrome’s CRLSets, Firefox’s CRLite) rather than querying per site.

  • CRL: signed list, fetched by the client, stale by design.
  • OCSP: per-certificate query, privacy leak, soft-fails when blocked — so an on-path attacker can suppress it.
  • OCSP stapling: server includes the response; Must-Staple makes its absence fatal.
  • Short-lived certificates make revocation nearly moot: the window is days, not a year.

Getting certificates: ACME and automation

Node.js

Before 2015 a certificate meant a purchase, a manual CSR, an email validation and a person pasting files onto a server once a year — and forgetting to, which is why expired certificates were the most common TLS outage. Let’s Encrypt made issuance free and the ACME protocol (RFC 8555) made it automatic: the client proves control of the domain by serving a token at /.well-known/acme-challenge/ (HTTP-01) or publishing a _acme-challenge TXT record (DNS-01, the only option for wildcards), and receives a certificate minutes later. Clients such as certbot, acme.sh, Caddy (built in) and Kubernetes’ cert-manager renew at two-thirds of the lifetime without human involvement.

Every publicly trusted certificate is also logged to Certificate Transparency logs before browsers accept it — the SCTs in the extensions above — so a misissued certificate for your domain is publicly visible; monitoring CT for your names is how you find out a CA or a colleague issued one you did not expect.

The remaining operational work is making renewal *observable*: an alert on days-to-expiry (from an external probe, not from the box that is supposed to renew) catches the case where the automation is broken and nobody noticed — a renewed certificate that was never reloaded into the running process is the modern form of the old outage.

Read what the server actually presented — the first step in any certificate incident
1import tls from 'node:tls'
2
3const socket = tls.connect({ host: 'api.example.com', port: 443, servername: 'api.example.com' }, () => {
4 const cert = socket.getPeerCertificate(true) // true → include the issuer chain
5 console.log(cert.subjectaltname) // 'DNS:api.example.com, DNS:www.example.com'
6 console.log(cert.valid_to) // 'Sep 10 09:14:32 2026 GMT'
7 console.log(cert.issuerCertificate?.subject) // the intermediate the server sent (or the leaf itself if none)
8 console.log(socket.authorized, socket.authorizationError) // false + 'CERT_HAS_EXPIRED' etc.
9 socket.end()
10})

The classic failures

Almost every certificate incident is one of six shapes, and each has a recognisable error string. Learn the strings; they tell you which check failed and therefore where to look.

Failure → what the client says → cause
FailureTypical errorCause and fix
ExpiredChrome ERR_CERT_DATE_INVALID; curl certificate has expired; Node CERT_HAS_EXPIREDRenewal did not run, or ran but the process never reloaded the file. Alert on expiry from outside.
Hostname mismatchChrome ERR_CERT_COMMON_NAME_INVALID; curl subject alternative name does not match; Node ERR_TLS_CERT_ALTNAME_INVALIDConnected to a name not in the SANs: bare domain vs www, wildcard depth, connecting by IP, SNI not sent so the default cert was served.
Missing intermediatecurl unable to get local issuer certificate; Java PKIX path building failed; browsers often succeedServer sends only the leaf. Concatenate leaf + intermediate(s) in the served chain (fullchain.pem).
Wrong system clockEverything looks "not yet valid" or "expired"Embedded devices, VMs resumed from snapshot, containers with a broken clock. Check date before blaming the CA.
Self-signed in developmentself-signed certificate / DEPTH_ZERO_SELF_SIGNED_CERTAdd the dev CA to the trust store (mkcert), never disable verification globally.
Corporate inspection proxyBrowser works; pip, npm, Docker and Go tools fail with unknown issuerThe proxy re-signs every site with its own CA, pushed to the OS store; tools with their own bundle (certifi, Java) do not have it. Add the proxy CA to those bundles.

Key points

  • A certificate is a signed binding of hostname(s) to a public key with a validity window; only SANs count for hostname matching.
  • The client verifies leaf → intermediate → root, where the root must already be in its trust store; servers send the leaf and intermediates, never the root.
  • Different clients consult different trust stores (OS, Mozilla, Java, certifi), which is why the browser and your script can disagree.
  • Wildcards match one label; IPs need IP: SANs; SNI decides which certificate a shared server presents.
  • Revocation via CRL/OCSP soft-fails and is easily suppressed; stapling helps, short-lived certificates help more.
  • ACME made issuance and renewal automatic; the remaining failure is renewal that silently stopped, so alert on expiry from outside.
  • Six failure shapes cover nearly every incident: expired, mismatch, missing intermediate, bad clock, self-signed, inspection proxy.

Why does this exist?

Mechanisms are answers to constraints. Open each question before reading the answer.

Why a chain instead of one signature from a root?

Roots must live offline and change rarely because every client ships them; signing millions of leaves needs an online key. Intermediates are the online, revocable, replaceable layer between the two.

Why does the server send the intermediates instead of the client fetching them?

Because the client should not need extra network round trips (or a plaintext HTTP fetch) in the middle of establishing a secure channel. AIA fetching exists as a browser fallback, which is exactly why a missing intermediate "works in Chrome" and fails everywhere else.

Why did the industry move to short-lived certificates rather than fixing revocation?

Every revocation check either adds latency and a privacy leak (OCSP) or fails open when blocked. Short validity bounds the damage of a leaked key without a runtime check, and automation made short lifetimes operationally free.

Why does `mkcert` or a dev CA beat disabling verification?

Disabling verification is usually a global switch that survives into staging and production configs; a local root trusted only by your machine keeps the same code path as production and fails loudly if the wrong certificate is served.

Certificate chain

Certificate chain validation
The client checks the leaf against the name it asked for, the clock, the chain of signatures back to a root it already trusts, and revocation. Break something and read what the browser and curl say.
Break
Client
Educational model
CN=engineer-atlas.devserved by the server
issuer CN=Atlas R3
SAN engineer-atlas.dev, www.engineer-atlas.dev
valid 2026-06-01 → 2026-08-30
↑ signed by
CN=Atlas R3served by the server
issuer CN=Atlas Root X1
valid 2024-03-01 → 2029-03-01
↑ signed by
CN=Atlas Root X1OS / browser trust store
issuer CN=Atlas Root X1 (self)
valid 2015-01-01 → 2035-01-01
  1. 0. Server presents its chainok
    2 certificate(s) in the Certificate message: CN=engineer-atlas.dev → CN=Atlas R3
  2. 1. Hostname matches a SAN?
  3. 2. Every certificate valid at the system clock?
  4. 3. Signatures verify up to a trusted root?
  5. 4. Not revoked (OCSP stapling)?
$ openssl s_client -connect engineer-atlas.dev:443 -servername engineer-atlas.dev </dev/null
Certificate chain
 0 s:CN=engineer-atlas.dev
   i:CN=Atlas R3
 1 s:CN=Atlas R3
   i:CN=Atlas Root X1
---
Verify return code: 0 (ok)
1/6 · Server presents its chain

How it fails

What the failure looks like from inside real software.

  • The certificate was renewed on disk but nginx/HAProxy/the Node process was never reloaded; probes from outside show the old expiry while the box says it is fine.
  • A new microservice fails with unable to get local issuer certificate while browsers load the same host — the served chain lacks the intermediate.
  • Mobile app with certificate pinning breaks when the CA rotates its intermediate; pin to your own key or to the root, and always ship a backup pin.
  • Internal CA root distributed to laptops but not to the CI runners, the Docker base image or the Java keystore; only some clients can talk to internal services.
  • Wildcard *.example.com deployed for api.v2.example.com; the extra label does not match.
  • A device with a dead RTC boots with the clock at 1970 or 2000 and rejects every certificate as not yet valid.