It happens every so often: some organization that sells publicly-trusted SSL
certificates does something monumentally stupid, like generating, storing,
and then intentionally disclosing all of their customers’ private keys (Trustico),
letting private keys be stolen via XSS (ZeroSSL),
or most recently, literally exploiting remote code execution in ACME clients
as part of their issuance process (HiCA aka QuantumCA).
When this happens, people inevitably refer to the certificate provider
as a certificate authority (CA), which is an organization with the
power to issue trusted certificates for any domain name. They fear that the integrity of
the Internet’s certificate authority system has been compromised by the “CA”‘s incompetence.
Something must be done!
But none of the organizations listed above are CAs – they just take
certificate requests and forward them to real CAs, who validate the
request and issue the certificate. The Internet is safe – from these
organizations, at least.
In this post, I’m going to define terms like certificate authority,
root CA, intermediate CA, and reseller, and explain whom you do and do
not need to worry about.
Note that I’m going to talk only about publicly-trusted SSL certificate
authorities – i.e those which are trusted by
mainstream web browsers.
Certificate Authority Certificates
“Certificate authority” is a label that can apply both to SSL certificates
and to organizations. A certificate is a CA if it can be used
to issue certificates which will be accepted by browsers.
There are two types of CA certificates: root and intermediate.
Root CA certificates, also known as “trust anchors” or just “roots”, are shipped with
your browser. If you poke around your browser’s settings,
you can find a list of root CA certificates.
Intermediate CA certificates, also known as “subordinate CAs” or just
“intermediates”, are certificates with the “CA” boolean set to true in
the Basic Constraints extension, and which were issued by a root or
another intermediate. If you decode an intermediate CA certificate with
openssl, you’ll see this in the extensions section:
X509v3 Basic Constraints: critical
CA:TRUE
When you connect to a website, your browser has to verify that the
website’s certificate was issued by a CA certificate. If the website’s
certificate was issued by a root, it’s easy because the browser already
knows the public key needed to verify the certificate’s signature.
If the website’s certificate was issued by an intermediate, the browser
has to retrieve the intermediate from elsewhere, and then verify that
the intermediate was issued by a CA certificate, recurring as necessary
until it reaches a root. The web server can help the browser out by sending
intermediate certificates along with the website’s certificate,
and some browsers are able to retrieve intermediates from the URL included
in the certificate’s AIA (Authority Information Access) field.
The purpose of this post is to discuss organizations, so for the rest
of this post, when I say “certificate authority” or “CA” I am referring
to an organization, not a certificate, unless otherwise specified.
Certificate Authority Organizations
An organization is a certificate authority if and only if they hold
the private key for one or more certificate authority certificates.
Holding the private key for a CA certificate is a big deal because it
gives the organization the power to create certificates that are valid
for any domain name on the Internet. These are essentially the Keys to
the Internet.
Unfortunately, figuring out who holds a CA certificate’s private key is
not straightforward. CA certificates contain an organization name in
their subject field, and many people reasonably assume that this must
be the organization which holds the private key. But as I explained
earlier this year, this is often not the case.
I spend a lot of time
researching the certificate ecosystem, and I am not exaggerating when I
say that I completely ignore the organization name in CA certificates.
It is useless. You have to look at other sources, like audit statements
and the CCADB, to figure out who really holds a CA certificate’s key.
Consequentially, just because you see a company’s name in a CA certificate,
it does not mean that it has a key which can be used to create certificates.
Internal vs External Intermediate Certificates
CAs aren’t allowed to issue website certificates directly from root certificates,
so any CA which operates a root certificate is going to have to issue themselves
at least one intermediate certificate. Since the CA retains control of the private key
for the intermediate, these are called internally-operated intermediate certificates.
Most intermediate certificates are internally-operated.
CAs are also able to issue intermediate certificates whose private key is
held by another organization, making the other organization a
certificate authority too. These are called externally-operated
intermediate certificates, and there are two reasons they exist.
The more legitimate reason is that the other organization operates, or intends to operate,
root certificates, and would like to issue certificates
which work in older browsers whose trust stores don’t include their roots.
By getting an intermediate from a more-established CA, they can issue certificates
which chain back to a root that is in more trust stores. This is called cross-signing,
and a well-known example is Identrust cross-signing Let’s
Encrypt.
The less-savory reason is that the other organization would like to become
a certificate authority without having to go through the onerous process
of asking each browser to include them. Historically, this was a huge loophole which allowed organizations to
become CAs with less oversight and thus less investment into security
and compliance. Thankfully, this loophole is closing, and nowadays Mozilla
and Chrome require CAs to obtain approval before issuing externally-operated
intermediates to organizations which aren’t already trusted CAs.
I would not be surprised if browsers eventually banned the practice outright,
leaving cross-signing as the only acceptable use case for externally-operated
intermediates.
Certificate Resellers
There’s no standard definition of certificate reseller, but I
define it as any organization which provides certificates which
they do not issue themselves. When someone requests a certificate from a
reseller, the reseller forwards the request to a certificate authority,
which validates the request and issues the certificate. The reseller
has no access to the CA certificate’s private key and no ability to cause
issuance themselves. Typically, the reseller will use an API (ACME or
proprietary) to request certificates from the CA. My company, SSLMate,
is a reseller (though these days we mostly focus on Certificate Transparency monitoring).
The relationship between the reseller and the CA can take many forms.
In some cases, the reseller may enter into an explicit reseller agreement
with the CA and get access to special pricing or reseller-only APIs. In other cases, the
reseller might obtain certificates from the CA using the same API and
pricing available to the general public. The reseller might not even
pay the CA – there’s nothing stopping someone from acting like a reseller
for free CAs like Let’s Encrypt. For example, DNSimple provides
paid certificates issued by Sectigo right alongside free certificates issued by
Let’s Encrypt. The CA might not know that their certificates are being resold – Let’s
Encrypt, for instance, allows anyone to create an anonymous account.
An organization may be a reseller of multiple CAs, or even be both
a reseller and a CA. A reseller might not get certificates
directly from a CA, but via a different reseller (SSLMate did this
in the early days because it was the only way to get good pricing
without making large purchase commitments).
A reseller might just provide certificates to customers, or they might use
the certificates to provide a larger service, such as web hosting or
a CDN (for example, Cloudflare).
White-Label / Branded Intermediate Certificates
Ideally, it would be easy to distinguish a reseller from a CA.
However, most resellers are middlemen who provide no
value over getting a certificate directly from a CA, and they don’t
want people to know this. So, for the right price, a CA will
issue themselves an internally-operated intermediate CA certificate with the reseller’s name in
the organization field, and when the reseller requests a certificate,
the CA will issue the certificate from the “branded” intermediate certificate
instead of from an intermediate certificate with the CA’s name in it.
The reseller does not have access to the private key of the branded
intermediate certificate. Except for the name, everything about the
branded intermediate certificate – like the security controls and the
validation process – is exactly the same as the CA’s non-branded intermediates.
Thus, the mere existence of a branded intermediate certificate does not in any
way affect the integrity of the certificate authority ecosystem, regardless
of how untrustworthy, incompetent, or malicious the organization named in the
certificate is.
What Should Worry You?
I spend a lot of time worrying about bad certificate authorities, but
bad resellers don’t concern me. A bad reseller can harm only their own
customers, but a bad certificate authority can harm the entire Internet.
Yeah, it sucks if you choose a reseller who screws you, but
it’s no different from choosing a bad hosting provider, domain registrar,
or any of the myriad other vendors needed to host a website. As always,
caveat emptor. In contrast, you can pick the best, most trustworthy
certificate authority around, and some garbage CA you’ve never heard of
can still issue an attacker a certificate for your domain. This is
why web browsers tightly regulate CAs, but not resellers.
Unfortunately, this doesn’t stop people from freaking out when a two-bit reseller
with a branded intermediate (such as Trustico, ZeroSSL, or HiCA/QuantumCA) does something awful. People flood
the mozilla-dev-security-policy mailing list to voice their concerns,
including those who have little knowledge of certificates and have never posted there before. These discussions
are a distraction from far more important issues. While the HiCA discussion was devolving into
off-topic blather,
the certificate authority Buypass disclosed in an
incident report that they have
a domain validation process which involves their employees manually looking up and interpreting CAA records.
They are far from the only CA which does domain validation manually despite it being easy to automate,
and it’s troubling because having a human involved gives attackers an opening to exploit
mistakes, bribery, or coercion to get unauthorized certificates for any domain.
Buypass’ incident response, which blames “human error” instead of addressing the root cause,
won’t make headlines
on Hacker News or be shared on Twitter with the popcorn emoji, but it’s actually far more concerning than
the worst comments a reseller has ever made.
About the Author
I’ve reported over 50 certificate authority compliance issues, and uncovered
evidence that led to the distrust of multiple certificate authorities and
Certificate Transparency logs. My research has prompted changes to ACME and
Mozilla’s Root Store Policy.
My company, SSLMate,
offers a JSON API for searching Certificate Transparency logs
and Cert Spotter, a service to monitor your domains for unauthorized, expiring,
or incorrectly-installed certificates.
If you liked this post, you might like:
Stay Tuned: Let’s Encrypt Incident
By popular demand, I will be blogging about how I found
the compliance bug
which prompted last week’s Let’s Encrypt downtime.
Be sure to subscribe by email or
RSS, or follow me on Mastodon or
Twitter.