Certificate Authority With
Multi-Tier Signing Hierarchy

Best practices of building & operating a Certificate Authority using OpenSSL and accommodating future use of cryptographic hardware

30 December 2003

Skip to OpenSSL commands and syntax


This is the least you need to know for construction and operation of a Certificate Authority (CA) using OpenSSL within an enterprise-wide deployment of X.509v3 Certificates for signing electronic documents, access control to remote computers and data encryption.

If the above paragraph makes little sense to you, please explore RFC 2459, www.openssl.org or more friendly material such as the OpenSSL book from O'Reilly (www.OReilly.com) as a starting point. This document supplies practical information and feature implementation where those leave off.

Topics explained here include multiple tier signing hierarchies. That is, counting the “end entity” certificates deployed to clients and the self-signed root CA, there are four certificates in our validation chain. This is useful for complex organizations securing their intranet, extranet and other resources.

Some terms are explained below. You are expected to have a basic understanding of PKI and Certificate concepts, even if only once installed a certificate within a Web server.

An imaginary bicycle manufacturing company is used to illustrate concepts, accounting for a distribution network of external dealers, out-sourced vendors providing parts and services, employees accessing servers, etc.

Fraud prevention and recovery from a technical and policy perspective are addressed. Human factors are also addressed– the balance of making it secure versus making it usable by non-geeks.

Incidental topics touched upon here may be useful as advocacy arguments to corporate management when pushing for PKI from within the IT trenches. (To that end, diagrams below have a corresponding Visio source document for you to modify freely.)

(Topics beyond the scope of this document, however, include mathematics and legislation.)


Public Key Infrastructure (PKI) has been gaining popularity over the past several years. There are many reasons, the least of which is expiration of the RSA algorithm patent, September of 2000.

More importantly, organizations are seeking to disconnect expensive leased circuits and eliminate the use of Virtual Private Network (VPN) equipment in favor of using Transport Layer Security (TLS), and its predecessor, Secure Socket Layer (SSL) enabled applications that use only encrypted content across the Internet.

Most examples within OpenSSL documentation or the O'Reilly book are too simplistic for practical use. While some applications may exist as an island– meaning, in isolation of any other application– most organizations prefer to have a consolidated signing hierarchy for better control and management of digital certificates.

Contemporary “best practices” from organizations including Microsoft and major automobile manufacturers are following techniques illustrated in this document.


Today, this is a systems integration effort rather than strictly software development. A systems programmer using a high level language such as Python or Perl is best for this project. Save the software engineers for creating custom work-flow unique to your organization or for enabling access to legacy applications.

Design of Certificate Authority

The general design for implementing your own CA uses the free, open source OpenSSL utility as a command-line black box.
(See www.openSSL.org)

Our approach works on Unix as well as Windows with Cygwin installed.
(See www.cygwin.com)

The core implementation here uses simple Borne shell scripts for extreme simplicity.

A validation chain is used here, which allows for express revocation of all descendant certificates in one step.

The validation chain gets bundled within a single certificate file for ease of installation on your clients.

There are both off-line and networked components of the CA. This is necessary in larger corporations when multiple levels of a signing hierarchy are used. Certain elements are always disconnected from any network, while others are best served via your intranet.

Physical access controls include a “signing ceremony” whereby upper level executives control passwords for higher echelons of the signing hierarchy yet are without login access to the server by which those passwords are used. This forces multiple people to be involved in creating new branches of certificates, further reducing potential fraud.

Application level validation uses the Online Certificate Status Protocol (OCSP) and the deprecated-but-still-widely-used Certificate Revocation List (CRL).

Finally, fraud recovery concerns are addressed through certificate policy such as including a Certificate Practice Statement (CPS) to keep the legal folks happy.

Example Company Deploying PKI

Our demonstration business is a bicycle company, referred to as ABC.

Franchised bicycle dealers often sell other brands as well. Consider dealers to be only minimally trusted. From a logistics perspective of deployment certificates, they exist as a relatively small, controlled population.

Activities of a dealer include using an embedded PC for tuning high-end bicycles and checking serial numbers for potentially stolen bikes. Outside of the service area, a desktop PC may be used for ordering parts and viewing newsletters. Because dealers must be kept informed ahead of the public, the newsletter is a restricted access web site.

The manufacturer follows contemporary practices of “just in time” parts delivery for their assembly line. This requires information to be passed between ABC and their upstream vendors. Here, we’ll use names of actual companies in the business for clarity even though the relationships are fictitious. These vendors include the maker of components (Shimano) such as gears and deraileurs plus others for each of frames (Giant), wheels (Araya), tires (Continental), etc.

Our model accounts for employees occasionally working from home and sales people who travel frequently, all needing access from outside of corporate headquarters.

All employees require certificates for multiple uses: signing documents electronically, TLS/SSL client access and data encryption. As with the dealers, employees are a small, controlled population.

Internal servers require TLS/SSL certificates yet purchasing one from, say, VeriSign was deemed an unnecessary expense during prior cost-reduction measures.

And of course, the general public needs access to a secure web site for supplying product registration information.

Finally, the company protects its certificate management system with certificates. (That is, you need a certificate issued from the off-line CA in order to access the on-line request & management system.)

Technology, Policy & Usability

Please understand that PKI is perhaps only 30% technology. At least 50% is policy and the rest, usability.

That is, whether you go with OpenSSL or the Microsoft CryptoAPI (CAPI) is a straight-forward choice based upon available criteria and experience of your software developer.

Policy dictates whether or not you use a single certificate with multiple “Key Usage” constraints for all of signing, encryption and access. (Whereas, you might use separate certificates for each purpose for stricter controls.) For example, do you use your house key to start your car? Probably not, but you may have them on the same key ring, yes?

Then there is the human factor. If you use multiple certificates for access to different servers, do you leave it up to employees to figure out which one to use? Or do you spend additional resources to make their lives easier while maintaining adequate security?

Best practices suggests that you bend the tool rather than the person. The alternative is that staff becomes lazy and you then have implicit trust in a system that is actually weaker than before.

These become more of a factor when using hardware tokens for client certificates when the human in the loop must have the appropriate USB token/key plugged in for everything to work.

Other best practices suggest that you first employ a low impact or weak security application where any compromise of the certificate system is less of a concern. That is, learn to crawl before you run.

Application Architecture

When security standards change as well as your organization’s understanding of how to use PKI matures, your implementation will change.

Your needs may iterate through a common path:

  1. Transparently using server certificates such as for Apache with mod_ssl
  2. Purchasing individual certificates from a trusted CA like VeriSign
  3. Quick & dirty CA, following OpenSSL examples from documentation
  4. Each application uses own Root certificate signing end-entities directly, creating new CA “island” for each project
  5. Introduction of a signing hierarchy (validation chain) so multiple applications share common infrastructure
  6. Use of hardware signing module for CA
  7. Use of hardware tokens for client authentication

Understand your non-PKI needs first. Isolate the PKI related functionality to its own black box, which in turn calls OpenSSL.

If, for example, you have a requirement for creating hundreds of certificates for a mass deployment to employees or outside vendors, keep batch processing features as a separate work-flow. Likewise, your organization’s own practices and procedures constraining things such as the handling of certificates once created should also be isolated or dealt with via “wrapper” scripts.

In the long run, this will save you much aggravation and improve maintenance.

But this goes far beyond the usual computer science topic of being modular as the separate pieces may need to be isolated for security or policy purposes (e.g., access control).

Components of the Certificate Authority

Isolate registration and request features from functions which actually issue certificates.

The Registration Authority (RA) is separate from the Issuing System. The RA contains access to databases of dealers, employees and vendors.

The Issuing System, sometimes referred to as “the certificate authority”, is where certificates are actually created and signed.

The Validation Authority (VA) is separate still and may in fact be just a feature within a third party product such as an TLS/SSL “termination” or “optimizer” in front of a web server. (Examples include the F5 Big/IP load-balancer with TLS/SSL acceleration options.)

Isolating the certificate features specifically is helpful with off-line components of a CA.

Off-Line Elements of the CA

Certain parts of your CA should be off-line while the rest is accessible from the network.

Your self-signed, root certificate and upper level subordinate CA certificates (signed by the root) should always live where physical access is severely restricted and network access is eliminated.

Then, depending upon the requirements of each application or project requiring certificates, the issuer certificates that sign end entities may or may not be on a networked server. The distinction will be obvious for certain high risk applications but subtle in one particular case.

For certificates issued to staff for creating HTTPS sessions to your Registration Authority, you want these to be issued from the off-line server as well. This provides safeguards against an unauthorized person from creating a fraudulent yet official request, should the RA host be compromised.

Registration Authority

The Registration Authority (RA) is essentially a request and management system.

Basic components of an RA include a back-end database and front-end user interface.

Minimal features are:

Note that request versus approve functions are separated.

Useful features to be supported should include:

As with any database, of course, construct access permissions to isolate data entry tasks from security-related functions such as certificate maintenance tasks. Handling a request versus actually issuing should also be separated.

In general, separate individuals would each have access to only one component.

That is, no one person should have complete access to data entry, request and issue functions at the same time.

Data Flow Between CA Components

As your application matures, you of course want your certificate operations to be completely replaced with improved features such as accessing the hardware signing module (HSM).

If your data flow consists of sharing flat files for requests and results, your work flow may move from a stand-alone application to being Web-based to be integrated with existing request and management systems you may already have.

Using comma separated values in text files, such as .csv files generated and readable by Microsoft Excel are best, especially since the CSV module support for Python and Perl work well.

This minimizes the actual interface– from a systems integration perspective– between the Registration Authority and the Certificate Authority to a pair of files: “request.csv” and “update.csv”. This, in turn, allows for manual transfer of requests to an off-line CA with simplicity of a portable USB storage device or even floppy disk.

More importantly, text-based files are easily archived within version control systems.

Signing Hierarchy

Signing Hierarchy with 4 tiers including Root and end entities; Click for larger view

Signing Hierarchy also referred to as Validation Chain, depending upon context

(View larger image or Visio document)

OpenSSL Commands & Syntax

The commands below work for the Bash or Borne shell, such as what is available with Cygwin, Linux, BSD Unix, Solaris, etc.

First, start with a file name for the certificate so you, the human, can readily locate it. Typical practices include using the dealer name and ID, as in our bicycle manufacturer example.

name="friendly name of recipient"

Next, create general purpose variables to make data flow easier to follow.



Make certificate request and generating key pair:

openssl req \
 -config openssl.config \
 -nodes -passin pass: \
 -newkey rsa \
 -days 1481 \
 -keyform PEM \
 -keyout "$KEY" \
 -outform PEM \
 -out "$REQUEST"

Sign request / making the certificate:

openssl ca \
 -config openssl.config \
 -batch \
 -in "$REQUEST" \
 -out "$PUBLIC"

Package certificate as PKCS#12 file: (DER encoded)

openssl pkcs12 \
 -export \
 -inkey "$KEY" \
 -in "$PUBLIC" \
 -nodes -passout pass: \
 -name "$name" \
 -certfile "$validationChain"

Finally, embed the public portions of certificates higher up in the validation chain.

cat "$validationChain"  >>  "$FINISHED"

Planning For Future

The various standards referenced in this document will most certainly change in future.

Fortunately, most repositories of RFCs such as https://tools.ietf.org/html/rfc5280 are annotated when each standard becomes superseded.

Accounting for a lengthy deployment, one client was given custom bootable FreeBSD installation media containing everything necessary, including:

Since the signing host was locked in a cabinet without network access, the installation media was used to perform a clean wipe & install. That same image was then copied to additional CD-R discs and along with paper instructions (for further redundancy), were put into lock-boxes in rotation for off-site secure storage.

Different lock-boxes contained USB keys with production private keys, and only a very limited group within the security organization had access.

The intent behind this kit is that the people involved in initial deployment are unlikely to be available in future when the system may need to be rebuilt or upgraded.

Leave a (secure) trail of breadcrumbs, and your successors will thank you for it.

Copyright © 2003 Daniel Joseph Pezely
May be licensed via Creative Commons Attribution.