Skip to content

External CA & Let's Encrypt certificates

FOG generates its own self-signed Certificate Authority at install time and uses
it for three things: the web server (HTTPS), the iPXE boot binaries (which are
compiled to trust that CA), and the fog-client, which pins the CA and uses it
to authenticate the server before acting on tasks.

Because of that pinning, you cannot simply drop a Let's Encrypt certificate onto
the Apache vhost and expect clients to keep working — the client validates the
server's certificate chain against the CA it pinned at registration time. This
document explains the supported way to use your own CA (including an internal
ACME / Let's Encrypt-style CA), the trade-offs of using public Let's Encrypt,
and the renewal caveats you must plan around.

TL;DR
- Use the installer's --external-ca support to sign FOG's server certificate
with your own intermediate CA. This is the supported, tested path.
- An internal ACME CA (e.g. step-ca / smallstep)
is the best fit — it gives you ACME automation without exposing FOG publicly,
and the CA you pin is stable.
- Public Let's Encrypt is possible but fragile: it requires a publicly
resolvable name (or DNS-01 automation), and LE rotates its intermediates,
which breaks the pinning model on renewal. Read the
caveats before going down this road.

Table of contents

How FOG uses certificates

Consumer What it uses Where it comes from
Web server (Apache/Nginx) srvpublic.crt + private key, served over HTTPS Generated by the installer, signed by FOG's CA
iPXE Trusts FOG's CA so it can fetch the boot file over HTTPS CA is compiled into the iPXE binaries at build time
fog-client Pins ca.cert.der and requires the server cert to chain to it Downloaded from /management/other/ca.cert.der

The critical detail is the pinned certificate. The client adds only
ca.cert.der to its validation store and requires that exact certificate to
appear in the server's chain. That means:

ca.cert.der must be the certificate that directly signs the server
certificate — i.e. the intermediate, not the root.

This is why "just point Apache at a Let's Encrypt cert" does not work: the client
never pinned LE's intermediate, so validation fails.

What --external-ca does

The installer (working-1.6 and later) can sign FOG's server certificate with a CA
you supply instead of generating its own. You provide three files:

Flag File Notes
--ca-cert Intermediate CA certificate (PEM) Must be a real CA cert (basicConstraints CA:TRUE)
--ca-key Intermediate CA private key (PEM) Must match --ca-cert
--ca-root Root CA certificate (PEM) --ca-cert must verify against this

Enable it with --external-ca, or answer the interactive prompt during install:

./installfog.sh \
    --external-ca \
    --ca-cert /root/pki/intermediate.crt \
    --ca-key  /root/pki/intermediate.key \
    --ca-root /root/pki/root.crt

What the installer does with them (see validateExternalCA() in
lib/common/functions.sh):

  1. Verifies the key matches the cert, that the cert is a CA, and that the
    intermediate chains to the root. Any failure aborts the install.
  2. Imports the files into FOG's CA directory (/opt/fog/snapins/ssl/CA/) as
    .fogCA.pem (intermediate), .fogCA.key, and .fogCAchain.pem (root +
    intermediate concatenated).
  3. Signs srvpublic.crt with your intermediate.
  4. Exports the intermediate as ca.cert.der — this is what the fog-client
    pins. (Pinning the root would break client validation, because the root is not
    what directly signs the server cert.)
  5. Passes the full chain (.fogCAchain.pem) to the iPXE build (TRUST=) and to
    the web server, so both trust root → intermediate → leaf.

The relevant values are persisted to .fogsettings (externalca, extcacert,
extcakey, extcaroot, sslcachain) so re-running the installer reuses them.
If the source files are no longer readable on a later run, the installer reuses
the already-imported CA in /opt/fog/snapins/ssl/CA/.

This is the cleanest way to get "Let's Encrypt-style" automation without any of
the public-LE downsides. You run a small internal CA that speaks ACME, point
acme.sh/certbot at it, and feed the resulting CA into FOG via --external-ca.

High-level setup:

  1. Stand up step-ca on a host
    you control. It issues you a root and an intermediate CA.
  2. Install FOG with --external-ca, passing step-ca's intermediate cert/key
    and root cert (see above). FOG's server cert is now signed by your
    intermediate; clients pin that intermediate.
  3. Issue / renew the server leaf cert via ACME against step-ca (e.g.
    acme.sh --server https://step-ca.internal/acme/acme/directory). Because the
    leaf is signed by the same intermediate the clients already pinned,
    renewing the leaf does not break client authentication.
  4. After each renewal, install the renewed leaf where Apache/Nginx serves it
    (a renewal hook — see Renewal and rotation).

Why this is better than public LE: the intermediate you pin is stable and under
your control, so leaf renewals are transparent to clients, and nothing needs to
be publicly resolvable.

Public Let's Encrypt: caveats

You can use the real public Let's Encrypt, but understand what you are signing
up for before you do.

  1. You need a publicly resolvable name. HTTP-01 validation requires LE to
    reach your server on port 80 over the public internet. Most FOG servers are
    internal imaging boxes and should not be exposed. Use DNS-01 validation
    (acme.sh/certbot with your DNS provider's API) to get a public cert without
    exposing the box.

  2. LE does not give you a CA key. With public LE you only ever receive leaf
    certificates — you never hold LE's intermediate private key. So you cannot use
    --external-ca to have FOG sign with LE. Instead you would pin LE's
    intermediate as the CA and let LE issue your leaf. This works only as long
    as the next point holds:

  3. LE rotates intermediates. Let's Encrypt periodically changes its
    intermediate CAs (e.g. R10/R11/R3…), and ACME clients can be handed certs from
    different chains. The moment your renewed leaf is signed by an intermediate
    your clients did not pin, client authentication breaks until every client
    re-pins. This is the core reason public LE is fragile for FOG and an internal
    ACME CA is preferred.

Bottom line: if you want ACME automation, run an internal ACME CA. Use
public LE only if you genuinely need publicly trusted certs (e.g. a
public-facing portal) and you have a plan for re-pinning clients when LE rotates
intermediates.

Renewal and rotation

FOG's certificate setup happens at install time. It does not auto-renew.
When a certificate is renewed you are responsible for putting it in place, and —
if the CA you pinned changes — for re-distributing trust:

  • Leaf renewal, same pinned CA (the normal step-ca case): just drop the new
    leaf where the web server reads it and reload the web server. Clients and iPXE
    are unaffected. Automate this with an ACME renewal hook (acme.sh --install-cert
    / certbot --deploy-hook).
  • Pinned CA (intermediate) changes (public LE rotation, or you rotate your
    internal intermediate): this is the disruptive case. You must:
  • Re-run the FOG installer so ca.cert.der, the iPXE TRUST bundle, and the web
    server config are regenerated against the new CA.
  • Have every host's fog-client re-pin the new ca.cert.der (re-run the
    client installer or whatever your re-registration flow is).
  • Have PXE clients pull the rebuilt iPXE binaries.

There is currently no automated client re-pinning or iPXE-rebuild trigger on
renewal — that is tracked as future work (true root-anchored intermediate support
in the client lives in FOGProject/zazzles#47). Plan your CA lifetimes
accordingly: long-lived, stable intermediates, short-lived leaves.

Switching an existing server to an external CA

If you re-run the installer with --external-ca on a server that already issued a
self-signed CA, the installer detects that the existing server cert no longer
verifies against the new chain and prints a warning: the web cert and iPXE
binaries are regenerated under the new CA, and any host whose fog-client already
pinned the old FOG CA will not trust the server until it re-pins. Re-run the
fog-client installer and reboot PXE clients after the switch.

Troubleshooting

  • The supplied CA private key does not match the supplied CA certificate —
    --ca-key and --ca-cert are not a pair. Confirm with:
    openssl x509 -noout -modulus -in cert | openssl md5 vs
    openssl rsa -noout -modulus -in key | openssl md5.
  • The supplied certificate is not a CA certificate — --ca-cert lacks
    basicConstraints CA:TRUE. You passed a leaf, not an intermediate CA.
  • The intermediate CA does not verify against the supplied root — --ca-cert
    does not chain to --ca-root. Check you exported the correct root.
  • Clients stop trusting the server after a renewal — the pinned CA changed.
    See Renewal and rotation; clients must re-pin the new
    ca.cert.der.

Related: this is the supported answer to the "Let's Encrypt support" request
(issue #633); the underlying external/intermediate CA installer support was added
for issue #794.