Title image: SSH access that expires, short-lived certificates instead of long-lived keys

SSH keys that expire: short-lived access with OpenBao and Keycloak

Long-lived SSH keys pile up on servers, and nobody can tell which ones are still needed. This post shows an alternative I use: a trusted CA that signs keys for 30 minutes, with Keycloak as the identity behind it and OpenBao as the CA. It needs nothing but OpenSSH on the hosts and works on-premise or air-gapped.

Note

Terminology

OpenSSH uses the term certificate when a public key is signed. This is not a X.509 certificate, but a different format that is specific to OpenSSH. The CA (Certificate Authority) is the entity that signs the user’s public ssh key, and the signed public key is called a certificate. Certificate may refer also the metadata content of the signed key, like the validity period, the principals, the key ID, signature key, etc. I’ll be using both terms - signed key and certificate - depending on context.

What are the benefits of this setup?

  • No long-lived keys on the hosts, only a public key of the CA.

  • Short-lived certificates, automatically renewed, tied to a single sign-on identity.

  • Access control is centralized in Keycloak, not on each host.

  • Scalable: adding or removing users does not require touching every host.

  • Auditable: logs show who logged in and when, based on the certificate’s key ID.

  • In case of a compromised signed key, the compromise has a limited time window. If the compromise is detected, it can also be mitigated by removing the trusted CA keys on the remote hosts, basically rotating the CA key in the vault and its public keys on the trusting hosts. The mitigation is only on infrastructure level, not the user level, which makes it much easier to perform quickly and reliably. OpenSSH can also block individual certificates (even by serial number) with a key revocation list (KRL, see the ssh-keygen man page), which sshd reads via RevokedKeys. As far as I can tell, OpenBao’s SSH engine neither generates nor distributes such a list, and neither does OpenSSH, so you would have to build it and push it to every host yourself. Short TTLs remain the main protection.

The building blocks for our experimental setup:

  • OpenBao (the open source fork of Hashicorp Vault) is the CA and signs my public key.

  • Keycloak is the identity provider. It knows who I am and which roles I have. We will define a new client role to enable key signing in OpenBao.

  • OpenSSH already supports all of this. No agent, no extra software on the hosts, just a few lines of sshd_config. This can be automated with Ansible, but that is not the focus of this post.

This post can also be used in conjunction with Hashicorp’s Vault.

Why signed keys?

There are two main benefits of using signed keys: scalability and security.

Scalability

With plain keys, every host has a list of public keys. Adding or removing a person means touching every host, and nobody can tell from the list whether a key is still needed.

With certificates, the remote host holds one thing: the public key of the CA. Whoever presents a key that the CA has signed, with a valid time window and an allowed principal, gets in. Access is then decided where the identity lives, not on each machine (in the OIDC provider, Keycloak in this case).

When a person should get access revoked, it is enough to remove the role in Keycloak. The next time the person tries to get a certificate, it will fail and they will not be able to log in anymore. No need to touch every host.

With the abstraction layer of principals, local accounts are detached from the ssh key. See below for details about principals.

Security

A stolen 30-minute certificate is a small problem, a stolen key from a year ago can be a big problem.

The certificate of a signed key contains a time limited validity (ttl: time to live). The CA can decide to sign the key for a short time only (like 30 minutes).

The certificate also contains a serial number, which makes it possible to correlate logins with the signing event. In this post’s setup it is even possible to know who requested the signature, because the requesting user (person or bot) authenticates with Keycloak.

The CA: where the trust comes from

Everything rests on one private key, the CA key. Whoever holds it can sign access to every host that trusts it. So the question is where it lives. In this setup the OpenBao vault is the CA.

Offline root key and online CA key, both trusted by the host

Two keys: an online one that signs daily, and an offline one for emergencies.

Note

Certificate chains

An important detail: OpenSSH CA keys have no chains, unlike X.509/TLS certificates. It is not possible to derive intermediate certificates. The host simply has a list of trusted CA public keys (TrustedUserCAKeys). Still we can mimic (to some extent) the behavior of a certificate chain by using multiple keys: an online and one offline “root” key. Scale the number of trusted offline keys to your paranoia level.

  • An online CA key in the vault that signs the daily certificates. It is generated inside the vault (generate_signing_key=true).

  • An offline CA key that is only a break-glass key. Its public half is trusted by the hosts too. If the online CA key is compromised or the vault is down, you bring the offline key out, sign a certificate by hand, and get in. It lets you remove the compromised online key without locking yourself out: trust both, swap, then remove the old one on each host (configuration management is encouraged).

How to keep the offline key(s) is a matter of tradeoff between paranoia and convenience:

  • On paper (printed key, in a safe or under grandma’s pillow): cheap and robust, can last centuries, not for productive use - or is it?

  • On an HSM or YubiKey: the key never leaves the device, signing happens on it. Nice and practical, fits corporate security policies, but expensive (the YubiKey is the poor man’s version, and not recommended in regulated environments).

  • A KeePass file with a strong password, stored in a safe place.

  • Split with Shamir’s Secret Sharing (for example 3 of 5 shares), distributed to different people or places. No single share reveals anything, any three reconstruct the key. Can be implemented with mentioned storage methods.

Preparing OpenBao for SSH signing

OpenBao comes with a built-in SSH secret engine. It can generate a signing key, and it can sign public keys. The signing key never leaves OpenBao, and the public signing key can be read out and trusted by the hosts by deploying them onto the remote hosts. Remote hosts can use the public signing key to verify incoming signed keys for their authenticity.

# once, on the OpenBao side
bao secrets enable -path=ssh ssh
bao write ssh/config/ca generate_signing_key=true

Note

Configuring OpenBao for SSH signing requires elevated privileges. A temporary root token can be generated from the unseal keys and dropped after the configuration is done. See also the OpenBao documentation for details.

Next comes the SSH role. It defines what values OpenBao will put into the signature. The policy created further below allows signing on ssh/sign/admin, and this role is what lives behind that path:

bao write ssh/roles/admin - <<'JSON'
{
  "key_type": "ca",
  "allow_user_certificates": true,
  "allowed_users": "infra-admin",
  "default_user": "infra-admin",
  "ttl": "30m",
  "max_ttl": "1h",
  "key_id_format": "{{token_display_name}}",
  "default_extensions": {"permit-pty": ""}
}
JSON
  • allowed_users is the list of principals that may appear in a certificate signed through this role (comma separated, no wildcard here on purpose). Asking for anything else fails at signing time.

  • default_user is used if the request contains no valid_principals.

  • ttl and max_ttl keep the certificates short-lived.

  • key_id_format writes the authenticated OpenBao identity into the key ID, which is what shows up in the sshd log later.

  • default_extensions grants a terminal (permit-pty). Without any extension, the certificate is valid but gives no interactive shell.

Planting the trust on the remote host

The remote host (the one that grants access to ssh connections) needs three things. All of them are public information:

CA public key, principal file and sshd drop-in on the host

What an Ansible role puts on the server (or any other deployment mechanism, which can be also a manual scp for testing).

The CA’s public key we already covered above. What is the principal file? It is a simple text file that maps local accounts on the remote host to principals that are allowed to log in as them. There can be one file per local account on the remote host with the list of allowed principals, which are provided in the signature during ssh login. More on that later.

The drop-in for sshd is a small configuration snippet that points to the CA public key and the principal file(s). It is basically the configuration for the ssh daemon to trust the CA and to check the principals. Details below.

How to get the CA public key from OpenBao? The bao CLI can read it:

bao read -field=public_key ssh/config/ca > ssh-user-ca.pub

If you didn’t log in to OpenBao yet you can do so with:

# bao login -method oidc requires additional policies to read the public key.
# out of scope for this post.
bao login # requires a root token.

Note

Permissions for downloading the CA’s public key

Reading the public key requires permissions. For simplicity’s sake, let’s use the root token for this, but a dedicated policy can be created that only allows reading the public key. This is out of scope for this post.

On the remote host create a drop-in for sshd that points to the CA public key and a file with the authorized principals:

# /etc/ssh/sshd_config.d/10-ssh-ca.conf
TrustedUserCAKeys /etc/ssh/trusted-user-ca.pem
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
  • sshd uses the first value it reads, so the drop-in must sort before 50-cloud-init.conf.

  • Always check the config with sshd -t before reloading, keep a second session open, and only switch off authorized_keys (AuthorizedKeysFile none) after the certificate login has been proven to work. I let Ansible do this and roll back the drop-in if validation fails.

Proving who I am: Keycloak, OIDC and a role claim

OpenBao does not know me. We want to delegate that to an OIDC (OpenID Connect) provider, like Keycloak. The trick is that Keycloak puts my client role into the token as a claim, and OpenBao decides based on that claim, if I am allowed to ask for a ssh-key signature.

Sequence: OIDC login at Keycloak, OpenBao token, signing, ssh login

From bao login to a shell, in eight steps.

The Keycloak side needs a one time configuration (in the realm):

  1. A client openbao (client in this context is the OpenBao instance, it’s a client application for the oidc provider). We assume it already exists, see the OpenBao documentation for details.

  2. A client role ssh-sign, assigned to any user that shall be allowed to sign SSH certificates with the specified principal.

  3. A mapper that writes the user’s client roles into a roles claim of the ID token. This is basically a configuration of how the ssh-sign claim will land in the auth token.

The OpenBao side: an OIDC auth method, and a role that only lets a user in if the claim contains the right role, and then gives them nothing but one policy. This requires elevated privileges, which can be achieved via a root token in the vault. From OpenBao v2.6.0 on it is possible to generate a root token from the unseal keys and then drop it, which is a very useful security feature in my opinion.

bao write auth/oidc/role/admin-ssh - <<'JSON'
{
  "role_type": "oidc",
  "user_claim": "preferred_username",
  "bound_audiences": "openbao",
  "bound_claims": {"roles": ["ssh-sign"]},
  "policies": "ssh-sign-admin",
  "ttl": "30m"
}
JSON

bao policy write ssh-sign-admin - <<'HCL'
path "ssh/sign/admin" {
  capabilities = ["update"]
}
HCL

This OpenBao role connects the claims in the oidc token with the ssh-sign-admin policy in the vault. The ssh-sign-admin policy allows asking for a signature on ssh/sign/admin, which is provided by the ssh secret store of OpenBao. If you want to support another user group with other permissions (e.g. partitioning access to different parts of the infrastructure), you add another Keycloak role, map it to the identity token in the OIDC provider (e.g. Keycloak) and another policy.

Signing, and a small helper

Now we’re at the finish line. Here is how I can generate a signed ssh key on my laptop to log into the remote host:

bao login -method=oidc role=admin-ssh
bao write -field=signed_key ssh/sign/admin \
  public_key=@$HOME/.ssh/id_ed25519_bao.pub \
  valid_principals=infra-admin > ~/.ssh/id_ed25519_bao-cert.pub
ssh-keygen -L -f ~/.ssh/id_ed25519_bao-cert.pub

The key pair stays the same, only the certificate next to it (-cert.pub) is replaced.

Now let’s setup the local ssh to use the signed key and renew it automatically on a new ssh session. Two things are needed: 1. a script that renews the signed key and 2. a Match exec line in the ~/.ssh/config that calls the script before connecting to the host.

Here the script, put it into your home path for this example as ~/bao-ssh-cert.sh.

bao-ssh-cert.sh
 1#!/usr/bin/env bash
 2# Renews the SSH user certificate from the OpenBao SSH CA when it is missing or about to expire.
 3# Logs in to OpenBao with OIDC (Keycloak) only if the current token is not valid. The key pair is reused, only the certificate changes.
 4# It requires the `bao` CLI to be installed and configured.
 5#
 6#   bao-ssh-cert            renew if less than MIN_SECONDS of validity are left
 7#   bao-ssh-cert --check    only report, exit 0 if the certificate is valid long enough, 1 if it needs renewal
 8#   bao-ssh-cert --force    renew now
 9#
10# Settings (environment): BAO_ADDR, SSH_KEY, MIN_SECONDS, BAO_SSH_SIGN_PATH, BAO_SSH_PRINCIPAL, BAO_OIDC_ROLE
11set -euo pipefail
12
13export BAO_ADDR="${BAO_ADDR:-http://localhost:8443}"
14key="${SSH_KEY:-$HOME/.ssh/id_ed25519_bao}"
15cert="${key}-cert.pub"
16min_seconds="${MIN_SECONDS:-300}"
17sign_path="${BAO_SSH_SIGN_PATH:-ssh/sign/admin}"
18principal="${BAO_SSH_PRINCIPAL:-infra-admin}"
19oidc_role="${BAO_OIDC_ROLE:-admin-ssh}"
20
21mode=renew
22case "${1:-}" in
23  "") ;;
24  --check) mode=check ;;
25  --force) mode=force ;;
26  *) echo "usage: ${0##*/} [--check|--force]" >&2; exit 2 ;;
27esac
28
29# Seconds until the certificate expires; negative when expired, empty when there is none
30seconds_left() {
31  [[ -f $cert ]] || return 0
32  local until
33  until=$(ssh-keygen -L -f "$cert" | sed -n 's/^ *Valid: from .* to \([^ ]*\)$/\1/p')
34  [[ -n $until ]] || return 0   # "Valid: forever" or unreadable: treat as unknown
35  echo $(( $(date -d "$until" +%s) - $(date +%s) ))
36}
37
38left=$(seconds_left)
39if [[ -n $left ]]; then
40  if (( left > 0 )); then echo "certificate $cert valid for another $(( left / 60 )) minutes"
41  else echo "certificate $cert expired $(( -left / 60 )) minutes ago"; fi
42else
43  echo "no usable certificate at $cert"
44fi
45
46needs_renewal=true
47if [[ $mode != force && -n $left ]] && (( left >= min_seconds )); then needs_renewal=false; fi
48
49if [[ $mode == check ]]; then
50  if $needs_renewal; then exit 1; else exit 0; fi
51fi
52$needs_renewal || exit 0
53
54[[ -f $key.pub ]] || { echo "public key $key.pub is missing (ssh-keygen -t ed25519 -f $key -N '')" >&2; exit 1; }
55command -v bao >/dev/null || { echo "bao CLI not found" >&2; exit 1; }
56
57if ! bao token lookup >/dev/null 2>&1; then
58  echo "OpenBao token missing or expired, logging in with OIDC (role $oidc_role)"
59  bao login -method=oidc role="$oidc_role" >/dev/null
60fi
61
62# Written to a temporary file first, so a failed signing does not destroy the old certificate
63umask 077
64tmp=$(mktemp "$cert.XXXXXX")
65trap 'rm -f "$tmp"' EXIT
66bao write -field=signed_key "$sign_path" public_key=@"$key.pub" valid_principals="$principal" > "$tmp"
67ssh-keygen -L -f "$tmp" >/dev/null   # refuses anything that is not a certificate
68chmod 644 "$tmp"
69mv "$tmp" "$cert"
70trap - EXIT
71
72echo "new certificate $cert:"
73ssh-keygen -L -f "$cert" | sed -n 's/^ *\(Valid:.*\)$/  \1/p; s/^ *\(Key ID:.*\)$/  \1/p'

Set the environment variable BAO_ADDR to the URL of your OpenBao instance (default: http://localhost:8443), and optionally SSH_KEY to the path of your private key (default: ~/.ssh/id_ed25519_bao).

To automatically call this script on new ssh sessions, we need to configure the local ssh settings. Add a host definition to your ~/.ssh/config like so (replace the hostname, keyfile and user with your own):

# ~/.ssh/config

# Define the hosts where you want to use the signed certificate and trigger a generation.
Host remote-host1
  HostName <your host url or ip>
  User <your user>

  # The private key, generate once with `ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_bao`
  IdentityFile ~/.ssh/id_ed25519_bao

  # The signed public key from the renewal script.
  CertificateFile ~/.ssh/id_ed25519_bao-cert.pub

  # avoids sending other keys to the host on failure.
  IdentitiesOnly yes

  # The automatic renewal of the signed public key before connecting.
  # Adapt the script path accordingly.
  Match exec "test -t 2 && ~/bao-ssh-cert.sh >&2; false"

The magic is the Match exec line that calls the renewal script before connecting.

Principals: who may become whom

A certificate does not create accounts on the remote host. It carries a list of principals: names the CA vouches for. The host then decides which local accounts those names may log into (“Allow”). A principal is something like an abstraction layer between the identity/CA world (Keycloak, OpenBao) and the Linux accounts on the remote hosts.

Chain from Keycloak role to Linux account, and two groups

Every layer narrows what the one before may do.

  • The OpenBao SSH role restricts which principals can be put into a signature at all (allowed_users). Asking for a principal that is not allowed fails at signing time.

  • The host file /etc/ssh/auth_principals/<account> lists which principals may log in as that account. A signed key for infra-admin works for user1 on the hosts whose user1 file lists it, and nowhere else.

  • Best practice is that a principal names a role or group (infra-admin, ops, dba), not a person. The key ID in the certificate records who actually asked, and it shows up in the sshd log.

On the host, the mapping is one small file per account, referenced by the AuthorizedPrincipalsFile line from the drop-in above. The file name is the local account, the content is the list of principals allowed to log in as it:

/etc/ssh/auth_principals/
├── user1      # principals allowed to log in as user1
├── deploy     # principals allowed to log in as deploy
└── root       # principals allowed to log in as root
# /etc/ssh/auth_principals/user1
infra-admin
ops

Logging in as user1 now works with any certificate that contains infra-admin or ops principals. Different hosts can map the same principal to different accounts, or not at all. An account without a file cannot be reached with a certificate. The files decide who gets in, so they should be owned by root and not writable by others.

The key signature says what you are allowed to be, the login name says what you want to be: you still choose the account yourself (ssh user1@host or User in ~/.ssh/config), and sshd only checks that the two match. This is a pure allow mechanism. Without a matching entry in the principals file, nobody gets in as that account with a certificate, and access is revoked by removing the entry.

Note

Without AuthorizedPrincipalsFile, sshd requires the principal to be identical to the Unix username. That works, but the CA then has to know the account names on every host and the abstraction is gone. Also, always set principals explicitly when signing: a signature without any principals can be accepted for arbitrary users in some configurations.

What can go wrong

  • OpenBao sealed or Keycloak down: no new signatures. Existing ones work until they expire and open sessions remain open.

  • Locking yourself out on the remote host: Test in a second terminal with an active session before switching off authorized_keys. Make sure that you can access the remote host in case of locking yourself out (e.g. physical access, a console of the Hypervisor or cloud provider).

  • Unattended jobs: a renewal opens a browser for the Keycloak login. Renew by hand before running Ansible or another config management tool.

The end

We have now a working setup with a PKI that allows to log into remote hosts with a signed key, renew signatures automatically and allow/revoke access centrally in Keycloak. The process is scalable and auditable, though it will require adaptions to real world environments.

I hope the goal of this post was achieved: To show the principal mechanisms, the chain of trust, clarify some terminology, give a working example, and show how to connect the dots between OpenBao, Keycloak and OpenSSH.