This is an automated archive made by the Lemmit Bot.
The original was posted on /r/selfhosted by /u/esiy0676 on 2025-02-16 00:53:31+00:00.
Whilst originally written for Proxmox VE users, this can be easily followed by anyone for standard Linux deployment - hosts, guests, virtual instances - when adjusted appropriately.
The linked OP of mine below is free of any tracking, but other than the limiting formatting options of Reddit, full content follows as well.
SSH certificates setup
TL;DR
PKI SSH setups for complex clusters or virtual guests should be a norm, one which improves security, but also manageability. With a scripted setup, automated key rotations come as a bonus.
ORIGINAL POST SSH certificates setup
Following an explanatory post on how to use SSH within Public-key Infrastructure (PKI), here is an example how to deploy it within almost any environment. Primary candidates are virtual guests, but of course also hosts, including e.g. Proxmox VE cluster nodes as those appear as if completely regular hosts from SSH perspective out-of-the-box (without obscure command-line options added) even when clustered - ever since the SSH host key bugfix.
Roles and Parties
There will be 3 roles mentioned going forward, the terms as universally understood:
- Certification Authority (CA) which will distribute its public key (for verification of its signatures) and sign other public keys (of connecting users and/or hosts being connected to);
- Control host from which connections are meant to be initiated by the SSH client or the respective user - which will have their public key signed by a CA;
- Target host on which incoming connections are handled by the SSH server and presenting itself with public host key equally signed by a CA.
Combined roles and parties
Combining roles (of a party) is possible, but generally always decreases the security level of such system.
IMPORTANT
It is entirely administrator-dependent where which party will reside, e.g. a CA can be performing its role on a Control host. Albeit less than ideal - complete separation would be much better - any of these setups are already better than a non-PKI setup.
One such controversial is combining a Control and Target into one - an architecture under which Proxmox VE falls under with its very philosophy of being able to control any host of the cluster (and guests therein), i.e. a Target, from any other node, i.e. an architecture without a designated Control host.
TIP
More complex setup would go the opposite direction and e.g. split CAs, at least one for signing Control user keys and another for Target host keys. That said, absolutely do AVOID combining the role of CA and a Target. If you have to combine Control and a Target, attempt to do so with a select one only - a master, if you will.
Example scenario
For the sake of simplicity, we assume one external Control party which doubles as a sole CA and multitude of Targets. This means performing signing of all the keys in the same environment as from which the control connections are made. A separate setup would only be more practical in an automated environment, which is beyond scope here.
Ramp-up
Further, we assume a non-PKI starting environment, as that is the situation most readers will begin with. We will intentionally - more on that below - make use of the previously described setup of strict SSH approach,^ but with a lenient alias. In fact, let's make two, one for secure shell ssh^ and another for secure copy scp^ (which uses ssh):
cat >> ~/.ssh/config <<< "StrictHostKeyChecking yes"
alias blind-ssh='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
alias blind-scp='scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
Blind connections
Ideally, blind connections should NOT be used, not even for the initial setup. It is explicitly mentioned here as an instrumental approach to cover two concepts:
blind-ssh as a pre-PKI setup way of executing a command on a target, i.e. could be instead done securely by performing the command on the host's console, either physical or with an out-of-band access, or should be part of installation and/or deployment of such host to begin with;blind-scp as an independent mechanism of distributing files across, i.e. shared storage or manual transfer could be utilised instead.
If you already have a secure environment, regular ssh and scp should be simply used instead. For virtual hosts, execution of commands or distribution of files should be considered upon image creation already.
Root connections
We abstract from privilege considerations by assuming any connection to a Target is under the root user. This may appear (and actually is) ill-advised, but is unfortunately a standard Proxmox VE setup and CANNOT be disabled without loss of feature set. Should one be considering connecting with non-privileged users, further e.g. sudo setup needs to be in place, which is out of scope here.
Setup
Certification Authority key
We will first generate CA's key pair in a new staging directory. This directory can later be completely dismantled, but of course the CA key should be retained elsewhere then.
(umask 077; mkdir ~/stage)
cd ~/stage
ssh-keygen -t ed25519 -f ssh_ca_key -C "SSH CA Key"
WARNING
From this point on, the ssh_ca_key is the CA's private (signing) key and ssh_ca_key.pub the corresponding public key. It is imperative to keep the private key as secure as possible.
Control key
As our CA resides on the Control host, we will right away create a user key and sign it:
TIP
We are marking the certificate with validity of 14 days (-V option), you are free to adjust or omit it.
ssh-keygen -f ssh_control_key -t ed25519 -C "Control User Key"
ssh-keygen -s ssh_ca_key -I control -n root -V +14d ssh_control_key.pub
We have just created user's private key ssh_control_key, respective public key ssh_control_key.pub and in turn signed it by the CA creating a user certificate ssh_control_key-cert.pub.
TIP
At any point, a certificate can be checked for details, like so:
ssh-keygen -L -f ssh_control_key-cert.pub
Target keys
We will demonstrate setting up a single Target host for connections from our Control host/user. This has to be repeated (automated) for as many targets as we wish to deploy. For the sake of convenience, consider the following script (interleaved with explanations), which assumes setting Target's hostname or IP address into the TARGET variable:
TARGET=
Sign host key for target
First, we will generate identity and principals (concepts explained previously) for our certificate that we will be issuing for the Target host, we can also do this manually, but running e.g. hostname^ command remotely and concatenating its comma-delimited outputs for -s, -f and -I switches allow us to list the hostname, the FQDN and the IP address all as principals without any risk of typos.
IDENT=`blind-ssh root@$TARGET "hostname"`
PRINC=`blind-ssh root@$TARGET "(hostname -s; hostname -f; hostname -I) | xargs -n1 | paste -sd,"`
We will now let the remote Target itself generate its new host key (in addition to whichever it already had prior, so as not to disrupt any other parties) and copy over its public key to the control for signing by the CA.
IMPORTANT
This demonstrates a concept which we will NOT abandon: Never transfer private keys. Not even over secure connections, not even off-band. Have the parties generate them locally and only transfer out the public key from the pair for signing, as in our case, by the CA.
Obviously, if you are generating new keys at the point of host image inception - as would be preferred, this issue is non-existent.
Note that we are NOT setting any validity period on the host key, but we are free to do so as well - if we are ready to consider rotations further down the road.
blind-ssh root@$TARGET "ssh-keygen -t ed25519 -f /etc/ssh/ssh_managed_host_key"
blind-scp root@$TARGET:/etc/ssh/ssh_managed_host_key.pub .
Now with the Target's public host key on the Control/CA host, we sign it with the affixed identity and principals as previously populated and simply copy it back over to the Target host.
ssh-keygen -s ssh_ca_key -h -I $IDENT -n $PRINC ssh_managed_host_key.pub
blind-scp ssh_managed_host_key-cert.pub root@$TARGET:/etc/ssh/
Configure target
The only thing left is to configure Target host to trust users that had their keys signed by our CA.
We will append our CA's public key to the remote Target host's list of (supposedly all pre-existing) trusted CAs that can sign user keys.
blind-ssh root@$TARGET "cat >> /etc/ssh/ssh_trusted_user_ca" < ssh_ca_key.pub
Still on the Target host, we create a new (single) partial configuration file which will simply point to the new host key, the corresponding certificate and the trusted user CA's key record:
blind-ssh root@$TARGET "cat > /etc/ssh/sshd_config.d/pki.conf" << EOF
HostKey /etc/ssh/ssh_managed_host_key
HostCertificate /etc/ssh/ssh_managed_host_key-cert.pub
...
***
Content cut off. Read original on https://old.reddit.com/r/selfhosted/comments/1iqgaz1/guide_on_ssh_certificates_signed_by_a_ca_ie_not/
