This is an automated archive made by the Lemmit Bot.
The original was posted on /r/selfhosted by /u/esiy0676 on 2025-02-09 00:45:02+00:00.
I would like to share the following "blogpost" of mine on use of SSH certificates, which for some reason evade popularity. It was originally written for Proxmox VE users, but only the last paragraph adds context to the case, the rest is all applicable to everyone deploying lots of systems.
The target behind the link contains no tracking, no ads and if you do not mind the limited formatting of Reddit, the same content follows inline in full as well.
Public Key Infrastructure with Secure Shell
TL;DR Secure your SSH infrastructure from the very first boot. Rotate keys and never trust a previously unknown machine. Never pass through a key-not-known prompt and do not get used to the identification-changed warning with a remote host.
ORIGINAL POST Public Key Infrastructure with Secure Shell
Lots of administration tasks are based on SSH, the ubiquitous protocol used to securely connect to remote hosts. But quite a minority of those is set up in a systematic way which would amount to a secure-by-design infrastructure approach. Perhaps it appears to be a hassle, but ease of manageability of a system is a good indication of its soundness in terms of security as well, so security should be made seamlessly easy. As environments grow, the question of trust across systems, real and especially virtual - all the way to the last containeraised workload - goes often unaddressed.
Investigate later
Trust everything from the get go. That sounds terrifying - at least to a security conscious administrator. But for some reason - not always, not with SSH, anyways:
The authenticity of host '10.10.10.10 (10.10.10.10)' can't be established.
ED25519 key fingerprint is SHA256:k95pBxp+arqCAfTTYDHhD63o6O0Sff7zgyzcglxbGaE.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?
And everyone is about to studiously paste in the fingerprint from another source - of course not. They would investigate after encountering an issue, but not before.
Even more relatable would be the pesky warning relating to the same:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:uQEwXegch2seMpndUTxkH9cv6qDqD+25Q2+uyZHldLA.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /root/.ssh/known_hosts:1
remove with:
ssh-keygen -f "/root/.ssh/known_hosts" -R "10.10.10.10"
Host key for 10.10.10.10 has changed and you have requested strict checking.
Host key verification failed.
Naturally, we will do as just been told - this is how most of us got intimate with ssh-keygen^ in the first place, unfortunately often only with its -R switch. The warning, while explicit, is also convoluted. After all, it is only "POSSIBLE" that there is some shenanigans going on, not likely. We are just re-using an IP address or hostname. And it cannot happen to us, anyways - everything else around is secure, so this does not matter, or does it?
Second thought: If it is a secure environment, why are we using encrypted communications within in the first place?
Trust is blind
The concept of trust on first use (TOFU) suddenly becomes familiar. It is, after all, the model of connections non-professionals use all of the time without giving it a second thought. They know this one time, it is just the new target machine being set up, or they are accessing it from this new client machine. But it's really bad, especially in terms of forming habits - since who knows which case is which and when. Every time this question is answered yes, our repertoire of known host keys grows by one more dubious entry - a host, that became trusted, just like that. A sophisticated attacker knows this as well.
Strict checking
It is so common, one might even make use of an option that SSH provides - StrictHostKeyChecking^ - and simply set it to no (perhaps counterintuitively, this causes the always assumed answer be yes - to continue connecting; the default for the option is to ask). But bad habits should not be reinforced, instead they need tackling in the opposite way and that is what the option invites us to do - set it to yes, which means our answer will always be no to the unknown unknowns - we do NOT want to connect to hosts we had not encountered so far. We can set this for the local user by appending a single configuration line file entry:
cat >> ~/.ssh/config <<< "StrictHostKeyChecking yes"
Now all these connections to unknown hosts will be failing, as they should - with a resounding relief of the security officer. Well, that's not too helpful. How do we go about connecting to all these new machines?
Cheating consciously
We will create an exception. Make a shell alias, an explicit invocation of which, will ignore the defaults, by overriding them through its command line.
alias blind-ssh='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'
NOTE Aliases defined on a command line like this do NOT survive through the end of the current shell session, but we do not worry about it in this demonstrative case. We are going to backtrack on it all, anyways.
This will accept blindly whatever key is being presented. If it has not been previously known, it will not affect the record of known hosts - unlike typing away yes did. Then a connection like this will "just work":
blind-ssh dubious@den.internal
NOTE The way this works is setting
StrictHostKeyCheckingtono, which means ayespresumed for every "sure you want to continue connecting?" prompt. That, however, also appends the newly approved key into the list of known host keys of the user. By redefining its location viaUserKnownHostsFile^ directive, it will be added there, which in this case, means nowhere.
But what kind of piece of advice is this? First forbid it, then bypass it. Exactly. But it actually IS better than the previous state in that it:
- forces one to only use such alias sporadically;
- no host keys' list is polluted;
- best of it all, such alias is undefined on a new system.
Because - one should NEVER need it after reading through the end. How so?
SSH - simple and secure
At its face value, SSH as a standard^ remains dependable and secure (not only) for the reason that it is conceptually simple. Clients connect to hosts. Clients authenticate (their users) to hosts. And what is less paid attention to - but equally true - hosts authenticate to clients. This is why there are the above-kind-of prompts in the first place.
So there's a key on each end, a public key cryptography kind of key - where the knowledge of public key of the one's counterpart (client/user or host) helps the system authenticate (both ends). The distribution of the public keys - so that the endpoints know each other - is left out of scope. And so TOFU became a habit.
IMPORTANT The scenario above assumes that keys (not passwords) are used to authenticate users - the primary side benefit that SSH provides. Some users opt to keep using passwords for the user authentication instead, which of course has its own implications, but even then the "distribution" problem remains out of scope - somehow the password had to be set on the target system prior to SSH connection is about to be established. That said, keys are always at play to authenticate hosts.
TIP Another interesting aspect of using client/user keys for SSH authentication instead of passwords is that such user does not even need to have ANY password set on the target system. What a better way to solve the issue of worrying about secure passwords than by having none in the first place.
Public Key Infrastructure
Current OpenSSH^ - implementation we will be mostly focused on - does NOT support Public Key Infrastructure (PKI) as established by the standard of X.509^ and employed with better known SSL/TLS setups. But that said, it DOES support PKI as such - it is just much simpler.
Public keys can be signed by (other, private) keys resulting in certificates, where additional information may be added. The signing party is equally known as certification authority - a familiar term, but these are not the kind of certificates you need to go obtain from the likes of Let's encrypt. You just issue them, distribute and manage them within your full control and they do not go through third-party (and their obscure validation), thus preserve confidentiality of the infrastructure setup as a whole.
NOTE There is a standard of X.509v3 Certificates for Secure Shell Authentication^ and alternative implementations which do go that route. This is entirely out of scope here.
OpenSSH
There is two sides to each authenticated SSH connection. The host tha...
Content cut off. Read original on https://old.reddit.com/r/selfhosted/comments/1il2fjx/public_key_infrastructure_with_secure_shell/