This document is a derivative of some operational work I did for a customer a while ago. They had 250 or so linux/unix servers. We setup a single server we call mgmt01 and distributed this server’s public key to the other servers so that we could ssh to them as root with no password.
In theory its easy. In practice there are many problems that can occur.
There are some that may think that having keys like this is a security risk. Maybe it is. However, trying to manage 250+ unix/linux servers without it is seriously less efficient. Whatever, make sure you protect your own accounts on mgmt01 and keep mgmt01 secure. There should be no generic logins on this box and only limited users should be able to sudo to root.
Some management and reporting tools on mgmt01 derive most of their functionality from having this key deployed everywhere.
These instructions and troubleshooting notes apply to using keys to ssh without a password to any usercode you like. Trying to do it as root has extra issues that you may come across.
In theory all you do is generate a private/public key pair using a command like
ssh-keygen -b 1024 -t rsa -N ""
Then all you do is distribute your public key to the servers that you need to passwordlessly login to.
Stage 1 – On the
Management Server, mgmt01
cat /.ssh/rsa.pub
which gives you
ssh-rsa A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String = root@mgmt01
This is the public key for root on mgmt01. This is the key we must deploy.
Stage 2 – On the
Target Server
Become root on serverX (where serverX is the host where we want to be able to passwordlessly login as root).
Then do this (some of it may already have been done of course).
cd
mkdir .ssh
cd .ssh
echo ssh-rsa A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String-A-Very-Long-String = root@mgmt01 >> authorized_keys
(MAKE SURE THE KEY LISTED ABOVE IS ALL ON ONE LINE OF THE FILE)
chmod 600 authorized_keys
chmod 700 .
chmod 755 ..
ls -la
which should look something like this
total 18
drwx------ 2 root other 512 Jan 23 2014 . <------------------ must be permissions 700
drwxr-xr-x 20 root root 1024 Aug 5 16:06 .. <------------------ must only be writeable by the user, not group or other
-rw------- 1 root root 222 Nov 20 1993 authorized_keys <------------------ must be permissions 600
We assume that sshd is already running on serverX. If not, you'll have to install/configure it.
Stage 3 – Back on
the Management Server, mgmt01
Login as root
Test the connection
ssh serverX uname -a
You would expect to see something like this the first time you make a connection:
The authenticity of host ServerX (10.0.0.99) can't be established.
DSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
Are you sure you want to continue connecting (yes/no)? yes
If its all working ok you should see the results of the uname -a
eg Linux ServerX 2.8.8-20.8PAE#1 Fri Aug 18 14:39:22 EDT 2013 i686 i686 i386 GNU/Linux
On subsequent attempts you should simply see the results of the commands you issue.
There are many things that stop this from working. I'll cover the most common ones.
From mgmt01 as root
Test the connection
ssh serverX uname -a
ERROR:
ssh: connect to host serverX port 22: Connection refused
SOLUTION:
sshd is not running or is not running on port 22. Install it or reconfigure it.
ERROR:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
Please contact your system administrator.
Add correct host key in /.ssh/known_hosts to get rid of this message.
Offending key in /.ssh/known_hosts:211
RSA host key for ServerX has changed and you have requested strict checking.
Host key verification failed.
SOLUTION:
The host key for serverX in mgmt01's /.ssh/known_hosts file is wrong/outdated.
Remove the entry (a single line) from this file and try again. In the example above its on line 211 of the /.ssh/known_hosts file
ERROR:
ssh: connect to host ServerX port 22: No route to host
SOLUTION:
The server exists in DNS and either there is no routing for it or its turned off or disconnected from the network.
Turn it on, connect it etc
ERROR:
ssh: ServerX: host/servname not known
SOLUTION:
There is no such host name so use the correct hostname
Or
It is in another domain eg .otherdomain.company.com and your DNS client (/etc/resolv.conf) is not configured to look in this domain, so use the fully name eg serverX.otherdomain.company.com or reconfigure your DNS client
ERROR:
root@ServerX's password:
i.e. you are prompted for a password – but aren’t we using passwordless authentication?
SOLUTION:
Well, now that all the easy problems are out of the way lets deal with the most common problem and also the problem with the most possbile causes!
· The first thing to check is go back to the basic instructions above and make sure the key is in the authorized_keys file, that you have an authorized_keys and not an authorised_keys file, that all the permissions are correct. This resolves most of the issues. I do remember one server had some kind of symbolic link to a shared directory where the server side keys were stored. This was never going to work because the shared directory violated one of the permissions rules "only the user should be able to write to the home directory".
· The next most common issue is that the sshd_config file specifically blocks root logins. This file normally lives in /etc/ssh or /usr/local/etc. You need to make sure the file contains the option "permitrootlogin yes". Solaris 9 by default is no. You need to restart the sshd after making this change.
· Sometime the sshd server is not configured correctly and won't allow you to use keys for authentication eg if "PubkeyAuthentication no" is set in the sshd_config then you'll have to change it and restart sshd. By default sshd servers such as openSSH allow public key authentication.
· The rest of the problems are less common and more difficult. Some I have come across are local firewall configurations, firewall blocking ssh etc. Local firewalls etc like tcpwrappers will need to allow ssh from mgmt01.
· The advanced debugging method below can help significantly
The best debugging comes from the server side debug messages.
You need to run the server in debug mode.
Most of the time, you must run another sshd server in parallel to the main one since you are probably using ssh to connect to the server in the first place.
To do this we need to find a port that is not blocked by firewalls that mgmt01 can reach on the problem server. Normally, if there is a firewall, it blocks everything, so this method will only work if you run the main server in debug mode on port 22 - you will then need console access to the box.
Run the sshd server in debug mode on a different port like this
sshd -D -ddd -p 24
After one connection from an ssh client it will exit.
It shows heaps of debug messages and sometimes its really obvious what the problem is because the messages can be quite explicit.
On mgmt01, run the ssh client in debug mode also, connecting to the different port. Some time this helps but its no where near as useful as the server debug.
ssh -vvv -p 24 serverX
You may need to compare the debug messages from a "good" server/client to a "bad" server/client to work out what the errors are.
If you are getting non-obvious error messages the following web site can be very useful http://www.google.com