5. Setting up password-less SSH connectivity between compute nodes for users

Password-less SSH movement around the cluster is not only useful for administrators, it also allows users to visit their compute nodes and inspect their calculations while they are running.

For this to work we need a key pair that is both readable by the user, but at the same time well enough protected to function with SSH. Any private key that doesn’t have proper permissions will be ignored.

A simple way to avoid this is to generate a password-less SSH key pair for each user account. Our user’s shouldn’t have to worry about this, these keys will need to be generated before or during a user’s first login.

Remember the SSH client configuration we set up before in /etc/ssh/ssh_config for root?

Host *.hpc
  IdentityFile ~/.ssh/id_rsa
  IdentityFile ~/.ssh/id_rsa_internal
  StrictHostKeyChecking yes

Host 192.168.*
  IdentityFile ~/.ssh/id_rsa
  IdentityFile ~/.ssh/id_rsa_internal
  StrictHostKeyChecking yes

Host *
  IdentityFile ~/.ssh/id_rsa
  IdentityFile ~/.ssh/id_rsa_internal

This same configuration will work for any other user, as long as we have a special id_rsa_internal key pair in that user’s home directory. We also need to add the public key to authorized_keys of that user.

Let’s consider our options of generating them:

  • We could make the key generation part of our account creation script. That is a viable option, with the downside of not self-repairing in case a user accidentally destroys them.

  • We could insert a modified .bashrc or .bash_profile into /etc/skel, which would include steps to generate these keys in case they are missing or damaged. However, this has the downside that user’s will be able to manipulate this part of the configuration. Changes like this also do not propagate to existing user accounts.

  • Finally, a good place for this kind of system-wide initial configurations that propagates to all users is /etc/profile and its children in /etc/profile.d/. These files will be executed whenever a user logs in. We can add logic to ensure that the necessary keys are generated if they are missing or damaged.

Files in /etc/profile.d/ are executed in alphabetic order. One way or running a script last is creating something like z01_firstlogin.sh. The number is useful if you want to control the order at the end of multiple such scripts.

Here is the script that automatically generates the necessary keys in a user’s home directory. Place it in /etc/profile.d/z01_firstlogin.sh on the master node. Since all our users will login to the master, this script will be executed during the first session and generate the keys.

if [ ! -r ~/.ssh/id_rsa_internal ] ; then
  echo -n "Generating RSA keys..."
  ssh-keygen -b 4096 -t rsa -f ~/.ssh/id_rsa_internal -N '' -C "`hostname -s` cluster internal key" -q
  if [ $? -eq 0 ] ; then
      echo -ne "\t\tdone\n"
      cat ~/.ssh/id_rsa_internal.pub >> ~/.ssh/authorized_keys
      chmod go-rwx ~/.ssh -R
  else
      echo -ne "\t\tFAILED\n"
  fi
fi