How-to: SSH to Multiple Machines Simultaneously

http://techthrob.com/2015/06/how-to-ssh-to-multiple-machines-simultaneously

I often find myself needing to run the same command on many boxes at the same time.  For example, I’ll sometimes need to check the memory usage of a fleet of boxes.  Other times, I’ll want to quickly grep the logs of multiple machines, for troubleshooting purposes.  Once you grow beyond managing one or two servers, you’ll find yourself wasting a ton of time if you don’t have a way to run commands on multiple Linux computers simultaneously.  In this post, I’ll explain how to quickly run commands on large numbers of Linux machines at once.

Tool of Choice: dsh

First, we’ll need to install dsh, which is the tool we’ll be using to ssh to multiple machines at once. On Debian-based distributions (such as Ubuntu), run:

sudo apt-get install dsh

At the time of this writing, there is no officially-supported DSH RPM for RedHat-derived distributions (such as CentOS or Fedora), but a quick Google search for “dsh x86_64 rpm” reveals multiple third-party downloads.

Configuration: Two Easy Steps

Once installed, there are two additional configuration steps you’ll need to take before you can really put dsh to use.  The first is to setup your ssh configuration such that you can ssh to remote machines without a password; this is covered in detail in various posts on the web, and you can simply google for passwordless ssh key.  Presumably, if you have a many-machine environment, you’ve already got ssh keys setup.
Once that’s done, you need to build out your machine list files.  DSH works by reading in a file that contains a list of the machines to ssh to.  There are two primary classifications of files: the machines.list file which can contain a list of all machines you will ever use DSH to access, and the group lists, which allow you specify subsets of machines via including them in groups.
If you always plan to use DSH to access all machines simultaneously, you can get away with just specifying a single machines.list file in ~/.dsh/machines.list, of the format:

username@hostname1
username@hostname2
#this is a comment
username@hostname3

If, however, you have a large number of machines that you wish to classify into groups, then you’ll probably want to do so by using the same file format as above, but placing them in files located at ~/.dsh/group/$groupname. For example, you may have two files, one called ~/.dsh/group/webservers and another called ~/.dsh/group/dbservers. Note that the filename you use here will determine the name of the group as specified on the command line when you execute DSH.

Example Usage

Once your configuration files are setup, you’re ready to begin using DSH.  For this example, let’s assume you created a group named “webservers” with two machines (one being 192.168.1.134, and the other being 192.168.1.101), and you want to find the uptime of both of those machines.  You might run a command like:

jdeprizio@horizons:~$ dsh -r ssh -g webservers -M -c “uptime”
192.168.1.134: 21:12:29 up 69 days, 47 min, 0 users, load average: 0.21, 0.31, 0.49
192.168.1.101: 21:13:18 up 69 days, 47 min, 1 user, load average: 0.01, 0.01, 0.00

Here, we specify that we want to use ‘ssh’ as the remote shell, that we want to use the ‘webservers’ group, that we want to list the hostnames of each machine along with the output (the ‘-M’ flag), and that we want to execute the command on all machines concurrently (the ‘-c’ option).
Typically, the -c option is incredibly useful, especially when you’re running a command on a very large number of machines.  Without it, dsh will execute the commands serially.  For example, imagine you have a command that takes 5 seconds — running it with the -c option will take about 5 seconds, whereas without the -c option, it will take 5 seconds multiplied by the number of machines.  One drawback of the -c option, however, is that I’ve noticed output from one machine occasionally clobbering the output from another, which can make piping the output to grep troublesome.

How Do You Manage Your Fleet?

I’d love to hear how you manage your environments where you have a large number of machines running.  While things like puppet are great for configuration management, I find one-off commands to be a huge part of my day, and dsh fits the bill for what I need.  Is there a better, or different, tool that you use?  Leave your feedback in the comments below!