Multiserver administration with Puppet

http://www.openlogic.com/wazi/bid/254233/Multiserver-administration-with-Puppet


Puppet software lets you manage the software configuration on all the hosts on your network. It simplifies repetitive operations and ensures state and configuration integrity among large number of machines. A centralized, single master server pushes configuration changes and commands to slaves nodes, with both sides relying on SSL certificates for security.
Puppet comes in both Enterprise and Apache-licensed open source versions; we worked with the latter. Its powerful object-oriented configuration language supports inheritance and code reuse. Puppet can run on and manage a wide range of operating systems – Linux, FreeBSD, Solaris, and even Windows (partially) – so it's suitable for heterogeneous server environments running different distributions of Linux or even different operating systems.
With Puppet, you specify a manifest that states the configuration you want on your systems, and Puppet finds the best way to reach this state. For instance, if you want to install the httpd package on a particular server, you don't tell Puppet to run the yum command; instead, you specify in a Puppet configuration file that the httpd package is required and let Puppet figure out the best way to install it.

How to install Puppet

If you want to try Puppet, the best way to install it is through its official repositories. Many Linux distributions are supported; for CentOS you can use the repository for Red Hat Linux and its derivatives.
To install the current version of Puppet repository on CentOS 6, run rpm -ivh http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-6.noarch.rpm. Once you have the Puppet repo, run the command yum -y install puppet on the Puppet slave nodes and yum -y install puppet puppet-server on the Puppet master server. Those commands install not only Puppet but all its dependencies as well, including the Ruby programming language, on which Puppet is based. Understanding Ruby may help you design advanced configurations with ERB templates, but Ruby is just the underlying basis and you don't need to know it in order to work with Puppet.
One noteworthy dependency package is Facter, a standalone cross-platform tool that gathers information such as host operating system, distribution, architecture, and network interfaces. The information from Facter is available both to the local node and to the master. Based on this information Puppet decides how to apply your specified state to each local machine.
To make sure puppet starts and stops automatically with your CentOS system, run the commands chkconfig puppet on on the slave nodes, and on the master server run chkconfig puppet on && checkconfig puppetmaster on.
If you use iptables for your firewall, make sure to allow the slave nodes to connect to the master. The master Puppet service listens by default on TCP port 8140. To allow it in iptables on CentOS, run these commands:

iptables -I INPUT -p TCP --dport 8140 -j ACCEPT
service iptables save

Initial Puppet setup

Puppet requires a few minor configuration adjustments before you can start using it. First, ensure that the slave nodes can connect to the master. By default, the slaves look for the master host at the fully qualified domain name "puppet." You can specify a different FQDN for the master by placing the directive server=somehost.example.org inside the [main] configuration block in the file /etc/puppet/puppet.conf on each node. Alternatively, you can simply hardcode the default 'puppet' address in the /etc/hosts file on each node by adding a line such as 192.168.1.200 puppet to each instance of that file.
The SSL certificate on the master must correspond to the FQDN to which the nodes connect. If you leave the default "puppet" FQDN for the master, add two lines to the file /etc/puppet/puppet.conf. The section heading [master] indicates that a directive for the master follows. The directive itself is certname=puppet.
Now the nodes should know where to find the master and be able to establish a valid secure connection, so you can manually start the Puppet services for the first time. On the master run service puppetmaster start && service puppet start. On the nodes run service puppet start.
When the Puppet services start on the nodes they try to connect to the master, but the master does not allow them to connect automatically. You have to authorize the connections and establish a trust relationship by signing the nodes' certificates.
To see the current signing requests on the master, run the command puppet cert --list. This should produce output similar to:

  "server2" (SHA256) 82:A8:FA:BB:CE:0D:D5:0A:DB:7A:3E:8D:A5:62:5B:AC:91:7D:9C:65:51:5F:80:50:F7:DB:ED:36:87:EC:B4:C0
  "server3" (SHA256) 7C:5C:05:58:CC:5A:1C:D7:7C:98:CC:C4:34:17:D5:35:1C:11:E8:DC:04:92:42:1C:8E:58:36:EA:5C:11:03:9B

To sign a node's certificate, run the command puppet cert --sign followed by the node's name: puppet cert --sign server2, for instance. Make sure that the hosts' names can be resolved from the master to avoid problems. The easiest way to do that is to add static records in the hosts file.
On the master, to verify certificates have been signed, use the command puppet cert --list --all. In the output, a plus sign in front of a certificate name shows that it has been signed: + "server1" (SHA256), for instance.
On the nodes, to verify they are able to connect to the master, check the file /var/log/messages, to which Puppet sends its logs by default. When you've created a successful relationship with the master, you should see an entry like:

Dec 11 12:03:20 server2 puppet-agent[2429]: Finished catalog run in 0.14 seconds

This means that the node can connect to the master and download and run the catalog. What's a catalog? Glad you asked!

Puppet manifests and catalogs

Puppet's configuration files are called manifests. They contain the instructions Puppet uses to bring nodes to a desired state. Manifests are compiled into a catalog. The compiling process resolves dependencies and correctly reorders the instructions. For example, suppose you want to install the httpd package and put a requirement that PHP package be present. In this case Puppet checks and installs PHP first if necessary, then proceeds with the httpd package installation.
When a node connects to the master it downloads the catalog and runs it locally. Running the catalog means checking to ensure the current node state corresponds to the state configured on the master. By default, Puppet slaves connect each 30 minutes to the master to synchronize. You can follow Puppet's activity in the /var/log/messages file.
Manifests are stored on the master in the directory /etc/puppet/manifests/ with, by default, an extension of .pp. When you want to include a file called something.pp you can drop the extension; Puppet automatically appends .pp when looking for a file to import.
The file site.pp is the main manifest that Puppet loads by default. To get started with a simple example, create it with the following contents:

node 'server2' {
    }
node 'server3' {
    include postfix
    }
import "postfix"

The node directive takes a node's FQDN and specifies the configuration for each node. And the import directive? Just as in a programming language, to make the configuration more readable and reusable you should separate atomic pieces of configuration (called Puppet classes) in separate files, and import them using the import directive. Here we import the contents of the file postfix.pp, which is located in /etc/puppet/manifests/, the same directory as site.pp. The postfix class might look like this:

class postfix {
    package { "postfix":
        ensure => installed,
    }
    service { "postfix":
       ensure => running,
       enable => true,
    }
}

First, the postfix package should be installed (ensure => installed). You don't have to tell Puppet what command to run; it automatically finds and uses the correct package manager based on the details provided by Facter.
The above manifest also shows that the postfix service should be started (ensure => running) and added to the default runlevels (enable => true), which means it is to be started and stopped with the system.

Once you have the above class you can include it in the definition for a node, as we've done with node server3, to ensure that the node complies with the manifest for postfix.
One last note on manifests: When you make changes in the file site.pp, Puppet automatically detects and enforces them by reloading its configuration. If you make changes to files imported into site.pp, you have to update the timestamp of site.pp with the command touch site.pp to make Puppet recognize the changes.
Through its manifests Puppet can edit text files, serve static files, perform installations, and adapt configurations on heterogeneous systems. You can learn more about manifests from the official documentation.

Reporting

Puppet supports reporting for its catalog runs. When configured, the nodes can send reports to the master about changes and operations that have been executed successfully, and those that have not. To enable reporting you have to configure the nodes to send reports and the master to accept them. On the nodes, edit the file /etc/puppet/puppet.conf and add in the [agent] section a new row containing report = true. On the server, edit /etc/puppet/puppet.conf, but this time go to the [master] section and add two rows. The first, reportdir = /etc/puppet/reports, specifies the directory in which the reports will be stored. The second, reports = store, defines the default action for the received logs.
After restarting Puppet both on the nodes and the master you should begin seeing reports in the directory /etc/puppet/reports. Each node creates its own directory there named after its hostname.
Puppet reports, which come in yaml format, are detailed and verbose. Each report starts with the file name of the applied manifest; for example, file: /etc/puppet/manifests/postfix.pp. A line with a message tag describes what has happened: message: "ensure changed 'stopped' to 'running'". This message means that Puppet has started the service in accordance to the instructions from the manifest. The last line specifies the time of the event, such as time: 2012-12-16 07:00:57.326020 +02:00.
Puppet reports provide much more information than in the simple example above. You won't want to analyze them manually; tools such as Puppet's dashboard can help you visualize the reports.
This article provides just a bare introduction to Puppet's many features. Puppet is very powerful but requires some knowledge to operate it well. Luckily, plenty of educational resources can help teach you Puppet; start by exploring the official Puppet documentation.