Category Archives: Configuration Management

Thoughts on Ansible variables

If you want to use Ansible to really empower your configuration management function, its important to have a solid understanding of how variables work.

Here’s a few must-knows:

Values in ansible.cfg are environment variables, not script variables

The ansible.cfg file is provided to allow the user set default values that are used when ansible is executed from a local environment. This file isn’t a YAML file, which is why assignments use “=” rather than “:”.

The values in this file are set as environment variables when Ansible runs. You cannot access these values directly as script variables eg

remote_user = root

does not provide you with a

{{ remote_user }}

variable in your playbooks.

Its important to create your variables in the right place: inventory or play

Generally, a variable will apply to either a host (or group of hosts), or to a task (play) within a playbook. Decide early where your variable applies and create it in the right place.

For variables that apply to hosts (eg a username to login with) create the variable in either:

Your inventory file:


server1 ansible_ssh_user=admin

Under your group_vars directory:

#file: ./groups_vars/server_group_1


Under your host_vars directory:

#file: ./host_vars/server1


You can also create host-related variables deeper in your playbook:

- hosts: webservers
  remote_user: admin

but I don’t recommend this. Ansible provides sufficient functionality to create an abstraction layer for variables above the play/task level, and it makes sense to use it.

For variables that are specific to plays, the value can be set closer to the point of execution, for example:

After the hosts specification:

- hosts: webservers
     app_version: 12.03

As a parameter for the role that is being applied to the hosts:

- hosts: webservers
    - { role: app, app_version: 12.03 }

Variables in Ansible have precedence rules

Particular care needs to be paid to precedence. In instances, you may want a variable to have an absolute value which cannot be changed by an assignment in any other part of the playbook or from the command line. In other instances you may wish to allow a variable to be changed. These behaviours are controlled by where you create the assignment of the variable.


Installing Passenger for Puppet on Amazon Linux


Puppet ships with a web server called Web Brick. This is fine for test and use with a small number of nodes, but will cause problems with larger fleets of nodes. It is recommended to use the Ruby application server, Passenger, to run Puppet in production environments.


Provision a new server instance.

Install required RPMs. Use Ruby 1.8 rather than Ruby 2.0. Both are shipped with the Amazon Linux AMI at the time of writing, but you need to set up the server to use version 1.8 by default.

sudo yum install -y ruby18 httpd httpd-devel mod_ssl ruby18-devel rubygems18 gcc mlocate
sudo yum install -y gcc-c++ libcurl-devel openssl-devel zlib-devel git

Make Ruby 1.8 the default

sudo alternatives --set ruby /usr/bin/ruby1.8

Set Apache to start at boot

sudo chkconfig httpd on

Install Passenger gem

sudo gem install rack passenger

Update the location DB (you will need this to find files later)

sudo updatedb

Find the path to the installer and add this to the path

locate passenger-install-apache2-module
sudo vi /etc/profile.d/
export PATH=$PATH:/usr/lib/ruby/gems/1.8/gems/passenger-5.0.10/bin/
sudo chmod 755 /etc/profile.d/

Make some Linux swap space (the installer will fail on smaller instances if this doesn’t exist)

sudo dd if=/dev/zero of=/swap bs=1M count=1024
sudo mkswap /swap
sudo chmod 0600 /swap
sudo swapon /swap

At this point, open a separate shell to the server (you should have 2 shells). This isn’t absolutely essential, but the installer will ask you to update an Apache file mid-flow, so if you want to do things to the letter of the law, a second shell helps.

Next, run the installer, and accept the default options.

sudo /usr/lib/ruby/gems/1.8/gems/passenger-5.0.10/bin/passenger-install-apache2-module

The installer will ask you to add some Apache configuration before it completes. Do this in your second shell. Add the config to a file called /etc/httpd/conf.d/puppet.conf. You can ignore warning about the PATH.

<IfModule mod_passenger.c>
  PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-5.0.10
  PassengerDefaultRuby /usr/bin/ruby1.8

Restart Apache after you add this and then press Enter to complete the installation

Next, make the necessary directories for the Ruby application

sudo mkdir -p /usr/share/puppet/rack/puppetmasterd
sudo mkdir /usr/share/puppet/rack/puppetmasterd/public /usr/share/puppet/rack/puppetmasterd/tmp

Copy the application config file to the application directory and set the correct permissions

sudo cp /usr/share/puppet/ext/rack/files/ /usr/share/puppet/rack/puppetmasterd/
sudo chown puppet:puppet /usr/share/puppet/rack/puppetmasterd/

Add the necessary SSL config for the ruby application to Apache. You can append this to the existing puppet.conf file you created earlier. Note that you need to update this file to specify the correct file names and paths for your Puppet certs (puppet.pem in the example below).The entire file should now look like below:

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-5.0.10/buildout/apache2/
<IfModule mod_passenger.c>
  PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-5.0.10
  PassengerDefaultRuby /usr/bin/ruby1.8
# And the passenger performance tuning settings:
# Set this to about 1.5 times the number of CPU cores in your master:
PassengerMaxPoolSize 12
# Recycle master processes after they service 1000 requests
PassengerMaxRequests 1000
# Stop processes if they sit idle for 10 minutes
PassengerPoolIdleTime 600
Listen 8140
<VirtualHost *:8140>
    # Make Apache hand off HTTP requests to Puppet earlier, at the cost of
    # interfering with mod_proxy, mod_rewrite, etc. See note below.
    PassengerHighPerformance On
    SSLEngine On
    # Only allow high security cryptography. Alter if needed for compatibility.
    SSLProtocol ALL -SSLv2 -SSLv3
    SSLHonorCipherOrder     on
    SSLCertificateFile      /var/lib/puppet/ssl/certs/puppet.pem
    SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/puppet.pem
    SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
    #SSLCARevocationCheck   chain
    SSLVerifyClient         optional
    SSLVerifyDepth          1
    SSLOptions              +StdEnvVars +ExportCertData
    # Apache 2.4 introduces the SSLCARevocationCheck directive and sets it to none
    # which effectively disables CRL checking. If you are using Apache 2.4+ you must
    # specify 'SSLCARevocationCheck chain' to actually use the CRL.
    # These request headers are used to pass the client certificate
    # authentication information on to the puppet master process
    RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
    DocumentRoot /usr/share/puppet/rack/puppetmasterd/public
    <Directory /usr/share/puppet/rack/puppetmasterd/>
      Options None
      AllowOverride None
      # Apply the right behavior depending on Apache version.
      <IfVersion < 2.4>
        Order allow,deny
        Allow from all
      <IfVersion >= 2.4>
        Require all granted
    ErrorLog /var/log/httpd/puppet-server.example.com_ssl_error.log
    CustomLog /var/log/httpd/puppet-server.example.com_ssl_access.log combined

The ruby application is now ready. Install the puppet master application. Note, do NOT start the puppetmaster service or set it to start at boot.

sudo yum install -y puppet-server

Restart Apache and test using a new puppet agent. You can also import the the ssl assets from an existing puppet master into /var/lib/puppet/ssl. This will allow you existing puppet agents to continue to work.

Allowing puppet agents manage their own certificates


Why would you want to allow a puppet agent manage the certificates the puppet master holds for that agent? Doesn’t that defeat the whole purpose of certificate based authentication in puppet?

Well, yes, it does, but there are situations in which this is useful, but only where security in not a concern!!

Enter Cloud Computing.

Servers in Cloud Computing environments are like fruit flies. There are millions of them all over the world being born and dying at any given time. In a an advanced Cloud configuration they can have lifespans of hours, if not minutes.

As puppet generally relies on fully qualified domain names to match agent requests to stored certificates, this can become a bit of a problem, as server instances that come and go in something like Amazon AWS can sometimes be required to have the same hostname at each launch.

Imagine the following scenario:

You are running automated performance testing, in which you want to test the amount of time if takes to re-stage an instance with a specific hostname and run some tests against it. Your script both launches the instance an expects the instance to contact a puppet master to obtain its application.

In this case, the first time the instance launches, the puppet agent will generate a client certificate signing request, send that to the master, get it signed and pull the necessary catalog. The puppet master will then have certificate for that agent.

Now, you terminate the instance and re-launch it. The agent presents another signing request, with the same hostname, but this time the puppet master refuses to play, telling you that it already has a certificate for that hostname, and the one you are presenting doesn’t match.

You’re snookered.

Or so you think. The puppet master has a REST api that is disabled by default but when you can open up to it receive HTTP requests to manage certificates. To enable the necessary feature, add the following to your auth.conf file

path /certificate_status
auth any
method find, save, destroy
allow *

Restart the puppet master when you’ve done this.

sudo service puppetmaster restart

Next, when you start you server instance, include the following script at boot. It doesn’t actually matter when this is run, provided it is run after the hostname of the instance has been set.


curl -k -X PUT -H "Content-Type: text/pson" --data '{"desired_state":"revoked"}' https://puppet:8140/production/certificate_status/$HOSTNAME

curl -k -X DELETE -H "Accept: pson"  https://puppet:8140/production/certificate_status/$HOSTNAME

rm -Rf /var/lib/puppet/ssl/*

puppet agent -t

This will revoke and delete the agent certificate on the master, delete the agent’s copy of the certificate and renew the signing process, giving you new certs on the agent and master and allowing the catalog to be ingested into the agent.

You can also pass a script like this as part of the Amazon EC2 process of launching an instance.

aws ec2 run-instances  --user-data file://./

Where is the name of the locally saved script file, and it is saved in the same directory as your working directory (otherwise include the absolute path).

With this in place, each time you launch a new instance, regardless of its hostname, it will revoke any existing cert that has the same hostname, and generate a new one.

Obviously, if you are launching hundreds of instances at the same time, you may have concurrency issues, and some other solution will be required.

Again, this is only a solution for environments where security is not an issue.