Objective
Create and use puppet environments to test new configuration before updating a live production system.
Operating System and Software Versions
- Operating System: Any major linux distribution e.g. Ubuntu, Debian, CentOS
- Software: puppet and puppet-master
Requirements
Privileged access to the puppet master server and the puppet client node.
Conventions
- # – requires given linux commands to be executed with root privileges either directly as a root user or by use of
sudocommand - $ – given linux commands to be executed as a regular non-privileged user
Introduction
Most Puppet installations begin life as a master server running a single branch. The master contains all the manifests and other configuration for all the Puppet agents that are synced to it. This is a fine place to begin but there will rapidly arrive a time when an update needs pushing that has the potential to break a production server. Hoping for the best is not the best way to proceed.
Puppet provides the tools to separate entire branches of configuration. These are called environments. A Puppet environment is a way to supply an isolated group of agent nodes with their own dedicated configuration. Each environment contains an entire Puppet configuration tree and can be considered as a separate Puppet master server.
How are Puppet environments used?
The typical scenario for environments, and is the one that we are exploring in this guide, is to create a testing environment, alongside the production environment, where new Puppet configuration is created.
One way to test the new configuration in the testing environment is by updating a copy of a production server, such as a VM snapshot. Any problems will be observed on the test machine and the Puppet configuration modified to correct this. However, it is not always possible to have a test server to check the changes in the testing environment.
Another method and the one we will explore here is to run the Puppet agent manually on the production server but use several options that will cause the Puppet agent to sync to the testing environment but only show what would have happened without making any actual changes. This will highlight any errors that would have occurred in a full update without actually causing any downtime.
Creating Puppet environments
In this guide, we will create a very simple Puppet instance with a Puppet Master and a Puppet agent node. The Puppet master server will be configured to have two environments; testing and development.
This guide assumes that you have a Puppet master server and a Puppet agent node that is able to connect to the Puppet master.
We are going to create two environments on the Puppet master and within these environments we will create a very simple Puppet manifest that creates a text file on the agent node.
The default location for Puppet’s configuration changes depending on which distribution you are using. On Ubuntu 18.04LTS, the version that will be used in this guide, the location is at /etc/puppet. Other distributions (and the official documentation) may place it at /etc/puppetlabs/. However, once you are in the main Puppet configuration directory all sub-directories are the same for all distributions.
Instructions
Create the environment directories
The environments and their configuration all exist under the /etc/puppet/code/ directory. On Ubuntu 18.04 this directory is empty on install so we will need to first create the two top-level environment directories with the following two commands:
# mkdir -p /etc/puppet/code/environments/testing # mkdir -p /etc/puppet/code/environments/development
Any new agent node will automatically connect to the development environment unless the environment variable is set to an alternative in the [agent] section of the puppet.conf file on the agent node.
Creating Two Simple site.pp Manifests
The site.pp file is the primary manifest from where the Puppet agent begins building a catalog of the desired machine state. We are going to create two very simple site.pp files in the two environments that create the same file on the agent node. The only difference is that they put different text into the file.
The first site.pp file will be the production environment at:
/etc/puppet/code/environments/development/manifests/site.pp
This file should have following contents:
file {'/tmp/example.txt':
ensure => present,
mode => "0644",
content => "From The Development Environment \n",
}
Use your favorite text editor to create and populate this file.
This manifest ensures that a file is present at /tmp/example.txt and contains the text “From The Development Environment” (the “\n” adds a new line at the end of the file which is good practice and stops Puppet showing a warning message when it is not present).
The second manifest will be under the testing environment at:
/etc/puppet/code/environments/testing/manifests/site.pp
This file contains the the following:
file {'/tmp/example.txt':
ensure => present,
mode => "0644",
content => "From The Testing Environment \n",
}
This is almost identical to the file in the development environment with the only difference being that the text in the file indicates that it has come from the testing environment.
Evaluating New Puppet Configuration From The Testing Environment
The agent node will by default only sync to the development environment. We will first manually instruct the Puppet agent to sync with the Puppet master server and create and apply the site.pp that we created in the development environment.
This is done with the following command:
# puppet agent --environment=production --test
The --test option makes the Puppet agent perform a catalog run in the foreground with verbose logging. Any updates or changes will get applied to the node.
The --environment=production option is there to make it clear that we are syncing from the production environment. Usually, this would be configured in the main Puppet agent configuration and would not need to be included in the command.
When the above command is run we get the following output:
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for digital-2.net
Info: Applying configuration version '1527680694'
Notice: /Stage[main]/Main/File[/tmp/example.txt]/ensure: defined content as '{md5}59f9ce1d4aad5fd155db7ccc2478a93b'
Notice: Applied catalog in 0.02 seconds
This output indicates that file /tmp/example.txt was not present so the Puppet agent created it as instructed in the site.pp manifest. Subsequent runs will not have the Notice: lines as the /tmp/example.txt file exists with the correct contents.
Now that the agent node’s state agrees with the development environment’s manifest we can test what would happen if we applied the alternative manifest from the testing environment.
In order to test and not commit the new configuration we need to run the following command:
# puppet agent --environment=testing --test --noop
As you can see the --environment option has been changed to testing and we have included the additional option --noop. This option makes the agent perform a dry-run. This means that the Puppet agent will not make any actual changes to the agent node but will produce all the output as if it had.
This allows us to evaluate what would have happened if the new configuration was applied to the server. In this case the output of the above command looks like:
Info: Using configured environment 'testing'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Applying configuration version '1527683748'
Notice: /Stage[main]/Main/File[/tmp/example.txt]/content:
--- /tmp/example.txt 2018-05-30 12:19:16.205774048 +0000
+++ /tmp/puppet-file20180530-21610-8ipzur 2018-05-30 12:35:48.740982652 +0000
@@ -1 +1 @@
-From The Development Environment
+From The Testing Environment
Notice: /Stage[main]/Main/File[/tmp/example.txt]/content: current_value '{md5}59f9ce1d4aad5fd155db7ccc2478a93b', should be '{md5}abbb8f68df144a5673d
62ae6c4a036ed' (noop)
Notice: Class[Main]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 0.04 seconds
The most interesting lines here are the following:
-From The Development Environment +From The Testing Environment
These indicate with the minus symbol ( - ) what is being changed from and with the plus symbol ( + ) what is being changed to. In this example it is the text in the file.
All of this output indicates that the new configuration would have been successfully applied and the contents of /tmp/example.txt would have been modified. If this is the desired state of the production server then the changes to the site.pp file can be safely made in the production environment.
Identifying An Error
New Puppet configuration does not always get applied without error and that is the reason that it should always be tested before being applied to a production system. We will force an error in this situation by making a deliberate mistake in the testing site.pp file. We will try to set the permissions of the file to 0944 which is not a valid permission and will cause an error.
Now, when we run:
# puppet agent --environment=testing --test --noop
We will see the following output:
Info: Using configured environment 'testing' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Error: Failed to apply catalog: Parameter mode failed on File[/tmp/example.txt]: The file mode specification is invalid: "0944" (file: /etc/puppetcode/environments/testing/manifests/site.pp, line: 1)
The following screen capture shows this output as it would be presented on the command line:
The colors immediately let us know that there would have been an error in attempting to use the new Puppet configuration from the testing environment. However, as we used the --noop option no errors were committed to the production server.
Conclusion
When running production systems that are managed by Puppet it is always important to test any new configuration before it gets applied. Using the tools Puppet provides to create alternative environments where new configuration can be safely created and evaluated against production systems will mean fewer errors and less downtime.
