Objective

Create and use puppet environments to test new configuration before updating a live production system.

Operating System and Software Versions

  • Operating System: Any major linux distribution e.g. Ubuntu, Debian, CentOS
  • Software: puppet and puppet-master

Requirements

Privileged access to the puppet master server and the puppet client node.

Difficulty

MEDIUM

Conventions

  • # - requires given linux commands to be executed with root privileges either directly as a root user or by use of sudo command
  • $ - given linux commands to be executed as a regular non-privileged user

Introduction

Most Puppet installations begin life as a master server running a single branch. The master contains all the manifests and other configuration for all the Puppet agents that are synced to it. This is a fine place to begin but there will rapidly arrive a time when an update needs pushing that has the potential to break a production server. Hoping for the best is not the best way to proceed.

Puppet provides the tools to separate entire branches of configuration. These are called environments. A Puppet environment is a way to supply an isolated group of agent nodes with their own dedicated configuration. Each environment contains an entire Puppet configuration tree and can be considered as a separate Puppet master server.

How are Puppet environments used?

The typical scenario for environments, and is the one that we are exploring in this guide, is to create a testing environment, alongside the production environment, where new Puppet configuration is created.

One way to test the new configuration in the testing environment is by updating a copy of a production server, such as a VM snapshot. Any problems will be observed on the test machine and the Puppet configuration modified to correct this. However, it is not always possible to have a test server to check the changes in the testing environment.

Another method and the one we will explore here is to run the Puppet agent manually on the production server but use several options that will cause the Puppet agent to sync to the testing environment but only show what would have happened without making any actual changes. This will highlight any errors that would have occurred in a full update without actually causing any downtime.

Creating Puppet environments

In this guide, we will create a very simple Puppet instance with a Puppet Master and a Puppet agent node. The Puppet master server will be configured to have two environments; testing and development.

This guide assumes that you have a Puppet master server and a Puppet agent node that is able to connect to the Puppet master.

We are going to create two environments on the Puppet master and within these environments we will create a very simple Puppet manifest that creates a text file on the agent node.

The default location for Puppet’s configuration changes depending on which distribution you are using. On Ubuntu 18.04LTS, the version that will be used in this guide, the location is at /etc/puppet. Other distributions (and the official documentation) may place it at /etc/puppetlabs/. However, once you are in the main Puppet configuration directory all sub-directories are the same for all distributions.

Instructions

Create the environment directories

The environments and their configuration all exist under the /etc/puppet/code/ directory. On Ubuntu 18.04 this directory is empty on install so we will need to first create the two top-level environment directories with the following two commands:
# mkdir -p /etc/puppet/code/environments/testing
# mkdir -p /etc/puppet/code/environments/development
Any new agent node will automatically connect to the development environment unless the environment variable is set to an alternative in the [agent] section of the puppet.conf file on the agent node.


$$$ Looking for LINUX ADMINISTRATOR ! $$$

BLUE SKY STUDIOS are looking for Linux Administrator to maintain and support the Studio's 450+ production Linux workstations, including daily interactions with the Studio’s digital animation artists.
LOCATION: Greenwich, Connecticut, USA

APPLY NOW

Creating Two Simple site.pp Manifests

The site.pp file is the primary manifest from where the Puppet agent begins building a catalog of the desired machine state. We are going to create two very simple site.pp files in the two environments that create the same file on the agent node. The only difference is that they put different text into the file.

The first site.pp file will be the production environment at: /etc/puppet/code/environments/development/manifests/site.pp This file should have following contents:

file {'/tmp/example.txt':
 ensure  => present,
 mode    => "0644",
 content => "From The Development Environment \n",
}
Use your favorite text editor to create and populate this file.

This manifest ensures that a file is present at /tmp/example.txt and contains the text "From The Development Environment" (the "\n" adds a new line at the end of the file which is good practice and stops Puppet showing a warning message when it is not present).

The second manifest will be under the testing environment at: /etc/puppet/code/environments/testing/manifests/site.pp This file contains the the following:

file {'/tmp/example.txt':
 ensure  => present,
 mode    => "0644",
 content => "From The Testing Environment \n",
}
This is almost identical to the file in the development environment with the only difference being that the text in the file indicates that it has come from the testing environment.

Evaluating New Puppet Configuration From The Testing Environment

The agent node will by default only sync to the development environment. We will first manually instruct the Puppet agent to sync with the Puppet master server and create and apply the site.pp that we created in the development environment.

This is done with the following command:
# puppet agent --environment=production --test
The --test option makes the Puppet agent perform a catalog run in the foreground with verbose logging. Any updates or changes will get applied to the node.

The --environment=production option is there to make it clear that we are syncing from the production environment. Usually, this would be configured in the main Puppet agent configuration and would not need to be included in the command.

When the above command is run we get the following output:
  Info: Using configured environment 'production'
  Info: Retrieving pluginfacts
  Info: Retrieving plugin
  Info: Retrieving locales
  Info: Loading facts
  Info: Caching catalog for digital-2.net
  Info: Applying configuration version '1527680694'
  Notice: /Stage[main]/Main/File[/tmp/example.txt]/ensure: defined content as '{md5}59f9ce1d4aad5fd155db7ccc2478a93b'
  Notice: Applied catalog in 0.02 seconds
This output indicates that file /tmp/example.txt was not present so the Puppet agent created it as instructed in the site.pp manifest. Subsequent runs will not have the Notice: lines as the /tmp/example.txt file exists with the correct contents.

Now that the agent node’s state agrees with the development environment’s manifest we can test what would happen if we applied the alternative manifest from the testing environment.

In order to test and not commit the new configuration we need to run the following command:
# puppet agent --environment=testing --test --noop
As you can see the --environment option has been changed to testing and we have included the additional option --noop. This option makes the agent perform a dry-run. This means that the Puppet agent will not make any actual changes to the agent node but will produce all the output as if it had.

This allows us to evaluate what would have happened if the new configuration was applied to the server. In this case the output of the above command looks like:
  Info: Using configured environment 'testing'
  Info: Retrieving pluginfacts
  Info: Retrieving plugin
  Info: Retrieving locales
  Info: Loading facts
  Info: Applying configuration version '1527683748'
  Notice: /Stage[main]/Main/File[/tmp/example.txt]/content:
  --- /tmp/example.txt    2018-05-30 12:19:16.205774048 +0000
  +++ /tmp/puppet-file20180530-21610-8ipzur      2018-05-30 12:35:48.740982652 +0000
  @@ -1 +1 @@
  -From The Development Environment
  +From The Testing Environment

  Notice: /Stage[main]/Main/File[/tmp/example.txt]/content: current_value '{md5}59f9ce1d4aad5fd155db7ccc2478a93b', should be '{md5}abbb8f68df144a5673d
  62ae6c4a036ed' (noop)
  Notice: Class[Main]: Would have triggered 'refresh' from 1 event
  Notice: Stage[main]: Would have triggered 'refresh' from 1 event
  Notice: Applied catalog in 0.04 seconds
The most interesting lines here are the following:
  -From The Development Environment
  +From The Testing Environment
These indicate with the minus symbol ( - ) what is being changed from and with the plus symbol ( + ) what is being changed to. In this example it is the text in the file.

All of this output indicates that the new configuration would have been successfully applied and the contents of /tmp/example.txt would have been modified. If this is the desired state of the production server then the changes to the site.pp file can be safely made in the production environment.



Identifying An Error

New Puppet configuration does not always get applied without error and that is the reason that it should always be tested before being applied to a production system. We will force an error in this situation by making a deliberate mistake in the testing site.pp file. We will try to set the permissions of the file to 0944 which is not a valid permission and will cause an error.

Now, when we run:
  # puppet agent --environment=testing --test --noop
We will see the following output:
  Info: Using configured environment 'testing'
  Info: Retrieving pluginfacts
  Info: Retrieving plugin
  Info: Retrieving locales
  Info: Loading facts
  Error: Failed to apply catalog: Parameter mode failed on File[/tmp/example.txt]: The file mode specification is invalid: "0944" (file: /etc/puppetcode/environments/testing/manifests/site.pp, line: 1)
The following screen capture shows this output as it would be presented on the command line:

Image showing a Puppet sync error message
Puppet will indicate any errors by printing them in red.


The colors immediately let us know that there would have been an error in attempting to use the new Puppet configuration from the testing environment. However, as we used the --noop option no errors were committed to the production server.

Conclusion

When running production systems that are managed by Puppet it is always important to test any new configuration before it gets applied. Using the tools Puppet provides to create alternative environments where new configuration can be safely created and evaluated against production systems will mean fewer errors and less downtime.
ARE YOU LOOKING FOR A LINUX JOB?
Submit your RESUME, create a JOB ALERT or subscribe to RSS feed on LinuxCareers.com.
LINUX CAREER NEWSLETTER
Subscribe to NEWSLETTER and receive latest news, jobs, career advice and tutorials.
DO YOU NEED ADDITIONAL HELP?
Get extra help by visiting our LINUX FORUM or simply use comments below.

You may also be interested in:



Comments and Discussions