Easy Puppet module development with PDK

By Saša Teković Sysbee Blog continuous-integration, pdk, puppet 0 Comments

If you’re using Puppet and you want to automate something other than basic OS configuration, package install or Unix user account management, Puppet Forge is the first place where you want to look. With over 6500 modules, chances are that you’ll find a module or two that automate the very thing you need. Sometimes, however, there won’t be an up-to-date or good enough module for the job, and you’ll end up writing your very own Puppet module. With the introduction of PDK (Puppet Development Kit) a couple of years back, module development has never been easier.

What is PDK?

PDK is a bundle of Puppet code development and testing tools. Essentially, PDK allows you to quickly set up the module directory structure, along with necessary metadata, CI/CD, Hiera, Rspec, and various other configuration files. Besides the initial module setup, PDK will also make your life easier when you need to create a new class, defined type, Puppet task, and perform various validations against your module.

Puppet modules 101

When you’re developing a Puppet module, you should keep a few things in mind. Firstly, a module should manage one thing and manage it well. In other words, if you need to manage multiple services (e.g. PostgreSQL and uWSGI that your application requires), you want to have multiple modules, one for each service.

Secondly, each module should be self-contained. Meaning, you want to steer clear of cross-module dependencies at all costs to avoid resource dependency problems or unexpected issues in one module caused by unrelated changes made in the dependent module.

Thirdly, you should stick with the basics. The main (init.pp) class should be the only parameterized class, and it should come with sane defaults. That way, the users can easily include the whole module and override desired parameters just via the main class.

The number of subclasses should be kept to a minimum. In most cases, besides the main class, you will need only three subclasses:

install – the class that handles the installation of software (the service you intend to manage along with its dependencies)
config – the class that handles the configuration of the managed service
service – the class that manages things related to the service’s running state

Getting started with PDK

Creating a new module using PDK is as simple as running the “pdk new module [module_name]” command and answering a couple of questions. E.g.

$ pdk new module sysbee-testmodule
pdk (INFO): Creating new module: sysbee-testmodule

We need to create the metadata.json file for this module, so we're going to ask you four questions.
If the question does not apply to this module, accept the default option shown after each question. You can modify any answers at any time by manually updating the metadata.json file.

[Q 1/4] If you have a Puppet Forge username, add it here.
We can use this to upload your module to the Forge when it's complete.
--> sysbee

[Q 2/4] Who wrote this module?
This is used to credit the module's author.
--> Sysbee

[Q 3/4] What license does this module code fall under?
This should be an identifier from https://spdx.org/licenses/. Common values are "Apache-2.0", "MIT", or "proprietary".
--> Apache-2.0

[Q 4/4] What operating systems does this module support?
Use the up and down keys to move between the choices, space to select and enter to continue.
--> RedHat based Linux, Debian based Linux

Metadata will be generated based on this information, continue? Yes
pdk (INFO): Using the default template-url and template-ref.
pdk (INFO): Module 'testmodule' generated at path '/home/user/testmodule'.
pdk (INFO): In your module directory, add classes with the 'pdk new class' command.

If you peek into the newly created “testmodule” directory, you should see that PDK has already created the necessary directory hierarchy, generated metadata.json, and a bunch of other documentation and configuration files.

To create the main class and “install” subclass, you will want to execute

$ pdk new class testmodule

---------------Files added--------------
/home/user/testmodule/spec/classes/testmodule_spec.rb
/home/user/testmodule/manifests/init.pp

and

$ pdk new class testmodule::install

---------------Files added--------------
/home/user/testmodule/spec/classes/install_spec.rb
/home/user/testmodule/manifests/install.pp

Besides classes, you can use PDK to generate defined_type, provider, task, test, and transport.

Validating Puppet modules

PDK comes with built-in module validation functionality, which utilizes several validators to check everything from Puppet code syntax and style to tasks, templates, metadata, etc.

If you validate previously created “testmodule”, you can see module validation in action, and there shouldn’t be any errors as the module is empty.

$ pdk validate
pdk (INFO): Using Ruby 2.5.8
pdk (INFO): Using Puppet 6.17.0
pdk (INFO): Running all available validators...
┌ [✔] Running metadata validators ...
├── [✔] Checking metadata syntax (metadata.json tasks/*.json).
└── [✔] Checking module metadata style (metadata.json).
┌ [✔] Running puppet validators ...
├── [✔] Checking Puppet manifest syntax (**/*.pp).
└── [✔] Checking Puppet manifest style (**/*.pp).
┌ [✔] Running ruby validators ...
└── [✔] Checking Ruby code style (**/**.rb).
┌ [✔] Running tasks validators ...
├── [✔] Checking task names (tasks/**/*).
└── [✔] Checking task metadata style (tasks/*.json).
┌ [✔] Running yaml validators ...
└── [✔] Checking YAML syntax (**/*.yaml **/*.yml).
info: puppet-epp: ./: Target does not contain any files to validate (["**/*.epp"]).
info: task-metadata-lint: ./: Target does not contain any files to validate (["tasks/*.json"]).

Ideally, you will want to validate the module before you commit changes to Git, Mercurial, or some other version control system.

Automated module validation

Here at Sysbee, we perform module validation whenever we merge branches in Git. That way, we are sure that no syntax and similar errors end up in the staging or production environment.

We have automated module validation with PDK using the GitLab CI/CD runner and Docker executor. As you can see, the GitLab CI/CD configuration is ridiculously small:

default:
  image:
    name: sysbee/puppet-pdk:latest

stages:
  - Test

pdk_validate:
  stage: Test
  tags:
    - docker
  script:
    - pdk validate --parallel

Versioning and documentation

The official Puppet documentation recommends versioning modules according to semantic versioning specification.

It’s a good practice to document all changes in a changelog file as well. When you create a new module, PDK conveniently generates a CHANGELOG.md template for you. To find out more about how to document changes, make sure to check out keepachangelog.com.

A good module must have equally good documentation. Documenting Puppet modules is satisfyingly easy thanks to Puppet Strings. Basically, Strings allow you to translate descriptive tags and comments for classes, defined types, functions, parameters, and other elements into properly formatted documentation. Strings supports multiple output formats – HTML, JSON, and Markdown.

Each module has to have a REFERENCE.md file in its version control repository. This is essentially reference documentation in Markdown format and generating one is as simple as running a single one command from your module’s directory:

$ puppet strings generate --format markdown

Bonus tips

To wrap things up, here are a few bonus tips that will help you boost your Puppet module development skills.

PDK convert

Do you already have a Puppet module and you want to make it PDK compatible? No problem, you can run pdk convert command from your module’s directory.

Parallel validation

The speed of module validation depends on the number of classes, functions, tasks, templates, etc. With small modules, the whole process takes roughly 20-30 seconds – enough to take a sip of tea or coffee.

If you run out of your beverage of choice, it’s not the end of the world. PDK supports parallel validation which can make the validation process almost twice as fast. Simply add –parallel switch next time you use the validate function.

$ pdk validate --parallel

PDK templates

By default, PDK distributes a lot of stuff when creating a new module. This might not be ideal, and in some cases, you will want to get rid of unnecessary configuration files or maybe add something specific to your development setup.

Thankfully, PDK allows you to use your own templates and you can reference them directly via URL to your Git repository. For example:

$ pdk new module sysbee-testmodule --template-url https://github.com/puppetlabs/pdk-templates

Code with style

Get to know Puppet language and Puppet Strings style guides. That way, you’ll minimize the amount of time spent fixing syntax and style guide errors post-module validation.

You can avoid making some mistakes by properly configuring your favourite text editor. Syntax highlighting, correct indentation, and automatic hash alignment, among other things go a long way. In case you use Vim as I do, you might want to check out vim-puppet plugin.

Make use of stdlib

Puppetlabs have created a standard library of resources for Puppet modules for a reason. Whenever you find yourself in a situation where you require some sort of data validation, transformation, calculation, or simply want to optimize the way of declaring resources, make sure to check out stdlib reference.

For example, stdlib is a lifesaver when you have to declare packages that might already exist in another module (as resources cannot be declared twice). By using ensure_packages function, you can bypass the duplicate declaration problem.

ensure_packages(
  ['python3-libs', 'python3-pip', 'python3-setuptools'],
  {'ensure' => 'installed'}
)

Make sure that you are using ensure_packages consistently in all modules where you reference the same packages. Declaring the same package using both package resource type and ensure_packages function will still produce an evaluation error.

What is your experience?

Have you published your module on Puppet Forge? What are the things you like and dislike most about Puppet? Share your Puppet success or fail story with us. Leave a comment, mention us on social media or get in touch !