Testing Ansible playbooks with Molecule

By Andrea Bjelogrlic Sysbee Blog Ansible playbooks, molecule, testing, testing framework 0 Comments

You wrote your playbook, congratulations. Unless you’re a brave hero that will check if the playbook works correctly in production you figure out it might be wise to check it in a testing environment first.
You spin up a VM on your computer or somewhere in the cloud, run your tests, destroy the VM, make changes to your playbook, spin up the VM again, run your tests, so, we will try to do testing Ansible playbooks with Molecule.

We can all agree that this is a pretty inefficient way of developing. You might keep the same VM and clean up after yourself every time you finish your tests, which is also annoying and prone to error.
Lucky for us, other people get annoyed by that as well and develop solutions to make all our lives easier.

In this post, we’re going to see how we can use Molecule to make the testing process simpler to manage and maybe automate it too.

What is Molecule?

Molecule is a testing framework that will help you in the development roles. You can use it with playbooks too, don’t worry. Molecule will provision the environment you need to run tests, based on the drivers you choose. The drivers can be Podman, Docker, Azure, Vagrant,…
In this post, we’re going to use the Podman driver to run tests locally and the default “delegated” driver to run it in a CI environment.
To get started you can run molecule init role testko this will create a new role named “testko” with the standard role structure and an extra directory named molecule.
By default it will use the delegated driver, you can choose another one by adding --driver-name [driver]

In the created molecule folder, you’ll find another one named “default”. This is the default scenario that molecule creates.
Inside you’ll find 3 files that you should care about:

default/
├── converge.yml
├── molecule.yml
└── verify.yml

molecule.yml contains the configuration for the scenario.
converge.yml is the playbook that will be executed during the test.
verify.yml will be used to do the verification after the test is done.

Using Molecule

The defaults

Let’s create a role with molecule molecule init role testko-role --driver-name podman

If we take a look into the molecule/default/molecule.yml file we will see the following

---
dependency:
  name: galaxy
driver:
  name: podman
platforms:
  - name: instance
    image: docker.io/pycontribs/centos:8
    pre_build_image: true
provisioner:
  name: ansible
verifier:
  name: ansible

This basically tells Molecule to use the Podman driver and use docker.io/pycontribs/centos:8 image.
You can specify your own image and parameter for Podman.
The Podman image will be provisioned with Ansible and will be verified with Ansible. You can use other verifiers such as Testinfra if you prefer it.
Here, we’re going to stick with the defaults.

The molecule/default/converge.yml file will be run during the testing phase

---
- name: Converge
  hosts: all
  tasks:
    - name: "Include testko-role"
      include_role:
        name: "testko-role"

For testing purposes, our role will just add a file on the remote machine in /file.txt location.

tasks/main.yml

---
# tasks file for testko-role

- name: "copy file in /file.txt"
  copy:
    src: file.txt
    dest: /file.txt

In the molecule/default/verify.yaml file we will check if the file exists.

---
- name: Verify
  hosts: all
  gather_facts: false
  tasks:
  - name: "stat /file.txt exists"
    stat:
      path: "/file.txt"
    register: file_txt_stat

  - name: "check if /file.txt exists"
    assert:
      that:
        - file_txt_stat.stat.exists == True
      success_msg: "/file.txt exists"
      fail_msg: "/file.txt doesn't exist"

Now that everything is setup, we can start our test, by simply running molecule test

The output will look like this

[prepration tasks ....]

INFO     Running default > converge
                                                                                                          
PLAY [Converge] ****************************************************************             
                                                                                                          
TASK [Gathering Facts] *********************************************************                                                                                                                                     ok: [instance]                                                                                                                                                                                                       

TASK [Include testko-role] *****************************************************
                                                                                                                                                                                                                     
TASK [testko-role : copy file in /file.txt] ************************************
changed: [instance]                                                                                       
                                                                                                          
PLAY RECAP *********************************************************************   
instance                   : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
                                                                                                                                                                                                                     
INFO     Running default > idempotence

PLAY [Converge] ****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [instance]

TASK [Include testko-role] *****************************************************

TASK [testko-role : copy file in /file.txt] ************************************
ok: [instance]

PLAY RECAP *********************************************************************
instance                   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

INFO     Idempotence completed successfully.
INFO     Running default > side_effect
WARNING  Skipping, side effect playbook not configured.
INFO     Running default > verify
INFO     Running Ansible Verifier

PLAY [Verify] ******************************************************************

TASK [stat /file.txt exists] ***************************************************
ok: [instance]

TASK [check if /file.txt exists] ***********************************************
ok: [instance] => {
    "changed": false,
    "msg": "/file.txt exists"
}

PLAY RECAP *********************************************************************
instance                   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

INFO     Verifier completed successfully.

[cleanup tasks....]

We can see that the converge playbook ran our role, and then it ran again. This is done to check if the role is idempotent.
If any change was made in the second run, our test would fail.
After the idempotency check, the verified playbook was executed and we see that our test was completed successfully.

Custom scenarios

When we initialized our role with Podman, the default scenario was used.
We may define multiple testing scenarios for a more complete test. First, we will modify our role to accept the destination as a variable:

---
# tasks file for testko-role

- name: "copy file in /file.txt"
  copy:
    src: file.txt
    dest: "{{ file_dest }}"

and define the default value in defautls/main.yml

---
# defaults file for testko-role
file_dest: /file.txt

Now that the role has been modified, we can create a new scenario named custom_location.
molecule init scenario custom_location --driver-name podman

You will notice that in the Molecule directory, you’ll have an extra directory named custom_location.
We will modify the custom_location/converge.yml

---
- name: Converge
  hosts: all
  tasks:
    - name: "Include testko-role"
      include_role:
        name: "testko-role"
      vars:
        - file_dest: "/file2.txt"

And reuse the same verify.yml file as before.

We can now run molecule test -s custom_location the -s flag allows us to choose a scenario.
Given that we reused the verify.yml from before, we expect the test to fail now, as the locations don’t match anymore.

PLAY [Verify] ******************************************************************

TASK [stat /file.txt exists] ***************************************************
ok: [instance]

TASK [check if /file.txt exists] ***********************************************
fatal: [instance]: FAILED! => {
    "assertion": "file_txt_stat.stat.exists == True",
    "changed": false,
    "evaluated_to": false,
    "msg": "/file.txt doesn't exist"
}

PLAY RECAP *********************************************************************
instance                   : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

If you have multiple scenarios, you can use molecule test --all to run them all.

One thing to note is that you can run only parts of the test. For example, if you wish to run only the converge part of the test you can with molecule converge [-s scenario_name ]

Molecule in a CI/CD environment

We use GitLab as our version control system ( and you should too 😉 ) , so everything you’ll see is GitLab specific.
I guess some of it is applicable to other testing platforms.

Some setup required

The official molecule documentation suggests using docker in docker to run molecule, but this doesn’t really fit our use-case.
The main reason is that almost all of our roles target CentOS 7 and Debian 11 platforms and we have to integrate Systemd into our tests. If you tried running Systemd in Docker, you’ll know it’s not as straightforward as it seems, running it in a Docker in Docker setup is even less simple.

However, Systemd in the container is a must to run a reliable test. After some work (not going to go into many details here, as this is a topic for a different blog post) we got Systemd working in both CentOS and Debian containers.
We can finally start with our tests.

What not to do

The first approach we tried was using the delegated driver for the Molecule test and running the test on localhost in the runner container. This attempt worked, kind of.
The problem here is the Systemd based image and the GitLab runner combination.
In our container image, we defined the first command to be executed as /bin/init. However, when the GitLab runner starts the container, this command gets overwritten with another one that’s needed to run the CI job.
This means that the init command won’t have PID 1 inside the container and any attempt to run Systemd services will fail (this took a while to figure out)

A working solution

GitLab offers a thing called services that we can use in out CI job. A service is just a container with a networking setup, that will be started when the job starts.
For example, if you need a MySQL database to run the tests, you can define a MySQL container as a service in you CI file and you’ll have a database to run your tests on.

This is great for our case, as we can run Molecule from one container and have them executed in another, just like the real thing.
It also solves the issue discussed in the previous paragraph, as the Docker image in the service runs just like we intended with Systemd as the first process.

So, what does the pipeline look like?

Here is a simplified version of the .gitlab.ci file we use to test the Redis install and configuration role in a Debian container.

---
stages:
  - test

.test-debian:
  image:
    name: molecule-ci:latest-debian
  services:
    - name: molecule-ci:latest-debian
      alias: debian-service
  tags:
    - docker
    - docker-runner

test-debian:
  extends: .test-debian
  stage: test
  before_script:
    - echo "$SSH_PRIVATE_KEY" > /root/.ssh/id_ecdsa
    - chmod 600 /root/.ssh/id_ecdsa
    - echo "Host *\n\tStrictHostKeyChecking no" >> ~/.ssh/config
  script:
    - ANSIBLE_ROLES_PATH=/builds/sysbee/ansible/roles/ molecule test --all

We’re running the test in the same custom image as our service container. This is an arbitrary choice, as long as your container can run ansible you should be fine.
A thing I would suggest is to alias the service you’re using. You can then use the alias name for your Molecule test.

To connect to our service container, we’re going to use SSH. For that reason, in the before script section of our job we’re importing the private key that matches the public key we have embedded in the service container.

Finally, we will testing Ansible playbooks with Molecule.

Test modifications

There was some modification needed for our test and finaly starting to do testing Ansible playbooks with Molecule, as the default Molecule configuration would not work.
First of all, we need to modify the molecule.yml file

-
dependency:
  name: galaxy
driver:
  name: delegated
platforms:
  - name: instance
provisioner:
  name: ansible
  inventory:
    links:
      hosts: inventory.ini
verifier:
  name: ansible

We have set the driver to delegated and added an inventory section that will be used by Ansible.
The inventory.ini file mentioned in the config, is your classic Ansible inventory file

[all]
debian-service

and lastly, we need to modify our playbooks to target the debian-service

---
- name: Converge
  hosts: debian-service
  roles:
    - role: redis

And that’s pretty much what’s needed to run the tests in GitLab CI.

Wrapping things up

Tests are an integral part of any development process, and with tools like molecule, they are easy to integrate into your workflow.
Whether you run your tests locally or in a CI environment, Molecule is here to make the whole process easier.
We started testing Ansible playbooks with Molecule recently and if you’ve been using it and don’t mind sharing your thoughts, feel free to drop us a comment.