Terraform codifies cloud, virtualization and many other APIs into declarative configuration files, allowing to define the infrastructure as code.

The combination of both is an excellent platform to manage resources in the Microsoft Azure cloud in an automated or semi-automated way.

Installation

Let’s start by installing Jenkins and Terraform. The official documentation at https://www.jenkins.io/doc/book/installing/linux/#debianubuntu describes how to install the Jenkins binaries on a Debian/ Ubuntu system, so we’ll not go further into details on that.

Once the required packages are installed as described above, go to http://localhost:8080/ to finalize the Jenkins setup. After installing the essential plugins required to run Jenkins, you should be directed to the following screen:

Next, install the required plugins.

Go the main dashboard and then click on “Manage Jenkins” -> “Manage Plugins“. In the “Available” tab, search for:

• Azure Credentials
• Terraform
• AnsiColor
• Git plugin

Check the checkbox of the required plugins and install them. Jenkins will restart at the end of the installation.

Pipeline Setup

After all plugins have been installed, the next step will be to add a “New Item” (see the menu on the right side).

Start by entering a valid name for the job and choose the type “Pipeline“. Hit “OK” to continue to the next screen.

Enter a description for the pipeline and scroll down to the “Pipeline” section and choose in the “Definition” section the option “Pipeline script from SCM“. Choose the SCM you want to use and fill in the repository URL of the code repository containing the Jenkinsfile pipeline file.

A bit further down it is possible to configure the SCM branch name to work on and the actual filename of the Jenkinsfile, which in most cases will be called just “Jenkinsfile“.

Once all options have been set, press “Save” to save the configuration.

Depending on whether the pipeline was created with or without parameters, either click on “Build” or “Build with Parameters” to start the pipeline.

Each build will appear in the “Build History” and contains links to for instance the “Console Output“, which contains the text output of all stages, steps and plugins executed by the pipeline.

Each build/ run contains information such as:

• Who started the pipeline
• The current SCM repository versions (git commit hash)
• An SCM log of the changes since the last build (git commit messages)
• Who approved the pipeline (if applicable)
• The console output of the plugins/ commands executed

The pipeline dashboard page, contains a stage view of pipeline runs, divided on the stages and their execution status.

Pipeline definitions

Finally, let’s have a look on how this pipeline file is actually built.

The above Jenkinsfile is divided in 4 sections:

• the agent section
• the tools section
• the environment section
• the stages section

The first 3 sections basically set some build options like:

• on which agent to run the pipeline (in this case, on “any” available node)
• which tools are required to be present on the build node
• which environment (shell) parameters should be set

The stages section contains the different stages and steps the pipeline is supposed to run.

Each stage, contains several steps separated per line:

In the above example, all steps are defined between the ‘steps { }’ block. Steps can be wrapped (enclosed between curly brackets), like in the above example line 15 calls the ansiColor() plugin to display a colorized output of the steps contained by it.

Next, the plugin dir() wraps the rest of the steps, which means that all enclosed steps will be executed in a specific directory.

In that specific directory, 3 plugins will be executed:

• git: line 17 in the above example will git clone the main branch of the supplied Github URL
• echo: line 19 will output some text
• sh: line 20 – 23 defines the shell commands to be executed

A complete example can be found at: https://github.com/insani4c/jenkins-terraform-datacenter-example/blob/main/JenkinsFile

AppArmor will write messages to the kernel log (visible with either the dmesg command or in kernel.log if available) regarding its actions.

If your libvirt guests are not starting up or failing, have a look at dmesg. Example:

In the above example AppArmor has denied (apparmor="DENIED") read access (requested_mask=r) to the file /data/vms/cluster_storage/base-os-ubuntu-focal.qcow2. This blocks of course the startup guest machines we have previously created in the article: Terraform and libvirtd nodes.

To fix the issue, edit the file: /etc/apparmor.d/libvirt/TEMPLATE.qemu

By default it has the following content:

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
}

In order to allow libvirt to use the guest image files, change the content to (or add a similar line if your file path is different):

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /data/vms/cluster_storage/**.qcow2 rwk,
}

The added line (line 9) will allow read (r), write (w) and lock (k) access to all qcow2 files in the directory /data/vms/cluster_storage.

Once added, all libvirt guests will start up again without any (AppArmor) issues.

After the default installation of Ubuntu 20.4-lts, the following packages are required to get started as a hypervisor:

apt install qemu-kvm libvirt-daemon bridge-utils virtinst libvirt-daemon-system virt-top libguestfs-tools libosinfo-bin qemu-system virt-manager qemu pm-utils

Once these are installed, the vnet_hosts module needs to be pre-loaded in /etc/modules:

echo vhost_net | tee -a /etc/modules
modprobe vhost_net

The hypervisor is now ready to start creating and deploying virtual machines.

In this article, Terraform will be used to manage the virtual machines in libvirtd. All example code snippets are available on Github, under https://github.com/insani4c/terraform-libvirt

Terraform has an excellent provider (dmacvicar/libvirtd) to manage the libvirtd nodes, which needs to be loaded and initialized:

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
    }
  }
}

provider "libvirt" {
  uri = "qemu+ssh://root@192.168.1.1/system"
}

In the above code snippet, Terraform will ensure the libvirt provider is loaded and it is configured to connect to the host with IP address 192.168.1.1 as the root user (this requires that a SSH public key is installed on the remote server, of the user who will be executing Terraform).

Next, the network will defined, where the virtual nodes will be deployed in.

resource "libvirt_network" "my_network" {
  name = "my_net"

  mode = "nat"

  addresses = ["10.1.2.0/24"]
  domain    = var.dns_domain

  autostart = true

  dhcp {
    enabled = false
  }

  dns {
    enabled = true

    local_only = false
    forwarders { address = "127.0.0.53" }

    hosts  {
        hostname = "host01"
        ip = "10.1.2.10"
      }
    hosts {
        hostname = "host02"
        ip = "10.1.2.20"
      }
  }  
}

The above code will ensure that a network of type NAT (to allow internal IPs, reachable from the hypervisor only) with network mask 10.1.2.0/24, will be created. It will ensure that DHCP is disabled and it will enable a DNS setup (the package dnsmasq must be installed) with two predefined hosts, host01 and host02.

Up next is the definition of the storage pools and volumes required for the virtual machines.

resource "libvirt_pool" "default" {
  name = "default"
  type = "dir"
  path = "/data/vms/cluster_storage"
}

resource "libvirt_volume" "local_install_image" {
  name   = var.local_install_image
  pool   = libvirt_pool.default.name
  source = var.os_img_url
  format = "qcow2"
}

The above defines the libvirt_pool, which basically configures the path on-disk for storing all sorts of volumes. Next it defines a volume called “local_install_image“, which will be used to set up the virtual machine as the volume will contain the “cloud image” for the installation. This volume requires two variables:

variable "os_img_url" {
  description = "URL to the OS image"
  default     = "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img"
}

variable "local_install_image" {
    description = "The name of the local install image"
    default     = "base-os-ubuntu-focal.qcow2"
}