Terraform codifies cloud, virtualization and many other APIs into declarative configuration files, allowing to define the infrastructure as code.

The combination of both is an excellent platform to manage resources in the Microsoft Azure cloud in an automated or semi-automated way.

Installation

Let’s start by installing Jenkins and Terraform. The official documentation at https://www.jenkins.io/doc/book/installing/linux/#debianubuntu describes how to install the Jenkins binaries on a Debian/ Ubuntu system, so we’ll not go further into details on that.

Once the required packages are installed as described above, go to http://localhost:8080/ to finalize the Jenkins setup. After installing the essential plugins required to run Jenkins, you should be directed to the following screen:

Next, install the required plugins.

Go the main dashboard and then click on “Manage Jenkins” -> “Manage Plugins“. In the “Available” tab, search for:

• Azure Credentials
• Terraform
• AnsiColor
• Git plugin

Check the checkbox of the required plugins and install them. Jenkins will restart at the end of the installation.

Pipeline Setup

After all plugins have been installed, the next step will be to add a “New Item” (see the menu on the right side).

Start by entering a valid name for the job and choose the type “Pipeline“. Hit “OK” to continue to the next screen.

Enter a description for the pipeline and scroll down to the “Pipeline” section and choose in the “Definition” section the option “Pipeline script from SCM“. Choose the SCM you want to use and fill in the repository URL of the code repository containing the Jenkinsfile pipeline file.

A bit further down it is possible to configure the SCM branch name to work on and the actual filename of the Jenkinsfile, which in most cases will be called just “Jenkinsfile“.

Once all options have been set, press “Save” to save the configuration.

Depending on whether the pipeline was created with or without parameters, either click on “Build” or “Build with Parameters” to start the pipeline.

Each build will appear in the “Build History” and contains links to for instance the “Console Output“, which contains the text output of all stages, steps and plugins executed by the pipeline.

Each build/ run contains information such as:

• Who started the pipeline
• The current SCM repository versions (git commit hash)
• An SCM log of the changes since the last build (git commit messages)
• Who approved the pipeline (if applicable)
• The console output of the plugins/ commands executed

The pipeline dashboard page, contains a stage view of pipeline runs, divided on the stages and their execution status.

Pipeline definitions

Finally, let’s have a look on how this pipeline file is actually built.

The above Jenkinsfile is divided in 4 sections:

• the agent section
• the tools section
• the environment section
• the stages section

The first 3 sections basically set some build options like:

• on which agent to run the pipeline (in this case, on “any” available node)
• which tools are required to be present on the build node
• which environment (shell) parameters should be set

The stages section contains the different stages and steps the pipeline is supposed to run.

Each stage, contains several steps separated per line:

In the above example, all steps are defined between the ‘steps { }’ block. Steps can be wrapped (enclosed between curly brackets), like in the above example line 15 calls the ansiColor() plugin to display a colorized output of the steps contained by it.

Next, the plugin dir() wraps the rest of the steps, which means that all enclosed steps will be executed in a specific directory.

In that specific directory, 3 plugins will be executed:

• git: line 17 in the above example will git clone the main branch of the supplied Github URL
• echo: line 19 will output some text
• sh: line 20 – 23 defines the shell commands to be executed

A complete example can be found at: https://github.com/insani4c/jenkins-terraform-datacenter-example/blob/main/JenkinsFile

In the required_providers section, the configuration_aliases must be configured first (usually in the main.tf file). This parameter must contain the same name (or names as the parameter takes a list of strings as input) as the alias parameter further below in the second provider section. Each provider section can have its own configuration parameters, such as for instance the subscription_id.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.4.0"
      configuration_aliases = [ azurerm.other-sub ]
    }
}

provider "azurerm" {
  subscription_id = var.subscription_id
  features {}
}

provider "azurerm" {
  alias           = "other-sub"
  subscription_id = var.other_subscription_id
  features {}
}

When calling a module, each provider containing an alias and which is required in the module, must be specified in the providers parameter. The key is the name configured in the configuration_aliases in the required_providers section of the module (see below), the value is the name of the provider alias in the playbook.

module "diagnostic_sa_private_endpoint" {
  source = "../terraform_modules/private_endpoint/"
  providers = {
      azurerm.other-sub = azurerm.other-sub
  }

  for_each = { for snet in data.azurerm_subnet.subnets : snet.id => snet }
  ...
}

The module itself (in ../terraform_modules/private_endpoint/) however must again define the alias in a “required_providers” section. This means that a file must be include in the directory ../terraform_modules/private_endpoint which includes:

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      configuration_aliases = [ azurerm.other-sub ]
    }
  }
}

Finally, all Azure resources which require the other Azure subscription, must include a “provider” line similar as:

data "azurerm_private_dns_zone" "dns_zone" {
  provider            = azurerm.other-sub
  name                = var.private_dns_zone_name
}

In this article, Packer images will created for Azure using azure-arm build type. The images will use an Ubuntu image available on Azure, as base image.

Let’s consider the following head of a Packer template file:

{  
  "builders": [{
      "type": "azure-arm",

      "use_azure_cli_auth": true,
      "subscription_id": "{{ user `subscription_id` }}",

      "managed_image_resource_group_name": "{{user `resource_group`}}",
      "managed_image_name": "{{user `managed_image_name`}}",

      "os_type": "{{user `basevm_os_type`}}",
      "image_publisher": "{{user `basevm_publisher`}}",
      "image_offer": "{{user `basevm_offer`}}",
      "image_sku": "{{user `basevm_sku`}}",

      "virtual_network_name": "{{user `virtual_network_name`}}",
      "virtual_network_resource_group_name": "{{user `virtual_network_resource_group_name`}}",
      "virtual_network_subnet_name": "{{user `virtual_network_subnet_name`}}",

      "azure_tags": {
        "dept": "DevOps",
        "task": "My custom base image"
      },

      "location": "{{user `location`}}",
      "vm_size": "{{user `vm_size`}}",

      "os_disk_size_gb": "{{user `os_disk_size_gb`}}",

      "shared_image_gallery_destination": {
        "subscription": "{{ user `subscription_id` }}",
        "resource_group": "{{user `resource_group`}}",
        "gallery_name": "{{user `gallery_name`}}",
        "image_name": "{{user `base_image`}}",
        "image_version": "{{user `base_image_version`}}",
        "replication_regions": "{{user `replication_regions`}}"
      },
      "shared_image_gallery_timeout": "2h1m1s"
    }],

All the variables surrounded by {{user `` }} are template variables, and must be configured in a separate file, example my_base_image_vars.json, which will be included later-on when initiating the Packer build command. The final image will be stored in Azure, in a Shared Image Gallery.

Packer however allows to configure extra provisioners, which will be executed once the initial virtual machine has been created and before the final image will be created.

One of those provisioner types is the ansible-local type. This provisioner allows to execute Ansible playbooks (and roles) once the virtual machine is booted, directly on the guest machine.

    "provisioners": [
      {
        "type": "shell",
        "inline_shebang": "/bin/sh -x",
        "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
        "script": "presetup.sh"
      },
      {
        "type": "ansible-local",
        "playbook_file": "./ansible/default_setup.yml",
        "playbook_dir": "./ansible",
        "role_paths": ["./ansible/defaults"],
        "clean_staging_directory": true,
        "staging_directory": "/tmp/packer-provisioner-ansible-local",
        "extra_arguments" : [ "--extra-vars", "ansible_python_interpreter=/usr/bin/python3" ]
      },

The first provisioner called, is the shell provisioner. This provisioner will upload and execute the script presetup.sh. This script could for instance ensure that the Ansible package is already installed before the next provisioner is called.

The second provisioner is the ansible-local type. The provisioner requires some parameters that must be configured:

• playbook_file: the Ansible playbook which will be executed
• playbook_dir: the base directory of the Ansible playbooks, roles, static_files, …
• role_paths: the path of the role which will be called in playbook_file (when required)

The above example will upload all files in the ./ansible directory (parameter playbook_dir) to the new guest virtual machine, to the directory configured in staging_directory, and will call the Ansible playbook default_setup.yml.

---
- name: Default post installation
  hosts: all
  connection: local
  become: yes
  tasks:
    - name: Import the Microsoft signing key into apt
      apt_key:
        url: "https://packages.microsoft.com/keys/microsoft.asc"
        state: present

    - name: Add the Azure CLI software repository
      apt_repository:
        repo: "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ {{ansible_distribution_release}} main"
        filename: azure-cli
        state: present

  roles:
    - defaults

The above Ansible playbook use the parameter connection: local. This is very important and must be included in each playbook executed by the ansible-local provisioner.

At first the playbook will execute two tasks which will ensure that the Microsoft Azure-CLI APT repository is installed and properly set up.

Afterwards, it will call the defaults Ansbile role. This role is set up as a typical Ansible role:

ansible/defaults:
tasks  templates  vars

ansible/defaults/tasks:
main.yml

ansible/defaults/templates:
modprobe_blacklist.j2  policy-rc.d.conf.j2  policy-rc.d.j2  securetty.j2  sysctl_base.j2

ansible/defaults/vars:
main.yml

The main.yml in the tasks sub-directory is used to configure the required post installation steps.

Example:

---
- name: Ensure default packages
  apt:
    name: "{{ default_packages }}"
    state: latest

- name: Ensure latest version of all packages
  apt:
    upgrade: dist
    force_apt_get: yes
    dpkg_options: 'force-confold,force-confdef'
    autoremove: yes
    autoclean: yes

- name: Update python alternatives (for Log Analytics)
  shell: |
    update-alternatives --remove-all python
    update-alternatives --install /usr/bin/python python /usr/bin/python2 1

- name: Forward Syslog-NG logs to Log Analytics
  ansible.builtin.copy:
    src: static_files/syslog-ng-lad.conf
    dest: /etc/syslog-ng/conf.d/syslog-ng-lad.conf
    owner: root
    group: root
    mode: '0644'

...

The YAML configuration hierarchy could be defined as the following file structure:

• common.yaml: Default settings no matter which role or host. These can be overridden in all the below.
• my_environment01.yaml: Environment specific configuration (example: development, staging, production, amsterdam, az01, az04, …). These can be overridden in all the below.
• common/roles/some_server_role.yaml: A server role, or type definition, which contains role specific configuration parameters. The roles could implement an extra hierarchy as for instance:
  
  • debian::databases::postgres
  • debian::databases::postgres::timescale
  • debian::databases::postgres::timescale::prometheus
  • debian::loadbalancer::internal
  • debian::application::request_processor
    
    The hierarchy steps are divided by :: in the above example, and need to be inherited accordingly, each with their own YAML file.
    These can be overridden in all the below.
• my_environment01/roles/some_server_role.yaml: override role configuration parameters per environment.
  These can even be overridden on host level below.
• my_environment01/hosts/my_hostname01.yaml: set host specific configuration parameters. This file is actually always required and should contain at least the IP address of the node and the server role string.

Let’s take the following example: The host vmazdbprm01 has the role debian::databases::postgres::timescale::prometheus and is deployed in the environment in my_cool_location03. The configuration management should search for parameters in the following file locations (and first verify if the file path exists):

1. common.yaml
2. my_cool_location01.yaml
3. common/roles/debian.yaml
4. common/roles/debian::databases.yaml
5. common/roles/debian::databases::postgres.yaml
6. common/roles/debian::databases::postgres::timescale.yaml
7. common/roles/debian::databases::postgres::timescale::prometheus.yaml
8. my_cool_location01/roles/debian.yaml
9. my_cool_location01/roles/debian::databases.yaml
10. my_cool_location01/roles/debian::databases::postgres.yaml
11. my_cool_location01/roles/debian::databases::postgres::timescale.yaml
12. my_cool_location01/roles/debian::databases::postgres::timescale::prometheus.yaml
13. my_cool_location01/hosts/vmazdbprm01.yaml

This means that any code which wants to implement the above configuration management, needs to verify if the above 13 files exists from top to bottom, and if yes loads the YAML file accordingly.

Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services.

Implementing the above YAML hierarchy in Terraform, could be done as follows:

locals {
  host_cfg             = yamldecode(fileexists("cfgmgmt/${var.environment}/hosts/${var.node}.yaml") ? file("cfgmgmt/${var.environment}/hosts/${var.node}.yaml") : "{server_role: debian}")

  roles_list           = split("::", local.host_cfg.server_role)
  all_roles_list       = [ for index in range(length(local.roles_list)): join("::",slice(local.roles_list, 0, index + 1))  ]

  common_cfg           = yamldecode(fileexists("cfgmgmt/common.yaml") ? file("cfgmgmt/common.yaml") : "{}")
  common_role_cfg_list = [ for file in local.all_roles_list:
      yamldecode(fileexists("cfgmgmt/common/roles/${file}.yaml") ? file("cfgmgmt/common/roles/${file}.yaml") : "{}" )]
    
  env_cfg              = yamldecode(fileexists("cfgmgmt/${var.environment}.yaml") ? file("cfgmgmt/${var.environment}.yaml") : "{}")
  env_role_cfg_list    = [ for file in local.all_roles_list:
      yamldecode(fileexists("cfgmgmt/${var.environment}/roles/${file}.yaml") ? file("cfgmgmt/${var.environment}/roles/${file}.yaml") : "{}") ]
        
  common_role_cfg_map  = merge(local.common_role_cfg_list...)
  env_role_cfg_map     = merge(local.env_role_cfg_list...)

  cfg                  = merge(local.common_cfg, local.env_cfg, local.common_role_cfg_map, local.env_role_cfg_map, local.host_cfg)
}

Let’s have a look what actually happens in the above code.

All YAML files are stored in a Git/ Hiera repository, accessible in the sub-directory cfgmgmt.

The code declares “local” variables by issuing the resource “locals“, starting from line 1.

Line 2 will check if a file called cfgmgmt/${var.environment}/hosts/${var.node}.yaml exists and if true, loads the YAML content as an map into the local variable host_cfg. If the file doesn’t exists, a default YAML code will be loaded. In theory, each node/ host must have a file defined as it should have at least data configuration such as:

• unique node host name
• IP address
• server role
• (optionally) VLAN/ subnet configuration
• …

Line 4 splits the server role string, stored in local.host_cfg.server_role, into a list, to build the server role hierarchy further below.

Line 5 creates a list of top level server roles which need to be imported too. Example: if the server role was set to debian::databases::postgres::timescale::prometheus , the list all_roles_list will contain the following elements:

• debian
• debian::databases
• debian::databases::postgres
• debian::databases::postgres::timescale
• debian::databases::postgres::timescale::prometheus

Line 7 loads the YAML content of common.yaml, if it exists.

Line 8 loops over the all_roles_list elements, created on line 5, and will load the YAML content of the server roles (if the file exists) into a list element. The result is a list called common_role_cfg_list.

Line 11 loads the general environment configuration YAML content (if it exists) into the local variable env_cfg.

Line 12 will do the same thing as line 8, but for environment specific roles. (for instance: when certain server roles have environment specific configuration parameters).

Lines 15 and 16 will merge the elements (which in theory are maps of YAML data) of the server role lists into one big map, in the order of the list. This allows that keys can be overridden. The expansion ... notation is explained at https://www.terraform.io/language/expressions/function-calls#expanding-function-arguments.

Finally on line 18, a local variable called cfg will be created, which merges the values of:

• local.common_cfg
• local.env_cfg
• local.common_role_cfg_map
• local.env_role_cfg_map
• local.host_cfg

By providing the environment name and the host/ node name to the above code (as var.environment and var.node ), all required configuration parameters can be loaded per node in Terraform, but since we’ve used a Git repository, this information can be loaded in any kind of automation tool (required is of course that each automation implements the same kind of hierarchy code).

After the default installation of Ubuntu 20.4-lts, the following packages are required to get started as a hypervisor:

apt install qemu-kvm libvirt-daemon bridge-utils virtinst libvirt-daemon-system virt-top libguestfs-tools libosinfo-bin qemu-system virt-manager qemu pm-utils

Once these are installed, the vnet_hosts module needs to be pre-loaded in /etc/modules:

echo vhost_net | tee -a /etc/modules
modprobe vhost_net

The hypervisor is now ready to start creating and deploying virtual machines.

In this article, Terraform will be used to manage the virtual machines in libvirtd. All example code snippets are available on Github, under https://github.com/insani4c/terraform-libvirt

Terraform has an excellent provider (dmacvicar/libvirtd) to manage the libvirtd nodes, which needs to be loaded and initialized:

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
    }
  }
}

provider "libvirt" {
  uri = "qemu+ssh://root@192.168.1.1/system"
}

In the above code snippet, Terraform will ensure the libvirt provider is loaded and it is configured to connect to the host with IP address 192.168.1.1 as the root user (this requires that a SSH public key is installed on the remote server, of the user who will be executing Terraform).

Next, the network will defined, where the virtual nodes will be deployed in.

resource "libvirt_network" "my_network" {
  name = "my_net"

  mode = "nat"

  addresses = ["10.1.2.0/24"]
  domain    = var.dns_domain

  autostart = true

  dhcp {
    enabled = false
  }

  dns {
    enabled = true

    local_only = false
    forwarders { address = "127.0.0.53" }

    hosts  {
        hostname = "host01"
        ip = "10.1.2.10"
      }
    hosts {
        hostname = "host02"
        ip = "10.1.2.20"
      }
  }  
}

The above code will ensure that a network of type NAT (to allow internal IPs, reachable from the hypervisor only) with network mask 10.1.2.0/24, will be created. It will ensure that DHCP is disabled and it will enable a DNS setup (the package dnsmasq must be installed) with two predefined hosts, host01 and host02.

Up next is the definition of the storage pools and volumes required for the virtual machines.

resource "libvirt_pool" "default" {
  name = "default"
  type = "dir"
  path = "/data/vms/cluster_storage"
}

resource "libvirt_volume" "local_install_image" {
  name   = var.local_install_image
  pool   = libvirt_pool.default.name
  source = var.os_img_url
  format = "qcow2"
}

The above defines the libvirt_pool, which basically configures the path on-disk for storing all sorts of volumes. Next it defines a volume called “local_install_image“, which will be used to set up the virtual machine as the volume will contain the “cloud image” for the installation. This volume requires two variables:

variable "os_img_url" {
  description = "URL to the OS image"
  default     = "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img"
}

variable "local_install_image" {
    description = "The name of the local install image"
    default     = "base-os-ubuntu-focal.qcow2"
}

Create a list of the existing virtual network subnets:

data azurerm_subnet "subnets" {
  count = length(data.azurerm_virtual_network.my_vnet.subnets)

  name                 = data.azurerm_virtual_network.my_vnet.subnets[count.index]
  virtual_network_name = var.vnet_name
  resource_group_name  = var.resource_group
}

locals {
  subnets = tomap({
      for snet in data.azurerm_subnet.subnets: snet.name => snet.id
  })
}

In the above example, we first loop over all subnet names, returned by data.azurerm_virtual_network.my_vnet.subnets, to create a list of Azure virtual network subnet objects.

Afterwards we create a locals map called subnets, which contains mapping like “subnet name points to subnet ID”.

Finally, when creating Azure network interfaces with an IP configuration, you can easily lookup the correct subnet ID based on the subnet name (which you might have configured per virtual machine)

resource "azurerm_network_interface" "my_nic" {
...
  ip_configuration {
    ...
    subnet_id = lookup(local.subnets, "my_subnet_name")
  }
...