The Docker installation is well documented at https://docs.docker.com/get-docker/ and is well supported amongst the most popular operating systems. The installation itself will not be covered in this article. If you want to get familiar with the details of Docker, start with the documentation at https://docs.docker.com/get-started/.

All code used in this article is available at: https://github.com/insani4c/docker-monitoring-stack.

In this article, we will see how to set up a monitoring solution based on:

• Prometheus
• Prometheus Node Exporter
• Prometheus Black Exporter
• Prometheus SNMP Exporter
• Loki
• Promtail
• Grafana

To monitor the deployed containers, we will also deploy Google’s cadvisor container, to get some interesting statistics and details in our Prometheus/ Grafana setup.

The Docker Compose file, called docker-compose.yml, contains all the information of the infrastructure such as:

• network information
• volumes
• services (the containers)
• …

Let’s start from the top of the file.

version: '3.8'

name: docmon

volumes:
  grafana-data: {}
  alertmanager-data: {}
  prometheus-data: {}
  loki-data: {}

At first at line 1, the version Docker Compose version is specified, to define which specifications are allowed. At line 3 a name for the container group or stack is set. And finally starting from line 5, data volumes (think disks) are defined, which will be used by the containers. These are persistent data volumes which will be reused unless the container has been completely removed.

Next, we will define the services in the docker-compose.yml:

services:
  cadvisor:
    image: 'gcr.io/cadvisor/cadvisor:latest'
    container_name: cadvisor
    restart: always
    mem_limit: 512m
    mem_reservation: 32m
    # ports: 
    #   - '8880:8080'
    volumes:
      - '/:/rootfs:ro'
      - '/var/run:/var/run:ro'
      - '/sys:/sys:ro'
      - '/var/lib/docker/:/var/lib/docker:ro'
      - '/dev/disk/:/dev/disk:ro'
    privileged: true
    devices: 
      - '/dev/kmsg:/dev/kmsg'

  prometheus:
    image: 'prom/prometheus:latest'
    container_name: prometheus
    restart: always
    mem_limit: 2048m
    mem_reservation: 256m
    cpus: 2
    # ports:
    #   - '9090:9090'
    volumes:
      - '$PROMETHEUS_HOME/config:/etc/prometheus'
      - 'prometheus-data:/prometheus'
    extra_hosts:
      myrouter: 192.168.1.1
      myswitch: 192.168.1.10
    depends_on:
      - cadvisor

Containers are defined as services. Each service will require at least:

• a service name (example line 2 and line 20)
• an image definition

All other options are optional or required by specific images.

The first image or container defined in the above example is cadvisor. This service provides statistics from Docker and the deployed containers to Prometheus. To be able to provide this information, the container must have read access to certain file paths or sockets on the hypervisor (read: the server where the Docker containers will be running). These are provided in the volumes section of the container. Here, directory paths on the hypervisor will be provided as mount partitions in the container, and they will be mounted with the readonly (:ro) parameter so that the container can’t make any changes to them.

Furthermore it provides access to a device (to read kernel messages), set memory and cpu limits and will run the container in privilege mode. The ports section has been put in comments, as it isn’t really require to expose ports, or make them available outside the Docker ecosystem. In our example, only Prometheus must be able to connect to it, and since Prometheus will be deployed as a container, we don’t need to be able to access the web service running on the container to read out the metrics or see the statistics.

The next container defined is called prometheus. For this container, volumes will be mounted to provide the Prometheus configuration files and to store the data to the volume called prometheus-data. It also defines an extra_hosts. These are entries that are typically defined in an /etc/hosts file, which Docker does not read from the hypervisor. And instead of deploying or mounting the hypervisor’s hosts file, extra host mappings can be defined or handed to the container, which is set up in the extra_hosts section as above.

At the end of the prometheus container definition, a depends_on section is configured, which means that the prometheus container won’t be deployed until the container names defined in that section are up and running.

Next we will define all other containers (see the second tab in the above code block, called the rest).

  hypervisor:
    image: 'prom/node-exporter:latest'
    container_name: hypervisor
    mem_limit: 128m
    mem_reservation: 32m
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'
      - '/proc:/host/proc:ro'
      - '/sys:/host/sys:ro'
    command:
      - '--path.rootfs=/host'
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.systemd'
      - '--collector.cgroups'
    depends_on:
      - cadvisor

  prom_snmp:
    image: 'prom/snmp-exporter:latest'
    container_name: prom_snmp
    restart: always
    mem_limit: 128m
    mem_reservation: 32m
    # ports: 
    #   - '9116:9116'
    volumes:
      - '$PROMSNMP_HOME/config:/etc/snmp_exporter'
    extra_hosts:
      myrouter: 192.168.1.1
      myswitch: 192.168.1.10
    depends_on:
      - cadvisor
      - prometheus

  alertmanager:
    image: 'prom/alertmanager:latest'
    container_name: alertmanager
    restart: always
    mem_limit: 256m
    mem_reservation: 32m 
    # ports:
    #   - 9093:9093
    volumes:
      - '$ALERTMANAGER_HOME/config/alertmanager.yml:/etc/alertmanager/config.yml'
      - 'alertmanager-data:/alertmanager'
    command:
      - '--config.file=/etc/alertmanager/config.yml'
      - '--storage.path=/alertmanager'
    depends_on:
      - cadvisor
      - prometheus

  loki:
    image: 'grafana/loki:latest'
    container_name: loki
    restart: always
    mem_limit: 32768m
    mem_reservation: 8192m
    cpus: 6 
    ports:
      - '3100:3100'
    volumes:
      - '$LOKI_HOME/config:/etc/loki'
      - 'loki-data:/loki'
    depends_on:
      - cadvisor
      - prometheus
      - alertmanager

  blackbox_exporter:
    image: 'prom/blackbox-exporter:latest'
    container_name: blackbox_exporter
    restart: always
    mem_limit: 128m
    mem_reservation: 32m
    dns:
      - 8.8.8.8
      - 8.8.4.4
    # ports:
    #   - 9115:9115
    volumes:
      - '$BLACKBOXEXPORTER_HOME/config:/etc/blackboxexporter/'
    command:
      - '--config.file=/etc/blackboxexporter/config.yml'
    depends_on:
      - cadvisor
      - prometheus

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: always
    mem_limit: 256m
    mem_reservation: 64m
    volumes:
      - $PROMTAIL_HOME/config:/etc/promtail/
      # to read container labels and logs
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
      - '/var/lib/docker/containers:/var/lib/docker/containers:ro'
      - '/var/log/ulog:/var/log/ulog/:ro'
    depends_on:
      - cadvisor
      - loki

  grafana:
    image: 'grafana/grafana:latest'
    container_name: grafana
    restart: always
    mem_limit: 2048m
    mem_reservation: 256m
    ports:
      - '3000:3000'
    volumes:
      - '$GRAFANA_HOME/config:/etc/grafana'
      - 'grafana-data:/var/lib/grafana'
      - '$GRAFANA_HOME/dashboards:/var/lib/grafana/dashboards'
    depends_on:
      - cadvisor
      - prometheus
      - loki
      - alertmanager

The rest of the code will deploy:

• a container called hypervisor, which is actually the Prometheus node-exporter for the hypervisor.
• a container called prom_snmp, which will retrieve SNMP statistics
• a container called blackbox_exporter, which mainly checks webservers and their SSL certificates
• a container called promtail, which collects logs and log statistics from the hypervisor
• a container called loki, which allows to store and index logs sent to the service by promtail (either container or as a service running on some external server)

Finally, the last container deployed is the grafana container. Besides its normal configuration file grafana.ini, the Docker container will also automatically provision (the provisioning sub directory in the config directory) datasources and dashboards so that no manual after-tasks are required once the containers are running.

The datasources can be preconfigured in a YAML file called default.yaml, stored in the provisioning/datasources/ sub directory.

apiVersion: 1

datasources:
 - name: Alertmanager
   type: alertmanager 
   access: proxy
   orgId: 1
   url: http://alertmanager:9093
   version: 1
   editable: false
   isDefault: false
   uid: DS_ALERTMANAGER
   jsonData:
    implementation: prometheus
 - name: Prometheus
   type: prometheus
   access: proxy
   orgId: 1
   url: http://prometheus:9090
   version: 1
   editable: false
   isDefault: true
   uid: DS_PROMETHEUS
   jsonData:
    alertmanagerUid: DS_ALERTMANAGER
    manageAlerts: true
    prometheusType: Prometheus
    prometheusVersion: 2.39.1
 - name: Loki
   type: loki 
   access: proxy
   orgId: 1
   url: http://loki:3100
   version: 1
   editable: false
   isDefault: false
   uid: DS_LOKI
   jsonData:
    alertmanagerUid: DS_ALERTMANAGER
    manageAlerts: true

Same thing goes for dashboards we want to have automatically deployed:

apiVersion: 1

providers:
 - name: 'default'
   orgId: 1
   folder: 'Custom'
   folderUid: ''
   type: file
   options:
     path: /var/lib/grafana/dashboards

Finally, if Docker is running on multiple network interfaces (for it is a hosted server, or it has internal and external IP addresses), you might want to limit access to the container to specific networks only.

Below is a netfilter example, which allows traffic only coming from 192.168.1.0/24 and from the network interface enp35so:

iptables -I DOCKER-USER -i enp35s0 ! -s 192.168.1.0/24 -m conntrack --ctdir ORIGINAL -j DROP

The chain DOCKER-USER is not flushed by Docker and thus can be created in a general firewall script of netfilter configuration, even at boot time:

-N DOCKER-USER
-I DOCKER-USER -i enp35s0 ! -s 192.168.1.0/24 -m conntrack --ctdir ORIGINAL -j DROP

Terraform codifies cloud, virtualization and many other APIs into declarative configuration files, allowing to define the infrastructure as code.

The combination of both is an excellent platform to manage resources in the Microsoft Azure cloud in an automated or semi-automated way.

Installation

Let’s start by installing Jenkins and Terraform. The official documentation at https://www.jenkins.io/doc/book/installing/linux/#debianubuntu describes how to install the Jenkins binaries on a Debian/ Ubuntu system, so we’ll not go further into details on that.

Once the required packages are installed as described above, go to http://localhost:8080/ to finalize the Jenkins setup. After installing the essential plugins required to run Jenkins, you should be directed to the following screen:

Next, install the required plugins.

Go the main dashboard and then click on “Manage Jenkins” -> “Manage Plugins“. In the “Available” tab, search for:

• Azure Credentials
• Terraform
• AnsiColor
• Git plugin

Check the checkbox of the required plugins and install them. Jenkins will restart at the end of the installation.

Pipeline Setup

After all plugins have been installed, the next step will be to add a “New Item” (see the menu on the right side).

Start by entering a valid name for the job and choose the type “Pipeline“. Hit “OK” to continue to the next screen.

Enter a description for the pipeline and scroll down to the “Pipeline” section and choose in the “Definition” section the option “Pipeline script from SCM“. Choose the SCM you want to use and fill in the repository URL of the code repository containing the Jenkinsfile pipeline file.

A bit further down it is possible to configure the SCM branch name to work on and the actual filename of the Jenkinsfile, which in most cases will be called just “Jenkinsfile“.

Once all options have been set, press “Save” to save the configuration.

Depending on whether the pipeline was created with or without parameters, either click on “Build” or “Build with Parameters” to start the pipeline.

Each build will appear in the “Build History” and contains links to for instance the “Console Output“, which contains the text output of all stages, steps and plugins executed by the pipeline.

Each build/ run contains information such as:

• Who started the pipeline
• The current SCM repository versions (git commit hash)
• An SCM log of the changes since the last build (git commit messages)
• Who approved the pipeline (if applicable)
• The console output of the plugins/ commands executed

The pipeline dashboard page, contains a stage view of pipeline runs, divided on the stages and their execution status.

Pipeline definitions

Finally, let’s have a look on how this pipeline file is actually built.

The above Jenkinsfile is divided in 4 sections:

• the agent section
• the tools section
• the environment section
• the stages section

The first 3 sections basically set some build options like:

• on which agent to run the pipeline (in this case, on “any” available node)
• which tools are required to be present on the build node
• which environment (shell) parameters should be set

The stages section contains the different stages and steps the pipeline is supposed to run.

Each stage, contains several steps separated per line:

In the above example, all steps are defined between the ‘steps { }’ block. Steps can be wrapped (enclosed between curly brackets), like in the above example line 15 calls the ansiColor() plugin to display a colorized output of the steps contained by it.

Next, the plugin dir() wraps the rest of the steps, which means that all enclosed steps will be executed in a specific directory.

In that specific directory, 3 plugins will be executed:

• git: line 17 in the above example will git clone the main branch of the supplied Github URL
• echo: line 19 will output some text
• sh: line 20 – 23 defines the shell commands to be executed

A complete example can be found at: https://github.com/insani4c/jenkins-terraform-datacenter-example/blob/main/JenkinsFile

In the required_providers section, the configuration_aliases must be configured first (usually in the main.tf file). This parameter must contain the same name (or names as the parameter takes a list of strings as input) as the alias parameter further below in the second provider section. Each provider section can have its own configuration parameters, such as for instance the subscription_id.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.4.0"
      configuration_aliases = [ azurerm.other-sub ]
    }
}

provider "azurerm" {
  subscription_id = var.subscription_id
  features {}
}

provider "azurerm" {
  alias           = "other-sub"
  subscription_id = var.other_subscription_id
  features {}
}

When calling a module, each provider containing an alias and which is required in the module, must be specified in the providers parameter. The key is the name configured in the configuration_aliases in the required_providers section of the module (see below), the value is the name of the provider alias in the playbook.

module "diagnostic_sa_private_endpoint" {
  source = "../terraform_modules/private_endpoint/"
  providers = {
      azurerm.other-sub = azurerm.other-sub
  }

  for_each = { for snet in data.azurerm_subnet.subnets : snet.id => snet }
  ...
}

The module itself (in ../terraform_modules/private_endpoint/) however must again define the alias in a “required_providers” section. This means that a file must be include in the directory ../terraform_modules/private_endpoint which includes:

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      configuration_aliases = [ azurerm.other-sub ]
    }
  }
}

Finally, all Azure resources which require the other Azure subscription, must include a “provider” line similar as:

data "azurerm_private_dns_zone" "dns_zone" {
  provider            = azurerm.other-sub
  name                = var.private_dns_zone_name
}

A small example of how to use the framework:

my $exporter = Prometheus::Exporter->new({
    listen_port => 9090, 
    listen_addr => "127.0.0.1", 
    max_threads => 5,
});

$exporter->register_metrics({
    test_metric        => {type => "gauge",     desc => "A test metric"},
    test_metric_labels => {type => "gauge",     desc => "A test metric", labels => ["code=42", "code=99"]},
    test_counter       => {type => "counter",   desc => "A test metric"},
    test_histogram     => {type => "histogram", buckets => ['0.3', '0.6', '1.2', '+Inf']},
});

$exporter->register_collector(sub {
    my $timeout = int(rand(5));
    sleep $timeout;

    $exporter->get_metric("test_metric")->value(rand(100));
    $exporter->get_metric("test_metric_labels")->value([rand(42), rand(99)]);

    $test_counter += int(rand(20));
    $exporter->get_metric("test_counter")->value($test_counter);

    $histo_buckets{"0.3"}  += rand(20);
    $histo_buckets{"0.6"}  += $histo_buckets{"0.3"} + rand(20);
    $histo_buckets{"1.2"}  += $histo_buckets{"0.6"} + rand(20);
    $histo_buckets{"+Inf"} += $histo_buckets{"1.2"} + rand(20);
    my $histo_sum = 2.0 * $histo_buckets{"+Inf"};
    my $histo_count = $histo_buckets{"+Inf"};
    $exporter->get_metric("test_histogram")->value(\%histo_buckets, $histo_sum, $histo_count);
});

$exporter->run;

The framework will start a small HTTP daemon once run() is called and will handle all client requests by using threads. On each request, the framework will call the subroutine or coderef defined at register_collector(). Currently, that coderef must store the observed values by using the construct seen in the above example, by calling the value() method on the registered metric objects.

Also currently, the histogram implementation is not yet supporting labels.

Many graphical interfaces are available for managing user accounts in OpenLDAP like PHPLDAPAdmin (http://phpldapadmin.sourceforge.net/wiki/index.php/Main_Page) or LAM (https://www.ldap-account-manager.org/lamcms/).

When generating a bulk amount of accounts with automation or just managing user details with a simple script, allows much more flexibility and can be even quicker.

LDAP passwords can be stored or changed by using an LDIF file. This LDIF file needs 3 required lines:

1. The “dn” you are about to change
2. the “changetype” set to “modify“
3. A “replace” line containing the field you want to change (in our case, since we are changing the password, this will be “userPassword“)

Your LDAP password can be stored either in clear-text (which is not advisable) or by sending a SHA-hash. The SHA-hash must include the salt at the end and must be base64 encoded.

The code snippit below will call a subroutine called generate_password() which comes from a previous article (Secure Password Generator in Perl).

At the end of the script, it will print out the LDIF file content, which needs to be saved to change.ldif. As last, it will print the ldapmodify command to make the actual change. You will need to know the admin password for this. Alternatively, you could also make this change using your own dn for authentication.

use Digest::SHA;
use MIME::Base64;

my $random_password = generate_password(24);
my $random_salt     = generate_password(3);

my $ctx = Digest::SHA->new;
$ctx->add($random_password);
$ctx->add($random_salt);
my $hashedPasswd = encode_base64($ctx->digest . $random_salt, '');

print "password: $random_password\n";
print "salt: $random_salt\n";
print <<EOF;
# LDIF
dn: uid=user1,ou=users,dc=shihai-corp,dc=at
changetype: modify
replace: userPassword
userPassword: {SSHA}$hashedPasswd
EOF

print "\n";
print q{LDAP cmd: ldapmodify -H "ldap://ldap_server01" -Z -x -W -D "cn=ldapadmin,ou=admins,dc=shihai-corp,dc=at" -f change.ldif} . "\n\n"

In this article we will have a closer look how to ensure that

• a default database has been created
• a set of configured extensions have been installed
• an initial database schema has been deployed

The user that will log on to the remote host using Ansible (and SSH), must be able to become the postgres user using sudo without a password (in this example, can be changed of course).

The tasks below are split in two big tasks, both containing a block. Blocks allow to group certain tasks which might need the same variables (for instance the same tags, become variables, …).

The first block will ensure that the default database has been created, based on a YAML variable called {{ db_name }}. This variable must be set in either a group_vars file, host_vars file or supplied at the command line.

In the same block, Ansible will also ensure that the pg_stat_statements extension is enabled and installed in the above database. pg_stat_statements is a very useful extension to debug or display statistics regarding the SQL statements executed on that specific database.

And the end of the first block, a task was added to loop over a YAML variable called {{ shared_preload_extensions }}. The variable is a supposed to be a list and should contain all extra extensions required to enabled to the above database. Shared preload extensions also need to be configured in the postgresql.conf file, which is not covered in this article.

- name: Default Database
  tags: default_database
  vars:
    ansible_become_user: postgres
    ansible_become_method: sudo
    ansible_become_pass: null
  block:
    - name: Ensure required database
      postgresql_db:
        name: "{{ db_name }}"
        encoding: UTF-8

    - name: Ensure pg_stat_statements extension
      postgresql_ext:
        name: "pg_stat_statements"
        db: "{{ db_name }}"
        schema: public
        state: present

    - name: Ensure shared_preload extensions
      postgresql_ext:
        name: "{{ item }}"
        db: "{{ db_name }}"
        state: present
      loop: "{{ shared_preload_libraries.split(',') }}"
      loop_control:
        label: " {{ item }}"

- name: Ensure initial database schema
  tags: default_database
  block:
    - name: Schema definition file
      template:
        src: "{{ db_name }}/schema_definition.sql.j2"
        dest: /var/lib/postgresql/initial_schema_definition.sql
        owner: postgres
        group: postgres
        mode: 0600
      when: ( role_path + "/templates/" + db_name + "/schema_definition.sql.j2" ) is file
      register: initial_schema

    - name: Apply the schema definition
      ignore_errors: True
      vars:
        ansible_become_user: postgres
        ansible_become_method: sudo
        ansible_become_pass: null
      community.postgresql.postgresql_query:
        db: "{{ db_name }}"
        path_to_script: /var/lib/postgresql/initial_schema_definition.sql
        encoding: UTF-8
        as_single_query: yes
      when: initial_schema.changed

  rescue:
    - debug:
        msg: "No schema definition found for {{db_name}}, skipping..."

The second block of tasks will ensure that an initial database schema is deployed. This block consists of two tasks only.

The first task will upload the schema definition SQL-based file to a specific path on the remote host.

The second task will try to execute that SQL file, if it was changed by the first task. This means of course that if the file was changed manually on the remote host (or even removed), Ansible will update that file again in the first task and it will deploy the full file again in the second task! This might overwrite or break your database.

In this article, Packer images will created for Azure using azure-arm build type. The images will use an Ubuntu image available on Azure, as base image.

Let’s consider the following head of a Packer template file:

{  
  "builders": [{
      "type": "azure-arm",

      "use_azure_cli_auth": true,
      "subscription_id": "{{ user `subscription_id` }}",

      "managed_image_resource_group_name": "{{user `resource_group`}}",
      "managed_image_name": "{{user `managed_image_name`}}",

      "os_type": "{{user `basevm_os_type`}}",
      "image_publisher": "{{user `basevm_publisher`}}",
      "image_offer": "{{user `basevm_offer`}}",
      "image_sku": "{{user `basevm_sku`}}",

      "virtual_network_name": "{{user `virtual_network_name`}}",
      "virtual_network_resource_group_name": "{{user `virtual_network_resource_group_name`}}",
      "virtual_network_subnet_name": "{{user `virtual_network_subnet_name`}}",

      "azure_tags": {
        "dept": "DevOps",
        "task": "My custom base image"
      },

      "location": "{{user `location`}}",
      "vm_size": "{{user `vm_size`}}",

      "os_disk_size_gb": "{{user `os_disk_size_gb`}}",

      "shared_image_gallery_destination": {
        "subscription": "{{ user `subscription_id` }}",
        "resource_group": "{{user `resource_group`}}",
        "gallery_name": "{{user `gallery_name`}}",
        "image_name": "{{user `base_image`}}",
        "image_version": "{{user `base_image_version`}}",
        "replication_regions": "{{user `replication_regions`}}"
      },
      "shared_image_gallery_timeout": "2h1m1s"
    }],

All the variables surrounded by {{user `` }} are template variables, and must be configured in a separate file, example my_base_image_vars.json, which will be included later-on when initiating the Packer build command. The final image will be stored in Azure, in a Shared Image Gallery.

Packer however allows to configure extra provisioners, which will be executed once the initial virtual machine has been created and before the final image will be created.

One of those provisioner types is the ansible-local type. This provisioner allows to execute Ansible playbooks (and roles) once the virtual machine is booted, directly on the guest machine.

    "provisioners": [
      {
        "type": "shell",
        "inline_shebang": "/bin/sh -x",
        "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
        "script": "presetup.sh"
      },
      {
        "type": "ansible-local",
        "playbook_file": "./ansible/default_setup.yml",
        "playbook_dir": "./ansible",
        "role_paths": ["./ansible/defaults"],
        "clean_staging_directory": true,
        "staging_directory": "/tmp/packer-provisioner-ansible-local",
        "extra_arguments" : [ "--extra-vars", "ansible_python_interpreter=/usr/bin/python3" ]
      },

The first provisioner called, is the shell provisioner. This provisioner will upload and execute the script presetup.sh. This script could for instance ensure that the Ansible package is already installed before the next provisioner is called.

The second provisioner is the ansible-local type. The provisioner requires some parameters that must be configured:

• playbook_file: the Ansible playbook which will be executed
• playbook_dir: the base directory of the Ansible playbooks, roles, static_files, …
• role_paths: the path of the role which will be called in playbook_file (when required)

The above example will upload all files in the ./ansible directory (parameter playbook_dir) to the new guest virtual machine, to the directory configured in staging_directory, and will call the Ansible playbook default_setup.yml.

---
- name: Default post installation
  hosts: all
  connection: local
  become: yes
  tasks:
    - name: Import the Microsoft signing key into apt
      apt_key:
        url: "https://packages.microsoft.com/keys/microsoft.asc"
        state: present

    - name: Add the Azure CLI software repository
      apt_repository:
        repo: "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ {{ansible_distribution_release}} main"
        filename: azure-cli
        state: present

  roles:
    - defaults

The above Ansible playbook use the parameter connection: local. This is very important and must be included in each playbook executed by the ansible-local provisioner.

At first the playbook will execute two tasks which will ensure that the Microsoft Azure-CLI APT repository is installed and properly set up.

Afterwards, it will call the defaults Ansbile role. This role is set up as a typical Ansible role:

ansible/defaults:
tasks  templates  vars

ansible/defaults/tasks:
main.yml

ansible/defaults/templates:
modprobe_blacklist.j2  policy-rc.d.conf.j2  policy-rc.d.j2  securetty.j2  sysctl_base.j2

ansible/defaults/vars:
main.yml

The main.yml in the tasks sub-directory is used to configure the required post installation steps.

Example:

---
- name: Ensure default packages
  apt:
    name: "{{ default_packages }}"
    state: latest

- name: Ensure latest version of all packages
  apt:
    upgrade: dist
    force_apt_get: yes
    dpkg_options: 'force-confold,force-confdef'
    autoremove: yes
    autoclean: yes

- name: Update python alternatives (for Log Analytics)
  shell: |
    update-alternatives --remove-all python
    update-alternatives --install /usr/bin/python python /usr/bin/python2 1

- name: Forward Syslog-NG logs to Log Analytics
  ansible.builtin.copy:
    src: static_files/syslog-ng-lad.conf
    dest: /etc/syslog-ng/conf.d/syslog-ng-lad.conf
    owner: root
    group: root
    mode: '0644'

...

The YAML configuration hierarchy could be defined as the following file structure:

• common.yaml: Default settings no matter which role or host. These can be overridden in all the below.
• my_environment01.yaml: Environment specific configuration (example: development, staging, production, amsterdam, az01, az04, …). These can be overridden in all the below.
• common/roles/some_server_role.yaml: A server role, or type definition, which contains role specific configuration parameters. The roles could implement an extra hierarchy as for instance:
  
  • debian::databases::postgres
  • debian::databases::postgres::timescale
  • debian::databases::postgres::timescale::prometheus
  • debian::loadbalancer::internal
  • debian::application::request_processor
    
    The hierarchy steps are divided by :: in the above example, and need to be inherited accordingly, each with their own YAML file.
    These can be overridden in all the below.
• my_environment01/roles/some_server_role.yaml: override role configuration parameters per environment.
  These can even be overridden on host level below.
• my_environment01/hosts/my_hostname01.yaml: set host specific configuration parameters. This file is actually always required and should contain at least the IP address of the node and the server role string.

Let’s take the following example: The host vmazdbprm01 has the role debian::databases::postgres::timescale::prometheus and is deployed in the environment in my_cool_location03. The configuration management should search for parameters in the following file locations (and first verify if the file path exists):

1. common.yaml
2. my_cool_location01.yaml
3. common/roles/debian.yaml
4. common/roles/debian::databases.yaml
5. common/roles/debian::databases::postgres.yaml
6. common/roles/debian::databases::postgres::timescale.yaml
7. common/roles/debian::databases::postgres::timescale::prometheus.yaml
8. my_cool_location01/roles/debian.yaml
9. my_cool_location01/roles/debian::databases.yaml
10. my_cool_location01/roles/debian::databases::postgres.yaml
11. my_cool_location01/roles/debian::databases::postgres::timescale.yaml
12. my_cool_location01/roles/debian::databases::postgres::timescale::prometheus.yaml
13. my_cool_location01/hosts/vmazdbprm01.yaml

This means that any code which wants to implement the above configuration management, needs to verify if the above 13 files exists from top to bottom, and if yes loads the YAML file accordingly.

Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services.

Implementing the above YAML hierarchy in Terraform, could be done as follows:

locals {
  host_cfg             = yamldecode(fileexists("cfgmgmt/${var.environment}/hosts/${var.node}.yaml") ? file("cfgmgmt/${var.environment}/hosts/${var.node}.yaml") : "{server_role: debian}")

  roles_list           = split("::", local.host_cfg.server_role)
  all_roles_list       = [ for index in range(length(local.roles_list)): join("::",slice(local.roles_list, 0, index + 1))  ]

  common_cfg           = yamldecode(fileexists("cfgmgmt/common.yaml") ? file("cfgmgmt/common.yaml") : "{}")
  common_role_cfg_list = [ for file in local.all_roles_list:
      yamldecode(fileexists("cfgmgmt/common/roles/${file}.yaml") ? file("cfgmgmt/common/roles/${file}.yaml") : "{}" )]
    
  env_cfg              = yamldecode(fileexists("cfgmgmt/${var.environment}.yaml") ? file("cfgmgmt/${var.environment}.yaml") : "{}")
  env_role_cfg_list    = [ for file in local.all_roles_list:
      yamldecode(fileexists("cfgmgmt/${var.environment}/roles/${file}.yaml") ? file("cfgmgmt/${var.environment}/roles/${file}.yaml") : "{}") ]
        
  common_role_cfg_map  = merge(local.common_role_cfg_list...)
  env_role_cfg_map     = merge(local.env_role_cfg_list...)

  cfg                  = merge(local.common_cfg, local.env_cfg, local.common_role_cfg_map, local.env_role_cfg_map, local.host_cfg)
}

Let’s have a look what actually happens in the above code.

All YAML files are stored in a Git/ Hiera repository, accessible in the sub-directory cfgmgmt.

The code declares “local” variables by issuing the resource “locals“, starting from line 1.

Line 2 will check if a file called cfgmgmt/${var.environment}/hosts/${var.node}.yaml exists and if true, loads the YAML content as an map into the local variable host_cfg. If the file doesn’t exists, a default YAML code will be loaded. In theory, each node/ host must have a file defined as it should have at least data configuration such as:

• unique node host name
• IP address
• server role
• (optionally) VLAN/ subnet configuration
• …

Line 4 splits the server role string, stored in local.host_cfg.server_role, into a list, to build the server role hierarchy further below.

Line 5 creates a list of top level server roles which need to be imported too. Example: if the server role was set to debian::databases::postgres::timescale::prometheus , the list all_roles_list will contain the following elements:

• debian
• debian::databases
• debian::databases::postgres
• debian::databases::postgres::timescale
• debian::databases::postgres::timescale::prometheus

Line 7 loads the YAML content of common.yaml, if it exists.

Line 8 loops over the all_roles_list elements, created on line 5, and will load the YAML content of the server roles (if the file exists) into a list element. The result is a list called common_role_cfg_list.

Line 11 loads the general environment configuration YAML content (if it exists) into the local variable env_cfg.

Line 12 will do the same thing as line 8, but for environment specific roles. (for instance: when certain server roles have environment specific configuration parameters).

Lines 15 and 16 will merge the elements (which in theory are maps of YAML data) of the server role lists into one big map, in the order of the list. This allows that keys can be overridden. The expansion ... notation is explained at https://www.terraform.io/language/expressions/function-calls#expanding-function-arguments.

Finally on line 18, a local variable called cfg will be created, which merges the values of:

• local.common_cfg
• local.env_cfg
• local.common_role_cfg_map
• local.env_role_cfg_map
• local.host_cfg

By providing the environment name and the host/ node name to the above code (as var.environment and var.node ), all required configuration parameters can be loaded per node in Terraform, but since we’ve used a Git repository, this information can be loaded in any kind of automation tool (required is of course that each automation implements the same kind of hierarchy code).

After the default installation of Ubuntu 20.4-lts, the following packages are required to get started as a hypervisor:

apt install qemu-kvm libvirt-daemon bridge-utils virtinst libvirt-daemon-system virt-top libguestfs-tools libosinfo-bin qemu-system virt-manager qemu pm-utils

Once these are installed, the vnet_hosts module needs to be pre-loaded in /etc/modules:

echo vhost_net | tee -a /etc/modules
modprobe vhost_net

The hypervisor is now ready to start creating and deploying virtual machines.

In this article, Terraform will be used to manage the virtual machines in libvirtd. All example code snippets are available on Github, under https://github.com/insani4c/terraform-libvirt

Terraform has an excellent provider (dmacvicar/libvirtd) to manage the libvirtd nodes, which needs to be loaded and initialized:

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
    }
  }
}

provider "libvirt" {
  uri = "qemu+ssh://root@192.168.1.1/system"
}

In the above code snippet, Terraform will ensure the libvirt provider is loaded and it is configured to connect to the host with IP address 192.168.1.1 as the root user (this requires that a SSH public key is installed on the remote server, of the user who will be executing Terraform).

Next, the network will defined, where the virtual nodes will be deployed in.

resource "libvirt_network" "my_network" {
  name = "my_net"

  mode = "nat"

  addresses = ["10.1.2.0/24"]
  domain    = var.dns_domain

  autostart = true

  dhcp {
    enabled = false
  }

  dns {
    enabled = true

    local_only = false
    forwarders { address = "127.0.0.53" }

    hosts  {
        hostname = "host01"
        ip = "10.1.2.10"
      }
    hosts {
        hostname = "host02"
        ip = "10.1.2.20"
      }
  }  
}

The above code will ensure that a network of type NAT (to allow internal IPs, reachable from the hypervisor only) with network mask 10.1.2.0/24, will be created. It will ensure that DHCP is disabled and it will enable a DNS setup (the package dnsmasq must be installed) with two predefined hosts, host01 and host02.

Up next is the definition of the storage pools and volumes required for the virtual machines.

resource "libvirt_pool" "default" {
  name = "default"
  type = "dir"
  path = "/data/vms/cluster_storage"
}

resource "libvirt_volume" "local_install_image" {
  name   = var.local_install_image
  pool   = libvirt_pool.default.name
  source = var.os_img_url
  format = "qcow2"
}

The above defines the libvirt_pool, which basically configures the path on-disk for storing all sorts of volumes. Next it defines a volume called “local_install_image“, which will be used to set up the virtual machine as the volume will contain the “cloud image” for the installation. This volume requires two variables:

variable "os_img_url" {
  description = "URL to the OS image"
  default     = "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img"
}

variable "local_install_image" {
    description = "The name of the local install image"
    default     = "base-os-ubuntu-focal.qcow2"
}

Create a list of the existing virtual network subnets:

data azurerm_subnet "subnets" {
  count = length(data.azurerm_virtual_network.my_vnet.subnets)

  name                 = data.azurerm_virtual_network.my_vnet.subnets[count.index]
  virtual_network_name = var.vnet_name
  resource_group_name  = var.resource_group
}

locals {
  subnets = tomap({
      for snet in data.azurerm_subnet.subnets: snet.name => snet.id
  })
}

In the above example, we first loop over all subnet names, returned by data.azurerm_virtual_network.my_vnet.subnets, to create a list of Azure virtual network subnet objects.

Afterwards we create a locals map called subnets, which contains mapping like “subnet name points to subnet ID”.

Finally, when creating Azure network interfaces with an IP configuration, you can easily lookup the correct subnet ID based on the subnet name (which you might have configured per virtual machine)

resource "azurerm_network_interface" "my_nic" {
...
  ip_configuration {
    ...
    subnet_id = lookup(local.subnets, "my_subnet_name")
  }
...