Terraform codifies cloud, virtualization and many other APIs into declarative configuration files, allowing to define the infrastructure as code.

The combination of both is an excellent platform to manage resources in the Microsoft Azure cloud in an automated or semi-automated way.

Installation

Let’s start by installing Jenkins and Terraform. The official documentation at https://www.jenkins.io/doc/book/installing/linux/#debianubuntu describes how to install the Jenkins binaries on a Debian/ Ubuntu system, so we’ll not go further into details on that.

Once the required packages are installed as described above, go to http://localhost:8080/ to finalize the Jenkins setup. After installing the essential plugins required to run Jenkins, you should be directed to the following screen:

Next, install the required plugins.

Go the main dashboard and then click on “Manage Jenkins” -> “Manage Plugins“. In the “Available” tab, search for:

• Azure Credentials
• Terraform
• AnsiColor
• Git plugin

Check the checkbox of the required plugins and install them. Jenkins will restart at the end of the installation.

Pipeline Setup

After all plugins have been installed, the next step will be to add a “New Item” (see the menu on the right side).

Start by entering a valid name for the job and choose the type “Pipeline“. Hit “OK” to continue to the next screen.

Enter a description for the pipeline and scroll down to the “Pipeline” section and choose in the “Definition” section the option “Pipeline script from SCM“. Choose the SCM you want to use and fill in the repository URL of the code repository containing the Jenkinsfile pipeline file.

A bit further down it is possible to configure the SCM branch name to work on and the actual filename of the Jenkinsfile, which in most cases will be called just “Jenkinsfile“.

Once all options have been set, press “Save” to save the configuration.

Depending on whether the pipeline was created with or without parameters, either click on “Build” or “Build with Parameters” to start the pipeline.

Each build will appear in the “Build History” and contains links to for instance the “Console Output“, which contains the text output of all stages, steps and plugins executed by the pipeline.

Each build/ run contains information such as:

• Who started the pipeline
• The current SCM repository versions (git commit hash)
• An SCM log of the changes since the last build (git commit messages)
• Who approved the pipeline (if applicable)
• The console output of the plugins/ commands executed

The pipeline dashboard page, contains a stage view of pipeline runs, divided on the stages and their execution status.

Pipeline definitions

Finally, let’s have a look on how this pipeline file is actually built.

The above Jenkinsfile is divided in 4 sections:

• the agent section
• the tools section
• the environment section
• the stages section

The first 3 sections basically set some build options like:

• on which agent to run the pipeline (in this case, on “any” available node)
• which tools are required to be present on the build node
• which environment (shell) parameters should be set

The stages section contains the different stages and steps the pipeline is supposed to run.

Each stage, contains several steps separated per line:

In the above example, all steps are defined between the ‘steps { }’ block. Steps can be wrapped (enclosed between curly brackets), like in the above example line 15 calls the ansiColor() plugin to display a colorized output of the steps contained by it.

Next, the plugin dir() wraps the rest of the steps, which means that all enclosed steps will be executed in a specific directory.

In that specific directory, 3 plugins will be executed:

• git: line 17 in the above example will git clone the main branch of the supplied Github URL
• echo: line 19 will output some text
• sh: line 20 – 23 defines the shell commands to be executed

A complete example can be found at: https://github.com/insani4c/jenkins-terraform-datacenter-example/blob/main/JenkinsFile

To prevent this issue, three steps are required:

• change the order kernel modules get loaded by initrd
• disable the mdadm auto-assembly of disks when initrd is loaded
• generate a new initrd image

The initrd configuration files can be found at /etc/initramfs-tools.

First define the order of module loading during boot (ensure that multipath is loaded before mdadm). Edit the file /etc/initramfs-tools/modules to match:

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax: module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
libfc
megaraid_sas
scsi_dh_alua
scsi_transport_fc
dm_multipath
dm_service_time
multipath
md_mod

This starts by loading the “fiber channel libs” (libfc) first, followed by the MegaRAID modules, SCSI over fiber channel, multipath and mdadm.

Now copy the mdadm initramfs hook to /etc to override the default one:

cp /usr/share/initramfs-tools/hooks/mdadm /etc/initramfs-tools/hooks/

Edit the hook file in /etc/initramfs-tools/hooks/mdadm and add an ‘exit 0‘ (see line 10):

#!/bin/sh
#
# Copyright © 2006-2008 Martin F. Krafft <madduck@debian.org>,
#
2012 Michael Tokarev <mjt@tls.msk.ru>
# based on the scripts in the initramfs-tools package.
# released under the terms of the Artistic Licence.
#
set -eu
exit 0
PREREQ="udev"
prereqs()
{
echo "$PREREQ"
}

Disable automatic startup of the mdadm service:

rm /lib/systemd/system/mdadm.service
systemctl daemon-reload
systemctl disable mdadm.service

Finally, update the initrd images for all kernel versions installed:

update-initramfs -u -k all

Reboot the server to see if modules are now loaded in the correct order, and multipath is loaded and started before mdadm.

AppArmor will write messages to the kernel log (visible with either the dmesg command or in kernel.log if available) regarding its actions.

If your libvirt guests are not starting up or failing, have a look at dmesg. Example:

In the above example AppArmor has denied (apparmor="DENIED") read access (requested_mask=r) to the file /data/vms/cluster_storage/base-os-ubuntu-focal.qcow2. This blocks of course the startup guest machines we have previously created in the article: Terraform and libvirtd nodes.

To fix the issue, edit the file: /etc/apparmor.d/libvirt/TEMPLATE.qemu

By default it has the following content:

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
}

In order to allow libvirt to use the guest image files, change the content to (or add a similar line if your file path is different):

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /data/vms/cluster_storage/**.qcow2 rwk,
}

The added line (line 9) will allow read (r), write (w) and lock (k) access to all qcow2 files in the directory /data/vms/cluster_storage.

Once added, all libvirt guests will start up again without any (AppArmor) issues.

The script itself is pretty straight-forward: it creates a server object using IO::Socket::INET and attaches that socket to an AnyEvent IO eventloop. Furthermore, upon every connection, it will call the subroutine fork_call (which is in the AnyEvent::Util module) to open up a client socket.

 
#!/usr/bin/env perl 
use strict;
use warnings;
use utf8;

use IO::Socket::INET;
use AnyEvent;
use AnyEvent::Util;
$AnyEvent::Util::MAX_FORKS = 15;

my $handled = 0;
$|++;

my $server = IO::Socket::INET->new(
    'Proto'     => 'tcp',
    'LocalAddr' => 'localhost',
    'LocalPort' => 1234,
    'Listen'    => SOMAXCONN,
    'Reuse'     => 1,
) or die "can't setup server: $!\n";
print "Listening on localhost:1234\n";

my $cv = AnyEvent->condvar;
my $w; $w = AnyEvent->io(
        fh   => \*{ $server }, 
        poll => 'r', 
        cb   => sub { 
                   $handled++;
                   $cv->begin; 
                   fork_call &handle_connections, 
                             $server->accept, 
                             sub { 
                               my ($client) = @_ ;
                               print " - Client $client closed\n"
                             } 
                    }
);
$cv->recv;

#
# Subroutines
# 
sub handle_connections {
    my ($client) =  @_;

    my $host = $client->peerhost;
    print "[Accepted connection from $host]\n";

    print $client "Hi, you're client #$handled\n";
    chomp ( my $input = <$client> );
    my $output = reverse $input;
    print $client $output, "\n";
    print $client "Bye, bye.\n";

    $cv->end;
    return $host;
}

One way of creating a simple backup, is by using the tar command with the incremental option.

You will also want to encrypt these backup files and one easy to do this, is by using GPG keys.

The following example assumes you have logged on the FTP server before and that you have created a backup subdirectory.
It also uses the ncftp tool.

#!/bin/bash

DIRS="/etc /root /home /var/cache/bind /var/spool/postfix /var/www"
INCFILE="/root/.backup/inc_file"
CURRENT_DATE=$(date '+%Y%m%d')
DSTFILE="/data/backup/backup-$CURRENT_DATE.gpg"
GPGUSER="gpg@localhost"
HETZNERUSER=ftpuser
HETZNERPASS=secret
HETZNERSERVER=ftpuser.your-backup.de
HETZNERDIR=backup

tar cvzg $INCFILE -f - $DIRS | gpg -r $GPGUSER -e > $DSTFILE
ncftpput -DD -u $HETZNERUSER -p $HETZNERPASS $HETZNERSERVER $HETZNERDIR $DSTFILE

This article assumes that the VM has been installed with just one root partition, but it can be applied to expand any kind of partition. If you are resizing a non-root partition, you do not need to reboot with a rescue disc. Unmounting that partition will be enough, but since this is impossible for the root partition, we will need to boot with a rescue disc.

The first screenshot shows the actual partition size, in this example the partition is called /dev/mapper/debian-root and is currently 7.3G. /dev/mapper/debian-root is actually just a mapper name (alias), the real LVM partition name is /dev/debian/root.

Shutdown the VM using the ‘shutdown -r now‘ or ‘init 0‘ command.
Then start your VMWare Player and hit the ‘Edit virtual machine settings‘ button.

Choose the ‘Hard Disk‘ option, then hit the ‘Utilities‘ button and afterwards ‘Expand‘.

Resize the partition, in this example the new partition size is set to 12G (initialy 8G). Then hit the ‘Expand‘ button. A popup will follow after a succesful resize.

Now go to the ‘CD/DVD‘ section and choose an ISO to boot from, in this example the original Debian ISO is used.

Press the ‘OK‘ button and boot your VM.
During boot, hit the ‘ESC’ button to go to the VMWare BIOS boot menu and choose to boot from ‘CD-ROM Drive’.

When the Debian boot loader appears, go to ‘Advanced Options‘ and choose ‘Rescue Mode‘ and then hit ‘Enter‘.

Go through the questions but at the end (you can answer with default questions, since we won’t be saving this settings. The only thing that is important is to choose to right kind of keyboard), choose not to use a root filesystem and finally execute a shell.

First a new partition needs to be created using the ‘fdisk‘ command. Since we will be adding a real partition to our LVM setup, we need to execute ‘fdisk /dev/sda‘.

A reboot is required to enable the new partition table, so you will need to boot the ISO again (hit the ‘ESC‘ button during boot to get the ‘Boot menu‘) and boot into ‘Rescue Mode‘ mode. Don’t enable a root filesystem and open a shell.

Once the shell is back, execute ‘pvdisplay‘ to find out the ‘VG Name‘.
Now set up a new PV using the ‘pvcreate‘ command and extend the VG with the ‘vgextend‘ command.

Use the ‘pvdisplay‘ command again to figure out the added size of the PV and then finally extend the LV.
Once the LV is extended, ‘e2fsck‘ needs to be run to clean up the filesystem and ‘resize2fs‘ to expand the filesystem.

The resize process is complete. Reboot the VM to see your changes in your Linux OS.