To prevent this issue, three steps are required:

• change the order kernel modules get loaded by initrd
• disable the mdadm auto-assembly of disks when initrd is loaded
• generate a new initrd image

The initrd configuration files can be found at /etc/initramfs-tools.

First define the order of module loading during boot (ensure that multipath is loaded before mdadm). Edit the file /etc/initramfs-tools/modules to match:

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax: module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
libfc
megaraid_sas
scsi_dh_alua
scsi_transport_fc
dm_multipath
dm_service_time
multipath
md_mod

This starts by loading the “fiber channel libs” (libfc) first, followed by the MegaRAID modules, SCSI over fiber channel, multipath and mdadm.

Now copy the mdadm initramfs hook to /etc to override the default one:

cp /usr/share/initramfs-tools/hooks/mdadm /etc/initramfs-tools/hooks/

Edit the hook file in /etc/initramfs-tools/hooks/mdadm and add an ‘exit 0‘ (see line 10):

#!/bin/sh
#
# Copyright © 2006-2008 Martin F. Krafft <madduck@debian.org>,
#
2012 Michael Tokarev <mjt@tls.msk.ru>
# based on the scripts in the initramfs-tools package.
# released under the terms of the Artistic Licence.
#
set -eu
exit 0
PREREQ="udev"
prereqs()
{
echo "$PREREQ"
}

Disable automatic startup of the mdadm service:

rm /lib/systemd/system/mdadm.service
systemctl daemon-reload
systemctl disable mdadm.service

Finally, update the initrd images for all kernel versions installed:

update-initramfs -u -k all

Reboot the server to see if modules are now loaded in the correct order, and multipath is loaded and started before mdadm.

The statistics part contains quite some interesting information for monitoring and alerting.

The below Perl code snippit will loop over a glob of socket files (for instance when you have multiple HAProxy configurations running as separate processes) and print out the values returned by the “show info” command.

use IO::Socket::UNIX;

foreach my $socket_file (glob("/run/haproxy/*.sock")){
    print "- Reading socket: $socket_file\n";
    my $client = IO::Socket::UNIX->new(
        Type => SOCK_STREAM(),
        Peer => $socket_file,
    );

    print "- show info\n";
    print $client "show info\n";
    my $header = <$client>;
    chomp($header);

    $header =~ s/^#\s+//;
    my @keys = split ',', $header;
    print "- header:$header\n";

    while (my $line = <$client>){
        next unless $line =~ /^.+/;

        chomp($line);
        my @values = split ',', $line;
        print " - Got $line\n";
        print "   $keys[$_]: ".($values[$_]//'')."\n" foreach 0..$#keys;
    }

    close $client;
}

A small example of how to use the framework:

my $exporter = Prometheus::Exporter->new({
    listen_port => 9090, 
    listen_addr => "127.0.0.1", 
    max_threads => 5,
});

$exporter->register_metrics({
    test_metric        => {type => "gauge",     desc => "A test metric"},
    test_metric_labels => {type => "gauge",     desc => "A test metric", labels => ["code=42", "code=99"]},
    test_counter       => {type => "counter",   desc => "A test metric"},
    test_histogram     => {type => "histogram", buckets => ['0.3', '0.6', '1.2', '+Inf']},
});

$exporter->register_collector(sub {
    my $timeout = int(rand(5));
    sleep $timeout;

    $exporter->get_metric("test_metric")->value(rand(100));
    $exporter->get_metric("test_metric_labels")->value([rand(42), rand(99)]);

    $test_counter += int(rand(20));
    $exporter->get_metric("test_counter")->value($test_counter);

    $histo_buckets{"0.3"}  += rand(20);
    $histo_buckets{"0.6"}  += $histo_buckets{"0.3"} + rand(20);
    $histo_buckets{"1.2"}  += $histo_buckets{"0.6"} + rand(20);
    $histo_buckets{"+Inf"} += $histo_buckets{"1.2"} + rand(20);
    my $histo_sum = 2.0 * $histo_buckets{"+Inf"};
    my $histo_count = $histo_buckets{"+Inf"};
    $exporter->get_metric("test_histogram")->value(\%histo_buckets, $histo_sum, $histo_count);
});

$exporter->run;

The framework will start a small HTTP daemon once run() is called and will handle all client requests by using threads. On each request, the framework will call the subroutine or coderef defined at register_collector(). Currently, that coderef must store the observed values by using the construct seen in the above example, by calling the value() method on the registered metric objects.

Also currently, the histogram implementation is not yet supporting labels.

AppArmor will write messages to the kernel log (visible with either the dmesg command or in kernel.log if available) regarding its actions.

If your libvirt guests are not starting up or failing, have a look at dmesg. Example:

In the above example AppArmor has denied (apparmor="DENIED") read access (requested_mask=r) to the file /data/vms/cluster_storage/base-os-ubuntu-focal.qcow2. This blocks of course the startup guest machines we have previously created in the article: Terraform and libvirtd nodes.

To fix the issue, edit the file: /etc/apparmor.d/libvirt/TEMPLATE.qemu

By default it has the following content:

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
}

In order to allow libvirt to use the guest image files, change the content to (or add a similar line if your file path is different):

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /data/vms/cluster_storage/**.qcow2 rwk,
}

The added line (line 9) will allow read (r), write (w) and lock (k) access to all qcow2 files in the directory /data/vms/cluster_storage.

Once added, all libvirt guests will start up again without any (AppArmor) issues.

In this article we will have a closer look how to ensure that

• a default database has been created
• a set of configured extensions have been installed
• an initial database schema has been deployed

The user that will log on to the remote host using Ansible (and SSH), must be able to become the postgres user using sudo without a password (in this example, can be changed of course).

The tasks below are split in two big tasks, both containing a block. Blocks allow to group certain tasks which might need the same variables (for instance the same tags, become variables, …).

The first block will ensure that the default database has been created, based on a YAML variable called {{ db_name }}. This variable must be set in either a group_vars file, host_vars file or supplied at the command line.

In the same block, Ansible will also ensure that the pg_stat_statements extension is enabled and installed in the above database. pg_stat_statements is a very useful extension to debug or display statistics regarding the SQL statements executed on that specific database.

And the end of the first block, a task was added to loop over a YAML variable called {{ shared_preload_extensions }}. The variable is a supposed to be a list and should contain all extra extensions required to enabled to the above database. Shared preload extensions also need to be configured in the postgresql.conf file, which is not covered in this article.

- name: Default Database
  tags: default_database
  vars:
    ansible_become_user: postgres
    ansible_become_method: sudo
    ansible_become_pass: null
  block:
    - name: Ensure required database
      postgresql_db:
        name: "{{ db_name }}"
        encoding: UTF-8

    - name: Ensure pg_stat_statements extension
      postgresql_ext:
        name: "pg_stat_statements"
        db: "{{ db_name }}"
        schema: public
        state: present

    - name: Ensure shared_preload extensions
      postgresql_ext:
        name: "{{ item }}"
        db: "{{ db_name }}"
        state: present
      loop: "{{ shared_preload_libraries.split(',') }}"
      loop_control:
        label: " {{ item }}"

- name: Ensure initial database schema
  tags: default_database
  block:
    - name: Schema definition file
      template:
        src: "{{ db_name }}/schema_definition.sql.j2"
        dest: /var/lib/postgresql/initial_schema_definition.sql
        owner: postgres
        group: postgres
        mode: 0600
      when: ( role_path + "/templates/" + db_name + "/schema_definition.sql.j2" ) is file
      register: initial_schema

    - name: Apply the schema definition
      ignore_errors: True
      vars:
        ansible_become_user: postgres
        ansible_become_method: sudo
        ansible_become_pass: null
      community.postgresql.postgresql_query:
        db: "{{ db_name }}"
        path_to_script: /var/lib/postgresql/initial_schema_definition.sql
        encoding: UTF-8
        as_single_query: yes
      when: initial_schema.changed

  rescue:
    - debug:
        msg: "No schema definition found for {{db_name}}, skipping..."

The second block of tasks will ensure that an initial database schema is deployed. This block consists of two tasks only.

The first task will upload the schema definition SQL-based file to a specific path on the remote host.

The second task will try to execute that SQL file, if it was changed by the first task. This means of course that if the file was changed manually on the remote host (or even removed), Ansible will update that file again in the first task and it will deploy the full file again in the second task! This might overwrite or break your database.

Monitoring the logs in real-time on the command line, can also be very useful when debugging either the rules themselves or when analyzing certain issues. Rather than just looking at the logs, in some situations it might be useful to track the rate of the log messages. A self-written Perl script can be useful as it allows to be flexible when it comes to:

• parsing logs
• formatting the output (with colors or tables or …)
• calculating statistics
• …

The following Perl script uses a few modules which need to be present:

use IO::Async::Timer::Periodic;
use IO::Async::Loop;
use Time::HiRes qw/time/;
use Term::ANSIColor qw(:constants);
use Getopt::Long;

The first two modules can be installed on Debian systems with:

apt install libio-async-perl

The others are part of the normal Perl packages and do not require any extra installation.

Next the script will use a polling mechanism to read from standard output at fixed intervals, to calculate the rate of the unique log lines. The default polling rate is set to 2 seconds but it can be managed through command line parameters:

my $last_poll_time = time;

my $poll_rate = 2;
GetOptions (
    'p|pollrate=i' => \$poll_rate,
);

my $loop = IO::Async::Loop->new;
my $timer = IO::Async::Timer::Periodic->new(
   interval => $poll_rate,
   on_tick  => \&log_rate
);

$timer->start;
$loop->add( $timer );
$loop->run;

Finally, the script will define a subroutine called log_rate, which will read from standard output (or even a file) at each poll interval. Important is of course that the log lines from standard output do not contain unique data such as timestamps. The output must be as generic as possible.

Example:

tail -qf /var/log/ulog/blocked_detailed.json /var/log/ulog/blocked.json /var/log/ulog/passed.json  | jq -r --unbuffered '."oob.prefix"' 
blocked: invalid state
blocked: invalid state
blocked: invalid state
blocked: invalid state
blocked: invalid state
action=blocked
action=blocked
action=blocked
action=blocked
action=blocked
action=passed
action=passed
action=passed
action=passed

The code snippit for log_rate could contain:

sub log_rate {
    local $SIG{ALRM} = sub { die time, " time exceeded to read STDIN\n" };

    alarm($poll_rate);
    my $h;
    eval {
        local $| = 1;
        while (my $line = <>) {
            chomp($line);
            $h->{$line}++;
        }
    };
    alarm(0);

    return unless keys %$h;

    my $delta_time = time - $last_poll_time;
    print DARK WHITE . sprintf("%d: ", time) . RESET;
    print( BOLD WHITE . $_ ." [" . GREEN . sprintf("%.2f/s", $h->{$_}/$delta_time) . BOLD WHITE "] | " . RESET) foreach keys %$h; 
    print "\n";

    $last_poll_time = time;
}

Line 2 will start with declaring the “ALARM” signal. This signal is called when the alarm timeout has been reached (see further below).

Line 4 defines the alarm timeout in seconds: meaning: if everything below line 4 (until the next alarm line) takes longer than the defined timeout in seconds, the “ALRM” signal handler defined at line 2 will be called, which basically stops the code execution with a die (which in theory should stop the script with an exit 1).

Line 5 defines a hash reference which is required down below, to temporarily store unique log lines.

Line 6 until 12 define an eval block. The eval block will catch the ALRM signal die (once reached) without stopping the script with an exit 1. Inside the eval block, the standard output will be read with the diamond operator (<>) and unique lines will be counted and stored in the $h hash reference.

Line 13, right after the eval block, sets to alarm timeout to 0 again, which means it is disabled. This allows that only execution of the eval block will be evaluated for timeout.

Line 15 ensures that only when log lines were discovered and stored in the temporary hash-ref $h, that rates will be printed to the screen.

The rest of the code will take care of printing the discovered log lines with their rates to the screen. Colors from Term::ANSIColor are used to make the output more vivid.

Example output:

The full version of the script can be found at: https://github.com/insani4c/perl_tools/tree/master/log_rate

The YAML configuration hierarchy could be defined as the following file structure:

• common.yaml: Default settings no matter which role or host. These can be overridden in all the below.
• my_environment01.yaml: Environment specific configuration (example: development, staging, production, amsterdam, az01, az04, …). These can be overridden in all the below.
• common/roles/some_server_role.yaml: A server role, or type definition, which contains role specific configuration parameters. The roles could implement an extra hierarchy as for instance:
  
  • debian::databases::postgres
  • debian::databases::postgres::timescale
  • debian::databases::postgres::timescale::prometheus
  • debian::loadbalancer::internal
  • debian::application::request_processor
    
    The hierarchy steps are divided by :: in the above example, and need to be inherited accordingly, each with their own YAML file.
    These can be overridden in all the below.
• my_environment01/roles/some_server_role.yaml: override role configuration parameters per environment.
  These can even be overridden on host level below.
• my_environment01/hosts/my_hostname01.yaml: set host specific configuration parameters. This file is actually always required and should contain at least the IP address of the node and the server role string.

Let’s take the following example: The host vmazdbprm01 has the role debian::databases::postgres::timescale::prometheus and is deployed in the environment in my_cool_location03. The configuration management should search for parameters in the following file locations (and first verify if the file path exists):

1. common.yaml
2. my_cool_location01.yaml
3. common/roles/debian.yaml
4. common/roles/debian::databases.yaml
5. common/roles/debian::databases::postgres.yaml
6. common/roles/debian::databases::postgres::timescale.yaml
7. common/roles/debian::databases::postgres::timescale::prometheus.yaml
8. my_cool_location01/roles/debian.yaml
9. my_cool_location01/roles/debian::databases.yaml
10. my_cool_location01/roles/debian::databases::postgres.yaml
11. my_cool_location01/roles/debian::databases::postgres::timescale.yaml
12. my_cool_location01/roles/debian::databases::postgres::timescale::prometheus.yaml
13. my_cool_location01/hosts/vmazdbprm01.yaml

This means that any code which wants to implement the above configuration management, needs to verify if the above 13 files exists from top to bottom, and if yes loads the YAML file accordingly.

Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services.

Implementing the above YAML hierarchy in Terraform, could be done as follows:

locals {
  host_cfg             = yamldecode(fileexists("cfgmgmt/${var.environment}/hosts/${var.node}.yaml") ? file("cfgmgmt/${var.environment}/hosts/${var.node}.yaml") : "{server_role: debian}")

  roles_list           = split("::", local.host_cfg.server_role)
  all_roles_list       = [ for index in range(length(local.roles_list)): join("::",slice(local.roles_list, 0, index + 1))  ]

  common_cfg           = yamldecode(fileexists("cfgmgmt/common.yaml") ? file("cfgmgmt/common.yaml") : "{}")
  common_role_cfg_list = [ for file in local.all_roles_list:
      yamldecode(fileexists("cfgmgmt/common/roles/${file}.yaml") ? file("cfgmgmt/common/roles/${file}.yaml") : "{}" )]
    
  env_cfg              = yamldecode(fileexists("cfgmgmt/${var.environment}.yaml") ? file("cfgmgmt/${var.environment}.yaml") : "{}")
  env_role_cfg_list    = [ for file in local.all_roles_list:
      yamldecode(fileexists("cfgmgmt/${var.environment}/roles/${file}.yaml") ? file("cfgmgmt/${var.environment}/roles/${file}.yaml") : "{}") ]
        
  common_role_cfg_map  = merge(local.common_role_cfg_list...)
  env_role_cfg_map     = merge(local.env_role_cfg_list...)

  cfg                  = merge(local.common_cfg, local.env_cfg, local.common_role_cfg_map, local.env_role_cfg_map, local.host_cfg)
}

Let’s have a look what actually happens in the above code.

All YAML files are stored in a Git/ Hiera repository, accessible in the sub-directory cfgmgmt.

The code declares “local” variables by issuing the resource “locals“, starting from line 1.

Line 2 will check if a file called cfgmgmt/${var.environment}/hosts/${var.node}.yaml exists and if true, loads the YAML content as an map into the local variable host_cfg. If the file doesn’t exists, a default YAML code will be loaded. In theory, each node/ host must have a file defined as it should have at least data configuration such as:

• unique node host name
• IP address
• server role
• (optionally) VLAN/ subnet configuration
• …

Line 4 splits the server role string, stored in local.host_cfg.server_role, into a list, to build the server role hierarchy further below.

Line 5 creates a list of top level server roles which need to be imported too. Example: if the server role was set to debian::databases::postgres::timescale::prometheus , the list all_roles_list will contain the following elements:

• debian
• debian::databases
• debian::databases::postgres
• debian::databases::postgres::timescale
• debian::databases::postgres::timescale::prometheus

Line 7 loads the YAML content of common.yaml, if it exists.

Line 8 loops over the all_roles_list elements, created on line 5, and will load the YAML content of the server roles (if the file exists) into a list element. The result is a list called common_role_cfg_list.

Line 11 loads the general environment configuration YAML content (if it exists) into the local variable env_cfg.

Line 12 will do the same thing as line 8, but for environment specific roles. (for instance: when certain server roles have environment specific configuration parameters).

Lines 15 and 16 will merge the elements (which in theory are maps of YAML data) of the server role lists into one big map, in the order of the list. This allows that keys can be overridden. The expansion ... notation is explained at https://www.terraform.io/language/expressions/function-calls#expanding-function-arguments.

Finally on line 18, a local variable called cfg will be created, which merges the values of:

• local.common_cfg
• local.env_cfg
• local.common_role_cfg_map
• local.env_role_cfg_map
• local.host_cfg

By providing the environment name and the host/ node name to the above code (as var.environment and var.node ), all required configuration parameters can be loaded per node in Terraform, but since we’ve used a Git repository, this information can be loaded in any kind of automation tool (required is of course that each automation implements the same kind of hierarchy code).

First, install the ulogd2 package (example is based on Debian/ Ubuntu):

apt install ulogd2

Example: log and drop packets which have an invalid state

# Log and drop all invalid packets                                                                                                                                                                                         
iptables -A INPUT -m conntrack --ctstate INVALID -j NFLOG --nflog-group 123 --nflog-prefix "packet with invalid state"
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

To log all those packets to a file in JSON format, ulogd2 requires the following configuration at /etc/ulogd.conf

[global]                                                                                                                                                                                                           
logfile="syslog"
loglevel=3                                                                                                                                                                                                                   
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_inppkt_NFLOG.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IFINDEX.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IP2STR.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IP2BIN.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_PRINTPKT.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_HWHDR.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_raw2packet_BASE.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_JSON.so"

stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,mac2str1:HWHDR,json1:JSON

[log1]
group=123

[json1]
sync=1
file="/var/log/ulog/netfilter_log.json"

After creating the configuration file, ensure that ulogd2 is restarted and that the directory /var/log/ulog exists

mkdir /var/log/ulog
chown ulog /var/log/ulog
systemctl restart ulogd2.service

Once the above created rule matches, a JSON log line will be written to disk:

tail -1 /var/log/ulog/netfilter_log.json | jq
{
  "timestamp": "2022-03-30T14:46:20.527282+0200",
  "dvc": "Netfilter",
  "raw.pktlen": 52,
  "raw.pktcount": 1,
  "oob.prefix": "packet with invalid state",
  "oob.time.sec": 1648644380,
  "oob.time.usec": 527282,
  "oob.mark": 0,
  "oob.ifindex_in": 2,
  "oob.hook": 1,
  "raw.mac_len": 14,
  "oob.family": 2,
  "oob.protocol": 2048,
  "raw.label": 0,
  "raw.type": 1,
  "raw.mac.addrlen": 6,
  "ip.protocol": 6,
  "ip.tos": 0,
  "ip.ttl": 116,
  "ip.totlen": 52,
  "ip.ihl": 5,
  "ip.csum": 41779,
  "ip.id": 16049,
  "ip.fragoff": 16384,
  "src_port": 58662,
  "dest_port": 445,
  "tcp.seq": 3872158206,
  "tcp.ackseq": 0,
  "tcp.window": 8192,
  "tcp.offset": 0,
  "tcp.reserved": 0,
  "tcp.urg": 0,
  "tcp.ack": 0,
  "tcp.psh": 0,
  "tcp.rst": 0,
  "tcp.syn": 1,
  "tcp.fin": 0,
  "tcp.res1": 0,
  "tcp.res2": 0,
  "tcp.csum": 60039,
  "oob.in": "eth0",
  "oob.out": "",
  "src_ip": "181.122.165.177",
  "dest_ip": "1.1.1.1",
  "mac.saddr.str": "94:f7:ad:4f:81:fc",
  "mac.daddr.str": "aa:aa:aa:aa:aa:aa",
  "mac.str": "aa:aa:aa:aa:aa:aa:94:f7:ad:4f:81:fc:08:00"
}

After the default installation of Ubuntu 20.4-lts, the following packages are required to get started as a hypervisor:

apt install qemu-kvm libvirt-daemon bridge-utils virtinst libvirt-daemon-system virt-top libguestfs-tools libosinfo-bin qemu-system virt-manager qemu pm-utils

Once these are installed, the vnet_hosts module needs to be pre-loaded in /etc/modules:

echo vhost_net | tee -a /etc/modules
modprobe vhost_net

The hypervisor is now ready to start creating and deploying virtual machines.

In this article, Terraform will be used to manage the virtual machines in libvirtd. All example code snippets are available on Github, under https://github.com/insani4c/terraform-libvirt

Terraform has an excellent provider (dmacvicar/libvirtd) to manage the libvirtd nodes, which needs to be loaded and initialized:

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
    }
  }
}

provider "libvirt" {
  uri = "qemu+ssh://root@192.168.1.1/system"
}

In the above code snippet, Terraform will ensure the libvirt provider is loaded and it is configured to connect to the host with IP address 192.168.1.1 as the root user (this requires that a SSH public key is installed on the remote server, of the user who will be executing Terraform).

Next, the network will defined, where the virtual nodes will be deployed in.

resource "libvirt_network" "my_network" {
  name = "my_net"

  mode = "nat"

  addresses = ["10.1.2.0/24"]
  domain    = var.dns_domain

  autostart = true

  dhcp {
    enabled = false
  }

  dns {
    enabled = true

    local_only = false
    forwarders { address = "127.0.0.53" }

    hosts  {
        hostname = "host01"
        ip = "10.1.2.10"
      }
    hosts {
        hostname = "host02"
        ip = "10.1.2.20"
      }
  }  
}

The above code will ensure that a network of type NAT (to allow internal IPs, reachable from the hypervisor only) with network mask 10.1.2.0/24, will be created. It will ensure that DHCP is disabled and it will enable a DNS setup (the package dnsmasq must be installed) with two predefined hosts, host01 and host02.

Up next is the definition of the storage pools and volumes required for the virtual machines.

resource "libvirt_pool" "default" {
  name = "default"
  type = "dir"
  path = "/data/vms/cluster_storage"
}

resource "libvirt_volume" "local_install_image" {
  name   = var.local_install_image
  pool   = libvirt_pool.default.name
  source = var.os_img_url
  format = "qcow2"
}

The above defines the libvirt_pool, which basically configures the path on-disk for storing all sorts of volumes. Next it defines a volume called “local_install_image“, which will be used to set up the virtual machine as the volume will contain the “cloud image” for the installation. This volume requires two variables:

variable "os_img_url" {
  description = "URL to the OS image"
  default     = "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img"
}

variable "local_install_image" {
    description = "The name of the local install image"
    default     = "base-os-ubuntu-focal.qcow2"
}

One simple (not full bullet-proof) way of doing this, is by setting up block rules on firewall level, which can be achieved on Linux servers with iptables and zone files of https://www.ipdeny.com/. These zone files contain the network ranges assigned to a specific country.

The network ranges are fed to a tool called ipset, which sets up of hash map that can be easily used within iptables rules.

On Debian/Ubuntu systems, ipset can be installed with the apt command:

apt install ipset

Next, create an iptables chain called “blocked_countries“, to which we will add rules afterwards. Add this chain to the beginning of the INPUT and FORWARD chain, to have early blocking in your ruleset.

iptables -N blocked_countries
iptables -I INPUT -j blocked_countries -m comment --comment "Blocked countries"
iptables -I FORWARD -j blocked_countries -m comment --comment "Blocked countries"

Finally, create a shell script which will download the required zone files from https://ipdeny.com/ and which feeds them to ipset. The example script will try to block the countries China, Russia and Belarus:

#!/bin/bash

COUNTRIES=('cn' 'ru' 'by')

for COUNTRY in "${COUNTRIES[@]}"; do
    ipset create "countries_${COUNTRY}" hash:net
done

iptables -v -F blocked_countries

for i in "${COUNTRIES[@]}"; do
    echo "Ban IP of country ${i}"
    ipset flush "countries_${i}"


    for IP in $(wget -O - https://www.ipdeny.com/ipblocks/data/countries/${i}.zone)
    do
        ipset add "countries_${i}" $IP
    done
    iptables -I blocked_countries   -m set --match-set "countries_${i}" src  -j LOGDROP -m comment   --comment "Block .${i}"
done


You can check the blocked_countries chain if packets are being blocked by your new rules:

iptables -v -n -L blocked_countries 
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain blocked_countries (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   104 LOGDROP    all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set countries_by src /* Block .by */
 2182  155K LOGDROP    all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set countries_ru src /* Block .ru */
  344 21370 LOGDROP    all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set countries_cn src /* Block .cn */