HAProxy provides a socket file which can be used to do maintenance (enable/ disable backends, retrieve information and statistics, …). The statistics part contains quite some interesting information for monitoring and alerting. The below Perl code snippit will loop over a glob of socket files (for instance when you have multiple HAProxy configurations running as…
I released a small project I wrote a while ago, to create quick Prometheus exporters in Perl for providing some custom data. The project itself can be found at https://github.com/insani4c/prometheus-exporter. Back then I decided not to use Net::Prometheus as I wanted to use HTTP::Daemon with threads and not Plack. A small example of how to…
With AppArmor enabled on Debian/ Ubuntu systems, starting up virtual machines with libvirt can cause startup failures if not AppArmor is not properly configured. AppArmor will write messages to the kernel log (visible with either the dmesg command or in kernel.log if available) regarding its actions. If your libvirt guests are not starting up or…
Ansible is a great automation tool to manage operating systems, but also to manage database like PostgreSQL. Many Ansible modules are available to create playbooks which execute various database administration tasks. In this article we will have a closer look how to ensure that a default database has been created a set of configured extensions…
In a previous article (IPTables Logging in JSON with NFLOG and ulogd2) we learned how to log certain IPTables rules to JSON log files. Monitoring the logs in real-time on the command line, can also be very useful when debugging either the rules themselves or when analyzing certain issues. Rather than just looking at the…
De-duplication of configuration information is key when managing large environments which use different types of automation (Terraform, Jenkins, Ansible, scripts executed as Systemd timers, Puppet…). Although many different configuration management tools exist (RDBMS, Consul, …), one of the easiest to use is Hiera or just a plain normal Git repository with YAML files, in some…
Logging with IPTables requires the use of an extra IPTables extension called NFLOG (https://manpages.debian.org/experimental/iptables/iptables-extensions.8.en.html#NFLOG) and a separate daemon process, called ulogd2 (https://www.netfilter.org/projects/ulogd/index.html). Ulogd2 reads out the packets sent to the above mentioned extension and stores them in local files or databases. First, install the ulogd2 package (example is based on Debian/ Ubuntu): Example: log and…
Libvirt (libvirtd) nodes (based on KVM and Qemu) are a great and cheap (read: free) alternative of deploying virtual nodes in a cloud. Required is a server which will act as a hypervisor, in our article we chose to use a Hetzner server installed with Ubuntu Linux 20.4-lts. After the default installation of Ubuntu 20.4-lts,…
Certain server setups do not require access for all countries or just want to block certain countries since they are know for their malicious activity. One simple (not full bullet-proof) way of doing this, is by setting up block rules on firewall level, which can be achieved on Linux servers with iptables and zone files…
Update: This article is updated thanks to Colin Keith his excellent comment. I was extremely inspired by it Maintaining a large number of servers cannot be done without proper programming skills. Each good system administrator must therefor make sure he knows how to automate his daily works. Although many many programming languages exist, most persons…
