chudnick.com chudnick.com en-us https://chudnick.com/rss.xml LUKS Block Device Encryption https://www.chudnick.com/articles/luks.html https://www.chudnick.com/articles/luks.html Sat, 18 Jun 2022 15:46:21 -0400

LUKS Block Device Encryption

Linux Unified Key Setup (LUKS) is a method for encrypting block devices that is built-in to the Linux Kernel. In this tutorial I will be showing how to create an encrypted USB drive with LUKS, but this process is applicable to other types of storage. This process will wipe all data on the device that you encrypt so make backups beforehand if needed.

Install Packages

Install cryptsetup and its dependencies

apt install cryptsetup

Prepare the Drive

If the device you are encrypting was previously used, you will want to completely overwrite any data on it before encryption. This can be done using dd as shown below. This could take a very long time depending on the size and type of your drive. Be sure that you enter the correct path to your drive as the argument to of, and do not specify a partition: /dev/sdb is correct, /dev/sdb1 is wrong. Obviously this will destroy any data on the device so be absolutely sure you specify the correct device in the command and have backups of data previously stored if needed.

dd if=/dev/zero of=/dev/sdX status=progress

Encrypt the Drive

cryptsetup is the main command used to perform LUKS-related tasks. This main command is followed by a subcommand that specifies which action to perform. To create a LUKS partition on a drive we need to use the subcommand luksFormat. We also use the type option to explicitly say to use LUKS version 2 encryption. Run the command and then enter and confirm the encryption password.

cryptsetup --type luks2 luksFormat /dev/sdX

Now we need to open and map the encrypted LUKS device. This is done by using the luksOpen subcommand. This subcommand takes the device as the first argument and then a map target as the second. The map target can be any string as long as a map does not already exist with that name. In the example I will use crypt. 

cryptsetup luksOpen /dev/sdX crypt

And then we just need to create a filesystem on the device; I will be making an ext4 filesystem in the example. To interact with an encrypted drive after it has been opened you need to refer to its device mapper target, which is found under /dev/mapper. Since we used crypt as the map target, our encrypted drive is /dev/mapper/crypt. 

mkfs.ext4 /dev/mapper/crypt

After creating the filesystem created the encrypted drive can now be mounted and used like any other device. Remember that when mounting the device you refer to the device mapper target (mount /dev/mapper/crypt /mnt/crypt).

In addition to unmounting the filesystem you should always close the device mapping before removing your encrypted drive. This is done with the luksClose subcommand and takes the device mapping as the argument.

cryptsetup luksClose /dev/mapper/crypt

The LUKS Header

The LUKS header sits at the front of your encrypted drive and is responsible for managing access to the device. If the header is damaged accessing your encrypted data will not be possible. Because of this, you may want to make backups of the header, but before doing so you need to weigh the risks and benefits.

The obvious benefit is that in the event of damage to the LUKS header, you can easily restore a backup and regain access to the data. However, having a backup of the header makes it harder to wipe the LUKS device. Without a header backup, overwriting the LUKS header on the device is enough to securely wipe the drive. With backups, you need to either destroy all header backups or overwrite all encrypted data on the device. The other main risk is that an encryption password that is valid at the time you make a backup will always be valid for that backup. For example, say you change the encryption password on your device after making a backup because it has been compromised. An attacker would be able to restore the header backup and then use the compromised password to access the data.

The cryptsetup documentation refers to creating header backups as making a trade-off between safety and security. You need to make a decision based on your individual use-case. It's a good idea to routinely update your header backups if you do choose to make them

When creating or restoring a header backup you always refer to the block device in the command, not the device mapper taget. To create a header backup:

cryptsetup luksHeaderBackup /dev/sdX --header-backup-file path/to/backup

To restore a header from a backup:

cryptsetup luksHeaderRestore /dev/sdX --header-backup-file path/to/backup

Note that is not necessary to store header backups on an encrypted device. I would recommend storing backups on at least one non-encrypted drive in case of an emergency.

Consider donating if this article was useful. [BTC]

]]> PAM OATH Two Factor Authentication https://www.chudnick.com/articles/pam-tfa.html https://www.chudnick.com/articles/pam-tfa.html Sat, 18 Jun 2022 15:45:01 -0400

PAM OATH Two Factor Authentication

In this article we are going to look at configuring two factor authentication via PAM using OATH. This is a simple and private way to increase the security of your systems. Even if you are not familiar with the term, it is likely that you have used OATH before. OATH (specifically TOTP) is the rotating 6 digit code that you get from scanning a QR code when setting up 2FA on an account.

This example will show how to configure 2FA for SSH logins to a server, but can easily be generalized to cover other programs or even all authentication on a system. The two factors here will be public key authentication and then the OATH/TOTP code. It is highly recommended that you remain SSHd into your server until after testing to avoid locking yourself out in the event of a configuration error.

Install Packages

You only need to install a single package on the server side.

apt install libpam-oath

On the client machine that will be SSHing to the server install these two packages.

apt install oathtool qrencode

Configure OATH

Create the OATH configuration file /etc/users.oath. This file will contain the OATH secret keys so permissions need to be set to only allow the root user to view it.

touch /etc/users.oath
chown root: /etc/users.oath
chmod 600 /etc/users.oath

Generate a secret key for the TOTP. Treat this secret key as you would your SSH or GPG private key. Anyone who has this key will be able to generate the code needed to authenticate.

openssl rand -hex 10

Now we define the TOTP configuration for our user. If you were setting this up for multiple users you would make one entry per line. Open /etc/users.oath and add this line. user is the username of the account you will SSH into. Replace the long string of numbers and letters with the secret key you just generated.

HOTP/T30/6 user - 00112233445566aabbcc

Configure PAM

Now we need to tell PAM to use OATH to authenticate sshd. Do that by opening /etc/pam.d/sshd and adding the following line to the top of the file.

auth sufficient pam_oath.so usersfile=/etc/users.oath window=30 digits=6

This tells PAM to consider a valid 6 digit code as fully authenticated and to skip any other processing that may normally occur, such as requesting a password.

Configure SSHD

We need to make a few changes to the sshd configuration to allow OATH to work properly. Open the sshd configuration file at /etc/ssh/sshd_config and make the following changes.

AuthenticationMethods publickey,keyboard-interactive
PubkeyAuthentication yes
PasswordAuthentication no
ChallengeResponseAuthentication yes
UsePAM yes

The AuthenticationMethods line specifically tells sshd that a user needs to both have an authorized SSH key and know the proper 6 digit code to login.

Restart sshd to apply the changes

systemctl restart sshd

Test the Changes

From your client ssh into your server as normal. Instead of connecting as you have been, you should now see a prompt for your one time password. You can use oathtool to get the code. Again, replace the long string of numbers and letters with the secret key you generated on the server.

oathtool --totp -d6 00112233445566aabbcc

Enter that 6 digit code into the prompt and you will be logged into your server.

Now, in the unlikely event that your SSH private key is stolen, an attacker still won't be able to access your server!

Managing your TOTP

You probably don't want to run the oathtool command everytime you need your code, and while you could make an alias, that would require storing your secret key in plaintext. Here are some better options.

  • pass otp is an extension to the command-line password manager pass for handling TOTP. Use this if you are already using pass
  • KeePassXC is a graphical password manager that can manage TOTP
  • Gnome Authenticator is a graphical TOTP manager for the GNOME desktop environment

You may also want to generate a QR code for easy setup on another device. Rerun the same oathtool command as before with the -v flag to get the base32 version of your secret key.

oathtool --totp -v -d6 00112233445566aabbcc
--------------------------------
Hex secret: 00112233445566aabbcc
Base32 secret: AAISEM2EKVTKVO6M
Digits: 6
Window size: 0
TOTP mode: SHA1
Step size (seconds): 30

Then use qrencode to generate the QR code image.

qrencode -o totp.png 'otpauth://totp/user@server?secret=AAISEM2EKVTKVO6M'

Consider donating if this article was useful. [BTC]

]]> FreeIPA Server Setup https://www.chudnick.com/articles/freeipa-server.html https://www.chudnick.com/articles/freeipa-server.html Sat, 18 Jun 2022 15:44:21 -0400

FreeIPA Server Setup

FreeIPA is a centralized idenity management solution developed by Redhat. It is in my opinion the most functional libre alternative to Microsoft's Active Directory. Like AD, FreeIPA integrates all of the pieces needed to setup a domain including LDAP, Kerberos, a Certificate Authority, and much more.

I will be using Fedora 35 in this tutorial. As of Debian 11, the FreeIPA server is still not in the Debian repos. You will need either a Fedora or a RHEL machine. A CentOS fork may work also but I have not tested that.

FreeIPA in an Enterprise

For readers exploring the use of FreeIPA in a business environment, note that FreeIPA documentation explicitly states that it is not a replacement for Active Directory. I have not personally tried to join a Windows computer to a FreeIPA domain, and so I can't speak to how well that would work. FreeIPA would also not be able to push out policy to Windows machines as is done with Group Policy. FreeIPA is though able to create inter-domain trusts with an existing AD infrastructure.

The Case for FreeIPA at Home

Using a full Kerberos and LDAP identity management server may seem like overkill at home. And if you only have a single computer then it probably is. But scaling up even slightly, to perhaps a small family each with their own computer, will make having FreeIPA advantageous (your family is all using Linux, right?). This will be especially apparent if you are hosting your own services. If you are for instance hosting a Jellyfin media server that everyone in your family accesses, you won't want them to juggle separate passwords for Jellyfin when you could just have them use the same password they do on the computer. This single/same sign-on capability is one of the most practically useful aspects of FreeIPA.

Install Packages

We start as usual by installing the required packages.

dnf install freeipa-server freeipa-dns

Set Hostname

The server will need to have a fully qualified hostname before setting up IPA. You will need both a hostname for the server itself and the domain name you will want for the FreeIPA domain. I will be using ipaserver.myhome.local, where ipaserver is the hostname and myhome.local is the domain name.

hostnamectl set-hostname ipaserver.home.local

We'll also need to add a hosts file entry to /etc/hosts. Open that file in an editor and add a new line with the IP of the server, the fully qualified name, and the hostname.

192.168.1.10 ipaserver.myhome.local ipaserver

Make sure to reboot the server before continuing to complete the hostname change.

Firewall Configuration

We'll need to allow several ports for FreeIPA to function properly. Fedora 35 uses firewalld by default but I am going to disable that in favor of UFW here.

#Install UFW
dnf install ufw
# Stop and disable firewalld
systemctl disable --now firewalld
# Configure UFW
ufw enable
ufw allow ssh
ufw allow dns
ufw allow 88 comment kerberos
ufw allow 389 comment ldap
ufw allow 443 comment webui
ufw allow 636 comment ldaps
ufw default deny incoming
ufw reload

Configure FreeIPA

Now we can run the FreeIPA setup script. This is an interactive but mostly automatic process that will configure all of the IPA components. The --mkhomedir flag will configure the server to create home directories for IPA users on their first login and would otherwise have to be done manually.

ipa-server-install --mkhomedir

That command will bring you into the install script. You will be prompted several times before the bulk of the configuration happens. Default values are show in brackets after the prompt. Let's run through those prompts.

Do you want to configure integrated DNS (BIND)?: yes

Sever host name: the default value should be showing ipaserver.myhome.local which is what we want. Simply hit enter to acecpt the default.

Please confirm the domain name: The default here should be correct myhome.local so hit enter to accept that.

Please provide a realm name: This should just be the domain name in all uppercase. If the default looks correct just hit enter.

Directory Manager password: This is the password for an administrator account used by system services. You will not need this for daily use so I recommend setting it to a long randomly generated string. I have found myself that using an extremely long password here will cause the installation to fail. A password under 40 characters should be safe.

IPA admin password: This is the password for your initial admin user. Make this a strong password as this user has full admin rights for the entire domain.

Do you want to configure DNS forwarders: This allows you to configure the IPA server to forward DNS requests to another DNS server for zones it is not authoratitve for. The DNS server is configured by default as a recursive DNS server so answering no does not prevent internet access. If you have another DNS server that should be used instead then answer yes and provide the IP address when prompted.

Do you want to configure chrony with NTP server or pool address? : Here you can configure a custom NTP server or pool for the NTP daemon chrony. If you already have an NTP server on your network answer yes and provide its IP. If you want to leave the deafult chrony configuration then answer no. Time synchronization is very important in Kerberos so you should consider how you want to achieve that on your network. If you do not have an NTP server you may want to configure the IPA server as one later.

Continue to configure the system with these values?: This is a final confirmation before the script takes over and configures the IPA components. Review the information printed and enter yes if it all looks correct.

The install script will now run through configuration. This process usually takes several minutes. When finished you should get a message saying The ipa-server-install command was successful.

To finish, run this command to receive a Kerberos TGT. Provide the password for the admin user when prompted.

kinit admin

Accessing the Web Interface

You are now able to manage FreeIPA through the web interface. You can browse either to the IP or the hostname if your DNS is configured correctly. You should see a screen similar to this.

FreeIPA Login Screen

Login with the username admin and the password you set during the insallation. You are now ready to begin configuring your IPA domain.

Consider donating if this article was useful. [BTC]

]]> Linux Software RAID https://www.chudnick.com/kb/mdadm-raid.html https://www.chudnick.com/kb/mdadm-raid.html Sun, 05 Jun 2022 15:19:11 -0400

Linux Software RAID

mdadm is a tool that allows for creation and management of software RAID arrays on Linux. Creating an array is a rather straightforward process.

Install packages

The only package needed is mdadm itself

apt install mdadm

Partition disks

We'll need to parition the disks to be used in the array before creating it. This isn't anything complicated, we will just be creating a single partition using all the space on each disk.

Use lsblk to get a list of disks attached to your system.

lsblk command

Then use fdisk to edit the partition table of the first disk. In my case this would be /dev/sdb. Be sure that you are selecting the correct disk as selecting the wrong one can result in data being lost.

fdisk /dev/sdb

Use g to create a new GUID partition table. Use n to create a new partition, and then just press enter at all of the prompts to accept the defaults. Finally use w to write the changes. You can use lsblk again to verify the change and you should see that /dev/sdb now has a partition /dev/sdb1.

fdisk command

Repeat this process for your other disks before continuing.

Create the array

Creating the array is done with a single command, but takes just a bit of planning.

  • Define an ID for the array, which is just a number identifier. I use 0 in the command.
  • Determine the RAID level you want to use. I am going to use RAID5 in this example. The argument to --level is the number of the RAID level
  • Determine the devices that will be used in the array. Somewhat contrary to the argument name, these devices will actually be the partitions of the disks and not the disks themselves. In the command, give the number of partitions (3) and then a space-separated list of the partitions that will be active in the array (/dev/sdb1 /dev/sdc1 /dev/sdd1).
  • If you want to have any spare devices in the array you will define them with the --spare-devices argument. These are defined in the exact same way as the active RAID devices. In the example I use 1 spare /dev/sde1. Spare devices are hot-spares that will automatically be inserted into the array if one of the disks fails.
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --spare-devices=1 /dev/sde1

After creating the array run the following command to get details and the status of the array. It will take a bit to initialize the array, you will know this is done when the state is clean. You do not need to wait for the array to completely intialize to continue.

mdadm --detail /dev/md0

Take note of the UUID and name values as we will need them in the next step.

Before moving on we need to make a filesystem on the array. I'm going to make a simple EXT4 filesystem here.

mkfs.ext4 /dev/md0

Configuration Files

Open the mdadm configuration file at /etc/mdadm/mdadm.conf and append this line. Replace uuid and name with the values you got when running mdadm --detail, replace 0 whatever ID you chose.

ARRAY /dev/md0 metadata=1.2 UUID=uuid name=name

Optionally, create an /etc/fstab entry for automounting of the array. Replace /mnt/raid with the directory where you want to mount the array. If you made a filesystem other than ext4 make sure to change that value.

/dev/md0 /mnt/raid ext4 defaults 0 1

Consider donating if this article was useful. [BTC]

]]> Integrating InfluxDB and Icinga https://www.chudnick.com/kb/icinga-influx.html https://www.chudnick.com/kb/icinga-influx.html Tue, 31 May 2022 06:22:43 -0400

Integrating InfluxDB and Icinga

Icinga2 has built-in support for writing monitoring data to InfluxDB. This makes Icinga quite extensible as it allows for other programs to read the gathered data from InfluxDB. For example, Icinga does not have built-in graphing support. But monitoring data can be written to InfluxDB and then consumed by a dedicated graphing tool like Grafana.

Install Packages

Let's install a few necessary packages

apt install influxdb influxdb-client ssl-cert

Generate self-signed certificate

Now generate a self-signed certificate for accessing InfluxDB over TLS

make-ssl-cert generate-default-snakeoil
usermod -aG ssl-cert influxdb

Create Database and Users

Start and enable the InfluxDB service

systemctl enable --now influxdb

Now we'll create our database and users. We'll be creating 3 users with different access rights. An admin user with full control over the database, a user with write access that Icinga will use to write data to the database, and a read-only user to allow external programs to read data from the database.

influx -ssl -unsafeSsl -execute "create database icinga2; create user admin with password 'changeme'; create user icingauser with password 'changeme'; create user readonly with password 'changeme'; grant all to admin; grant write on icinga2 to icingauser; grant read on icinga2 to readonly;"

Configuration Files

Then right InfluxDB's configuration file at /etc/influxdb/influxdb.conf

reporting-enabled = false
[meta]
  dir = "/var/lib/influxdb/meta"
[data]
  dir = "/var/lib/influxdb/data"
  wal-dir = "/var/lib/influxdb/wal"
[coordinator]
[retention]
[shard-precreation]
[monitor]
[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = true
  https-enabled = true
  https-certificate = "/etc/ssl/certs/ssl-cert-snakeoil.pem"
  https-private-key = "/etc/ssl/private/ssl-cert-snakeoil.key"
[ifql]
[logging]
[subscriber]
[[graphite]]
[[collectd]]
[[opentsdb]]
[[udp]]
[continuous_queries]
[tls]
  min-version = "tls1.2"

And restart InfluxDB to pickup the changes

systemctl restart influxdb

Then we need to configure Icinga to write data to our database. Start by enabling the influxdb feature in Icinga

icinga2 feature enable influxdb

Now we tell Icinga how to write to our database. Open the configuration file at /etc/icinga2/features-available/influxdb.conf and replace with the following

object InfluxdbWriter \"influxdb\" {
  host = "127.0.0.1"
  port = 8086
  username = "icingauser"
  password = "icinga_password"
  ssl_enable = true
  database = "icinga2"
  flush_threshold = 1024
  flush_interval = 10s
  host_template = {
  measurement = "$host.check_command$"
    tags = {
      hostname = "$host.name$"
    }
  }
  service_template = {
    measurement = "$service.check_command$"
    tags = {
      hostname = "$host.name$"
      service = "$service.name$"
    }
  }
}

And finally restart Icinga to make those changes live.

systemctl restart icinga2

Icinga2 will now be writing the data it collects to your InfluxDB instance.

Consider donating if this article was useful. [BTC]

]]> Postfix and Dovecot Mail Server https://www.chudnick.com/kb/mail-server.html https://www.chudnick.com/kb/mail-server.html Mon, 30 May 2022 08:03:44 -0400

Postfix and Dovecot Mail Server

Postfix and dovecot will be the two primary pieces of our mail sever. Postfix is the mail transport agent that handles the sending and receiving of mail and dovecot is the IMAP server that will allow us to access our mail from a mail client such as mutt. The server will also have several other supporting components, a complete list of which is:

  • SpamAssassin for spam filtering
  • OpenDKIM for DKIM verification and signing
  • Postgrey for greylisting
  • Policyd-SPF for SPF verification
  • OpenDMARC for DMARC verification

You can use this script I have written to automate this process, but I would recommend that you run through the tutorial first to understand what is being done.

Please note that this tutorial is loosely intended for small personal mail servers. Using PAM for authentication, as is done here, is not a scalable solution for working with a large number of users. I do plan on covering Dovecot LDAP authentication at some point which would be a better solution in an enterprise setting.

Install Packages

Let's start by installing the required packages. Note that if you already have Apache installed on the server, replace python3-certbot-nginx with python3-certbot-apache.

apt install postfix dovecot-imapd dovecot-sieve opendkim opendkim-tools spamassassin gnupg postgrey postfix-policyd-spf-python opendmarc dbconfig-no-thanks certbot python3-certbot-nginx

During the installation of Postfix you will get a Debconf prompt in which you need to select "Internet Site" and then provide your domain name, example.com.

Get a certificate

Now we'll use Certbot to get a certificate for our server. If you are using Apache replace nginx with apache2.

systemctl stop nginx
certbot certonly --standalone -d mail.example.com 
systemctl start nginx

Postfix Main Configuration

In this section we will be doing the bulk of the postfix configuration. The postconf command used throughout appends (or changes) the specified configuration item in /etc/postfix/main.cf

Network Configuration

Let's start by configuring some network and domain information.

postconf -e "myorigin = example.com"
postconf -e "mydestination = \$myhostname, \$mydomain, localhost"
postconf -e "mynetworks = 127.0.0.0/8 [::1]/128"
postconf -e "myhostname = mail.example.com"

Next, point postfix to the cerbot key and certificate, as well as the distro's CA certificates.

postconf -e "smtpd_tls_key_file=/etc/letsencrypt/live/mail.example.com/privkey.pem"
postconf -e "smtpd_tls_cert_file=/etc/letsencrypt/live/mail.example.com/fullchain.pem"
postconf -e "smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt"

Harden the TLS configuration by forcing strong protocols and ciphers, and requiring that authentication occur only over an encrypted session.

# Require authentication over TLS and optionally use it for sending and receiving mail
postconf -e "smtpd_tls_auth_only = yes"
postconf -e "smtpd_tls_security_level = may"
postconf -e "smtp_tls_security_level = may"

# Force the use of TLSv1.2 or TLSv1.3
postconf -e 'smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1'
postconf -e 'smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1'
postconf -e 'smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1'
postconf -e 'smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1'

# Prefer server ciphers
postconf -e "tls_preempt_cipherlist = yes"

# Force strong ciphers
postconf -e "smtpd_tls_ciphers = high"
postconf -e "smtpd_tls_mandatory_ciphers = high"
postconf -e "smtp_tls_ciphers = high"
postconf -e "smtp_tls_mandatory_ciphers = high"
postconf -e "smtpd_tls_exclude_ciphers = aNULL, eNULL, EXP, LOW, MEDIUM, PSK, SRP, SHA1, kRSA, CAMELLIA, ARIA, DSS, RSA+AES, ADH, AECDH"
postconf -e "smtp_tls_exclude_ciphers = aNULL, eNULL, EXP, LOW, MEDIUM, PSK, SRP, SHA1, kRSA, CAMELLIA, ARIA, DSS, RSA+AES, ADH, AECDH"

Local Recipients and Aliases

Here we configure the bulk of the postfix built-in security settings which are structured as a series of access restrictions. Do not edit these settings without first reading the Postfix documentation as an incorrect change could inadvertently make your server an open relay.

postconf -e "smtpd_helo_required = yes"
postconf -e "smtpd_sender_login_maps = proxy:hash:/etc/postfix/login_maps"
postconf -e "smtpd_helo_restrictions = reject_unknown_helo_hostname, reject_non_fqdn_helo_hostname" 
postconf -e "smtpd_sender_restrictions = reject_sender_login_mismatch, reject_non_fqdn_sender, reject_unknown_sender_domain"
postconf -e "smtpd_recipient_restrictions = reject_non_fqdn_recipient, reject_unknown_recipient_domain, permit_sasl_authenticated, reject_unauth_destination, check_policy_service unix:private/postgrey, check_policy_service unix:private/policyd-spf, reject_rbl_client zen.spamhaus.org"
postconf -e "smtpd_relay_restrictions = permit_sasl_authenticated, reject_unauth_destination"
postconf -e "smtpd_data_restrictions = reject_unauth_pipelining"

# Disable VRFY command to prevent harvesting of user accounts on system
postconf -e "disable_vrfy_command = yes"

# Change smptd banner (hide distribution)
postconf -e "smtpd_banner = \$myhostname ESMTP \$mail_name"

Now, configure the local mail recipients and some aliases. We'll create an account called mailadmin to receive mail addressed to several other accounts. This is to keep administrative mail separate, but you can certainly alias these to your main account later if you would prefer to see it there.

# Set a custom local_recipient_maps here in order to avoid accepting mail for all local accounts
postconf -e "local_recipient_maps = proxy:hash:/etc/postfix/local_maps \$alias_maps"

# You will need to manually set a password later to login as mailadmin
adduser --disabled-login --shell /usr/sbin/nologin --gecos "" mailadmin
echo "# postfix aliases
postmaster:     mailadmin
root:           mailadmin
dmarc:          mailadmin
" > /etc/aliases

# Update address databases
echo "mailadmin@mail.example.com    mailadmin" > /etc/postfix/login_maps
echo "mailadmin mailadmin" > /etc/postfix/local_maps
newaliases
postmap /etc/postfix/login_maps
postmap /etc/postfix/local_maps

Mail Delivery

These commands configure our mail delivery preferences. Mail will be delivered inside a user's home folder with a maildir-style mailbox using dovecot.

# Maildir delivery to $HOME/Mail/Inbox/
postconf -e "home_mailbox = Mail/Inbox/"
# Deliver mail with Dovecot
postconf -e "mailbox_command = /usr/lib/dovecot/deliver"

Header and Body Checks

Header and body checks allow for some simple content filtering within Postfix. This is done by scanning a message line by line for a configured regex string, nothing more. For example, the first header check listed will reject a message with an attachment of ransomware.exe but will not block it if sent with no extension. This is mostly a protection against uneducated users and poorly written mail clients. Other checks block vulnerabilities and improve privacy.

Create a new file /etc/postfix/header_checks, then open it in a text editor and add the following

# Block files with common executable extensions
/name=[^>]*\.(exe|pif|com|dll|vbs|bat|sh|bash|so|zip|tar|gz|cpio)/ REJECT

# Block message/partial vulnerability
/message\/partial/ REJECT

# Remove Received string that is created when spamassassin reinjects message into postfix
# This is to prevent leaking the userid of the spamassassin user
/^Received:.*userid.*/  IGNORE

# Remove User-Agent strings from headers
/^User-Agent: .*/       IGNORE

Create another new file /etc/postfix/body_checks, and add this

# Block messages with iframes
/<iframe/ REJECT" > /etc/postfix/body_checks

And then run these commands to point postfix to the check files.

postconf -e "header_checks = regexp:/etc/postfix/header_checks"
postconf -e "body_checks = regexp:/etc/postfix/body_checks"

Postfix Master Configuration

SMTP client

This simple command configures the SMTP client process that is responsible for sending your mail to other mail servers.

postconf -M "smtp/unix=smtp unix - - y - - smtp"

Postscreen and SMTP Recipient

Postscreen is a kind of firewall that sits in front of the Postfix SMTPD process and receives all incoming traffic. Postscreen will drop connections from IPs on a DNS blacklst, or from clients that violate the SMTP protocol by speaking out of turn or sending non-SMTP commands. This adds up to less spam connections and therefore a much lighter workload for your server.

postconf -M "smtp/inet=smtp inet n - y - 1 postscreen"
postconf -M "smtpd/pass=smtpd pass - - y - - smtpd"
postconf -P "smtpd/pass/content_filter=spamassassin"
postconf -M "tlsproxy/unix=tlsproxy unix - - y - 0 tlsproxy"
postconf -M "dnsblog/unix=dnsblog unix - - y - 0 dnsblog"
postconf -e "postscreen_dnsbl_sites = zen.spamhaus.org"
postconf -e "postscreen_dnsbl_action = enforce"
postconf -e "postscreen_greet_action = enforce"

Submission over TLS (submissions)

Submission over TLS (aka submissions) is the process you will use to submit mail to your server from a mail client. These commands configure submissions to use a fully-encrypted session, as opposed to STARTTLS, and to only allow access to authenticated clients.

postconf -M "submissions/inet=submissions inet n - y - - smtpd"
postconf -P "submissions/inet/smtpd_tls_wrappermode=yes"
postconf -P "submissions/inet/smtpd_tls_security_level=encrypt"
postconf -P "submissions/inet/smtpd_tls_auth_only=yes"
postconf -P "submissions/inet/smtpd_sasl_auth_enable=yes"
postconf -P "submissions/inet/smtpd_client_restrictions=permit_sasl_authenticated,reject"
postconf -P "submissions/inet/smtpd_helo_restrictions="
postconf -P "submissions/inet/smtpd_sender_restrictions=reject_sender_login_mismatch"
postconf -P "submissions/inet/smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject"
postconf -P "submissions/inet/syslog_name=postfix/submissions"
postconf -P 'submissions/inet/smtpd_tls_protocols=!SSLv2,!SSLv3,!TLSv1,!TLSv1.1'
postconf -P 'submissions/inet/smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3,!TLSv1,!TLSv1.1'

OPTIONAL - submission with mandatory STARTTLS

Having configured submission over TLS on port 465 this step is optional. STARTTLS is considered by some to be less secure than full-session TLS and may be vulnerable to exploitation.

postconf -M "submission/inet=submission inet n - y - - smtpd"
postconf -P "submission/inet/smtpd_tls_security_level=encrypt"
postconf -P 'submission/inet/smtpd_tls_protocols=!SSLv2,!SSLv3,!TLSv1,!TLSv1.1'
postconf -P 'submission/inet/smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3,!TLSv1,!TLSv1.1'
postconf -P "submission/inet/smtpd_sasl_auth_enable=yes"
postconf -P "submission/inet/smtpd_tls_auth_only=yes"
postconf -P "submission/inet/syslog_name=postfix/submission"
postconf -P "submission/inet/smtpd_helo_restrictions="
postconf -P "submission/inet/smtpd_client_restrictions=permit_sasl_authenticated,reject"
postconf -P "submission/inet/smtpd_helo_restrictions="
postconf -P "submission/inet/smtpd_sender_restrictions=reject_sender_login_mismatch"
postconf -P "submission/inet/smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject"

SpamAssassin Configuration

Finally, this command tells Postfix how to interact with SpamAssassin.

postconf -M "spamassassin/unix=spamassassin unix - n n - - pipe user=debian-spamd argv=/usr/bin/spamc --socket=/var/run/spamd.sock -e /usr/sbin/sendmail -oi -f \${sender} \${recipient}"

Dovecot Configuration

Dovecot configuration is usually split up into many different files under /etc/dovecot/conf.d/ but here will be doing all of the configuration in the primary config file /etc/dovecot/dovecot.conf. Open that file with your editor of choice, clear all of its contents, and then replace it with the following.

# /etc/dovecot/conf.d/10-auth.conf
disable_plaintext_auth = yes
auth_username_format = %n
auth_mechanisms = plain
userdb {
	driver = passwd
}
passdb {
	driver = pam
}

# /etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:~/Mail:INBOX=~/Mail/Inbox:LAYOUT=fs
namespace inbox {
	type = private
	prefix = 
	separator = /
	inbox = yes
	subscriptions = yes
	list = yes
}

# /etc/dovecot/conf.d/10-master.conf
service imap-login {
# Run login processes in high-security mode (see: LoginProcess.txt in dovecot docs)
service_count = 1
# Disable unencrypted IMAP by setting port for plain IMAP to 0
	inet_listener imap {
		port = 0
	}
	inet_listener imaps {
		port = 993
		ssl = yes
	}
}

# Allow postfix to use dovecot SASL
service auth {
	unix_listener /var/spool/postfix/private/auth {
		mode = 0660
		user = postfix
		group = postfix
	}
}

# /etc/dovecot/conf.d/10-ssl.conf
ssl = required
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = </usr/share/dovecot/dh.pem

# Mozilla intermediate compatibility
ssl_min_protocol = TLSv1.2
ssl_cipher_list = ECDHE+ECDSA+AESGCM:ECDHE+aRSA+AESGCM:ECDHE+ECDSA+CHACHA20:ECDHE+aRSA+CHACHA20:DHE+aRSA+AESGCM:!aNULL:!eNULL

ssl_prefer_server_ciphers = yes
ssl_client_require_valid_cert = yes


# /etc/dovecot/conf.d/15-lda.conf
protocol lda {
	mail_plugins = \$mail_plugins sieve
}

# /etc/dovecot/conf.d/15-mailboxes.conf
namespace inbox {
	mailbox Sent {
			special_use = \Sent
			auto = subscribe
	}
	mailbox Trash {
			special_use = \Trash
			auto = create
			autoexpunge = 30d
	}
	mailbox Drafts {
			special_use = \Drafts
			auto = subscribe
	}
	mailbox Spam {
			special_use = \Junk
			auto = create
			autoexpunge = 30d
	}
	mailbox Archive {
			special_use = \Archive
			auto = create
	}
}

# /etc/dovecot/conf.d/20-imap.conf
imap_capability = +SPECIAL-USE

# /etc/dovecot/conf.d/90-sieve.conf
plugin {
	sieve = ~/.dovecot.sieve
	sieve_default = /var/lib/dovecot/sieve/default.sieve
	sieve_global = /var/lib/dovecot/sieve/
}

Then create the default sieve filtering script at /var/lib/dovecot/sieve/default.sieve

require ["fileinto", "mailbox"];
/*
* Discard mail that has a spam score greater than or equal to 10
*/
if header :contains "X-Spam-Level" "**********" {
	discard;
	stop;
}
/*
* Discard messages marked as infected by a virus scanner
*/
if header :contains "X-Virus-Scan" "infected" {
	discard;
	stop;
}
/*
* If message is marked as spam (and falls below discard threshold) put into spam mailbox
*/
if header :contains "X-Spam-Flag" "YES" {
	fileinto "Spam";
}

And compile the script

sievec /var/lib/dovecot/sieve/default.sieve

Finally, configure PAM authentication for dovecot at /etc/pam.d/dovecot. Append these changes leaving any include statements intact.

auth    required        pam_unix.so
account required        pam_unix.so

OpenDKIM

DKIM is a mail-verification method that cryptographically signs mail to allow receivers to verify the authenticity of the sender. Our mail server will use DKIM to validate signatures on incoming mail and sign outgoing mail. DKIM requires a public key to be published via DNS, which will be done near the end of the guide.

Start by generating the DKIM key

opendkim-genkey -D /etc/dkimkeys -d example.com -s mail
chown opendkim: /etc/dkimkeys/*
chmod 600 /etc/dkimkeys/*
mv /etc/dkimkeys/mail.private /etc/dkimkeys/mail.pem

Here we make a directory for the opendkim socket inside the postfix chroot and make it accessible to the postfix user.

mkdir /var/spool/postfix/opendkim
chmod 770 /var/spool/postfix/opendkim
chown opendkim:opendkim /var/spool/postfix/opendkim
usermod -aG opendkim postfix

Edit the configuration file at /etc/opendkim.conf to be as follows:

On-BadSignature		reject
On-Security		reject
Syslog			yes
SyslogSuccess		yes
LogResults		yes
Canonicalization		simple
Mode			sv
OversignHeaders		From
Domain			example.com
Selector			mail
KeyFile			/etc/dkimkeys/mail.pem     
UserID			opendkim
UMask			007
Socket			local:/var/spool/postfix/opendkim/opendkim.sock
PidFile			/run/opendkim/opendkim.pid
TemporaryDirectory		/run/opendkim
InternalHosts		127.0.0.1
TrustAnchorFile		/usr/share/dns/root.key
RequireSafeKeys		True
AlwaysAddARHeader		True

OpenDMARC

DMARC is another mail-verification technology that provides verification of the address seen by end-users and either or both of SPF and DKIM.

Like with OpenDKIM, we need to make a directory inside the postfix chroot for the socket and assign proper permissions.

mkdir /var/spool/postfix/opendmarc
chmod 770 /var/spool/postfix/opendmarc
chown opendmarc:opendmarc /var/spool/postfix/opendmarc
usermod -aG opendmarc postfix

Now we write the configuration file at /etc/opendmarc.conf

PidFile 		/run/opendmarc/opendmarc.pid
PublicSuffixList	/usr/share/publicsuffix/public_suffix_list.dat
RejectFailures  	True
Socket  		local:/var/spool/postfix/opendmarc/opendmarc.sock
Syslog  		True
SyslogFacility  	mail
UMask   		002
UserID  		opendmarc
HistoryFile	/var/run/opendmarc/opendmarc.hist
SPFIgnoreResults	True
SPFSelfValidate	True

Then create the history file and set permissions.

touch /var/run/opendmarc/opendmarc.hist
chown opendmarc:opendmarc /var/run/opendmarc/opendmarc.hist
chmod 664 /var/run/opendmarc/opendmarc.hist

Now that both OpenDKIM and OpenDMARC are configured we can define them as milters in postfix. This will tell postfix to route mail through one or both of these milters depending on whether it is incoming or outgoing.

postconf -P "smtpd/pass/smtpd_milters=unix:opendkim/opendkim.sock,unix:opendmarc/opendmarc.sock"
postconf -P "submissions/inet/smtpd_milters=unix:opendkim/opendkim.sock"
# If you enabled submission on port 587 run this too
postconf -P "submission/inet/smtpd_milters=unix:opendkim/opendkim.sock"

Postgrey

Postgrey implements a spam-filter technique known as greylisting, which always rejects mail on the first try and for a period of time afterwards known as the greylist period. The idea behind this being that legitimate senders will send the mail again later, while spammers, in a rush to send as many messages as possible before being blacklisted, will not.

Postgrey ships with an extensive whitelist domains that are known to cause issues (mainly large providers that constantly send from different addresses). This whitelist file is located at /etc/postgrey/whitelist_clients and can be appended to include any domain you do not wish to be subject to greylisting.

The configuration needed here is minimal, just open /etc/default/postgrey and make these changes

POSTGREY_OPTS="--unix=/var/spool/postfix/private/postgrey --privacy"
POSTGREY_TEXT="Greylisted - see https://www.greylisting.org"

And then enable the service

systemctl enable --now postgrey

Policyd-SPF

SPF is yet another mail-verification technology that uses DNS records to delegate specific servers as being authorized to send mail for the domain (and implicitly all other servers as unauthorized). Policyd-SPF will perform SPF checking of received mail and reject mail that fails SPF verfication.

First, tell postfix how to access Policyd-SPF

postconf -e "policyd-spf_time_limit = 3600"
postconf -M "policyd-spf/unix=policyd-spf unix - n n - 0 spawn user=policyd-spf argv=/usr/bin/policyd-spf"

And then edit the configuration file at /etc/postfix-policyd-spf-python/policyd-spf.conf

debugLevel = 1
TestOnly = 1
HELO_reject = Fail
Mail_From_reject = Fail
Header_Type = AR
# These settings increase false-positive risk
# Comment them if you want to reduce that risk
PermError_reject = True
TempError_Defer = True

SpamAssassin

SpamAssassin is a spam-filter that will scan all received mail and assign a spam score based on configured rules. SpamAssassin is much heavier and more resource-intensive than any of the previous spam-filtering/verification programs we have configured. The postfix spam-filtering philosophy emphasizes the use of lightweight checks before passing to an external content filter such as SpamAssassin. Ideally, non-legitimate mail will have already been caught by one of the previous methods, and SpamAssassin will only have to operate on a much smaller subset of the mail that is sent to our server.

We have actually already told postfix to use SpamAssassin as a content filter so in this section we just need to edit the configuration file /etc/spamassassin/local.cf.

# Clearly indicate message is spam to user
rewrite_header Subject *****SPAM*****
rewrite_header From *****SPAM*****

# Set required score to be marked as spam, 5.0 is default.
# Lower to make policy more strict or raise to be more lenient.
required_score 5.0

# Attach original messages as text/plain instead of message/rfc822 to spam reports
report_safe 2

Do not implicitly trust mail based on IP address except localhost
trusted_networks       127.0.0.1/32

And finally make a few changes to the defaults file at /etc/default/spamassassin

OPTIONS="--listen /var/run/spamd.sock --max-children 5"
PIDFILE=/var/run/spamd.pid
CRON=1

Wrapping Up

At this point we have done all of the necessary configuration of the mail server programs. We have just a few more minor tasks before your mail server is operational.

Configure Firewall

We need to open the proper ports in the firewall. This example uses UFW.

ufw allow 25 comment "smtp"
ufw allow 465 comment "submission over TLS"
# Run this next command only if you enabled submission on port 587
ufw allow 587 comment "mail submission"
ufw allow 993 comment "IMAP over TLS"
ufw reload

Restart services

Now let's restart the services to pick up any configuration changes.

systemctl restart postfix
systemctl restart dovecot
systemctl restart opendkim
systemctl restart opendmarc
systemctl enable --now spamassassin
systemctl restart spamassassin
systemctl restart postgrey

DNS Entries

Finally, we needs to set some required DNS records to enable mail flow and verification. Begin by logging into your registrar or DNS host and editing your DNS records.

A Record

If you did not set a wildcard A record earlier, you will need to set one now for mail. Alternatively, if you are running the mail server on the same server as your website, you may want to instead make a CNAME record pointing mail to www.

MX Record

MX records tell servers attempting to send you mail where to send it. Open the MX records section on your registrar and add a new record. An MX record consists of a priority and a destination. Set the priority to 10 and the destination to mail, or whatever your subdomain for this mail server is. The host value can be left blank or may need to be set to "@" depending on your registrar.

DKIM TXT Record

Now we will set the three TXT records we need. Open the TXT records tab on your registrar.

We'll set the DKIM record first. The command we ran to generate our DKIM keys also generates a DNS record for us which will be helpful here. Print that to the screen with:

cat /etc/dkimkeys/mail.txt

You should get a lengthy output that looks something like the following. The bolded portion is the value.

mail._domainkey	IN	TXT	( "v=DKIM1; h=sha256; k=rsa; "
	  "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAz50PSYL0Ob+OlF/0B77rwlzLe7zF6JKnxQNtMqcOCZ0Dar2FPhSUSz1FR0YmNuoShjMogdgKeojIzgRUqwK5GZ5Lz456qiXWkfAtLPc6UQ/WPoyEBGbJpRBYPGWdN4VoNcHkk/I4csvXW6MOI55ghPOwDmootPkCzNPR6gmNAXMe0duS4Lb+bIjy9QMOxGYVUaQ/b+7xar+fWw"
	  "bA3DjQa3jTLCydzzJpjEMfVaKqNhQ4N+ve7O2Mb3LF5k5B977mtok/6POjVG5HY8g6Pba+GzMFItR6nJO5EE2fyfv6cNbRLsZiM+WQmqvDBst5ejaeapy86F5PdJFlX/TUgXjtuwIDAQAB" )  ; ----- DKIM key mail for example.com

You can cleanup the spacing of the value as your registrar should automatically handle any needed splitting of the record. The parts you need to paste into your registrar's web interface should then look like this.

# Name/Host 
mail._domainkey
# TXT Value
"v=DKIM1; h=sha256; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAz50PSYL0Ob+OlF/0B77rwlzLe7zF6JKnxQNtMqcOCZ0Dar2FPhSUSz1FR0YmNuoShjMogdgKeojIzgRUqwK5GZ5Lz456qiXWkfAtLPc6UQ/WPoyEBGbJpRBYPGWdN4VoNcHkk/I4csvXW6MOI55ghPOwDmootPkCzNPR6gmNAXMe0duS4Lb+bIjy9QMOxGYVUaQ/b+7xar+fWwbA3DjQa3jTLCydzzJpjEMfVaKqNhQ4N+ve7O2Mb3LF5k5B977mtok/6POjVG5HY8g6Pba+GzMFItR6nJO5EE2fyfv6cNbRLsZiM+WQmqvDBst5ejaeapy86F5PdJFlX/TUgXjtuwIDAQAB"

DMARC TXT Record

The DMARC record should be as follows:

# Name/Host 
_dmarc
# Value 
"v=DMARC1; p=reject; rua=mailto:dmarc@example.com; fo=1"

SPF Record

Your SPF record will look like this. Remember to replace mail.example.com with your server name.

# Name/Host 
@
# Value
"v=spf1 a:mail.example.com -all"

PTR Record

Many mail servers rely on PTR records for verification purposes so we need to make sure our server's IP address resolves to the proper domain name. If your mail server is residing on a VPS, you will need to add this record on your VPS provider's interface, consult their documentation for details.

Creating your own Mail User

Your mail server is now up and running. Let's create an email for you to receive mail.

useradd --shell /usr/sbin/nologin --create-home --user-group user
echo "user@example.com	user" >> /etc/postfix/login_maps
echo "user	user" >> /etc/postfix/local_maps
postmap /etc/postfix/login_maps
postmap /etc/postfix/local_maps
postfix reload

I have a script available for adding and removing users that you can find here.

Connecting From a Mail Client

When connecting your account to a mail client you need to use these settings.

  • Username: user@example.com
  • Password: the password for user@example.com
  • Server name: mail.example.com
  • IMAP Port: 993
  • IMAP Connection: SSL/TLS
  • SMTP Port: 465
  • SMTP Connection Type: SSL/TLS

Consider donating if this article was useful. [BTC]

]]> Icinga2 Agent Configuration https://www.chudnick.com/kb/icinga-agent.html https://www.chudnick.com/kb/icinga-agent.html Mon, 30 May 2022 08:03:44 -0400

Icinga Agent Node Installation and Configuration

With the Icinga master node configured, the servers we want to monitor can now be added as agent nodes. As the names suggest, the Icinga master node pushes the desired configuration to agent nodes, while agent nodes report the configured status checks back to the master. Communication between the master and agent nodes is encrypted via TLS, with the master node acting as a certificate authority.

You can find my script to automate this process here.

Install Pakcages

Start by installing the required packages on the server to be monitored.

apt install icinga2 monitoring-plugins 
monitoring-plugins-contrib

Initialize PKI with master

Now we need to setup the PKI that will be used for the communication with the master node. The first step is to generate a certificate signing request. Replace hostname with the FQDN of the server.

icinga2 pki new-cert --cn "hostname" --cert "/etc/icinga2/pki/hostname.crt" --csr "/etc/icinga2/pki/hostname.csr" --key "/etc/icinga2/pki/hostname.key"

Next we save the master node's public key certificate. Replace master with the FQDN of your master node.

icinga2 pki save-cert --host "master" --port 5665 --key "/etc/icinga2/pki/hostname.key" --trustedcert "/etc/icinga2/pki/trusted-master.crt"

Receive signed certificate from the master node.

icinga2 pki request --host "master" --port 5665 --key "/etc/icinga2/pki/hostname.key" --cert "/etc/icinga2/pki/hostname.crt" --trustedcert "/etc/icinga2/pki/trusted-master.crt" --ca "/etc/icinga2/pki/ca.crt"

Deploy configuration files

Write Icinga configuration.

/etc/icinga2/icinga2.conf 
include "constants.conf"
const NodeName = "$nodename"
include "zones.conf"
include "features-enabled/*.conf"
include <itl>
include <plugins>
include <plugins-contrib>
include <manubulon>
include <windows-plugins>
include <nscp>"

Write zones configuration.

/etc/icinga2/zones.conf 
echo "object Endpoint "hostname" {}
object Zone "hostname" {
  parent = "master"
  endpoints = [ "hostname" ]
}
object Zone "master" {
  endpoints = [ "master" ]
}
object Endpoint "master" {
  host = "master"
}
object Zone "director-global" {
  global = true
}

Write API configuration file.

/etc/icinga2/features-available/api.conf 
echo "object ApiListener \"api\" {
  accept_commands = true
  accept_config = true
}

Enable API

Next, we need to enable the API on the agent.

icinga2 feature enable api

mkdir -p /var/lib/icinga2/certs

cp /etc/icinga2/pki/hostname.crt /etc/icinga2/pki/hostname.key /etc/icinga2/pki/ca.crt /var/lib/icinga2/certs/

chown -R nagios: /var/lib/icinga2/certs/

Sign agent CSR on Master

The only action needed on the master node is to sign the agent's CSR. Logon to your master node and run the following:

fpr="$(icinga2 ca list | tail -1 | cut -d '|' -f 1)"
icinga2 ca sign $fpr

Configure Firewall

Before finishing we need to open the proper firewall port. I will use UFW in the example here and allow traffic only only from the master node for best security.

ufw allow proto tcp from master-ip to any port 5665

Restart Icinga on Agent

Finally, restart the icinga service on the agent node.

systemctl restart icinga2

The Icinga agent node will now pull down configuration from the master. You will know that this worked if /var/lib/icinga2/api/zones begins to populate with new files.

Consider donating if this article was useful. [BTC]

]]> Icinga2 Director https://www.chudnick.com/kb/icinga-director.html https://www.chudnick.com/kb/icinga-director.html Mon, 30 May 2022 08:03:44 -0400

Icinga Director

Icinga Director is the web-based configuration tool for Icinga2. Director provides a simple interface for configuring the various parts of your monitoring environment. Even if you would rather do all of the configuration from a terminal, I still recommend using Director for its self-service API which allows new nodes to register with no interaction required on the node (i.e. via a script).

Start by logging into your Icinga instance. If you followed the master installation guide you should have a tab labeled Icinga Director. Click on that, and you should see something similar to this:

Hosts

Let's start by looking at the hosts section. Clicking on the hosts button from the Director menu will present you with several options. Click on host templates, and then click add to define our first host template. Host templates are the building blocks of Icinga, they allow for your nodes to be structured however you see fit, and then to have monitoring checks automatically applied to them based on that structure.

I like to structure my host templates by operating system. For example, I have a template called Linux Server, which is designed to encompass all of the Linux servers in my environment. I then get more specific, creating templates based on distro. These templates are children of the Linux Server template, so they inherit whatever is applied to the parent, but then can have distro-specific checks applied to themselves.

Start by giving your template a name - I will use Linux Server here. Groups can be left empty for now, but you may want to add groups and apply them to templates later. The check command should be set to hostalive. Expand Icinga Agent and zone settings and set Icinga2 Agent, Establish connection, and Accepts config to Yes. Click store to save the template. Your template should look like this:

Service templates

Let's turn to services now. Return to the Director menu and select Services. The services menu is structured similar to hosts, and we will start with the service templates section. The idea behind service templates is very similar to host templates. Typically, a service template corresponds to a single monitoring command.

As an example, we'll create a service template for a monitoring command that checks the status of a web server. Give the template a name, set the check command to http, and finally expand Icinga Agent and zone settings and set Run on agent to no. We set this to no because we want Icinga to query the web server externally instead of from the web server itself.

Service sets

Service sets are simply groups of service templates. They can be structured however you see fit. Service sets can then be applied to hosts/host templates to have the checks be automatically applied.

Add a new service set and give it a name. Then click on the services tab and add all of the services you want to group into that set. Here is an example of a service set Linux Standard that has service checks that should be applied to all Linux servers.

To bring service sets and host templates together, return to your host templates, select Linux Server, select the services tab, and then select add service set and choose your desired set from the dropdown menu.

Render your config

When you have made all of the changes you need you will need to render the Director configuration. Return to the Director menu, select Config Deployment, and then select Render config.

Self Service API

In this last section we will look at what I think is the best feature of Director which is the self service API. To enroll a host template in the self service API, select the host template, select the agent tab, and select generate self service api key. That's it! The string of letters and numbers is the API key associated with this host template. Hosts can be enrolled with this API key and Icinga will automatically assign the host to this template. With proper structuring, you can have hosts be completely provisioned without touching Director. In the next article, we will use this to enroll a host in Icinga with a shell script.

Consider donating if this article was useful. [BTC]

]]> Icinga2 Master Setup https://www.chudnick.com/kb/icinga-master.html https://www.chudnick.com/kb/icinga-master.html Mon, 30 May 2022 08:03:44 -0400

Icinga2 Master Installation

This tutorial will cover the installation of the Icinga2 monitoring application master node. This includes the base program, the web frontend, and the web-based configuration tool. This guide was made for Debian but should be similar on other distributions.

I have a script available to automate the steps described in this tutorial available from my git repo.

Install Packages

Here we will install the required packages. Icinga can use either MySQL or PostgreSQL, however this tutorial will use MySQL/MariaDB.

apt install icinga2 icingaweb2 icinga2-ido-mysql icingaweb2-module-director monitoring-plugins monitoring-plugins-contrib default-mysql-server

Secure MySQL

This step is optional but strongly recommended. The mysql_secure_installation script will harden your MySQL instance.

mysql_secure_installation

I recommend the following responses:

  • Switch to unix_socket authentication? Y
  • Change the root password? Y
  • Remove anonymous users? Y
  • Disallow root login remotely? Y
  • Remove the test database and access to it? Y
  • Reload privilege tables now? Y

Create Monitoring Database

The next several sections will cover creating databases for the various parts of Icinga. We'll start with the monitoring database. The following command creates a MySQL database named icinga2 and grants permissions to a user named ido_admin. These values are arbitrary, but I use them throughout the tutorial so I recommend leaving them as is. You should definitely change the password though, which in the command is change me. You will need this password and the passwords for the other databases later, so make sure you save them.

mysql -u root -e "CREATE DATABASE icinga2; GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON icinga2.* TO ido_admin@'localhost' IDENTIFIED BY 'change me'; FLUSH PRIVILEGES;

We then need to import the ido schema into the database.

mysql -u root icinga2 </usr/share/icinga2-ido-mysql/schema/mysql.sql

After importing the schema, we then write the configuration file that tells the monitoring module how to connect to the database.

/etc/icinga2/features-available/ido-mysql.conf
library "db_ido_mysql"
object IdoMysqlConnection "ido-mysql" {
	user = "ido_admin",
	password = "ido_password",
	host = "localhost",
	database = "icinga2"
}"

And finally we enable the monitoring module in Icinga.

icinga2 feature enable ido-mysql

Create Icingaweb2 Database

This step is nearly identical to the last. This time we create a database named icingaweb2 and grant permissions to the user named icingaweb2_admin.

mysql -u root -e "CREATE DATABASE icingaweb2;GRANT ALL ON icingaweb2.* TO 'icingaweb2_admin'@'localhost' IDENTIFIED BY 'changeme'; FLUSH PRIVILEGES;

Again we will need to import required schema into the database.

mysql -u root icingaweb2 </usr/share/icingawbe2/etc/schema/mysql.schema.sql

In this step we create the initial admin user that will be used to login to the web interface. As is, this would create a user named admin with the password changme. You should at least change the password.

passhash="$(php -r "echo password_hash(\"changeme\", PASSWORD_DEFAULT);")"
mysql -u root -e "USE icingaweb2; INSERT INTO icingaweb_user (name, active, password_hash) VALUES (\"admin\", 1, \"$passhash\"); FLUSH PRIVILEGES;"

Create Icinga Director Database

Here we create the database for Director. Director will require more configuration later, so for now we will just be creating the database.

mysql -u root -e "CREATE DATABASE director CHARACTER SET 'utf8'; GRANT ALL on director.* TO 'director'@'localhost' IDENTIFIED BY '$director_password';FLUSH PRIVILEGES;"

Setup Icinga2 API

Run the following command to initialize the Icinga API.

icinga2 api setup

And then restart Icinga to apply the changes.

systemctl restart icinga2

Configure Web Server

In this section we will configure the web server for accessing Icinga's web interface and Director configuration tool. This tutorial will use nginx but apache could be used as well. We'll start by installing the necessary packages.

apt install nginx php-fpm

Then we need to create the site configuration file.

/etc/nginx/sites-available/icingaweb2.conf
server {
  listen 80;
  server_name monitoring.example.com
  location ~ ^/icingaweb2/index\.php(.*)$ {
    fastcgi_pass unix:/var/run/php/php-fpm.sock;
    fastcgi_index index.php;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /usr/share/icingaweb2/public/index.php;
    fastcgi_param ICINGAWEB_CONFIGDIR /etc/icingaweb2;
    fastcgi_param REMOTE_USER $remote_user;
  }

  location ~ ^/icingaweb2(.+)? {
    alias /usr/share/icingaweb2/public;
    index index.php;
    try_files $1 $uri $uri/ /icingaweb2/index.php$is_args$args;
  }

  # Not strictly necessary but allows you to get to icinga without 
  # specifying /icingaweb2 in the URL.
  location = / {
    return 302 http://$host/icingaweb2;
  }

}

And then restart nginx to pick up the changes.

systemctl restart nginx

At this point we are done with the Icinga setup module and so we can disable it.

icingacli module disable setup

Write Configuration Files

In this section we will write several configuration files. Icinga uses the INI format for its web interface configuration files.

In this first file we tell Icinga about the various resources it should have access to. These resources are the three databases created previously. Replace the password in each section with the corresponding password you set for that database earlier.

/etc/icingaweb2/resources.ini
[icinga2]
type = "db"
db = "mysql"
host = "localhost"
port = ""
dbname = "icinga2"
username = "ido_admin"
password = "ido password"
charset = ""
use_ssl = "0"

[icingaweb2]
type = "db"
db = "mysql"
host = "localhost"
port = ""
dbname = "icingaweb2"
username = "icingaweb2_admin"
password = "ido password"
charset = ""
use_ssl = "0"


[director]
type = "db"
db = "mysql"
host = "localhost"
port = ""
dbname = "director"
username = "director"
password = "director password"
charset = "utf8"
use_ssl = "0"

This file controls the authentication settings for the web interface. Here we tell Icinga to look at the icingaweb2 database for authentication purposes.

/etc/icingaweb2/authentication.ini
[icingaweb2]
backend = "db"
resource = "icingaweb2"

Now we tell icinga which users should have admin permissions. If you changed the username value from admin previously, be sure to update it here.

/etc/icingaweb2/roles.ini
[admins]
users = "admin"
resource = "icingaweb2"

Enable the web interface monitoring module.

icingacli module enable monitoring

Then write the configuration file pointing the monitoring module to the monitoring database.

/etc/icingaweb2/modules/monitoring/backends.ini
[icinga]
type = "ido"
resource = "icinga2"

Here we configure Icinga to use the API for communication. You will need to get your unique API password generated during the API setup from from /etc/icinga2/conf.d/api-users.conf. hostname should be the FQDN of the server.

/etc/icingaweb2/modules/monitoring/commandtransports.ini
[icinga2]
transport = "api"
host = hostname
port = "5665"
username = "root"
password = "api password"

Lastly, tell Icinga to protect variables with potentially sensitive values.

/etc/icingaweb2/modules/monitoring/config.ini
[security]
protected_customvars = "*pw*,*pass*,*community*"

Configure Director

This section will cover configuring Director configuration tool.

Create Director module configuration directory.

mkdir -p /etc/icingaweb2/modules/director

Write the Director configuration file.

/etc/icingaweb2/modules/director/config.ini
[db]
resource = "director"

Enable Director module and run the initial migration.

icingacli module enable director
icingacli director migration run

Write Director kickstart configuration file.

/etc/icingaweb2/modules/director/kickstart.ini
[config]
endpoint = "hostname"
username = "root"
password = "api password"

Kickstart Director, then render and deploy the configuration.

icingacli director kickstart run
icingacli director config render
icingacli director config deploy

Director is setup at this point so we will shred the unneeded configuration file containing sensitive information.

shred -uz /etc/icingaweb2/modules/director/kickstart.ini

Login to your Monitoring Instance

You are now ready to login to your monitoring instance with the admin user created previously. Open a web browser and go to http://hostname/icingaweb2. You should see a screen similar to this:

Icinagweb2 Login Screen

Next Steps

In the following articles we will go through setting up Icinga2 agents on servers, and configure your monitoring instance through Icinga Director.

Consider donating if this article was useful. [BTC]

]]>