Excerpt

Encrypted servers and workstations using Fedora or Red Hat Linux can be unlocked at boot time using Tang and Clevis. In this article I will show how the client setup done using Cockpit on Fedora 32.

Part one covers the installation and setup of Tang service. This is part two which covers the client setup for two different scenarios: Unlocking encrypted root volumes at boot time and unlocking encrypted volumes at the end of the boot process.

Content

  1. Configuration of Tang service (Part 1)
  2. The problem of network dependency
  3. Making networking available in early boot phase
  4. Writing a custom systemd unit

2 The problem of network dependency

A problem of unlocking volumes with Tang is the dependency on the network. At the time when the cryptsetup is running, the network is not yet available. In systemd different targets (units) are defined and executed during boot. The cryptsetup-target is reached very early during the boot process, while the network-online target is reached very late.

Therefore network based unlocking of volumes can only be done while still in dracut or very late in systemd but not just before the cryptsetup.

A flowchart showing the some important steps in the boot process of Fedora 32

You can use systemd-analyze to check which services have to be started before a target can be reached or a service can be started:

$ systemd-analyze critical-chain cryptsetup.target
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

cryptsetup.target @461ms
└─systemd-ask-password-wall.path @460ms
  └─-.mount
    └─system.slice
      └─-.slice

$ systemd-analyze critical-chain network-online.target
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

network-online.target @7.167s
└─NetworkManager-wait-online.service @4.032s +3.135s
  └─NetworkManager.service @3.699s +331ms
    └─network-pre.target @3.699s
      └─firewalld.service @2.998s +700ms
        └─polkit.service @5.136s +476ms
          └─basic.target @2.991s
            └─dbus-broker.service @3.068s +109ms
              └─dbus.socket @2.973s
                └─sysinit.target @2.969s
                  └─systemd-update-utmp.service @2.962s +6ms
                    └─auditd.service @2.927s +33ms
                      └─systemd-tmpfiles-setup.service @2.887s +38ms
                        └─systemd-journal-flush.service @739ms +2.146s
                          └─systemd-journald.service @478ms +260ms
                            └─systemd-journald.socket
                              └─-.mount
                                └─system.slice
                                  └─-.slice

Because of a complex web of dependencies, the cryptsetup.target cannot simply be waiting for the network to come up. Changing the boot order by defining new dependencies leads to some chaos.

Screenshot: Boot process is stuck, unlock with tang does not work because of missing network.

There are two possible solutions I like to discuss here.

  1. Making network available in the early boot phase and automatically unlock an encrypted root volume.
  2. Writing a custom systemd-unit to mount a volume after boot when the network is available. This can be useful when a volume is not needed during the boot process.

The solution you need depends on your setup: If your linux root-volume is encrypted and you want to unlock it automatically then solution 1 is your only option. If you want to unlock an additional volume besides your system’s root volume, then solution 2 is for you.

3 Making networking available in early boot phase

This chapter only applies to the decryption of root-volumes. If you selected encryption during installation and your computer is connected to the network over ethernet, then this is the right solution for you. This solution cannot be used when you are using Wi-Fi.

Example: Encryption of volume selected during installation of Fedora 32
Example: Encryption of volume selected during installation of Fedora 32

3.1 Find the ethernet driver used by the machine’s kernel.

$ lspci -k | grep Ethernet -A 3
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16)
	Subsystem: Lenovo Device 3709
	Kernel driver in use: r8169
	Kernel modules: r8169

The above example shows the driver in use by the kernel on a physical machine. The driver is loaded as a kernel module named “r8169”.

3.2 Add networking and driver to dracut

Dracut is the tool used to create the initramfs image which is loaded at early boot during the start of Linux. If the network shall be brought up before systemd takes over, Dracut is the way to go.

Note: Dracut can not connect to Wi-Fi networks!

First create a configuration file for the network settings in Dracut:

$ sudo nano /etc/dracut.conf.d/networking.conf

Add the following content:

add_dracutmodules="network"
add_drivers="r8160"

Note: If the driver of your Ethernet chip is not loaded as a module but is part of the Kernel itself then leave line 2 away.

To force Dracut to regenerate initramfs, run

$ sudo dracut -f

Finally two additional kernel arguments are needed. Update the grub bootloader with this command:

$ sudo grubby --update-kernel=ALL --args="rd.neednet=1 ip=dhcp" --make-default

3.3 Bind your luks-encrypted root volume to Tang

To bind your encrypted root volume to clevis you need to install the package clevis-luks and clevis-dracut:

$ sudo dnf -y install clevis-luks clevis-dracut

There are two ways to bind an existing root volume to a tang server: On the shell command line or in cockpit.

3.3.1 In Cockpit

To install and enable cockpit run these commands:

$ sudo dnf -y install cockpit
$ sudo systemctl enable --now cockpit.socket

Cockpit will be available on https://machine-ip:9090

In Cockpit select “Storage” on the left pane and then your hard drive or ssd on the right side.

Open the encrypted partition, select the tab “Encryption” and click the plus-symbol to add another key.

In the “Add Key” dialog choose “Tang keyserver” and type in the ip address and port of your tang server.

Click “Trust key”.

3.3.2 In shell command line

First find out, which partition on your machine contains the encrypted lvm group containing the root filesystem. Use the lsblk-command to get an overview over all block devices, their volumes and mount points:

In this example the partition is /dev/vda3 and it contains a luks encrypted volume group.

Run the following command to bind the volume to a Tang server:

$ sudo clevis luks bind -d /dev/your_crypt_volume tang '{"url":"server-ip:port"}'
Screenshot showing an example output of binding tang to a luks keyslot with clevis
Screenshot showing an example output of binding tang to a luks keyslot with clevis

Finally, to check the binding:

$ sudo clevis luks list -d /dev/vda3
Screenshot showing an example output of clevis luks list
Screenshot showing an example output of clevis luks list

4 Writing a custom systemd unit

This solution applies if you want to unlock and mount a single encrypted volume after boot automatically.

4.1 Disable automatic mounting for encrypted volume

Fedora 32 will try to unlock volumes at boot time depending on the configuration in /etc/fstab and /etc/crypttab. To make sure you are not asked for a password during boot, disable “Unlock at boot” in the volume’s encryption settings.

4.2 Bind the encrypted volume to Tang

In the encryption settings of the volume press the plus-sign in the table of keys and select “Tang keyserver”. Type in your keyserver’s address like http://<ip-address>:<port> and provide one of the current passphrases you need to unlock the volume.

If the connection to the keyserver works, you will be asked to verify the key.

Write your own service to unlock the volume

The first step is to write a script to unlock the volume and mount it to the filesystem. This script can be very short.

Here is for comparison the logical volume setup in cockpit:

Screenshot from cockpit showing the configuration of example volume

Store your script in /usr/local/bin/ . It could look like this:

$ sudo nano /usr/local/bin/mount-example.sh
#!/bin/bash
clevis luks unlock -d /dev/fedora_example-server/example -n example
mount /dev/mapper/example /example

After creating the script, make it executable:

$ sudo chmod +x /usr/local/bin/mount-example.sh

You might also have to change the SELinux-context if you copied the script from another location to the /usr/local/bin/folder:

$ sudo restorecon -v /usr/local/bin/mount-example.sh

Finally a systemd service has to be created:

$ sudo systemctl edit --force --full cryptsetup-example.service
[Unit]
Description=Run custom startup script after boot
Requires=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/mount-example.sh

[Install]
WantedBy=multi-user.target

Finally enable the new service:

$ sudo systemctl enable cryptsetup-example.service

Reference list

H. (2016, February 18). Create private networks with libvirt. Retrieved June 11, 2020, from https://gist.github.com/atomtigerzoo/d6929b5e42cab5909ee6

Red Hat. (n.d.). Chapter 9. Configuring automated unlocking of encrypted volumes using policy-based decryption Red Hat Enterprise Linux 8. Retrieved June 11, 2020, from https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/configuring-automated-unlocking-of-encrypted-volumes-using-policy-based-decryption_security-hardening

Tags:

Leave a Reply

Your email address will not be published. Required fields are marked *