Part 3: Adding Longhorn for Persistent Storage on Our Talos-Powered Kubernetes Cluster

In this post, we’ll walk through adding Longhorn to our Kubernetes cluster for persistent storage, specifically focusing on how to deploy it with Talos, an immutable operating system. Talos brings some extra steps compared to a typical Longhorn installation, but we’ll cover everything needed to get Longhorn up and running smoothly.

Generating a New Talos Image with System Extensions

To get started, we need to generate a new Talos image with system extensions that support Longhorn’s dependencies, specifically for iSCSI and utility tools. Talos provides an Image Factory that allows us to customize and generate the image.

First, create a configuration file called bare-metal.yaml:

customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/iscsi-tools
      - siderolabs/util-linux-tools

Now, use Talos’s Image Factory to generate the image by running the following command:

curl -X POST --data-binary @bare-metal.yaml https://factory.talos.dev/schematics

The Image Factory will provide an ID that we will use to upgrade the nodes. Use the following command to upgrade Talos with the new image:

talosctl upgrade --talosconfig=./talosconfig --nodes cp-001.ord.k8s.example.com --image factory.talos.dev/installer/<ID from last step>:v1.7.1

At this point, I encountered an issue where my x509 certificates didn’t include the domain ord.k8s.lambdacoffiellc.com for the API server. If you run into this issue, here’s how to fix it.

Fixing the x509 Certificate Error

To resolve the x509 certificate error, we need to modify the machine configuration for each control plane node. You can do this by editing the machine configuration for each affected node:

talosctl -n cp-001.ord.k8s.example.com -e cp-001.ord.k8s.example.com edit mc

Within that editor, add your domain to the machine.certSANs key in the YAML machine config. This ensures that the API server certificates include the correct domain, preventing further x509 certificate errors.

Preparing Worker Nodes for Longhorn

Once the x509 issue is resolved, we’ll move on to configuring our worker nodes for Longhorn. This involves ensuring that the necessary volumes are mounted for Longhorn to use. First, we’ll create a patch file named longhorn-volume.patch.yaml:

machine:
  kubelet:
    extraMounts:
      - destination: /var/lib/longhorn
        type: bind
        source: /var/lib/longhorn
        options:
          - bind
          - rshared
          - rw

Then, apply this patch to your worker nodes:

talosctl patch mc --nodes worker-001.ord.k8s.example.com,worker-002.ord.k8s.example.com,worker-003.ord.k8s.example.com --patch @longhorn-volume.patch.yaml

I only applied this to my worker nodes since I don’t have much storage on my control plane nodes. Now that the worker nodes are configured, we can move on to deploying Longhorn.

Installing Longhorn with ArgoCD

We’ll install Longhorn using ArgoCD. To do this, add a new file to your cluster state repository under apps/kube-system/longhorn-app.yaml. The contents of this file should look like the following:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: longhorn
  namespace: argocd
spec:
  project: default
  sources:
    - chart: longhorn
      repoURL: https://charts.longhorn.io/
      targetRevision: v1.6.2 # Replace with the Longhorn version you'd like to install or upgrade to
      helm:
        values: |
          preUpgradeChecker:
            jobEnabled: false
  destination:
    server: https://kubernetes.default.svc
    namespace: longhorn-system

After syncing the ArgoCD application, Longhorn will be installed on the cluster.

Accessing the Longhorn Web UI

Longhorn includes a useful web UI for managing volumes, backups, jobs, and general monitoring. To access it, you have a couple of options:

1. kubectl port-forward:

   Bash

   kubectl port-forward svc/longhorn-frontend -n longhorn-system 8080:80

2. Cloudflare Tunnel (if you want to set up ingress and expose it publicly). However, note that the Longhorn web UI doesn’t have built-in authentication by default, which could be a security risk.

For additional security, Longhorn’s documentation suggests setting up basic auth with an NGINX ingress controller. If you’re using a Cloudflare Tunnel like me, I recommend using Cloudflare Access Policies to protect the interface. You can find these settings under the Cloudflare Zero Trust dashboard. This will let you authenticate using your own identity provider (IdP) or a one-time pin sent to approved emails.

Personally, I’m sticking to kubectl port-forward for now since it’s simpler and fits my needs.

Conclusion

Adding Longhorn to our Talos-powered Kubernetes cluster provides us with a robust solution for managing persistent storage. While Talos adds a few extra steps to the process, the combination of Longhorn’s ease of use and Talos’ security features ensures a reliable setup.

What’s Next?

In our upcoming blog posts, we will delve into several exciting topics designed to further enhance our Kubernetes environment:

1. Kubernetes Database Operator: Look forward to a post on deploying a Kubernetes database operator, which can simplify the management of database instances within our cluster.
2. WordPress Demo Project: We will demonstrate setting up WordPress on Kubernetes, providing a practical example of managing web applications in our configured environment.
3. Security Hardening: Future posts will cover additional security measures to harden our Kubernetes setup against potential vulnerabilities.

If there are specific topics you are interested in or questions you’d like us to address in future posts, please leave a comment below. We value your input and look forward to making our content as helpful and relevant as possible.