Deploy Kestra On Kubernetes With Argo CD: A Step-by-Step Guide

by Alex Johnson 63 views

In this comprehensive guide, we'll walk you through deploying the Kestra orchestration platform on a Kubernetes lab cluster using Argo CD. Our approach focuses on leveraging existing GitOps patterns, reusing storage and database operators, and ensuring the Kestra UI is accessible via the Tailscale network. This setup guarantees a fully GitOps-managed Kestra deployment, making it easy to manage and scale your workflows.

Introduction to Kestra and Argo CD

Before diving into the implementation, let's briefly introduce the key players. Kestra is a powerful open-source orchestration platform that allows you to define, schedule, and monitor complex data pipelines. It provides a user-friendly interface and robust features for managing workflows. Argo CD, on the other hand, is a declarative, GitOps-based continuous delivery tool for Kubernetes. It enables you to manage your Kubernetes applications and infrastructure through Git repositories, ensuring consistency and reproducibility.

By combining Kestra and Argo CD, you can achieve a seamless and automated deployment process, where any changes to your Kestra configuration are automatically applied to your Kubernetes cluster. This approach not only simplifies deployment but also enhances the overall manageability and reliability of your orchestration platform.

Current State of the Lab Cluster

To provide context, let's outline the current state of our lab cluster. Argo CD is already bootstrapped and managing applications using the app-of-apps and overlays patterns. This means that applications are defined under the argocd/ directory, with overlays for cluster-specific configurations. Persistent storage and databases are provisioned by operators and storage classes managed elsewhere in the repository, specifically under tofu/ for OpenTofu configurations and argocd/ and kubernetes/ for operators and manifests. Tailscale is also configured to expose services on the tailnet with stable hostnames.

The key piece missing is Kestra itself. Currently, there is no Argo CD application defined for Kestra, which is what we aim to address in this guide. By the end of this process, Kestra will be fully integrated into our GitOps workflow, managed declaratively through Argo CD.

Desired Outcome: A GitOps-Managed Kestra Deployment

Our goal is to deploy Kestra into the cluster in a dedicated namespace, such as kestra, and manage it entirely through GitOps using Argo CD. This means that all Kestra-related configurations and deployments will be defined in our Git repository, and Argo CD will automatically synchronize these changes with the cluster.

Persistent Backing Services: Kestra will utilize persistent backing services, including a real database (Postgres or comparable) provisioned via existing operators in the repository. We will not introduce any new database operators. Additionally, Kestra will use persistent volumes bound to existing storage classes, ensuring data durability and availability.

Tailscale Integration: The Kestra web UI will be reachable exclusively on the Tailscale network under the hostname kestra. This mirrors the existing Tailscale patterns in our repository, providing secure access to Kestra from our private network.

Manifest Management: All manifests will reside under the argocd/ directory, adhering to the existing layout and overlays conventions. This ensures that bun run k:bootstrap combined with Argo CD synchronization is sufficient to bring Kestra up from scratch, without requiring any imperative kubectl or Helm steps after Argo CD is in place.

Implementation Plan: Step-by-Step Guide

To achieve our desired outcome, we'll follow a structured implementation plan, breaking down the deployment process into manageable steps.

1. Namespace and Labels

First, we need to create a dedicated namespace for Kestra. This namespace will isolate Kestra's resources and provide a clear boundary for management and monitoring.

  • Create a Namespace: We'll create a Kubernetes namespace, typically named kestra. This manifest will be managed by Argo CD, either as part of the Kestra Application or within a shared “system” Kustomization if such a pattern already exists in the repository.
  • Apply Standard Labels: We'll apply standard labels and annotations commonly used in other application namespaces. These labels are crucial for monitoring, backups, and other operational aspects of the application. Consistency in labeling ensures that Kestra integrates seamlessly with our existing infrastructure and tooling.

2. Argo CD Application Structure

Next, we'll define the Argo CD application structure, ensuring it aligns with the existing patterns used for other services in our cluster. This consistency simplifies management and promotes a unified approach to deployments.

  • Follow Existing Patterns: We'll adhere to the established conventions for Argo CD applications, typically found under the argocd/applications/* directory. This involves creating a directory structure that includes base configurations and cluster-specific overlays.
  • Proposed Layout: Here’s a proposed layout for our Kestra application:
    • argocd/applications/kestra/
      • base/
        • application.yaml: This file defines the Argo CD Application resource or a part of an ApplicationSet, depending on how other applications are defined.
        • kustomization.yaml: If we're wrapping Helm with Kustomize, this file manages the Kustomize configuration.
        • values.yaml: This file contains the default values for the Kestra Helm chart.
      • overlays/cluster/
        • kustomization.yaml: This file manages cluster-specific customizations.
        • values-cluster.yaml: This file contains values and patches specific to our cluster, such as storage class, node selectors, and resource limits.
  • App-of-Apps Integration: If we're using the app-of-apps pattern, we'll add a reference from the relevant root application (e.g., the cluster “apps” Application that already manages other services like observability, graf, and facteur).

Key Points

We'll reuse the patterns established for other applications, particularly those with multi-component stacks and storage requirements (like observability) and those using overlays for cluster-specific customizations (like facteur under overlays/cluster). This ensures that our Kestra deployment is consistent with the rest of our infrastructure.

3. Official Kestra Helm Chart

To simplify the deployment and ensure we're using the latest best practices, we'll leverage the official Kestra Helm chart. Helm charts provide a standardized way to package and deploy applications on Kubernetes.

  • Reference the Helm Chart: We'll reference the official Kestra Helm chart from our Argo CD Application resource.
    • spec.source.chart: Set this to the name of the Kestra chart.
    • spec.source.repoURL: Point this to the official chart repository.
    • spec.source.targetRevision: Pin this to a specific version (not latest) and include a comment on how to update it.
  • Manage Configuration: We'll manage Kestra's configuration through:
    • A base values.yaml file in argocd/applications/kestra/base/.
    • Overlay-specific values and patches in argocd/applications/kestra/overlays/cluster/.

4. Persistence: Reusing Existing Operators and Storage

One of our key goals is to reuse existing operators and storage solutions within our cluster. This avoids introducing unnecessary complexity and ensures consistency across our infrastructure.

Database

  • Reuse Existing Operator: We will not install a new database operator. Instead, we'll reuse the existing operator (e.g., Postgres) that is already provisioned in the cluster and used by other services.
  • Add a Database for Kestra: We'll add a dedicated database for Kestra using either:
    • The existing database operator’s Custom Resource Definitions (CRDs), which is the preferred method, or
    • An existing “shared” Postgres instance with a dedicated database and user for Kestra.
  • Define DB Resources: We'll define the database and database user in the same way as other applications:
    • Database CRDs and/or Helm values under the relevant argocd/applications/<db-operator>/... directory.
    • Credentials via sealed secrets or secret generators, consistent with existing patterns (using scripts/generate-* if applicable).

Storage

  • Configure Helm Chart: We'll configure the Kestra Helm chart to use existing storage classes.
    • Identify the standard storage class used for other stateful sets (e.g., for observability and databases).
    • Set persistence.storageClass (or equivalent chart values) for:
      • Kestra configuration and data volumes.
      • Embedded components (if the chart optionally bundles Postgres, Elasticsearch, etc.).
        • Disable these and point to the existing operators, or
        • Enable them but still use the same storage class and PersistentVolume (PV) patterns as other applications.
  • Ensure Persistent Volumes: We'll ensure that all stateful workloads run with PersistentVolumeClaims (PVCs) backed by these storage classes. We will not use emptyDir for any data that needs to survive pod restarts.

5. Tailscale Exposure: Hostname kestra

To provide secure access to the Kestra UI, we'll expose it on the Tailscale network using the hostname kestra. This approach leverages our existing Tailscale setup and ensures consistent access patterns.

  • Reuse Existing Pattern: We'll reuse the exact pattern already used in the repository for exposing services on the Tailscale network.
    • Find an existing example (e.g., for ArgoCD or Coder) and mirror:
      • The annotations used to instruct the Tailscale operator to expose the service.
      • The way hostnames are assigned on the tailnet.
      • Any Service or Ingress configurations used to map HTTP(S) traffic.
  • Kestra Configuration: For Kestra:
    • Expose the UI Service (HTTP, typically port 8080 or the chart default) via the Tailscale operator.
    • Set the Tailscale hostname to kestra (e.g., through an annotation like tailscale.com/hostname: kestra or the operator’s equivalent, matching the existing pattern).
    • Ensure that any TLS/HTTP routing matches how other internal services are exposed, reusing existing patterns for tls secrets.

6. Secrets and Configuration

Managing secrets securely is crucial. We'll configure Kestra to read database credentials from a Kubernetes Secret and ensure no plaintext secrets are committed to our repository.

  • Configure Helm Values: We'll configure the Kestra Helm values to:
    • Read database credentials from a Kubernetes Secret whose lifecycle is already managed via sealed secrets or scripts/generate-* helpers.
    • Set any required environment variables (e.g., KESTRA_*) through Values or envFrom referencing ConfigMaps/Secrets instead of hardcoding sensitive values.
  • Ensure Security: We'll ensure that:
    • No plaintext secrets are committed.
    • Any new SealedSecret manifests reside under argocd/, consistent with existing secrets patterns.

7. Observability and Health

Integrating Kestra with our existing observability stack is essential for monitoring and maintaining its health.

  • Align with Existing Stack: We'll align with the observability stack already deployed under argocd/applications/observability:
    • Add standard labels and annotations so that logs and metrics are captured in the existing stack.
    • Expose readiness and liveness probes for Kestra pods, using chart defaults where available.
  • Optional Enhancements: We can also:
    • Add a simple synthetic check or dashboard panel for Kestra in our existing dashboard tooling.

8. Documentation and Operations

Finally, we'll update our documentation to reflect the Kestra deployment and provide guidance for future operations.

  • Update Documentation: We'll add a short section to argocd/README.md (or a dedicated docs/ location) describing:
    • Where Kestra manifests reside.
    • How to update the Helm chart version.
    • Any runtime assumptions (e.g., database operator, storage class, tailnet name).
  • Workflow Documentation: If needed, we'll add a one-liner to docs/codex-workflow.md or a relevant document explaining how Kestra will be used as an orchestrator for lab workflows.

Acceptance Criteria: Ensuring a Successful Deployment

To ensure our deployment meets our requirements, we'll use the following acceptance criteria:

  • Manifest Structure: New Kestra Argo CD application manifests exist under argocd/applications/kestra/... and match the existing base/ + overlays/cluster/ structure used elsewhere.
  • Automated Deployment: Applying bun run k:bootstrap (and allowing Argo CD to sync) is sufficient to:
    • Create the kestra namespace (or equivalent).
    • Deploy Kestra components (server, workers, etc.) at the desired version.
  • Persistent Database: Kestra is configured to use a persistent database:
    • The database is provisioned via an existing operator/stack in our repository (no new operator introduced).
    • Connection information is provided via a Kubernetes Secret managed the same way as other applications.
  • Persistent Storage: All stateful parts of Kestra use PVCs backed by existing storage classes (no new storage operator or ad-hoc storage class).
  • Tailscale Access: A Tailscale-exposed endpoint exists:
    • The Kestra UI is reachable on the tailnet using the hostname kestra.
    • The Tailscale manifests/annotations follow an existing pattern from the repository (only the hostname is changed).
  • Basic Smoke Tests: Basic smoke tests pass:
    • The Kestra UI loads and shows healthy components.
    • A trivial test workflow can be created and executed successfully.
  • Documentation Updates: Documentation is updated:
    • argocd/README.md (or equivalent) includes a short section on Kestra deployment and upgrades.
    • Any new SealedSecrets or scripts are documented where relevant.

Scope Guardrails: Defining Boundaries

To keep our implementation focused, we'll adhere to the following scope guardrails:

In Scope

  • Adding Argo CD definitions and Kustomizations for Kestra under argocd/.
  • Wiring Kestra to existing database/storage operators and Tailscale patterns.
  • Minimal smoke testing of Kestra (UI up, simple workflow runs).

Out of Scope

  • Migrating existing workflows from other orchestrators into Kestra.
  • Designing multi-cluster or High Availability (HA) Kestra topologies beyond what the chart already supports by default.
  • Any changes to the underlying Tailscale, OpenTofu, or cluster bootstrap modules beyond what’s strictly needed to expose the kestra service.

Validation Checklist: Ensuring Correct Configuration

Before considering the deployment complete, we'll validate the following items:

  • Argo CD Status: Argo CD shows the Kestra application as Healthy and Synced.
  • PVC Status: PVCs for Kestra are bound and use the expected storage class(es).
  • Database Resources: Database resources for Kestra are visible under the existing database operator namespace and are in a Ready state.
  • Tailscale Hostname: The kestra hostname appears on the Tailscale network and correctly proxies to the Kestra UI.
  • Operator Reuse: No new infrastructure operators (database, storage, or Tailscale) were introduced; all reuse existing operators/resources.
  • Documentation: Documentation is updated and committed.

Rollout and Operations Notes: Planning for the Future

To ensure a smooth rollout and ongoing operation of Kestra, we'll consider the following:

  • Initial Rollout: The initial rollout can be targeted at the primary lab cluster only.
  • Integration with Existing Flows: If we use progressive syncs or app-of-apps, we'll integrate the Kestra application into the same promotion flow as other cluster applications.
  • Monitoring: We'll monitor:
    • Database CPU/memory and disk consumption.
    • Kestra pod resource usage.
    • Any Tailscale operator logs related to the kestra service.
  • Resource Management: If Kestra causes resource pressure, we'll follow existing patterns to add resource requests/limits in the Helm values overlay and re-sync.

Conclusion

By following this step-by-step guide, you can successfully deploy Kestra on your Kubernetes lab cluster using Argo CD. This approach ensures a fully GitOps-managed deployment, leveraging existing infrastructure and best practices. With Kestra now integrated into your cluster, you can begin orchestrating complex workflows and managing your data pipelines with ease.

For further information on Kestra, visit the official website at Kestra.io. You can also learn more about Argo CD and GitOps principles from Argo CD Documentation. This integration empowers you to streamline your data operations and achieve greater efficiency in your workflows.