Kumoss

Deploy with Helm

Install Kumoss on Kubernetes with the published Helm chart, try it on a local kind cluster, and know what the chart changes from the Compose stack.

The chart in charts/kumoss deploys the same stack as docker-compose.yml on Kubernetes: the proxy, the core, the IaC and notifications sidecars, and the bundled backing services (two PostgreSQL databases, Redis, RustFS object storage, Phoenix and SearXNG). Every backing service can be switched off and replaced by an external one.

Each release publishes the chart as an OCI artifact, oci://ghcr.io/inditextech/charts/kumoss, with the same version as the release; its Kumoss image tags default to that version. Install it without a checkout:

helm upgrade --install kumoss oci://ghcr.io/inditextech/charts/kumoss \
  --version <version> -n kumoss --create-namespace \
  --set core.existingSecret=<secret with LLM / Git credentials>

The examples below use charts/kumoss from a checkout; the OCI reference works in its place everywhere.

The chart does not make a deployment production-ready on its own. The decisions in Deploy to production still apply; this page covers only how the chart maps them to Kubernetes.

What the chart deploys

Component Kind Notes

proxy

Deployment + Service

The single entry point, as in Compose: port 80 serves the SPA, /api (including the event stream) and /monitoring; port 9000 fronts the object store for presigned artifact URLs.

core

Deployment (1 replica, Recreate)

Mounts config.yaml from a ConfigMap and the shared workspaces volume. Probed on GET /healthz.

iac

Deployment (1 replica, Recreate)

Mounts the same workspaces volume as the core.

notifications

Deployment

Off by default: the service refuses to start without SLACK_WEBHOOK_URL.

core-db, phoenix-db

StatefulSet (postgres:17)

Single instance each. Use a managed database or an operator in production.

redis, object-storage, phoenix, searxng

Deployment / StatefulSet

Same images and settings as Compose. SearXNG runs the upstream image with the repository’s settings.yml.

The repository’s config.yaml, nginx/nginx.conf and services/searxng/settings.yml are symlinked into charts/kumoss/files, so the chart always ships what Compose ships. At render time the chart replaces the Compose hostnames (core, iac, phoenix, …​) with the release’s Service names, then applies your config overrides on top.

Values that matter

Value Purpose

publicUrl

URL the browser uses to reach the proxy. It becomes the core’s only CORS origin (http.cors_origins).

storagePublicUrl

URL the browser uses to reach the proxy’s port 9000. The core signs artifact download links for this host (storage.public_endpoint_url).

config

Any config.yaml key, merged over the repository file. See Configuration.

core.existingSecret

Secret loaded whole into the core: LLM credentials (see LLM providers and models), GIT_USER, GIT_TOKEN. core.secretEnv creates one from values instead, for local testing.

iac.existingSecret

Cloud provider credentials for plan and apply.

caBundle.existingConfigMap, caBundle.certificates

Extra CA roots (PEM) trusted for outbound HTTPS by every pod that calls out: the core, the IaC and notifications sidecars, and SearXNG. Needed behind a TLS-inspecting proxy; see Behind a TLS-inspecting proxy.

secrets.*

Service tokens and bundled-service passwords. Leave them blank: the chart generates them on first install and reuses them on upgrade. If the chart is rendered offline, see Rendering offline (Argo CD, Flux, helm template).

workspaces.accessMode, workspaces.storageClass

The core and the IaC sidecar share this volume. With ReadWriteOnce (the default) the IaC pod is required to run on the core’s node and the core prefers the IaC pod’s node, so a rollout returns to the node that holds the volume. Setting iac.affinity or core.affinity replaces those rules: keep the equivalent podAffinity in yours. On a multi-node cluster, a ReadWriteMany class removes the constraint.

<service>.enabled, <service>.externalUrl

Replace a bundled backing service. postgres.core.externalUrl and redis.externalUrl take connection URLs; for object storage, disable it and set config.storage.* instead.

<component>.serviceAccount, <component>.podLabels

The Kubernetes identity each of proxy, core, iac and notifications runs as; see Cloud credentials with workload identity.

notifications.enabled

Off by default. Turning it on requires SLACK_WEBHOOK_URL in notifications.secretEnv or notifications.existingSecret; the chart refuses to render without it.

<component>.resources

CPU and memory requests and limits. Empty by default; set them for production, and on clusters that require them (GKE Autopilot fills in its own defaults otherwise).

phoenix.exposeViaProxy

On by default, as in Compose: the proxy serves the Phoenix UI under /monitoring/, and so does any Ingress in front of it. Phoenix has no authentication and its traces hold full prompts and repository content, so in production set it to false and reach Phoenix with kubectl port-forward or a separately protected route. The admin panel’s Phoenix link then answers 502.

phoenix.enabled

With false, point config.telemetry.collector_url at your own Phoenix (the core seeds its prompts there at startup). The proxy does not route /monitoring/ to an external Phoenix; open it at its own URL.

ingress.*

Optional Ingress in front of the proxy. ingress.storageHost routes a second host to port 9000 for downloads; set storagePublicUrl to match.

<component>.extraEnv, extraVolumes and extraVolumeMounts cover everything else.

Cloud credentials with workload identity

On a managed cluster, prefer workload identity over long-lived keys in Secrets: each pod runs as a Kubernetes ServiceAccount that the cloud maps to one of its identities, and the pod receives short-lived tokens automatically.

The chart creates one ServiceAccount per Kumoss component, so permissions stay separate. Two of them need cloud access:

  • core: the LLM provider (Vertex AI, Azure AI, Bedrock) and, with S3 storage, the artifacts store: leave the static keys blank and the default AWS credential chain picks up the identity. STORAGE_ACCOUNT storage does not use the pod identity; it still needs STORAGE_ACCOUNT_KEY (for example in core.existingSecret), because the core signs download links with the account key.

  • iac: the permissions plan and apply need on the target infrastructure. This is the most privileged identity in the deployment; scope it accordingly.

Bind a ServiceAccount by annotating it (and, on AKS, labelling the pods):

# AKS — Azure Workload Identity. The webhook injects AZURE_* variables;
# the azurerm provider and import discovery read the ARM_* ones.
iac:
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: <managed-identity-client-id>
  podLabels:
    azure.workload.identity/use: "true"
  env:
    ARM_USE_AKS_WORKLOAD_IDENTITY: "true"
    ARM_CLIENT_ID: <managed-identity-client-id>
    ARM_TENANT_ID: <tenant-id>

# EKS — IRSA (EKS Pod Identity needs no annotation: associate the
# ServiceAccount with the role in EKS instead). The webhook injects the
# role and token; only the region is yours to set.
iac:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<account>:role/<role>
  env:
    AWS_REGION: <region>

# GKE — Workload Identity Federation, impersonating a service account.
# The provider finds the identity through the metadata server.
iac:
  serviceAccount:
    annotations:
      iam.gke.io/gcp-service-account: <name>@<project>.iam.gserviceaccount.com

The per-cloud variables come from services/iac/PROVIDERS.md, which lists every supported credential form.

The cloud side (the managed identity or role, its permissions, and the trust that lets this ServiceAccount assume it) is set up outside the chart. The trust is keyed on the namespace and ServiceAccount name: <release>-kumoss-<component> by default (kumoss-iac for a release named kumoss), or set serviceAccount.name. To use a ServiceAccount you manage yourself, set serviceAccount.create: false and serviceAccount.name.

Kumoss does not call the Kubernetes API, so the pods do not mount a Kubernetes API token (serviceAccount.automount: false). The workload-identity token is injected separately and is unaffected.

Rendering offline (Argo CD, Flux, helm template)

The chart reads its generated credentials back from the cluster with Helm’s lookup, which only works when Helm talks to the cluster (helm install / helm upgrade). Tools that render the chart offline get nothing back, so every render would generate new passwords, and the databases, initialised with the first ones, would reject them.

Manage that Secret yourself instead (with Sealed Secrets, External Secrets, SOPS…​) and set secrets.existingSecret to its name. The chart then renders no credentials of its own. The Secret needs every key the generated one has:

Key Value

KUMOSS_IAC_TOKEN, KUMOSS_NOTIFICATIONS_TOKEN

Bearer tokens between the core and the sidecars (any random string)

CORE_DB_PASSWORD, PHOENIX_DB_PASSWORD

Passwords the bundled databases are initialised with

KUMOSS_SQL_DATABASE_URL, PHOENIX_SQL_DATABASE_URL

Connection URLs, e.g. postgresql://postgres:<password>@<release>-kumoss-core-db:5432/kumoss. Percent-encode the password.

KUMOSS_REDIS_URL

e.g. redis://<release>-kumoss-redis:6379/0

RUSTFS_ACCESS_KEY, RUSTFS_SECRET_KEY

Credentials of the bundled object store

SEARXNG_SECRET

SearXNG’s secret key (any random string)

Keys for components you disabled can be omitted. Pods do not restart when this Secret changes; restart them after rotating it.

Try it on kind

kind runs a Kubernetes cluster as Docker containers on your workstation. The steps below give a single-node cluster whose host ports 80 and 9000 reach the proxy, so the browser uses the same URLs as with Compose and the default publicUrl / storagePublicUrl work unchanged. Run them from the repository root.

charts/kind-up.sh runs all of them in one idempotent command; re-run it to upgrade. It reads the credentials from a Compose-style .env (stripping the quotes kubectl --from-env-file would keep), and with CA_DIR set it also does the TLS-inspecting proxy steps:

ENV_FILE=core/.env charts/kind-up.sh
# Behind a TLS-inspecting proxy:
ENV_FILE=core/.env CA_DIR=/usr/local/share/ca-certificates charts/kind-up.sh
# Other host ports, or the published chart instead of the checkout:
HTTP_PORT=8080 STORAGE_PORT=9900 charts/kind-up.sh
CHART=oci://ghcr.io/inditextech/charts/kumoss CHART_VERSION=<version> charts/kind-up.sh

Its header lists every setting; arguments after them go to helm upgrade --install. The manual steps:

  1. Create the cluster. The port mappings forward host ports 80 and 9000 to the node ports the proxy is exposed on in step 4:

    cat <<'EOF' | kind create cluster --name kumoss --config -
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    nodes:
      - role: control-plane
        extraPortMappings:
          - { containerPort: 30080, hostPort: 80 }
          - { containerPort: 30900, hostPort: 9000 }
    EOF
  2. Make the images available. The chart’s image tags follow the chart version, so a chart from a release pulls that release’s images and nothing needs doing. To run your checkout instead, build the images you changed and load them into the node, which has its own image store:

    docker build -f core/Dockerfile --build-context host-certs=docker/certs \
      -t kumoss-core:dev .
    kind load docker-image kumoss-core:dev --name kumoss

    Then point the component at it with --set core.image.repository=kumoss-core,core.image.tag=dev,core.image.pullPolicy=Never.

  3. Store the LLM credentials the configured models need (here, the default Vertex AI models) and, optionally, the Git credentials:

    kubectl create namespace kumoss
    kubectl -n kumoss create secret generic kumoss-credentials \
      --from-literal=VERTEXAI_PROJECT=my-project \
      --from-literal=VERTEXAI_LOCATION=europe-west1 \
      --from-file=VERTEXAI_CREDENTIALS=./service-account.json \
      --from-literal=GIT_USER=<user> \
      --from-literal=GIT_TOKEN=<token>
  4. Install, exposing the proxy on the node ports mapped in step 1:

    helm upgrade --install kumoss charts/kumoss -n kumoss \
      --set proxy.service.type=NodePort \
      --set proxy.service.nodePorts.http=30080 \
      --set proxy.service.nodePorts.storage=30900 \
      --set core.existingSecret=kumoss-credentials \
      --wait --timeout 10m
    helm test kumoss -n kumoss
  5. Open http://localhost. Phoenix is at http://localhost/monitoring/.

Behind a TLS-inspecting proxy

The proxy re-signs HTTPS with a corporate root that neither the kind node nor the images trust. Two things change, matching the Compose setup; charts/kind-up.sh does both when CA_DIR is set.

  • Image pulls. The node fails to pull with x509: certificate signed by unknown authority (ImagePullBackOff). Pull the images on the host and copy them in, all in one archive. Save a single platform, because kind load docker-image fails on multi-platform images:

    images=$(helm template kumoss charts/kumoss | sed -n 's/^ *image: *"\{0,1\}\([^"]*\)"\{0,1\}$/\1/p' | sort -u)
    for i in $images; do docker pull -q "$i"; done
    docker save --platform linux/amd64 -o /tmp/kumoss-images.tar $images
    kind load image-archive /tmp/kumoss-images.tar --name kumoss

    Render with the same --set flags you install with: enabling notifications adds its image. The list includes the curlimages/curl image helm test runs.

  • Outbound HTTPS from the pods. Put your corporate roots (PEM) in a ConfigMap and point caBundle.existingConfigMap at it:

    kubectl -n kumoss create configmap corporate-ca-roots \
      --from-file=/usr/local/share/ca-certificates
    helm upgrade --install kumoss charts/kumoss -n kumoss ... \
      --set caBundle.existingConfigMap=corporate-ca-roots

    Or pass the certificates inline with --set-file caBundle.certificates=corp-root.pem and the chart creates the ConfigMap. Every pod that calls out gets an init container that combines its image’s public roots with yours into one bundle, pointed at by SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS and AWS_CA_BUNDLE. A file may hold several PEM certificates:

    Pod Outbound HTTPS

    core

    LLM providers, Git hosts, web pages read by web_search. The roots are also mounted at KUMOSS_CA_DIR, so the entrypoint imports them into Chromium’s store.

    iac

    The OpenTofu registry and provider downloads (tofu init). Without the roots, the pod starts and passes its probes, but every plan fails.

    notifications

    Slack.

    searxng

    The search engines.

    Pods do not restart when an existingConfigMap changes; restart them after editing it. Earlier versions of this guide mounted the roots through core.extraVolumes and searxng.extraVolumes; remove those when switching to caBundle.

    When building the core image, pass the same directory as the host-certs build context instead of docker/certs.

Troubleshooting kind

kube-proxy crash-looping with too many open files, followed by CoreDNS and the local-path-provisioner failing, means the host ran out of inotify instances. Every kind node on the host shares the per-user limit, so it shows up with several clusters. Raise it, and persist it if your distribution reads /etc/sysctl.d:

sudo sysctl fs.inotify.max_user_instances=512
echo 'fs.inotify.max_user_instances=512' | sudo tee /etc/sysctl.d/99-kind.conf

Then delete the crashed pods so they restart immediately, for example kubectl -n kube-system delete pod -l k8s-app=kube-proxy.

Upgrades and removal

  • config.yaml changes roll the core automatically (the pod carries a checksum of the rendered file). Changes to secrets.*, the external service URLs, or a component’s secretEnv roll every pod that reads them the same way. A Secret you reference through existingSecret is managed outside the chart: after rotating it, restart its readers (kubectl rollout restart deploy/<release>-kumoss-core). The same goes for caBundle.existingConfigMap; inline caBundle.certificates roll the pods on change.

  • Generated passwords and tokens are read back from the existing Secret on every upgrade. Changing a database password in values after the first install does not change it inside an initialised database.

  • helm uninstall keeps the data: the workspaces claim, the StatefulSets' data claims, and the <release>-kumoss-internal Secret holding the passwords that data was initialised with (both carry helm.sh/resource-policy: keep). Reinstalling the same release in the same namespace picks all three up again. To start from scratch, delete them explicitly:

    kubectl -n kumoss delete pvc -l app.kubernetes.io/instance=kumoss
    kubectl -n kumoss delete secret kumoss-internal