Skip to main content
ALTCHA®

Install on Kubernetes

Run Sentinel on Kubernetes with a few plain manifests: the configuration in a ConfigMap, the secrets in a Secret, and a StatefulSet that gives each pod a stable name and its own data volume. This page starts with one replica on embedded SQLite, then extends it to several replicas with PostgreSQL and Redis.

For the license, secrets, database, and Redis requirements of a production install, see Install Sentinel.

Prerequisites

  • A Kubernetes cluster with a default StorageClass, and kubectl configured for it.
  • A Sentinel license key. To get a trial, register at cloud.altcha.org and create a Sentinel V2 trial license in the Licenses section.

Step 1 — Create the secret files

Keep each secret in its own file. The Secret you create from them is mounted at /run/secrets in the pod, and Sentinel reads each file with a ${file:/run/secrets/…} reference in the configuration.

mkdir -p sentinel/secrets && cd sentinel
printf '%s' 'SENT-…' > secrets/sentinel_license_key   # your license key
openssl rand -base64 24 > secrets/root_password        # first root password

Step 2 — Write sentinel.toml

Create sentinel.toml next to the secrets folder:

[server.admin_ui]
enabled = true   # stays on 127.0.0.1 in the pod; reach it with kubectl port-forward

[license]
key = "${file:/run/secrets/sentinel_license_key}"

[bootstrap]
root_password = "${file:/run/secrets/root_password}"   # applied on first boot only

Every other key keeps its default: the main API listens on port 3000, data goes to /home/nonroot/data in the container, and the signing and encryption secrets are generated into the data volume on first start. The admin UI listens on the pod's loopback interface only, so nothing outside the pod reaches it; kubectl port-forward connects to it from inside the pod. All keys are listed in the Configuration reference.

If Docker is available on your machine, validate the file before you deploy it, as shown in Install with Docker. Otherwise, an invalid file stops the pod at startup, and kubectl logs names the key and the reason:

Caused by:
    unknown field `enabeld`, expected one of `enabled`, `host`, `port`, …
    in `server.admin_ui`

Step 3 — Create the namespace, Secret, and ConfigMap

kubectl create namespace sentinel

kubectl -n sentinel create secret generic sentinel-secrets \
  --from-file=secrets/ --dry-run=client -o yaml | kubectl apply -f -

kubectl -n sentinel create configmap sentinel-config \
  --from-file=sentinel.toml --dry-run=client -o yaml | kubectl apply -f -

--from-file=secrets/ adds every file in the folder under its file name. Piping through kubectl apply lets you run the same commands again after you change a file.

Step 4 — Deploy Sentinel

Create sentinel.yaml:

apiVersion: v1
kind: Service
metadata:
  name: sentinel
  namespace: sentinel
spec:
  selector:
    app.kubernetes.io/name: sentinel
  ports:
    - name: http
      port: 3000
      targetPort: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sentinel
  namespace: sentinel
spec:
  serviceName: sentinel
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: sentinel
  template:
    metadata:
      labels:
        app.kubernetes.io/name: sentinel
    spec:
      automountServiceAccountToken: false   # Sentinel doesn't call the Kubernetes API; frees /run/secrets
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        fsGroup: 65532
      containers:
        - name: sentinel
          image: ghcr.io/altcha-org/sentinel:2.0.0-beta.1
          ports:
            - name: http
              containerPort: 3000
            - name: admin
              containerPort: 3002
          volumeMounts:
            - name: config
              mountPath: /home/nonroot/sentinel.toml
              subPath: sentinel.toml
              readOnly: true
            - name: secrets
              mountPath: /run/secrets
              readOnly: true
            - name: data
              mountPath: /home/nonroot/data
          startupProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 5
            failureThreshold: 60
          livenessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 10
          resources:
            requests:
              cpu: "1"
              memory: 512Mi
            limits:
              memory: 1Gi
      volumes:
        - name: config
          configMap:
            name: sentinel-config
        - name: secrets
          secret:
            secretName: sentinel-secrets
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 5Gi
  • StatefulSet. Each pod gets a stable name (sentinel-0, sentinel-1, …), which Kubernetes also sets as HOSTNAME. Sentinel uses it as the node ID of its license instance slot, so a restarted pod takes its slot back at once. Each pod also gets its own data volume from volumeClaimTemplates.
  • automountServiceAccountToken: false. Kubernetes mounts the service account token at /var/run/secrets/kubernetes.io/serviceaccount. In the image, /var/run is a link to /run, so with the Secret mounted read-only at /run/secrets the pod fails to start with read-only file system. Sentinel doesn't use the token.
  • fsGroup: 65532. The container runs as 65532:65532 and needs write access to the data volume. fsGroup gives that group ownership of the volume when it's mounted. The Secret's files are mounted with mode 0644, readable by the same user.
  • Probes. All three call GET /, which answers {"ok":true} without touching the database or Redis, so a storage outage doesn't restart or unpublish every pod. The startup probe allows up to 5 minutes for the first start. Don't probe GET /.health; see Liveness and readiness.
  • Configuration file. The ConfigMap is mounted at the default path, so the container needs no arguments. Sentinel reads the file only at startup, and a subPath mount doesn't receive ConfigMap updates, so restart the pods after you change it.
  • Resources. The requests match the evaluation size in Sizing. Raise them for production, and raise the memory limit when you enable the local similarity model or memory-hard proof of work.
  • Use an exact 2.0.0-beta.x tag; during the beta, latest still points to Sentinel v1.

Apply it and wait for the pod:

kubectl apply -f sentinel.yaml
kubectl -n sentinel rollout status statefulset/sentinel
kubectl -n sentinel logs sentinel-0

The log shows the license status (for example licensed to "…"), created root user from bootstrap.root_password, web app listening addr=127.0.0.1:3002, and listening addr=0.0.0.0:3000. If you didn't set [bootstrap] root_password, the log contains the generated root password instead.

Step 5 — Sign in

Forward the admin UI port to your machine:

kubectl -n sentinel port-forward pod/sentinel-0 3002:3002

Open http://localhost:3002 and sign in with the username root and the password from secrets/root_password.

Verify it works

Forward the main API in a second terminal and call its status endpoint:

kubectl -n sentinel port-forward svc/sentinel 3000:3000
curl -s http://localhost:3000/
{"ok":true}

To check the configuration, storage, and network access from inside the pod, run doctor in it:

kubectl -n sentinel exec sentinel-0 -- /usr/local/bin/altcha-sentinel doctor

doctor prints one line per check and exits with 1 if any check fails. security.root_mfa warns until you enable TOTP or a passkey for root. See the CLI reference for the options.

Expose Sentinel

The Service is ClusterIP, so backends inside the cluster call http://sentinel.sentinel.svc:3000. To serve widgets on the internet, put the main API behind an Ingress that terminates TLS. This example uses ingress-nginx and a TLS Secret named sentinel-tls:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sentinel
  namespace: sentinel
spec:
  ingressClassName: nginx
  tls:
    - hosts: [sentinel.example.com]
      secretName: sentinel-tls
  rules:
    - host: sentinel.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sentinel
                port:
                  name: http

Then tell Sentinel about the proxy in sentinel.toml, update the ConfigMap as in Step 3, and restart the pods with kubectl -n sentinel rollout restart statefulset/sentinel:

[server]
trusted_proxies = ["10.42.0.0/16"]           # your cluster's pod network, where the ingress controller runs
base_url        = "https://sentinel.example.com"
  • Without trusted_proxies, every request appears to come from the ingress controller, which breaks rate limits, penalties, and IP rules. The ingress controller itself must also see the client IP, for example with externalTrafficPolicy: Local on its Service or the PROXY protocol from your cloud load balancer.
  • To check the result, run the doctor through the admin UI and look at security.client_ip; see Listeners: Verify it works.
  • To publish only the widget endpoints, enable [server.public_api] (port 3003), add the port to the Service, and point the Ingress at it. To publish the admin UI instead of port-forwarding, set its host to "0.0.0.0", restrict it with allowed_ips, and give it its own trusted_proxies and base_url. See Listeners & reverse proxy.

Run several replicas with PostgreSQL and Redis

Several replicas keep Sentinel available while a pod restarts or a node drains. Every replica must share an external database, a Redis-compatible KV store, and the same secrets; see Clustering & high availability. Use a managed PostgreSQL and Redis, or ones running in your cluster. Size Redis with Redis or Valkey and set maxmemory-policy noeviction.

  1. Add the secrets. With an external database, Sentinel no longer generates its signing and encryption keys, and every replica must use the same values. Generate them next to the existing files, and add the database and Redis passwords:

    openssl rand -hex 32 > secrets/jwt_secret
    openssl rand -hex 32 > secrets/altcha_hmac_secret
    openssl rand -hex 32 > secrets/encryption_key
    printf '%s' '<database password>' > secrets/db_password
    printf '%s' '<redis password>' > secrets/redis_password

    Back up these files with the database: a different encryption_key can't read the API key and TOTP secrets stored in it. See Server secrets.

  2. Point sentinel.toml at the database and Redis:

    [server.admin_ui]
    enabled = true
    
    [database]
    url = "postgres://sentinel:${file:/run/secrets/db_password}@postgres.example.internal:5432/sentinel?sslmode=verify-full"
    
    [redis]
    nodes = ["redis://:${file:/run/secrets/redis_password}@redis.example.internal:6379"]
    
    [secrets]
    jwt_secret         = "${file:/run/secrets/jwt_secret}"
    altcha_hmac_secret = "${file:/run/secrets/altcha_hmac_secret}"
    encryption_key     = "${file:/run/secrets/encryption_key}"
    
    [license]
    key = "${file:/run/secrets/sentinel_license_key}"
    
    [bootstrap]
    root_password = "${file:/run/secrets/root_password}"

    Use rediss:// for Redis over TLS. A password with characters that aren't allowed in a URL must be percent-encoded. Each replica opens up to [database] pool_max connections (default 20), so keep pool_max × replicas below the database's connection limit. See Databases & KV store.

  3. Update the Secret and ConfigMap with the commands from Step 3.

  4. Scale the StatefulSet. Set replicas: 2 (or more) in sentinel.yaml, then apply it and restart the pods so they read the new configuration:

    kubectl apply -f sentinel.yaml
    kubectl -n sentinel rollout restart statefulset/sentinel
    kubectl -n sentinel rollout status statefulset/sentinel

    The replicas migrate the schema one at a time under a database lock: the first logs schema migrations complete applied=…, and the others find the schema current and log applied=0. Each log shows kv store ready pubsub=true. To see all replicas, open Cluster in the admin UI, or see List the replicas.

  5. Limit voluntary disruptions so a node drain never stops every replica at once:

    apiVersion: policy/v1
    kind: PodDisruptionBudget
    metadata:
      name: sentinel
      namespace: sentinel
    spec:
      maxUnavailable: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: sentinel

    Spread the replicas across nodes or zones with topologySpreadConstraints in the pod template.

The replicas start with an empty PostgreSQL database; the SQLite database in the first pod's data volume stays untouched. To carry over accounts, keys, and security groups, follow Restore into a fresh deployment: the restored API keys work only with the old encryption_key, which the single instance generated into secrets.json in its data volume.

Each replica keeps its own data volume for downloaded remote files and caches. Don't share one volume between replicas.

Update Sentinel

Before an update, read the release notes and take a snapshot, as described in Before you update. Then change the image tag:

kubectl -n sentinel set image statefulset/sentinel sentinel=ghcr.io/altcha-org/sentinel:2.0.0-beta.2
kubectl -n sentinel rollout status statefulset/sentinel

Update the tag in sentinel.yaml too, so the next kubectl apply doesn't roll it back. The StatefulSet replaces one pod at a time, from the highest ordinal down, and waits for each new pod to pass its readiness probe. The first new pod migrates the schema, and the older pods keep working against it meanwhile. Each new pod reuses its predecessor's HOSTNAME, so it takes over the same license instance slot without a burst slot. With a single replica, Sentinel is unavailable for a few seconds while the pod restarts. Rolling updates and rollback are covered in Updating Sentinel.

Next steps

Start typing to search...

↑ ↓ Navigate ↵ Select