# Install on Kubernetes

> Install a self-hosted Botyard bot on Kubernetes: verify the signed image and chart, keep the enrollment token in a Secret, run helm install, check it is online.

Canonical URL: https://docs.botyard.io/docs/self-hosted/install-kubernetes

This guide installs one self-hosted Botyard bot's runtime (`botyardd`) in your own Kubernetes
cluster with the `botyard-runtime-selfhosted` Helm chart. For its own control-plane traffic, the
runtime makes only outbound connections, over HTTPS and WSS, to Botyard, and listens on no network
port for Botyard. The chart creates no Service and no Ingress. This covers the runtime's own
transport only. Commands it starts through `exec`, and services it supervises, can open listeners
and connect on other ports. Having no Service or Ingress does not stop direct connections to the
Pod IP. Use the chart's [NetworkPolicy](#egress-networkpolicy-networkpolicy) (off by default) or
your own ingress controls.

Read the [self-hosted overview](/docs/self-hosted) first. It covers what runs where and what you
need before you start.

> [!NOTE]
> **Availability**
>
> Self-hosted bots are enabled per organization by Botyard. If your organization is not enabled
> yet, contact Botyard to request access. Kubernetes is the only supported install target today.
> Linux VM and macOS installs are not yet available.

## Before you start

You need:

* A Kubernetes cluster at version **1.25 or later**, on `linux/amd64` or `linux/arm64` nodes.
* A default StorageClass, or the name of one, that can provision a **ReadWriteOnce** volume.
* Outbound HTTPS (TCP 443) from the pod to Botyard, either direct or through an HTTP proxy.
* `kubectl`, [Helm](https://helm.sh/docs/intro/install/) 3.8 or later (for OCI charts),
  [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) 2.x or later, and `jq`.
* A self-hosted bot in Botyard, which you will enroll this cluster as. See
  the **Enrollment and devices** page.

The chart runs a single-replica StatefulSet. Botyard allows **one active device per bot**, and the
pod's volume holds the device key, so the volume *is* the device. There is no `replicaCount`.

The examples below use the namespace `botyard`, the release name `bot` and version `1.7.2414`.
With release name `bot`, the pod is `bot-botyard-runtime-selfhosted-0`.

```sh
VERSION=1.7.2414
kubectl create namespace botyard
```

The chart's pod meets the Kubernetes `restricted` Pod Security Standard: a non-root user, a
read-only root filesystem, all capabilities dropped and no service-account token. You can enforce
that standard on the namespace:

```sh
kubectl label namespace botyard pod-security.kubernetes.io/enforce=restricted
```

## 1. Verify what you pull

Botyard publishes the image and the chart to GitHub Container Registry. Both are public:

| Artifact                       | Reference                                                                           |
| ------------------------------ | ----------------------------------------------------------------------------------- |
| Container image (amd64, arm64) | `ghcr.io/botyard-ai/runtime-selfhosted:<version>`                                   |
| Helm chart (OCI)               | `oci://ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted`, `--version <version>` |

Each release is signed **keyless** with [cosign](https://docs.sigstore.dev/). The signing
certificate is issued to Botyard's GitHub Actions release workflow, and every signature is
recorded in the public Rekor transparency log. The image also carries a signed SPDX SBOM
attestation. Botyard holds no long-lived signing key, so you verify the signer's *identity*, not
a key:

```sh
IDENTITY='^https://github\.com/Botyard-AI/botyard/\.github/workflows/release-runtime-selfhosted\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+$'
ISSUER=https://token.actions.githubusercontent.com
```

`IDENTITY` accepts only the `release-runtime-selfhosted.yml` workflow in the `Botyard-AI/botyard`
repository, run from a release tag such as `v1.7.2414`. `ISSUER` is GitHub Actions' OIDC issuer.

Verify the image by tag, and keep the digest that cosign verified. The block takes the digest
only from a successful `cosign verify`, and accepts it only if it is a well-formed `sha256`
digest:

```sh
IMAGE_DIGEST=
if IMAGE_VERIFIED=$(cosign verify "ghcr.io/botyard-ai/runtime-selfhosted:${VERSION}" \
    --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER"); then
  IMAGE_DIGEST=$(printf '%s' "$IMAGE_VERIFIED" | jq -r '.[0].critical.image."docker-manifest-digest"')
fi
if printf '%s' "$IMAGE_DIGEST" | grep -Eqx 'sha256:[0-9a-f]{64}'; then
  echo "image verified: $IMAGE_DIGEST"
else
  IMAGE_DIGEST=
  echo "STOP: image verification failed. Do not install." >&2
fi
```

From here on, refer to the image **by that digest**. A tag can be moved; a digest cannot. Verify
the SBOM attestation on the same digest:

```sh
if [ -n "$IMAGE_DIGEST" ] && cosign verify-attestation --type spdxjson \
    "ghcr.io/botyard-ai/runtime-selfhosted@${IMAGE_DIGEST}" \
    --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" > /dev/null; then
  echo "SBOM attestation verified"
else
  echo "STOP: SBOM attestation verification failed. Do not install." >&2
fi
```

Verify the chart the same way, then pull it and check that the pulled chart has the digest you
verified:

```sh
CHART_DIGEST=
if CHART_VERIFIED=$(cosign verify "ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted:${VERSION}" \
    --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER"); then
  CHART_DIGEST=$(printf '%s' "$CHART_VERIFIED" | jq -r '.[0].critical.image."docker-manifest-digest"')
fi

if ! printf '%s' "$CHART_DIGEST" | grep -Eqx 'sha256:[0-9a-f]{64}'; then
  echo "STOP: chart verification failed. Do not install." >&2
elif ! PULL_OUTPUT=$(helm pull oci://ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted \
    --version "$VERSION" 2>&1); then
  printf '%s\n' "$PULL_OUTPUT" >&2
  echo "STOP: helm pull failed." >&2
elif printf '%s\n' "$PULL_OUTPUT" | grep -Fqx "Digest: ${CHART_DIGEST}"; then
  echo "chart verified and pulled: $CHART_DIGEST"
else
  printf '%s\n' "$PULL_OUTPUT" >&2
  echo "STOP: the pulled chart is not the chart you verified. Delete the .tgz. Do not install." >&2
fi
```

`helm pull` saves `botyard-runtime-selfhosted-<version>.tgz` in the current directory. You install
from that file in step 3, so Helm uses exactly the chart you verified.

Each block above ends by printing either a success line or a line starting with `STOP`. If you see
`STOP`, or any `cosign`, `jq` or `helm` error, for any reason (not only `no matching signatures`),
do not install. Fix the cause and run the whole block again.

> [!NOTE]
> **A 403 from ghcr.io on a public package**
>
> cosign, Helm and Docker read registry credentials from `~/.docker/config.json`. An expired
> `ghcr.io` login there makes even anonymous pulls of public packages fail with `403 Forbidden`.
> Run `docker logout ghcr.io` (or `helm registry logout ghcr.io`) and retry, or run the commands
> with `DOCKER_CONFIG` pointing at an empty directory.

### Mirroring into a private registry

You can mirror both artifacts into your own registry, for example Amazon ECR, and install from
there. `cosign copy` copies the image or chart together with its signatures and attestations, so
the same `cosign verify` commands, pointed at your mirror, still check Botyard's identity:

```sh
MIRROR=registry.example.com/botyard   # your registry and path

cosign copy "ghcr.io/botyard-ai/runtime-selfhosted@${IMAGE_DIGEST}" \
  "${MIRROR}/runtime-selfhosted:${VERSION}"
cosign copy "ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted:${VERSION}" \
  "${MIRROR}/charts/botyard-runtime-selfhosted:${VERSION}"
```

Mirroring does not change the image digest. Then set `image.repository` to your mirror. If your
nodes need credentials to pull from it, also set `imagePullSecrets`:

```yaml
image:
  repository: registry.example.com/botyard/runtime-selfhosted
  digest: sha256:<IMAGE_DIGEST>
imagePullSecrets:
  - name: my-registry-credentials
```

## 2. Store the enrollment token in a Secret

The pod enrolls itself as the bot's device on first start, using a **single-use enrollment
token** that is valid for 7 days. Mint it right before you install:

1. In Botyard, open **User menu** › **Settings** › **Bots** › *‹your bot›* › **Device**.
2. Select **Generate install command**.
3. Botyard shows a one-line install command for a workstation, ending in `--token <token>`.
   &#x2A;*Do not run that command.** Copy only the token, the value after `--token`.

The **Enrollment and devices** page covers tokens, revocation and
re-enrollment in detail.

Put the token in a Kubernetes Secret. Don't put it in a values file, and don't type it on a
command line where it would land in your shell history. This reads it from a hidden prompt:

```sh
read -rs BOTYARD_TOKEN   # paste the token, then press Enter
printf '%s' "$BOTYARD_TOKEN" | kubectl -n botyard create secret generic botyard-enrollment \
  --from-file=token=/dev/stdin
unset BOTYARD_TOKEN
```

If you manage Secrets with a secret store (for example External Secrets Operator), create a
Secret named `botyard-enrollment` with the token under the key `token` instead.

What happens to the token:

* On the **first** start, the container enrolls with it, then removes it from its environment
  before the daemon starts. Neither the container's init process nor the daemon holds it.
* A shell you open with `kubectl exec` **does** see `BOTYARD_ENROLLMENT_TOKEN`. Kubernetes sets
  a container's environment once, when the container starts, and does not update it later. So
  every `kubectl exec` into that container sees the token it started with, **even after you
  delete or change the Secret**.
* On **every later** start, the volume is already enrolled, so the token is not used again. A
  token that was already used or has expired is inert, so you can leave the Secret in place.

To remove the token from the running pod, wait until the bot is online (step 4), then delete the
Secret and replace the pod:

```sh
kubectl -n botyard delete secret botyard-enrollment
kubectl -n botyard rollout restart statefulset/bot-botyard-runtime-selfhosted
```

The chart marks the Secret reference optional, so the new pod starts without the variable.
Replacing the pod keeps its PersistentVolumeClaim, so the bot comes back already enrolled. Do the
same after any later change to the Secret.

## 3. Install the chart

Create a minimal values file. It pins the image by the digest you verified, points at the
Secret, and turns on the chart's egress-only NetworkPolicy:

```yaml title="values.yaml"
image:
  digest: sha256:<IMAGE_DIGEST>   # from step 1
enrollment:
  existingSecret: botyard-enrollment
networkPolicy:
  enabled: true
```

Install from the chart file you pulled and verified:

```sh
helm install bot "./botyard-runtime-selfhosted-${VERSION}.tgz" -n botyard -f values.yaml
```

When `image.digest` is set, the chart pulls the image by digest and ignores `image.tag`.

You can also install straight from the registry with
`helm install bot oci://ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted --version "$VERSION" -n botyard -f values.yaml`.
Helm then resolves the version tag again at install time, rather than using the chart file whose
digest you checked.

The pod needs no registry credentials: the image and chart are public. The NetworkPolicy only
takes effect if your cluster's network plugin enforces NetworkPolicy. On Amazon EKS, that means
the VPC CNI with network policy enabled.

## 4. Check that it worked

Wait for the pod to be ready. The chart defines no probes: the pod is ready once `botyardd` is
running, and the daemon reconnects to Botyard by itself.

```sh
kubectl -n botyard rollout status statefulset/bot-botyard-runtime-selfhosted
```

Read the logs:

```sh
kubectl -n botyard logs statefulset/bot-botyard-runtime-selfhosted
```

On a first start, look for these lines:

* `enrolled this device device_id=<id> custody_tier=file …`, then
  `botyard-selfhosted: enrollment complete`. The token was accepted, and this volume is now the
  bot's device.
* `connected to the Botyard runtime gateway gateway=wss://runtime.botyard.io …`. The daemon is
  connected.

Some INFO and WARN lines about hardware key storage are expected. They say that no Secure
Enclave, TPM or similar is available, so the device key is kept on the volume
(`custody_tier=file`).

On every **later** start, such as after the pod is deleted or rescheduled, these lines are
expected and are **not** an error:

```text
refusing to enroll: this machine is already bound and --replace was not given
botyard-selfhosted: state directory already enrolled; the enrollment token was not used
```

The volume is already this bot's device, so the token is left unused and the daemon starts and
reconnects.

Ask the daemon directly:

```sh
kubectl -n botyard exec bot-botyard-runtime-selfhosted-0 -- botyardd status
```

The `Botyard` line should read `connected`. kubectl may also print a line like
`Unknown stream id 1, discarding message`. That message comes from kubectl itself and is harmless.

Finally, in Botyard, **User menu** › **Settings** › **Bots** › *‹your bot›* › **Device** should show the device as **Online**. The device is
named after the pod unless you set `enrollment.deviceLabel`.

Before the bot can answer a message, it needs a model credential. Assign one with the `llm` scope
on **User menu** › **Settings** › **Bots** › *‹your bot›* › **Credentials**. Without one, a turn fails with
`No credential is configured for provider scope 'llm'`.

If the pod is running but the bot stays offline:

* `this machine is not enrolled` in the logs means no token reached a new volume. Check that
  `enrollment.existingSecret` names your Secret, then restart the pod.
* `enrollment failed` means the token was already used or has expired. Mint a new one, replace
  the Secret's value, and restart the pod.
* No `connected to the Botyard runtime gateway` line and no error usually means the pod can't
  reach Botyard: a NetworkPolicy, firewall or proxy is in the way, or the proxy URL uses an
  unsupported scheme. Today the daemon logs failed connection attempts only at debug level, so
  they don't appear in the default logs. Check outbound TCP 443 from the pod and the
  [proxy settings](#outbound-proxy-proxy).

The **Operations and troubleshooting** page covers crashloops, upgrades and
logs.

## Values that matter

The chart's
[`values.yaml`](https://github.com/Botyard-AI/botyard/blob/main/runtime/selfhosted/chart/botyard-runtime-selfhosted/values.yaml)
is the source of truth for every value and its default. These are the ones most installs touch.
Add them to `values.yaml` and run `helm upgrade bot ./botyard-runtime-selfhosted-${VERSION}.tgz
-n botyard -f values.yaml`.

### Host policy (`hostPolicy`)

`hostPolicy` is this machine's own limit on what the bot may do on the pod: read and write files,
run commands, and so on. The bot can only use an operation that Botyard requests *and* this
policy allows.

**Empty, the default, means the runtime performs no host operation.** No command runs and no file
is read or written. The policy does not decide which tools the model is offered. That depends on
what the bot is assigned in Botyard, and Botyard does not use your local policy to filter the
assignment. So under an empty policy:

* a host tool the bot is not assigned is not offered. The bot may answer that the tool (for
  example `process_start`) isn't available;
* a host tool the bot *is* assigned can still be offered. If the bot calls it, the call reaches the
  runtime, which **refuses** it and writes an audit line with `decision` `deny` and `reason`
  `local_policy`.

To let the bot use host tools, list the operations in `hostPolicy.enabled`:

```yaml
hostPolicy:
  enabled: [read, write, patch]
  root: /var/lib/botyard/workspaces
```

* `root` must be on the pod's volume (under `/var/lib/botyard/`) or under `/tmp`. It must not be
  the daemon's state directory, `/var/lib/botyard/state`, which holds the device key. The chart
  refuses those paths at install time and creates the root for you.
* `root` limits the *file* operations. It does not confine `exec`: a command runs as the same
  user and can read the device key. Enable `exec` only for a bot you would trust with this
  device's identity.
* Changing `hostPolicy` restarts the pod.
* To manage the policy yourself, set `hostPolicyConfigMap.name` to an existing ConfigMap instead.
  The root directory must then already exist.

The **Host policy and security** page explains each operation
and the trust model.

### Outbound proxy (`proxy`)

If the pod must reach the internet through a proxy, it must be an **HTTP proxy** (HTTP
`CONNECT`) with an `http://` URL. `https://` and `socks5://` proxies are not supported. The URL
needs a real host and either no port (meaning 80) or a port from 1 to 65535.

Set **one** of these, not both:

* `proxy.url` — a proxy without credentials, such as `http://proxy.corp.example:3128`. The chart
  checks the URL against the rules above when you install, and refuses it with an error naming
  `proxy.url`. It may not contain a user name or password.
* `proxy.existingSecret` — the name of a Secret whose `HTTPS_PROXY` key holds the URL. Use this
  for a proxy that needs credentials. The chart can't read Secrets, so **nothing checks this value
  at install time**. Check the URL before you store it. A wrong scheme, such as `https://` or
  `socks5://`, does not stop the pod: it starts and stays running, but never comes online in
  Botyard, and the failed connections are not logged at the default log level. See
  the **Operations and troubleshooting** page if the pod is running but the bot
  stays offline.

`proxy.noProxy` lists hosts to reach directly, with curl semantics: domains, `.suffix`, IP
addresses, CIDRs or `*`.

```yaml
proxy:
  url: http://proxy.corp.example:3128
  noProxy: .svc,.cluster.local
```

### Extra CA bundle (`extraCABundle`)

If your proxy inspects TLS, give the pod your proxy's CA certificate. The bundle is **added to**
the built-in public roots; it doesn't replace them. Put the PEM bundle in a ConfigMap or a Secret
and set exactly one of these:

```yaml
extraCABundle:
  configMapName: corp-ca   # or secretName: corp-ca
  key: ca.pem
```

If the bundle can't be read or isn't valid PEM, the daemon refuses to start.

### Egress NetworkPolicy (`networkPolicy`)

The chart can create an egress-only NetworkPolicy. It denies all ingress, and allows DNS plus
TCP 443. Your cluster's network plugin must enforce NetworkPolicy.

```yaml
networkPolicy:
  enabled: true
  # Optional: allow 443 only to these CIDRs, for example your proxy.
  egressCIDRs: [10.20.0.15/32]
  # Optional: extra ports, for example your proxy's port.
  extraEgressPorts:
    - port: 3128
      protocol: TCP
```

`egressCIDRs` restricts only the 443 and extra-port rule. DNS stays allowed to any destination.

### Storage (`persistence`)

The pod keeps its state on a ReadWriteOnce volume: the device key, the enrollment and the bot's
workspaces.

```yaml
persistence:
  storageClassName: gp3   # empty = the cluster default
  size: 10Gi
```

* `persistence.existingClaim` reuses an existing PVC, for example to restore a device after
  recreating the release.
* `helm uninstall` **keeps** the volume. Deleting the PVC deletes the device's identity: you
  would then revoke the device in Botyard and enroll again with a new token.

### Resources (`resources`)

The defaults request 250m CPU and 512Mi memory, with a 2Gi memory limit. Raise them if the bot
runs heavy commands on the pod:

```yaml
resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    memory: 2Gi
```

### Other values

* `enrollment.deviceLabel` — the device's name in Botyard. Defaults to the pod name.
* `extraEnv` — extra, non-secret environment variables. For example, set
  `BOTYARD_HOST_TOOL_AUDIT` to `off` to turn off the per-call audit lines. The chart refuses
  variables it manages itself, and the identity variables described below.
* `podLabels`, `podAnnotations`, `nodeSelector`, `tolerations`, `affinity`, `priorityClassName`
  and `terminationGracePeriodSeconds` work as in most charts.

## Don't connect the pod to a service mesh identity

Don't inject `SPIFFE_ENDPOINT_SOCKET` into this pod, and don't run your own SPIRE agent or
SPIFFE-based mesh identity for it.

A self-hosted bot proves who it is with the **device key it enrolled with**. Botyard's own hosted
bots use a different, Botyard-run identity system, which is found through that variable. If the
variable is present, today's daemon assumes it is running inside Botyard's hosted platform, tries
that identity path instead of its device key, and **will not connect**.

The chart refuses to set `SPIFFE_ENDPOINT_SOCKET` (and the related hosted-identity variables)
through `extraEnv`. A mesh admission webhook can still inject it, so exclude this namespace or
pod from any SPIFFE/SPIRE injection.

## Known limitations

* **One replica.** One active device per bot. There is no high-availability mode; restarts are
  covered by the daemon reconnecting and resuming.
* **Enrollment is by token only.** Workload-identity federation (such as IRSA or a service
  account's OIDC token) is not supported yet.
* **The device key is stored on the volume.** KMS- and TPM-backed key storage are not supported
  yet.
* **No customer SPIRE or SPIFFE mesh identity on this pod**, as explained above.
* **No self-update.** You upgrade by installing a newer, verified chart and image version.