# Operations and troubleshooting > Run a self-hosted Botyard runtime day to day: status, logs and audit lines, upgrades, restarts and storage, offline behaviour, and start-up refusals. Canonical URL: https://docs.botyard.io/docs/self-hosted/operations This page is for the platform team that runs the self-hosted runtime (`botyardd`) in its own Kubernetes cluster. It assumes the chart is installed as described in [Install on Kubernetes](/docs/self-hosted/install-kubernetes). The [self-hosted overview](/docs/self-hosted) explains what runs where. The commands below use `` for the namespace and `` for the Helm release name. The pod is always `-botyard-runtime-selfhosted-0`. ## Check the runtime ```sh kubectl -n exec -botyard-runtime-selfhosted-0 -- botyardd status ``` `botyardd status` prints a report and exits `0` only when the runtime is enrolled, the daemon is running and it is reaching Botyard. The first line is the summary: | First line | Meaning | | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- | | `Connected to Botyard.` | Healthy. | | `Enrolled, but not currently reaching Botyard.` | Enrolled and running, but the connection is down. It retries on its own. Check egress and the proxy. | | `Not connected to Botyard — this machine is not enrolled.` | No enrollment on this volume. See [Running, but the bot stays offline](#running-but-the-bot-stays-offline). | | `Not connected — the daemon started before this machine enrolled.` | Restart the pod so the daemon picks up the enrollment. | The report also shows the bot and device IDs, the daemon version and the state directory (`/var/lib/botyard/state`). If the device key line reads `MISSING — re-enroll to mint a new one`, the volume has lost the key. See [Restarts and storage](#restarts-and-storage). `botyardd status` works even when the proxy or CA settings are broken. When the pod is crashlooping, `kubectl exec` usually misses the container, so run the command in a one-off copy of the pod instead. The copy mounts the same volume and doesn't start the daemon: ```sh kubectl -n debug -botyard-runtime-selfhosted-0 --copy-to=botyard-diag \ --same-node --container=botyardd -it -- botyardd status kubectl -n delete pod botyard-diag ``` In the copy, the report says the daemon isn't running. That's expected, because the copy doesn't start it. Read the enrollment and device-key lines instead. `botyardd config` prints the effective configuration, and where each value came from, the same way. ## Logs ```sh kubectl -n logs -botyard-runtime-selfhosted-0 # current container kubectl -n logs -botyard-runtime-selfhosted-0 --previous # the one that just crashed ``` The container writes two streams. Route them separately. | Stream | What it is | How to recognise it | | ---------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | **stderr** | The daemon's own log: start-up, connection state, errors. | One JSON object per line, with `timestamp`, `level`, `fields.message` and `target`. Lines from the entrypoint script start `botyard-selfhosted: `. | | **stdout** | The host-tool **audit** log: one line per host-tool call Botyard asked this runtime to make. | Each line starts `{"botyard_audit":"host_tool",`. | A healthy start logs `botyardd starting`, then `connected to the Botyard runtime gateway`. Some lines look alarming but are expected: * **WARN `stronger key custody was unavailable`.** A container has no hardware key store, so the runtime keeps its device key on the volume and reports `custody_tier=file`. * **The enrollment refusal on every restart.** See [Restarts and storage](#restarts-and-storage). * **`Unknown stream id 1, discarding message`.** kubectl prints this during `kubectl exec … botyardd status`. It is harmless. Daemon log settings, set through the chart's `extraEnv`: * `BOTYARD_LOG` sets the filter. The default is `info`. Use `debug` while troubleshooting connectivity. * `BOTYARD_LOG_FORMAT` is `json` (the default) or `text`. Any other value stops the daemon with `BOTYARD_LOG_FORMAT: expected one of json, text, got ""`. ### Audit lines An audit line records what was asked and how it ended, never secrets or output. Fields, in order: * `botyard_audit` * `ts_ms` * `bot_id` * `turn_id`, when Botyard supplied one * `source`: `control_plane` or `local_relay` * `tool` * the target, as one of: * an exec's `exe` (basename only) and `argc` (argument count, never the arguments); * a file tool's `path`. * `decision`: `allow` or `deny` * `reason`, on a denial: `local_policy`, `unknown_tool` or `not_on_this_seam` * `is_error` * `exit_code` * `duration_ms` Lines appear only for calls that reach the runtime. With the chart's default `hostPolicy` (nothing enabled), every command or file call that reaches it is refused, so any such line is a refusal (`decision` `deny`, `reason` `local_policy`). Whether you see any at all depends on which host tools the bot is assigned in Botyard. An audit line records a request; only `decision` `allow` means it ran. Missing fields are left out, not written as `null`. Don't build alerting that assumes `turn_id` is always present. If your log pipeline stalls, `botyardd` drops audit lines rather than slow a tool call down. The next line it writes is preceded by `{"botyard_audit":"host_tool_dropped", … "dropped":N}`, so the gap is visible. Audit lines are on by default. `BOTYARD_HOST_TOOL_AUDIT=off` turns them off, and any value other than `on` or `off` stops the daemon with `BOTYARD_HOST_TOOL_AUDIT must be "on" or "off"`. The **Host policy and security** page explains which tools can appear here. ## Upgrades You choose when to upgrade. The runtime has no self-update, and Botyard never changes the version you run. To upgrade, verify the new release as described in [Verify what you pull](/docs/self-hosted/install-kubernetes#1-verify-what-you-pull), then: ```sh helm upgrade oci://ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted \ --version -n --reuse-values \ --set image.digest=sha256: ``` > [!WARNING] > If you pinned `image.digest`, change it on every upgrade. A pinned digest wins over the chart's > own image version, so bumping only `--version` (with `--reuse-values`) upgrades the chart and > keeps the **old** image. What happens during an upgrade: * **The pod is replaced.** The chart runs one replica. The old pod is stopped completely before the new one starts, so two runtimes never hold the device at once. Changing an inline `hostPolicy` value also replaces the pod, because the runtime reads its policy only at start-up. * **Changes outside the chart's values don't restart anything.** The runtime reads its settings once, at start-up. Editing your own `hostPolicyConfigMap`, the proxy Secret, or the CA bundle doesn't replace the pod, and the running daemon keeps its old settings. Restart it yourself: `kubectl -n rollout restart statefulset/-botyard-runtime-selfhosted`. * **The old daemon gets time to stop.** It receives `SIGTERM`, logs `botyardd shutting down`, and has `terminationGracePeriodSeconds` (default `60`) to stop cleanly. * **A reply in progress is cut short.** The turn is interrupted. When the new pod starts, the runtime reports it to Botyard as failed, so the conversation doesn't wait forever. The reply ends with: `The bot's runtime restarted while this reply was in progress, so it was stopped. Any tools it had started may already have run; check before sending the message again.` Tools the bot had started may have finished, so check what already happened before you send the message again. * **Anything started with `watch_process` stops.** Those processes run inside the container, so they stop with the pod. Botyard tells the bot the watched process ended without a recorded exit, or was lost to a runtime restart. The process may have done part of its work before it stopped, so check what it left behind before you start it again. * **Messages sent during the restart get the offline answer.** See [While the runtime is offline](#while-the-runtime-is-offline). * **Identity is kept.** The new pod uses the same volume, so it doesn't re-enroll and doesn't need a new token. Botyard doesn't currently enforce a minimum runtime version on self-hosted runtimes. Stay on a published release, and upgrade when Botyard asks you to. ## Restarts and storage The volume claim (`state--botyard-runtime-selfhosted-0`, or your `persistence.existingClaim`) holds the device key and the enrollment. **The volume is the device.** * **Pod restarts, node moves and upgrades** keep the device, as long as the same volume comes back. On every start after the first, the log shows the enrollment step refusing to run, then the daemon starting normally. This is expected and doesn't spend the token in the Secret: ```text botyardd: this machine is already Botyard device for bot . … refusing to enroll: this machine is already bound and --replace was not given botyard-selfhosted: state directory already enrolled; the enrollment token was not used ``` The token stays in the pod spec, so a `kubectl exec` shell can read it from its own environment. It can't enroll anything once spent, but you can delete it from the Secret after the first successful start. * **`helm uninstall` keeps the volume.** Reinstalling with the same release name and namespace reuses it. * **Losing the volume means enrolling again.** Deleting the claim or the namespace, or losing the underlying disk, loses the device key. Botyard still lists the old device as the bot's active device. To recover: 1. Revoke the old device on **User menu** › **Settings** › **Bots** › *‹your bot›* › **Device**. 2. Mint a new enrollment token. 3. Update the Secret with the new token. 4. Delete the claim (if anything is left of it) and the pod. Revoke **before** enrolling. While the old device is active, enrollment is refused with `This bot already has an active device. Revoke the current device, generate a fresh enrollment token, and run the installer again…`. * **Moving the runtime to a different bot is never done by a restart.** Revoke the device, delete the claim, and enroll with the other bot's token. See the **Enrollment** page. Back up the volume as you would any secret: whoever holds a copy of it holds the device identity. Revoke the device in Botyard if a copy may have leaked. ## While the runtime is offline A self-hosted bot only works while its runtime is connected. The runtime is offline while the pod is stopped, restarting, or can't reach Botyard. **In the app.** **User menu** › **Settings** › **Bots** › *‹your bot›* › **Device** shows **Online** while the device has been seen in the last 5 minutes, and **Offline** with **Last seen …** after that. The badge is a liveness hint and can lag a short outage. What decides whether a message is sent is the check below, made at send time. **Chat messages.** Botyard waits up to 5 seconds for the runtime to reconnect. If it doesn't, the message isn't sent. A card under it reads **The bot's machine is offline**, followed by **This message wasn't sent. Retry once the machine is back online.** and **Last seen …**. Nothing ran, so choosing **Retry** once the runtime is back is safe. **Schedules and awakes.** If the runtime is offline when a scheduled run or awake is due, Botyard keeps retrying for a reconnect window: * the window is the schedule's own wait, or **3 minutes** if it sets none, with a maximum of 1 hour; * for a recurring schedule, the window also ends when the next run is due. If the runtime reconnects within the window, the run happens late. If it doesn't, the run is recorded as missed, with the error `the bot's machine did not reconnect within the wake's reconnect window`, and isn't caught up later. **Watches.** The runtime renews each `watch_process` watch about every 20 seconds. If it can't reach Botyard for longer than the lease (60 seconds) plus a grace period, Botyard ends the watch and tells the bot its watched work was lost to a runtime restart. A process that keeps running through a short network outage still reports its real exit when the runtime reconnects, if the watch hasn't been ended by then. **Reconnecting.** The runtime reconnects by itself, with backoff. You don't need to restart the pod after a network outage. A device revoked in Botyard is different: the runtime logs `Botyard refused this device; the daemon will not reconnect until it is enrolled again` and stays disconnected until you enroll it again. ## Troubleshooting Find the symptom, then read the pod's logs (`--previous` for a crashloop). Every message quoted below is printed by the chart, the entrypoint or the runtime as shown. ### Install or upgrade fails (`helm install` / `helm upgrade`) The chart checks your values before anything is deployed. The error starts `Error: execution error at (botyard-runtime-selfhosted/templates/statefulset.yaml…)`, followed by one of these messages: | Message | Cause | Fix | | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `proxy.url "": botyardd only accepts an http:// proxy URL (not https://, socks5://, ...)` | `proxy.url` isn't `http://`. | Use your proxy's `http://` address. The runtime tunnels TLS through it with `CONNECT`. | | `proxy.url "": expected http://HOST[:PORT] with a real host (bracketed IPv6 allowed) and a port of 1-65535, or none for 80` | Empty host, bad port, or a path. | Write it as `http://proxy.example:3128`. | | `proxy.url must not contain credentials (user:pass@). …` | Credentials in a plain value. | Put the full URL in a Secret and set `proxy.existingSecret`. | | `hostPolicy.root "" must not be, contain or sit inside the daemon state directory /var/lib/botyard/state: …` | The host-tool root overlaps the directory holding the device key. | Use e.g. `/var/lib/botyard/workspaces`. | | `hostPolicy.root "" is not a canonical path (repeated '/', '.' or '..' segments); write it as ""` | The root contains `//`, `.` or `..`. | Use the path the message suggests. | | `hostPolicy.root "" must be an absolute path` | Relative root. | Start it with `/`. | | `hostPolicy.root "" must be on the state volume (/var/lib/botyard/...) or /tmp: …` | The root filesystem is read-only, so the chart can't create the root anywhere else. | Use a path under `/var/lib/botyard/`. To use your own volume, set `hostPolicyConfigMap` and create the root yourself. | | `extraEnv: is managed by the chart (or must not be set on a self-hosted device); use the dedicated value` | `extraEnv` sets a variable the chart owns, another proxy spelling, or a hosted-identity variable such as `SPIFFE_ENDPOINT_SOCKET`. | Use the dedicated value (`proxy`, `extraCABundle`, `hostPolicy`, `enrollment`). Don't set the identity variables at all. | | `… set X OR Y, not both` | Two mutually exclusive values are set, for example `proxy.url` and `proxy.existingSecret`. | Keep one. | ### The pod never starts (`ContainerCreating`) If the CA ConfigMap or Secret named in `extraCABundle`, or the key in it, doesn't exist, the volume can't be mounted. The pod stays in `ContainerCreating`, `botyardd` never runs, and there are no container logs. `kubectl -n describe pod -botyard-runtime-selfhosted-0` shows a `FailedMount` event from Kubernetes. Create the ConfigMap or Secret, or set `extraCABundle.key` to the key that holds the PEM (default `ca.pem`). ### The pod crashloops | Symptom in the logs | Cause | Fix | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cannot resolve the outbound network settings` with `HTTPS_PROXY is set but is not a usable proxy URL (expected e.g. http://proxy.example:3128)` | The proxy URL from `proxy.existingSecret` has no host, or a port that isn't a number from 1 to 65535. `HTTP_PROXY` and `ALL_PROXY` are checked the same way. The value itself is never logged, because it may contain a password. | Fix the URL in the Secret, then delete the pod. | | `cannot resolve the outbound network settings` with `BOTYARD_EXTRA_CA_BUNDLE=: cannot read the CA bundle: …` | The runtime can't read the bundle file. The chart mounts it at `/etc/botyard/ca/` with Kubernetes' default mode `0644`, which uid `10001` can read. With the chart, this means something changed the file's permissions, such as an admission policy. Without the chart, `BOTYARD_EXTRA_CA_BUNDLE` points at a missing or unreadable path. | Without the chart, fix the path in `BOTYARD_EXTRA_CA_BUNDLE`. With the chart, make the mounted file readable by uid `10001`. Then delete the pod. | | `cannot resolve the outbound network settings` with `BOTYARD_EXTRA_CA_BUNDLE=: no certificate in the bundle`, `not a valid PEM certificate: …` or `not usable as a trust anchor: …` | The bundle is empty, or isn't PEM certificates. | Store PEM-encoded CA certificates (`-----BEGIN CERTIFICATE-----`). | | `botyard-selfhosted: enrollment failed (exit 1); mint a fresh token in Botyard and update the Secret`, after `botyardd: This enrollment token is no longer valid. It may have already been used, been replaced by a newer token, or expired. …` | The token was already spent, replaced by a newer token, or is older than 7 days. Botyard answers `403` (or `404` for a token it never issued) and records the attempt in its audit log. | Mint a new token, put it in the Secret, delete the pod. | | The same, but you never started the pod with this token before | **Generate install command** on **User menu** › **Settings** › **Bots** › *‹your bot›* › **Device** gives a `curl … \| sh -s -- --token ` command. Running that command anywhere (for example, to try it on a workstation) installs the desktop runtime there and spends the token. | Don't run the install command. Mint a new token and copy only the token value into the Secret. | | `botyardd: unexpected response from Botyard: This bot already has an active device. …` | The bot still has an active device, for example after the volume was lost or replaced. | Revoke the old device on **User menu** › **Settings** › **Bots** › *‹your bot›* › **Device**, mint a new token, update the Secret, delete the pod. | | `botyardd: cannot reach Botyard: error sending request for url (https://api.botyard.io/v1/public/device-enrollment/redeem)` | First enrollment can't reach Botyard: egress blocked, DNS, proxy or TLS inspection. | See [Egress is blocked](#egress-is-blocked). | | `cannot load host policy` with `invalid BOTYARD_HOST_POLICY JSON: …` | Malformed JSON in your `hostPolicyConfigMap`. | Fix the JSON. The chart only checks `hostPolicy`, not a ConfigMap you provide. | | `cannot initialize host tools` with `host file operation failed: No such file or directory (os error 2)` | The host-policy `root` doesn't exist. The message doesn't name the path. The chart creates the root for `hostPolicy`, but not for `hostPolicyConfigMap`. | Create the `root` directory named in your policy, on the volume and owned by uid `10001`, then delete the pod. | | `cannot bind the control socket` with `cannot restrict the Botyard control socket at to this user` | The state directory isn't owned by the runtime's user (`10001`). This happens when the image runs without the chart and `BOTYARD_HOME` points at the root of a volume, which usually arrives owned by root. | Point `BOTYARD_HOME` and `HOME` at a *child* of the mount, as the chart does (`/var/lib/botyard/state`). The runtime creates it with the right owner and mode. | ### Running, but the bot stays offline | Symptom | Cause | Fix | | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ``this machine is not enrolled; run `botyardd enroll --token ...` `` | A new volume, and no token reached the pod. | Set `enrollment.existingSecret` to a Secret holding a fresh token, then delete the pod. | | `pod identity source not ready; not connecting to Botyard`, about 10 seconds after start | The volume isn't enrolled, no token reached the pod, and `SPIFFE_ENDPOINT_SOCKET` is in the pod (typically injected by a service mesh or SPIRE webhook). An unenrolled runtime takes that variable as Botyard's hosted identity and doesn't connect. | Provide a token (see the row above) and exclude the namespace or pod from mesh identity injection, then delete the pod. Once enrolled, the runtime ignores the variable and logs `this daemon is enrolled, so it runs as a self-hosted device; pod identity variables in its environment are ignored`. | | `Botyard refused this device; the daemon will not reconnect until it is enrolled again` | The device was revoked. | Enroll again: delete the claim, mint a token, update the Secret, delete the pod. | | No errors, `botyardd status` says `Enrolled, but not currently reaching Botyard.` | The connection to Botyard is failing. | See [Egress is blocked](#egress-is-blocked). | ### The bot is online, but replies fail | Symptom | Cause | Fix | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The reply is empty, with `No credential is configured for provider scope 'llm'. Add one under Settings -> Credentials in Botyard. This will not resolve on its own.` | The runtime is connected and took the turn, but no model (LLM) credential is bound to this bot. The app may still show the bot as ready. | Add a model credential under **User menu** › **Settings** › **Credentials** › **LLM Providers** if the organization has none, bind it to this bot for the `llm` scope with a default model, then send the message again. | | The bot says a host tool such as `process_start` or `file_read` "isn't available" | The tool isn't **assigned** to the bot in Botyard. Assignment decides what the bot is offered; your local policy doesn't. | Assign the tool to the bot in Botyard. Also allow the operation in `hostPolicy.enabled`, or the call is refused (next row). | | The bot calls a host tool and it's refused | The operation isn't in `hostPolicy.enabled`. The chart's default `hostPolicy` enables nothing. The audit log has a line for the call with `decision` `deny` and `reason` `local_policy`. | Add the operation to `hostPolicy.enabled` and set `hostPolicy.root` to an absolute directory on the volume, then run `helm upgrade`. The runtime needs `root` whenever anything is enabled, including only `exec` or `process_start`. Without it, the runtime stops at start-up with `cannot initialize host tools`. See the **Host policy and security** page. | ### Egress is blocked For its own control-plane traffic, the runtime needs DNS and outbound HTTPS (TCP 443), directly or through your proxy, and Botyard never connects in to it. (Commands it starts through `exec` and services it supervises are ordinary processes in the pod and can listen or use other ports; the chart's NetworkPolicy is off by default.) It connects to `api.botyard.io`, and to the runtime gateway that Botyard names at connect time (today `wss://runtime.botyard.io`). Moving files between the bot and Botyard can also reach Botyard's object storage, which today is `hel1.your-objectstorage.com` (or a bucket host under it), not a `botyard.io` host. If you restrict egress by name, allow those hosts, then check the `debug` log for anything still blocked. Today, a running runtime that can't connect retries quietly. At the default log level it writes **no** error for this, and `botyardd status` is the only sign. To see why it can't connect, add `BOTYARD_LOG=debug` to `extraEnv` and look for `cannot open a Botyard session; will retry` and `cannot reach the Botyard runtime gateway`. Remove it afterwards. Common causes: * **`networkPolicy.enabled` with a proxy.** The policy allows TCP 443 only. Add your proxy's port to `networkPolicy.extraEgressPorts` (e.g. `[{port: 3128, protocol: TCP}]`), and its address to `networkPolicy.egressCIDRs` if you restrict destinations. * **A proxy URL that isn't `http://`.** A credentialed proxy set through `proxy.existingSecret` isn't checked by Helm. The runtime currently **starts** with an `https://` or `socks5://` proxy, then fails every connection to Botyard with `proxy scheme "https" is not supported for WebSocket dials; use an http:// proxy URL`, visible only at `debug`. Use the proxy's `http://` address. * **`NO_PROXY` covers Botyard.** Hosts that match `proxy.noProxy` are dialled directly. Make sure `botyard.io` isn't in it if direct egress is blocked. * **TLS inspection.** If your proxy re-signs TLS, add its CA with `extraCABundle`. Without it, connections fail certificate checks. ### Image pulls | Symptom | Cause | Fix | | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ImagePullBackOff` from a private mirror such as ECR | The mirror needs credentials or the repository path is wrong. The `host-tool-root` init container uses the same image. | Set `image.repository` to the mirror, add `imagePullSecrets` (or node IAM for ECR), and keep `image.digest`. `cosign copy` keeps the digest and signatures. | | `403` when pulling or verifying the public image or chart, from your workstation | Stale `ghcr.io` credentials in `~/.docker/config.json`, or in the Helm or cosign registry config. They are sent even for public packages and are rejected. | Run `docker logout ghcr.io` and `helm registry logout ghcr.io`, or use a clean `DOCKER_CONFIG`. No credentials are needed. | ## Contacting Botyard support Send: * the chart version and image digest (`helm -n list`, and the `image` of the pod); * the output of `botyardd status`; * the stderr log around the problem, with `--previous` for a crashloop; * what you changed last: values, proxy, network policy, volume. > [!CAUTION] > Never send an enrollment token, a proxy URL that contains a password, the contents of the state > volume, or any other secret. Redact them from logs and values before you send anything. If a token > was exposed, mint a new one, which replaces the old token. If the device key may have been > exposed, revoke the device.