Operations and troubleshooting

This page is for the platform team that runs the self-hosted runtime (botyardd) in its own Kubernetes cluster. It assumes the chart is installed as described in Install on Kubernetes. The self-hosted overview explains what runs where.

The commands below use <ns> for the namespace and <release> for the Helm release name. The pod is always <release>-botyard-runtime-selfhosted-0.

Check the runtime

kubectl -n <ns> exec <release>-botyard-runtime-selfhosted-0 -- botyardd status

botyardd status prints a report and exits 0 only when the runtime is enrolled, the daemon is running and it is reaching Botyard. The first line is the summary:

The report also shows the bot and device IDs, the daemon version and the state directory (/var/lib/botyard/state). If the device key line reads MISSING — re-enroll to mint a new one, the volume has lost the key. See Restarts and storage.

botyardd status works even when the proxy or CA settings are broken. When the pod is crashlooping, kubectl exec usually misses the container, so run the command in a one-off copy of the pod instead. The copy mounts the same volume and doesn't start the daemon:

kubectl -n <ns> debug <release>-botyard-runtime-selfhosted-0 --copy-to=botyard-diag \
  --same-node --container=botyardd -it -- botyardd status
kubectl -n <ns> delete pod botyard-diag

In the copy, the report says the daemon isn't running. That's expected, because the copy doesn't start it. Read the enrollment and device-key lines instead. botyardd config prints the effective configuration, and where each value came from, the same way.

Logs

kubectl -n <ns> logs <release>-botyard-runtime-selfhosted-0            # current container
kubectl -n <ns> logs <release>-botyard-runtime-selfhosted-0 --previous # the one that just crashed

The container writes two streams. Route them separately.

A healthy start logs botyardd starting, then connected to the Botyard runtime gateway. Some lines look alarming but are expected:

• WARN stronger key custody was unavailable. A container has no hardware key store, so the runtime keeps its device key on the volume and reports custody_tier=file.
• The enrollment refusal on every restart. See Restarts and storage.
• Unknown stream id 1, discarding message. kubectl prints this during kubectl exec … botyardd status. It is harmless.

Daemon log settings, set through the chart's extraEnv:

• BOTYARD_LOG sets the filter. The default is info. Use debug while troubleshooting connectivity.
• BOTYARD_LOG_FORMAT is json (the default) or text. Any other value stops the daemon with BOTYARD_LOG_FORMAT: expected one of json, text, got "<value>".

Audit lines

An audit line records what was asked and how it ended, never secrets or output. Fields, in order:

• botyard_audit
• ts_ms
• bot_id
• turn_id, when Botyard supplied one
• source: control_plane or local_relay
• tool
• the target, as one of:
  • an exec's exe (basename only) and argc (argument count, never the arguments);
  • a file tool's path.
• decision: allow or deny
• reason, on a denial: local_policy, unknown_tool or not_on_this_seam
• is_error
• exit_code
• duration_ms

Lines appear only for calls that reach the runtime. With the chart's default hostPolicy (nothing enabled), every command or file call that reaches it is refused, so any such line is a refusal (decision deny, reason local_policy). Whether you see any at all depends on which host tools the bot is assigned in Botyard. An audit line records a request; only decision allow means it ran.

Missing fields are left out, not written as null. Don't build alerting that assumes turn_id is always present.

If your log pipeline stalls, botyardd drops audit lines rather than slow a tool call down. The next line it writes is preceded by {"botyard_audit":"host_tool_dropped", … "dropped":N}, so the gap is visible. Audit lines are on by default. BOTYARD_HOST_TOOL_AUDIT=off turns them off, and any value other than on or off stops the daemon with BOTYARD_HOST_TOOL_AUDIT must be "on" or "off".

The Host policy and security page explains which tools can appear here.

Upgrades

You choose when to upgrade. The runtime has no self-update, and Botyard never changes the version you run. To upgrade, verify the new release as described in Verify what you pull, then:

helm upgrade <release> oci://ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted \
  --version <X.Y.Z> -n <ns> --reuse-values \
  --set image.digest=sha256:<new-digest>

What happens during an upgrade:

• The pod is replaced. The chart runs one replica. The old pod is stopped completely before the new one starts, so two runtimes never hold the device at once. Changing an inline hostPolicy value also replaces the pod, because the runtime reads its policy only at start-up.
• Changes outside the chart's values don't restart anything. The runtime reads its settings once, at start-up. Editing your own hostPolicyConfigMap, the proxy Secret, or the CA bundle doesn't replace the pod, and the running daemon keeps its old settings. Restart it yourself: kubectl -n <ns> rollout restart statefulset/<release>-botyard-runtime-selfhosted.
• The old daemon gets time to stop. It receives SIGTERM, logs botyardd shutting down, and has terminationGracePeriodSeconds (default 60) to stop cleanly.
• A reply in progress is cut short. The turn is interrupted. When the new pod starts, the runtime reports it to Botyard as failed, so the conversation doesn't wait forever. The reply ends with: The bot's runtime restarted while this reply was in progress, so it was stopped. Any tools it had started may already have run; check before sending the message again. Tools the bot had started may have finished, so check what already happened before you send the message again.
• Anything started with watch_process stops. Those processes run inside the container, so they stop with the pod. Botyard tells the bot the watched process ended without a recorded exit, or was lost to a runtime restart. The process may have done part of its work before it stopped, so check what it left behind before you start it again.
• Messages sent during the restart get the offline answer. See While the runtime is offline.
• Identity is kept. The new pod uses the same volume, so it doesn't re-enroll and doesn't need a new token.

Botyard doesn't currently enforce a minimum runtime version on self-hosted runtimes. Stay on a published release, and upgrade when Botyard asks you to.

Restarts and storage

The volume claim (state-<release>-botyard-runtime-selfhosted-0, or your persistence.existingClaim) holds the device key and the enrollment. The volume is the device.

• Pod restarts, node moves and upgrades keep the device, as long as the same volume comes back. On every start after the first, the log shows the enrollment step refusing to run, then the daemon starting normally. This is expected and doesn't spend the token in the Secret:

  botyardd: this machine is already Botyard device <device-id> for bot <bot-id>.
  … refusing to enroll: this machine is already bound and --replace was not given
  botyard-selfhosted: state directory already enrolled; the enrollment token was not used

  The token stays in the pod spec, so a kubectl exec shell can read it from its own environment. It can't enroll anything once spent, but you can delete it from the Secret after the first successful start.

• helm uninstall keeps the volume. Reinstalling with the same release name and namespace reuses it.

• Losing the volume means enrolling again. Deleting the claim or the namespace, or losing the underlying disk, loses the device key. Botyard still lists the old device as the bot's active device. To recover:

  1. Revoke the old device on User menu › Settings › Bots › ‹your bot› › Device.
  2. Mint a new enrollment token.
  3. Update the Secret with the new token.
  4. Delete the claim (if anything is left of it) and the pod.

  Revoke before enrolling. While the old device is active, enrollment is refused with This bot already has an active device. Revoke the current device, generate a fresh enrollment token, and run the installer again….

• Moving the runtime to a different bot is never done by a restart. Revoke the device, delete the claim, and enroll with the other bot's token. See the Enrollment page.

Back up the volume as you would any secret: whoever holds a copy of it holds the device identity. Revoke the device in Botyard if a copy may have leaked.

While the runtime is offline

A self-hosted bot only works while its runtime is connected. The runtime is offline while the pod is stopped, restarting, or can't reach Botyard.

In the app. User menu › Settings › Bots › ‹your bot› › Device shows Online while the device has been seen in the last 5 minutes, and Offline with Last seen … after that. The badge is a liveness hint and can lag a short outage. What decides whether a message is sent is the check below, made at send time.

Chat messages. Botyard waits up to 5 seconds for the runtime to reconnect. If it doesn't, the message isn't sent. A card under it reads The bot's machine is offline, followed by This message wasn't sent. Retry once the machine is back online. and Last seen …. Nothing ran, so choosing Retry once the runtime is back is safe.

Schedules and awakes. If the runtime is offline when a scheduled run or awake is due, Botyard keeps retrying for a reconnect window:

• the window is the schedule's own wait, or 3 minutes if it sets none, with a maximum of 1 hour;
• for a recurring schedule, the window also ends when the next run is due.

If the runtime reconnects within the window, the run happens late. If it doesn't, the run is recorded as missed, with the error the bot's machine did not reconnect within the wake's reconnect window, and isn't caught up later.

Watches. The runtime renews each watch_process watch about every 20 seconds. If it can't reach Botyard for longer than the lease (60 seconds) plus a grace period, Botyard ends the watch and tells the bot its watched work was lost to a runtime restart. A process that keeps running through a short network outage still reports its real exit when the runtime reconnects, if the watch hasn't been ended by then.

Reconnecting. The runtime reconnects by itself, with backoff. You don't need to restart the pod after a network outage. A device revoked in Botyard is different: the runtime logs Botyard refused this device; the daemon will not reconnect until it is enrolled again and stays disconnected until you enroll it again.

Troubleshooting

Find the symptom, then read the pod's logs (--previous for a crashloop). Every message quoted below is printed by the chart, the entrypoint or the runtime as shown.

Install or upgrade fails (helm install / helm upgrade)

The chart checks your values before anything is deployed. The error starts Error: execution error at (botyard-runtime-selfhosted/templates/statefulset.yaml…), followed by one of these messages:

The pod never starts (ContainerCreating)

If the CA ConfigMap or Secret named in extraCABundle, or the key in it, doesn't exist, the volume can't be mounted. The pod stays in ContainerCreating, botyardd never runs, and there are no container logs. kubectl -n <ns> describe pod <release>-botyard-runtime-selfhosted-0 shows a FailedMount event from Kubernetes. Create the ConfigMap or Secret, or set extraCABundle.key to the key that holds the PEM (default ca.pem).

The pod crashloops

Running, but the bot stays offline

The bot is online, but replies fail

Egress is blocked

For its own control-plane traffic, the runtime needs DNS and outbound HTTPS (TCP 443), directly or through your proxy, and Botyard never connects in to it. (Commands it starts through exec and services it supervises are ordinary processes in the pod and can listen or use other ports; the chart's NetworkPolicy is off by default.) It connects to api.botyard.io, and to the runtime gateway that Botyard names at connect time (today wss://runtime.botyard.io). Moving files between the bot and Botyard can also reach Botyard's object storage, which today is hel1.your-objectstorage.com (or a bucket host under it), not a botyard.io host. If you restrict egress by name, allow those hosts, then check the debug log for anything still blocked.

Today, a running runtime that can't connect retries quietly. At the default log level it writes no error for this, and botyardd status is the only sign. To see why it can't connect, add BOTYARD_LOG=debug to extraEnv and look for cannot open a Botyard session; will retry and cannot reach the Botyard runtime gateway. Remove it afterwards.

Common causes:

• networkPolicy.enabled with a proxy. The policy allows TCP 443 only. Add your proxy's port to networkPolicy.extraEgressPorts (e.g. [{port: 3128, protocol: TCP}]), and its address to networkPolicy.egressCIDRs if you restrict destinations.
• A proxy URL that isn't http://. A credentialed proxy set through proxy.existingSecret isn't checked by Helm. The runtime currently starts with an https:// or socks5:// proxy, then fails every connection to Botyard with proxy scheme "https" is not supported for WebSocket dials; use an http:// proxy URL, visible only at debug. Use the proxy's http:// address.
• NO_PROXY covers Botyard. Hosts that match proxy.noProxy are dialled directly. Make sure botyard.io isn't in it if direct egress is blocked.
• TLS inspection. If your proxy re-signs TLS, add its CA with extraCABundle. Without it, connections fail certificate checks.

Image pulls

Contacting Botyard support

Send:

• the chart version and image digest (helm -n <ns> list, and the image of the pod);
• the output of botyardd status;
• the stderr log around the problem, with --previous for a crashloop;
• what you changed last: values, proxy, network policy, volume.