# Host policy and security > What a self-hosted Botyard runtime can and cannot do in your cluster: trust boundaries, the local host policy, sudo, the host-tool audit log and network controls. Canonical URL: https://docs.botyard.io/docs/self-hosted/host-policy-and-security This page is for the platform and security teams who run a Botyard bot's runtime (`botyardd`) in their own Kubernetes cluster. It describes what Botyard can and cannot do on that compute, the local policy you control, and what the audit log records. It also says where each control **stops**. Read those parts before you enable anything. For what runs where, read the [self-hosted overview](/docs/self-hosted) first. > [!NOTE] > **Availability** > > Self-hosted runtimes are enabled per organization by Botyard. If your organization does not see > the self-hosted option when it creates a bot, contact Botyard to request access. Kubernetes is the > only supported platform today. Linux VMs and macOS are not yet available. ## Trust boundaries | Boundary | What holds | Where it stops | | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Outbound-only control plane** | The runtime itself listens on no network port for Botyard: it opens its own HTTPS/WSS connection to Botyard, and Botyard sends work over that connection. The chart creates no Service and no Ingress. | Botyard still sends work over the connection the runtime opened. Anything the bot is assigned **and** your local policy allows runs when Botyard asks for it. This covers the runtime's own transport only. Commands it starts through `exec`, and services it supervises, can open listeners and connect on other ports. Having no Service or Ingress does not stop direct connections to the Pod IP. Use the chart's NetworkPolicy (off by default) or your own ingress controls. See [Network](#network). | | **No model-provider credentials on your compute** | Model calls are brokered. The runtime sends only a request body over its Botyard connection, and Botyard calls the model provider with its own credentials. | This covers Botyard's provider credentials. Credentials you or the bot place in the workspace, or inject through secret leases, are on your compute. | | **Secrets through short-lived leases** | The runtime holds references, never secret values. When a command needs a secret, Botyard resolves the lease server-side and returns the value once, for one process. The runtime keeps it in memory only and never logs it. | After it is injected, the value is in that process's environment. Other processes running as the same user can generally read it, including a command started through `exec`. | | **Enrollment token kept out of the runtime** | The single-use enrollment token is read from a Kubernetes Secret. The container's entrypoint removes it from the environment before it starts the runtime, so neither the runtime nor any command it starts holds it. On every restart after the first, the token is not used. | The token stays in the Secret, and a shell you open with `kubectl exec` gets it from the pod spec. After the first successful start, delete it from the Secret if you want it gone. | | **Device key on your volume** | The key that identifies the runtime to Botyard is created on the persistent volume, in a directory the runtime owns (`0700`). The volume **is** the device. | The key is a file on disk (the `file` custody tier). Anyone who can read the volume can impersonate the device, and so can `exec` in the pod (see [What `root` does not bound](#what-root-does-not-bound)). KMS and TPM custody are not yet supported. | | **Locked-down container** | Runs as UID/GID `10001`, never as root. Meets Pod Security Admission `restricted`: read-only root filesystem, no privilege escalation, all capabilities dropped, `RuntimeDefault` seccomp. No service-account token is mounted. The image has no `sudo`, and setuid/setgid bits are stripped from every binary. | This is container isolation inside your cluster. It does not limit what the runtime's own user can do inside its own pod. | ## The local host policy The **local host policy** is the operator's ceiling on what the runtime may do on your compute. It is a JSON document in the `BOTYARD_HOST_POLICY` environment variable. Set it in one of two ways: * **`hostPolicy`** in the Helm chart's values. The chart renders it into a ConfigMap and adds a checksum annotation to the pod, so changing it rolls the pod. * **`hostPolicyConfigMap.name`** (and `.key`, default `policy.json`), pointing at a ConfigMap you manage. The chart cannot see its contents, so changing it does **not** roll the pod. Restart the pod yourself. The runtime reads the policy **once, at startup**. A change takes effect only after a restart. If the value is not valid JSON, is larger than 64 KiB, or contains an unknown field, the runtime refuses to start. It never falls back to a more permissive default. ```yaml title="values.yaml" hostPolicy: enabled: [read, write, patch] root: /var/lib/botyard/workspaces ``` | Field | Default | Meaning | | --------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | `[]` | Operations the runtime may perform. Empty means **none**, not all. Values: `read`, `write`, `patch`, `exec`, `process_status`, `process_cancel`, `sudo_exec`, `service_register`, `service_start`, `service_stop`, `service_restart`, `service_status`, `service_list`, `service_logs`, `service_unregister`. | | `root` | none | Absolute path of the file-tool root. **Required for every operation**, including `exec`. Without it nothing is allowed. | | `service_owner` | `disabled` | `native` lets the runtime run supervised services. The `service_*` operations are refused unless this is `native`. | | `allow_sudo` | `false` | Operator opt-in for `sudo_exec`. See [sudo](#sudo). | | `sudo_executables` | `[]` | Absolute executables `sudo_exec` may run. **Empty means unrestricted**, not none. | | `child_oom_score_adj` | `500` | OOM-killer preference for commands the runtime starts (1–1000), so a runaway command is killed before the runtime is. | | `browser` | none | The runtime-owned browser. Requires `service_owner: native`. | ### Deny by default With no policy, or with `enabled` empty, the runtime performs no host operation. That is the chart's default (`hostPolicy: {}`). What the policy guarantees is that the runtime **performs** nothing. It does not decide which tools the model is offered. That depends on what the bot is assigned in Botyard, and Botyard does not use your local policy to filter the assignment. So under an empty policy: * a host tool the bot is not assigned is not offered. The bot may answer that the tool (for example `process_start`) isn't available; * a host tool the bot *is* assigned can still be offered. If the bot calls it, the call reaches the runtime, which **refuses** it and writes an audit line with `decision` `deny` and `reason` `local_policy`. For a well-formed request, that line includes the same metadata as for an allowed call: `exe` and `argc` for a command, or `path` for a file tool. Either way, no command runs and no file is read or written. To let the bot use host tools, set `hostPolicy.enabled` and `hostPolicy.root` ([Host policy values](/docs/self-hosted/install-kubernetes#host-policy-hostpolicy)). A few other operations are gated by `enabled` members, so they are refused unless the member is listed: | Operation | Allowed only when the policy allows | | -------------------------------------------------------------------------- | ----------------------------------- | | Brokered Git, loopback readiness probe, process watches | `exec` | | Viewing an image from the workspace, uploading a workspace file to Botyard | `read` | | Downloading a file into the workspace | `write` | | The browser | the `browser` block | The one exception is the runtime's in-memory busy markers, which Botyard uses to tell whether the runtime is busy. They touch no file, process or network resource, so they are served under any policy. ### Effective policy: assigned in Botyard ∩ allowed locally A host tool runs only when both of these are true: 1. **Botyard asks for it.** The tool must be assigned to the bot in Botyard. For self-hosted bots, Botyard never fills in a default policy: it asks only for what the bot's configuration says. 2. **Your local policy allows it.** The runtime checks the local policy on every call, whatever Botyard asked for. Botyard cannot change your local policy. Nothing it sends over the connection modifies it. The only place to change it is your cluster. ### What `root` bounds `root` bounds the **file tools** (`read`, `write`, `patch`). File paths are resolved relative to the root and opened one component at a time, without following links. Symlinks are refused, even ones that point inside the root. Only regular files are read or written. The chart also refuses a `root` that is, contains or sits inside the runtime's state directory (`/var/lib/botyard/state`, which holds the device key). The root must be on the volume (`/var/lib/botyard/...`) or under `/tmp`. The chart creates it with an unprivileged init container. With `hostPolicyConfigMap`, the chart cannot see the root, so it must already exist, and the chart cannot check where it is. ### What `root` does not bound **`root` does not confine `exec`.** A command started with `exec` runs as the runtime's own user (`10001`) with that user's full permissions in the pod. It can: * read the state directory, including the **device key**; * generally read the environment of other processes running as the same user, including injected secrets; * reach anything the pod's network egress allows. The host policy is not an OS sandbox. Enable `exec` only for bots you would trust with the device's identity. To limit what `exec` can reach on the network, use the [NetworkPolicy](#network) or your own egress controls. ### sudo `allow_sudo` is an operator opt-in. The runtime reads it only from `BOTYARD_HOST_POLICY`, so Botyard cannot turn it on for you, and Botyard never applies a default policy to a self-hosted bot. For `sudo_exec` to run a command, all of these must hold: * `sudo_exec` is assigned to the bot in Botyard; * `sudo_exec` is in `enabled`, `allow_sudo` is `true`, and `root` is set; * the executable is in `sudo_executables`, or `sudo_executables` is empty, which means unrestricted; * the OS `sudo` policy allows it. The runtime runs `/usr/bin/sudo -n -- `. `sudo_exec` is served only to Botyard. A harness on the same machine cannot call it through the local relay. > [!WARNING] > **sudo does not work on the Helm chart** > > The self-hosted image ships no `sudo` binary and no setuid binaries, and the pod runs with > privilege escalation disabled. Setting `allow_sudo: true` in the chart therefore cannot elevate > anything: `sudo_exec` calls fail. Leave `allow_sudo` off. The **Local host policy** panel still > lists `sudo_exec` as allowed if you configure it, because the panel shows your policy, not what > the image can do. ### Seeing the policy in Botyard For a self-hosted bot, **User menu** › **Settings** › **Bots** › *‹your bot›* › **Device** has a **Local host policy** card, marked **Reported by the runtime**. It shows **Allowed operations**, **File root**, **Sudo**, **Services** and **Browser**. * **Allowed operations** lists what the runtime would actually admit, not everything you wrote: `sudo_exec` appears only with `allow_sudo`, service operations only with `service_owner: native`, and nothing appears without a `root`. * The runtime sends the report each time it connects. It is read-only and Botyard cannot edit it. * It is **self-reported** by the runtime. Botyard does not verify it and does not use it to decide anything: the runtime enforces its own policy on every call. * If the report had to be shortened to fit its size limits, the card says so. In that case the file root and the sudo list may be incomplete. ## Host-tool audit log The runtime writes one JSON line to **stdout** for every host-tool call it handles, whether the call ran or was refused. Its own diagnostic logs go to **stderr**. The audit log is produced on your compute and is **not sent to Botyard**. Botyard keeps its own audit of the calls it makes. This log exists so you do not have to rely on Botyard's. Lines appear only for calls that reach the runtime. With the chart's default empty policy every command or file call that reaches it is refused, so any such line is a refusal. Whether you see any at all depends on what the bot is assigned. See [Deny by default](#deny-by-default). `BOTYARD_HOST_TOOL_AUDIT` controls it: * unset: **on** for a self-hosted runtime; * `on` or `off` (case-insensitive) to choose explicitly. Set it through the chart's `extraEnv`; * any other value: the runtime refuses to start. ### Line format Every audit line starts with `{"botyard_audit":"host_tool"`: ```json {"botyard_audit":"host_tool","ts_ms":1793738763311,"bot_id":"…","turn_id":"…","source":"control_plane","tool":"process_start","exe":"git","argc":2,"decision":"allow","is_error":false,"exit_code":0,"duration_ms":412} ``` Fields appear in this order. Fields marked *optional* are **left out** when they have no value. They are never `null`. | Field | Meaning | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `botyard_audit` | Always `host_tool`. | | `ts_ms` | Unix time in milliseconds. | | `bot_id` | *Optional.* The enrolled bot's ID. | | `turn_id` | *Optional.* The Botyard turn the call belongs to, when Botyard attributed it to one. Never present on `local_relay` calls. It is a label only and never authorizes anything. | | `source` | `control_plane` (Botyard named the tool) or `local_relay` (a harness on this machine named it). | | `tool` | The tool name, for example `file_read`, `process_start` or `sudo_exec`. | | `exe` | *Optional.* For `process_start` and `sudo_exec`: the **basename** of the executable only. | | `argc` | *Optional.* For `process_start` and `sudo_exec`: how many arguments were passed. Never the arguments themselves. | | `path` | *Optional.* For `file_read`, `file_write` and `file_patch`: the root-relative path, cut at 512 characters. Never the content. | | `decision` | `allow` (it ran, though it may still have failed) or `deny` (refused before running). | | `reason` | *Optional, on `deny`.* `local_policy` (your policy does not allow it), `unknown_tool`, or `not_on_this_seam` (for example `sudo_exec` from the local relay). | | `is_error` | *Optional, on `allow`.* Whether the tool reported failure. | | `exit_code` | *Optional, on `allow`.* The process exit code, when the result carries one. | | `duration_ms` | How long the runtime spent on the call. | `exe`, `argc` and `path` are filled in only when the request was valid for that tool. Any other tool, an unknown name or a malformed request gets none of them. **Never in the log:** command arguments, file contents, tool output, or secret values. ### What the audit log does not cover The log records **calls to host tools**. It does not record what a command does after it starts: child processes, files it touches, or network connections it opens. To audit those, use your own runtime security tooling. ### Delivery and overflow Writing an audit line never delays a tool call: * Lines go into a bounded in-memory queue of **1,024** lines, written by a dedicated thread. * If the queue is full, the line is **dropped** and counted. Before the next line it writes, the writer emits a marker with the count, so a gap is always visible: ```json {"botyard_audit":"host_tool_dropped","ts_ms":1793738763999,"dropped":17} ``` * If writing to stdout fails, a warning goes to stderr and that line is lost. The runtime keeps serving. * At shutdown the runtime waits up to **two seconds** to flush. Lines still queued after that are lost. The audit log is therefore best effort, not a guaranteed record. Alert on `host_tool_dropped`. ### Shipping to a SIEM The runtime does not ship the log anywhere itself. Your cluster's log pipeline collects the container's stdout like any other container log (for example Fluent Bit, Vector, or the CloudWatch agent on EKS). Route lines whose message starts with `{"botyard_audit":` to your audit sink. That prefix matches both `host_tool` and `host_tool_dropped` lines. Treat everything else as diagnostic logging. To check the lines by hand: ```sh kubectl -n logs -botyard-runtime-selfhosted-0 | grep '^{"botyard_audit":' ``` ## Supply chain Releases of the image (`ghcr.io/botyard-ai/runtime-selfhosted`) and the Helm chart (`ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted`) are signed **keyless** with cosign: * The signing identity is the GitHub Actions workflow `release-runtime-selfhosted.yml`, running on a `v*` release tag. * Signatures are recorded in the public Rekor transparency log. * Botyard holds no long-lived signing key. * The image carries an SPDX SBOM attestation. Verify the signature and SBOM before you install, and pin the image by digest. The verification commands are in [Verify what you pull](/docs/self-hosted/install-kubernetes#1-verify-what-you-pull). ## Network For its own control-plane traffic, the runtime makes only outbound connections, over TCP 443 (HTTPS and WSS), and needs DNS. It connects to: * the Botyard API, `https://api.botyard.io`; * the runtime gateway. Botyard names its origin on each connection; today it is `wss://runtime.botyard.io`; * object storage, when a workspace file is uploaded to or downloaded from Botyard. The runtime uses a presigned URL that Botyard supplies for that one transfer, so the host is whatever Botyard's storage uses. Botyard does not yet publish a complete, stable hostname list. If you allowlist by hostname, expect it to need updating. The runtime also works through an HTTP CONNECT proxy (`proxy.url` or `proxy.existingSecret`). The chart can create an **egress-only NetworkPolicy**. It is **off by default**: ```yaml title="values.yaml" networkPolicy: enabled: true egressCIDRs: ["10.20.0.15/32"] # e.g. your proxy; empty = any destination extraEgressPorts: - port: 3128 protocol: TCP ``` When enabled, the policy: * denies all ingress; * allows DNS (UDP and TCP 53) to any destination; * allows TCP 443, plus any `extraEgressPorts`, to **any destination**. If you set `egressCIDRs`, only to those CIDRs. Keep in mind: * The TCP 443 list above is the runtime's own traffic. Commands started through `exec`, and services the runtime supervises, are ordinary processes in the pod. They can open listening sockets and connect on any port your network allows. With the NetworkPolicy off (the default), the chart does not restrict ingress to the Pod IP. The absence of a Service or Ingress only means nothing advertises the pod. To block inbound connections, enable this policy or apply your own NetworkPolicy or firewall controls. * The policy is per pod. It applies to everything in the pod, including commands started through `exec`. With `egressCIDRs` empty, those commands can reach any host on port 443. * It filters by IP address, not hostname. To allow only Botyard's hostnames, send traffic through your proxy and allowlist the hostnames there, then set `egressCIDRs` to the proxy's address. * A NetworkPolicy only takes effect if your cluster's network plugin enforces NetworkPolicy. On EKS with the Amazon VPC CNI, network-policy support must be turned on.