Troubleshooting

Session stuck in provisioning

• kubectl -n holtdroid describe pod <session>: usually the redroid image is still being pulled (about 1.5 GB) or no node matches nodeSelector/ tolerations with a free slot.
• Enable the pre-pull DaemonSet to avoid the pull on first use.

Session fails in booting

• CreateContainerError: … path escapes from parent: the cluster doesn’t honour supplementalGroupsPolicy: Strict (Kubernetes 1.31+, containerd 2.x).
• Android never boots: the node’s kernel needs binder (CONFIG_ANDROID_BINDERFS). The android container’s log is Android’s own (kubectl -n holtdroid logs <session> -c android); Android also writes to the node’s kernel log.
• No screen, or a black one: the node needs a GPU with a Mesa driver at redroid.gpu (/dev/dri/renderD128).
• The agent’s log: kubectl -n holtdroid logs <session> -c agent.
• Images built for one architecture: the session and redroid images are linux/amd64 only. arm64 nodes can’t run them.

Install fails with INSTALL_FAILED_NO_MATCHING_ABIS

The redroid images are x86_64 with ARM translation (libndk_translation), which runs arm64-v8a libraries. APKs that ship only 32-bit ARM libraries (armeabi-v7a without arm64-v8a) can’t be installed: no Android 12–16 x86_64 image translates 32-bit ARM. Ask for a build that includes arm64-v8a (or x86_64).

App installs but crashes immediately

• ARM-only native libraries run through Android’s ARM translation layer; a few libraries (heavy NEON assembly, JIT engines, some DRM) don’t work under it. adb logcat from your connected computer shows libndk_translation in the crash’s backtrace.
• The app may detect root, an emulator or Frida and exit. The agent log shows the Frida spawn and the app’s exit. Holtdroid never repackages an APK, so such apps need a build without the checks.

Frida doesn’t attach

The agent starts the app under Frida to turn on WebView debugging; nothing in the APK changes.

• The session fails with adb root: … or start frida-server: …: after boot the agent runs adb root, pushes frida-server and starts it. The redroid images are userdebug builds, so adb root works; a different image in redroid.images must be one too.
• Only the app’s main process is instrumented. A WebView in another process of the app may show up without debugging enabled.
• Apps started from the launcher are instrumented too: the agent notices the new process within about a second and attaches to it. A WebView created in that first second gets debugging once the script loads.
• Upgrading Frida means bumping frida-go and FRIDA_VERSION in images/session.Dockerfile together: the agent’s devkit and the device’s frida-server must be the same version.
• Building the session image on Apple Silicon works, running the agent there under Rosetta doesn’t (Frida needs eventfd). Test on an amd64 node.

DevTools shows no targets

• The app has not created a WebView yet; targets appear as soon as it does.
• If the app starts WebViews in a separate process, pick that process in the target selector.
• Check the agent’s /v1/devtools/targets through the portal: https://<portal>/s/<session>/v1/devtools/targets.

DevTools panel stays blank

The portal fetches the DevTools frontend from Google’s frontend host and keeps it in devTools.storage (the api’s storage when unset; a cache volume there means a crawl after every restart, so point it at an S3 prefix). It does so ahead of sessions for every Chromium and WebView build it lists, and on the first open of any other revision, which can take a minute. The api needs outbound HTTPS for that; session Pods don’t.

VPN location doesn’t change the IP

• bypassCIDRs must not contain the address range of the VPN servers.
• The sidecar logs lookup ….cluster.local: … mdns: query timeout: sing-box sends *.local names to mDNS. Set vpn.resolver to your cluster DNS IP (kubectl -n kube-system get svc kube-dns).
• A WireGuard tunnel that drops whenever another session starts: the same key is in use from several places (other session Pods, or a client outside Holtdroid on the same server). Move WireGuard into a shared VPN gateway and give it servers nobody else uses with that key.
• WireGuard logs sendmmsg: message too long: lower its mtu to 1280 on overlay networks.

No metrics or audit lines

• Metrics are off until telemetry.metrics.listen is set (chart: telemetry.metrics.enabled). curl <pod>:9090/metrics from inside the cluster; session pods only accept scrapes from telemetry.metrics.from.
• The PodMonitor needs the Prometheus Operator CRDs, and your Prometheus usually only picks monitors with its own release label (telemetry.metrics.podMonitor.labels).
• Audit lines go to stdout, regular logs to stderr. A collector that only reads one stream misses the other. Filter on "log":"audit".
• holtdroid_audit_dropped_total{sink="webhook"} growing: the webhook is down or slow; the api log says why (audit webhook dropped a batch).