Kubernetes Deployment

Deploy weavers on Kubernetes for remote code execution environments

Overview

Weavers are ephemeral Kubernetes pods that provide isolated execution environments for Loom agents. Each weaver runs a containerized instance of the Loom CLI with network isolation, resource limits, and audit logging.

Loom uses k3s (lightweight Kubernetes) for weaver provisioning. The server communicates with the k3s API to create, manage, and cleanup weaver pods.

Architecture

┌─────────────────┐
│  loom-server    │
│                 │
│  Provisioner    │───────┐
└─────────────────┘       │
                          │ kubectl API
                          ▼
                ┌──────────────────┐
                │   k3s cluster    │
                │                  │
                │  ┌────────────┐  │
                │  │ Weaver Pod │  │  Namespace: loom-weavers
                │  │            │  │
                │  │ ┌────────┐ │  │
                │  │ │ CLI    │ │  │  Image: ghcr.io/user/weaver:latest
                │  │ └────────┘ │  │
                │  │ ┌────────┐ │  │
                │  │ │ Audit  │ │  │  Sidecar: eBPF monitoring
                │  │ └────────┘ │  │
                │  └────────────┘  │
                └──────────────────┘

k3s Installation

Loom uses k3s for Kubernetes. The NixOS module handles installation and configuration.

NixOS Configuration

configuration.nix

services.loom-k3s = {
  enable = true;
  role = "server";  # "server" or "agent"
  clusterInit = true;  # true for first server
  
  # Disable built-in components (we use custom ingress)
  disableTraefik = true;
  disableServiceLB = false;
  disableLocalStorage = false;
  
  # Bind to localhost only (security)
  bindAddress = "127.0.0.1";
  
  # kubeconfig location
  kubeconfigPath = "/etc/rancher/k3s/k3s.yaml";
};

Namespace Setup

The k3s module automatically creates the loom-weavers namespace:

kubectl get namespace loom-weavers

If you need to create it manually:

kubectl create namespace loom-weavers

Image Pull Secrets

For private container registries (e.g., GitHub Container Registry):

1. Generate a GitHub Personal Access Token

   1. Go to GitHub Settings → Developer settings → Personal access tokens
   2. Create a token with read:packages scope
   3. Save the token to a file: /run/secrets/github-token
2. Enable ghcr.io secret in NixOS

   configuration.nix

   services.loom-k3s.ghcrSecret = {
     enable = true;
     username = "ghuntley";
     tokenFile = "/run/secrets/github-token";
   };
   
3. Verify the secret exists

   kubectl get secret ghcr-secret -n loom-weavers
   

Or create manually:

kubectl create secret docker-registry ghcr-secret \
  --namespace=loom-weavers \
  --docker-server=ghcr.io \
  --docker-username=ghuntley \
  --docker-password=ghp_xxxxxxxxxxxx

Weaver Configuration

Configure the weaver provisioner in loom-server:

configuration.nix

services.loom-server.weaver = {
  enable = true;
  
  # Kubernetes namespace for weavers
  namespace = "loom-weavers";
  
  # Path to kubeconfig (k3s default)
  kubeconfigPath = "/etc/rancher/k3s/k3s.yaml";
  
  # Cleanup interval for expired weavers
  cleanupIntervalSecs = 1800;  # 30 minutes
  
  # Weaver TTL limits
  defaultTtlHours = 4;
  maxTtlHours = 48;
  
  # Concurrency limits
  maxConcurrent = 64;
  
  # Pod readiness timeout
  readyTimeoutSecs = 60;
  
  # Image pull secrets for private registries
  imagePullSecrets = [ "ghcr-secret" ];
  
  # Audit sidecar for eBPF monitoring
  audit = {
    enable = true;
    image = "ghcr.io/ghuntley/loom-audit-sidecar:latest";
    batchIntervalMs = 100;
    bufferMaxBytes = 268435456;  # 256 MB
    metricsPort = 9090;
    healthPort = 9091;
  };
};

Environment Variables

Alternatively, configure via environment variables:

export LOOM_SERVER_WEAVER_ENABLED=true
export LOOM_SERVER_WEAVER_K8S_NAMESPACE=loom-weavers
export LOOM_SERVER_WEAVER_CLEANUP_INTERVAL_SECS=1800
export LOOM_SERVER_WEAVER_DEFAULT_TTL_HOURS=4
export LOOM_SERVER_WEAVER_MAX_TTL_HOURS=48
export LOOM_SERVER_WEAVER_MAX_CONCURRENT=64
export LOOM_SERVER_WEAVER_READY_TIMEOUT_SECS=60
export LOOM_SERVER_WEAVER_IMAGE_PULL_SECRETS=ghcr-secret
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Weaver Images

Weavers run container images with the Loom CLI pre-installed.

Building a Weaver Image

Use the provided Dockerfile:

docker/Dockerfile.weaver

FROM rust:1.83-slim-bookworm AS rust-builder

WORKDIR /app

RUN apt-get update && apt-get install -y \
    pkg-config \
    libssl-dev \
    && rm -rf /var/lib/apt/lists/*

COPY Cargo.toml Cargo.lock ./
COPY crates ./crates

RUN cargo build --release --package loom-cli

# Runtime image
FROM debian:bookworm-slim AS runtime

WORKDIR /workspace

RUN apt-get update && apt-get install -y \
    ca-certificates \
    git \
    curl \
    && rm -rf /var/lib/apt/lists/* \
    && useradd -m -u 1000 loom

COPY --from=rust-builder /app/target/release/loom /usr/local/bin/loom
COPY docker/weaver-entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

USER loom

ENTRYPOINT ["/entrypoint.sh"]

Building with Nix

nix build .#weaver-image
docker load < result

Pushing to Registry

# Build
docker build -f docker/Dockerfile.weaver -t ghcr.io/your-org/weaver:latest .

# Login to GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u your-username --password-stdin

# Push
docker push ghcr.io/your-org/weaver:latest

Managing Weavers

CLI Commands

List weavers

loom --server-url https://loom.example.com weaver ps

loom --server-url https://loom.example.com new --image ghcr.io/your-org/weaver:latest

loom --server-url https://loom.example.com attach <weaver-id>

loom --server-url https://loom.example.com weaver delete <weaver-id>

Kubernetes Commands

List pods

sudo kubectl get pods -n loom-weavers

sudo kubectl describe pod <pod-name> -n loom-weavers

sudo kubectl logs <pod-name> -n loom-weavers

sudo kubectl delete pod <pod-name> -n loom-weavers

sudo kubectl exec -it <pod-name> -n loom-weavers -- /bin/bash

Troubleshooting

ErrImagePull - Image not found

The container image doesn’t exist or is private.

Check:

1. Verify image exists: docker pull ghcr.io/your-org/weaver:latest
2. Check image pull secret: kubectl get secret ghcr-secret -n loom-weavers
3. Verify secret is configured in weaver settings

Fix:

# Recreate the secret
kubectl delete secret ghcr-secret -n loom-weavers
kubectl create secret docker-registry ghcr-secret \
  --namespace=loom-weavers \
  --docker-server=ghcr.io \
  --docker-username=your-username \
  --docker-password=$GITHUB_TOKEN

Pod immediately exits (Succeeded status)

The container has no long-running entrypoint.

Weaver images must run a persistent process. The default entrypoint script keeps the container alive:

weaver-entrypoint.sh

#!/bin/bash
set -e

# Start loom REPL
exec loom repl --server-url $LOOM_SERVER_URL

Verify your entrypoint doesn’t exit immediately.

401 Unauthorized when creating weavers

You need to authenticate first:

loom --server-url https://loom.example.com login

This creates a session token in ~/.loom/credentials.json.

Namespace not found

The loom-weavers namespace doesn’t exist.

Create it:

kubectl create namespace loom-weavers

Or wait for the NixOS service to create it (runs after k3s starts).

Permission denied accessing kubeconfig

The loom-server user needs read access to /etc/rancher/k3s/k3s.yaml.

Fix permissions:

sudo chmod 640 /etc/rancher/k3s/k3s.yaml
sudo chown root:loom-server /etc/rancher/k3s/k3s.yaml

The NixOS module handles this automatically via the loom-k3s group.

Audit Sidecar

Weavers include an optional eBPF audit sidecar for monitoring system calls, network activity, and file access.

Configuration

configuration.nix

services.loom-server.weaver.audit = {
  enable = true;
  image = "ghcr.io/ghuntley/loom-audit-sidecar:latest";
  batchIntervalMs = 100;        # Event batching interval
  bufferMaxBytes = 268435456;   # 256 MB buffer
  metricsPort = 9090;           # Prometheus metrics
  healthPort = 9091;            # Health check endpoint
};

Viewing Audit Logs

# View audit sidecar logs
kubectl logs <pod-name> -c audit-sidecar -n loom-weavers

# Metrics endpoint
kubectl port-forward <pod-name> 9090:9090 -n loom-weavers
curl http://localhost:9090/metrics

Resource Limits

Set resource limits on weaver pods:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: weaver-quota
  namespace: loom-weavers
spec:
  hard:
    requests.cpu: "32"
    requests.memory: 64Gi
    limits.cpu: "64"
    limits.memory: 128Gi
    pods: "64"

Apply with:

kubectl apply -f weaver-quota.yaml

Next Steps

Server Configuration

Complete environment variable reference

Database Migrations

Manage SQLite schema changes