Skip to content

Execution nodes

Experiments don't run on the control node — they run on execution nodes (runners): machines you attach. A node clones the project's private code and runs scheduled experiments in an isolated git worktree per run. Connect whatever compute you have; nodes are heterogeneous by design.

Nothing runs until a node is attached

A live project with no execution node will queue experiments but never run them. Attaching a node is what turns idle compute into results.

Turn this machine into a node

autolab serve --project alice/nanochat

serve starts a background daemon and returns once the node is registered — the node stays connected after you close the terminal or SSH session. No tmux, nohup, or systemd unit required.

Because it fetches private code, registration needs a researcher+ token (researcher, admin, or owner). The token is read from, in order: --token, then AUTOLAB_TOKEN, then your stored autolab login.

Manage the background node

autolab serve status     # running? project, control-node view, recent log
autolab serve stop       # disconnect this machine (idempotent; --name for one node)
autolab serve log        # this node's entire runner log (-n N for a tail, -f to follow)

serve status exits 0 when a daemon is running on this machine and 1 otherwise, so it's scriptable. It also shows what the control node currently thinks of the node (online/offline, last heartbeat, running experiment) when you're signed in.

Daemon state and logs live under ~/.local/state/autolab/serve/ (one slot per node identity — machine + node name). If that directory is on a shared home (NFS), records written by other machines show up in status as read-only entries — manage those daemons on their own machine. You can also disconnect a node from the dashboard — the daemon notices and exits; autolab serve status on the machine will say it was disconnected.

Keeping a node current. The control node stamps a fresh CLI build on every deploy; a running daemon keeps its old code and logs a warning when it drifts. Update and restart:

autolab update
autolab serve stop && autolab serve --project <owner>/<slug>   # or: serve … --replace

The runner keeps a timestamped narration of everything the node does — registration and reconnects, accepted jobs, agent bash commands, failures, shutdowns. autolab serve log prints it on the machine, and the node ships the same log to the control node with each heartbeat, so it also renders live in the Node Log panel on the node's dashboard page — you can debug a remote node without SSHing into it.

One daemon per node identity. A node's identity is its machine plus its node name, and the default name is unique per GPU allocation (<host>-gpu<pci>, falling back to <host>-job<slurm-job-id>, then <host>). In practice:

  • Re-running serve on the same resource is a no-op that reports the existing daemon.
  • Starting a different project on the same resource errors and points you at autolab serve stop --name <node>; pass --replace to switch it in one step.
  • Separate GPU allocations each get their own node — two interactive Slurm sessions on one host (or several machines sharing an NFS home) can all run autolab serve, and each becomes its own node. serve prints what else the machine is serving so nothing runs behind your back; serve stop disconnects all of this machine's nodes, --name just one.
  • Two shells that see the same GPUs derive the same name, so an accidental duplicate is still caught — use --name only to deliberately run two nodes on one resource (e.g. MPS sharing).

serve options

Option Default Meaning
--project (required) Project slug owner/slug to run jobs for.
--name derived Node name shown in the dashboard (default is unique per machine / GPU allocation).
--poll-interval 5 Seconds between polls for new work.
--token — Researcher+ token to register with.
--autolab-home ~/.autolab Root dir for node workspaces.
--foreground off Stay attached to the terminal (Ctrl-C stops).
--replace off Stop the same-named background node first, then start.
--type local slurm connects a Slurm cluster (see below).
--max-parallel-jobs 4 Concurrent sbatch runs a Slurm node accepts (local is always 1).

Node workspaces live under <AUTOLAB_HOME>/<hostname>/projects/<owner>--<slug>/. The hostname namespacing makes a shared NFS root collision-safe.

Headless / remote GPU box

An agent (or you) can bootstrap a remote node over SSH with a dedicated, revocable token — no browser on the remote box:

TOK=$(autolab token create --name gpu-box)        # a fresh researcher+ token
ssh gpubox 'curl -fsSL https://app.autolab.ai/install.sh | sh'   # install the CLI
ssh gpubox "AUTOLAB_TOKEN=$TOK ~/.local/bin/autolab serve --project alice/nanochat"

The last command returns once the node is online; the daemon keeps running on the box after the SSH session ends. Check on it later with ssh gpubox "~/.local/bin/autolab serve status".

Revoke the token with autolab token revoke gpu-box. See Tokens & API keys.

Slurm clusters

On a Slurm login node,

autolab serve --type slurm --project alice/nanochat

scans the cluster (partitions, node states, GPUs, your accounts/QoS), shows the inventory, then connects the login node as an execution node whose experiments run as sbatch jobs on the cluster — the daemon on the login node only codes, submits, and streams logs/status. Requirements and behavior:

  • AUTOLAB_HOME must be on a filesystem the compute nodes share (experiment worktrees are prepared on the login node and executed by the batch job).
  • QoS/account handling: many clusters reject jobs without an explicit --qos/--account. AutoLab derives them from your sacctmgr associations automatically when unambiguous; otherwise set --slurm-qos/--slurm-account (or let the agent pin resources.qos per run). Standard SBATCH_* environment variables exported before autolab serve are honored too.
  • --slurm-partition, --slurm-account, --slurm-qos, --slurm-gres set submission defaults; unset values fall through to the derived values, then cluster defaults.
  • Environment setup (setup_command, e.g. uv sync) re-runs inside the batch job on the compute node, so the venv matches the compute architecture.
  • The job's time limit is the experiment's expected runtime; Slurm state maps back honestly (TIMEOUT, OUT_OF_MEMORY, CANCELLED, …) and the run log carries [autolab]-prefixed submit/finish lines with the Slurm job id.
  • Restarting the daemon (or the login node) re-attaches to still-running sbatch jobs and delivers verdicts of jobs that finished in between.
  • The cluster node runs up to --max-parallel-jobs experiments concurrently (default 4): the agent codes the next experiment on the login node while earlier ones execute as sbatch jobs.
  • The agent requests resources per run (partition, GPUs, CPUs, memory, time limit, QoS/account — plus any other sbatch flag, e.g. --exclusive or --constraint=a100) from the cluster inventory it sees — including each partition's allowed accounts/QoS, so restricted partitions get valid submissions; your --slurm-* flags are the fallback defaults. Ask the agent in chat to schedule differently and it records the preference in the node notes for future runs.
  • A preempted or node-failed run is requeued automatically (no failure analysis) and reschedules on the next pass — up to 3 automatic reruns, after which it is handed to the agent as a failure.
  • The job page shows the exact submit command (sbatch --time=… --partition=… trees/<id>/.autolab-run.sh) and the Slurm job id for every run, live while it executes — with the submitted script (workspace env + setup + run) one click away under the command.
  • The dashboard marks the node with a slurm badge and shows the cluster panel — partitions, GPU inventory, your accounts/QoS, derived submit defaults — plus a Check GPU availability button that asks the node for live idle counts.

You can still run plain autolab serve inside an allocation (salloc/srun) to attach a single allocation as its own node — including several allocations at once, each with its own serve daemon ("one daemon per node identity" above).

List & remove nodes

autolab nodes              # list the project's execution nodes
autolab nodes add          # how to attach a machine (prints the instructions)
autolab nodes rm <id>      # request shutdown + deregister a node (id or name)

Removing a node (CLI or the dashboard's Disconnect button) asks the runner to shut down: it exits on its next poll, and a mid-run experiment is cancelled and requeued for another node.

Keep the agent and queue in view while nodes run with autolab status.