> ## Documentation Index > Fetch the complete documentation index at: https://openpipe-art-austin-monarch-multinode-training.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Multi-node deployment ART's distributed runtime consumes a Monarch host mesh. SkyPilot can provision that mesh, but it is a deployment tool rather than an ART dependency. The ART process does not launch another SkyPilot cluster from inside its allocation. ## Controller program The bootstrap accepts a source script or import path to a top-level async function. A script reference such as `train.py:main` adds the script's directory to every ART-owned worker's Python path, so source synchronized by SkyPilot does not need to be packaged first. SkyPilot runs the bootstrap on every node, but only rank 0 imports the module and invokes the controller. `examples/multinode/program.py` is a complete CPU-runnable Yes/No/Maybe smoke: ```python theme={null} import asyncio import os import socket import art from art.distributed import ( ArtLaunchContext, ArtRuntime, InstalledAsyncCallable, compile_topology, ) REWARDS = {"yes": 0.5, "no": 0.75, "maybe": 1.0} async def rollout( _model: art.TrainableModel, answer: str, _config: None ) -> art.Trajectory: messages: art.MessagesAndChoices = [ {"role": "user", "content": f"Respond with {answer}."}, {"role": "assistant", "content": answer}, ] return art.Trajectory( messages_and_choices=messages, reward=REWARDS[answer], metadata={ "answer": answer, "hostname": socket.gethostname(), "process_id": os.getpid(), }, ) async def main(launch: ArtLaunchContext) -> None: host_count = launch.host_count runtime = await ArtRuntime.start( launch.host_mesh, compile_topology( cluster=launch.homogeneous_cluster( cpu_slots=1, startup_timeout_s=90, rpc_timeout_s=30, ) ), ) try: workers = tuple(range(host_count)) executor = runtime.rollout_executor( InstalledAsyncCallable.from_callable(rollout), target_workers=host_count, ) executor.set_workers(workers) model = art.TrainableModel( name="multinode-smoke", project="art", base_model="not-loaded", run_name="multinode-smoke", ) trajectories = [] for answer in REWARDS: trajectories.extend( await asyncio.gather( *( executor.run(worker, rollout, model, answer, None) for worker in workers ) ) ) answers = [ str(trajectory.metadata["answer"]) for trajectory in trajectories ] expected = [answer for answer in REWARDS for _ in workers] placements = { (trajectory.metadata["hostname"], trajectory.metadata["process_id"]) for trajectory in trajectories } if answers != expected or len(placements) != host_count: raise RuntimeError( f"distributed rollout mismatch: {answers=}, {placements=}" ) print(f"ART_MULTINODE_SMOKE_PASS hosts={host_count} answers={answers}") finally: await runtime.close() ``` The controller receives an `ArtLaunchContext` once, while the top-level `rollout` runs in one process on each host. The context owns the attached host mesh and builds a homogeneous typed cluster without exposing provider environment variables. Both functions must be installed or synchronized at the same import paths on every node; ART sends verified import references and never ships opaque closures. The dummy `TrainableModel` is serialized for the rollout contract but never loaded, so this validates the package, host admission, process placement, public trajectory types, and cleanup without a GPU or inference server. The distributed service APIs are opt-in. Existing single-node programs continue to construct and use `LocalBackend` exactly as before: ```python theme={null} from art.local import LocalBackend backend = LocalBackend() ``` ## SkyPilot Start with `examples/multinode/skypilot.yaml`. It is an intentionally CPU-only two-node smoke that runs all three bounded rollouts on each host without reserving training GPUs or provisioning the managed vLLM runtime. For source-based GPU training, replace its resources and setup with the desired topology and run the CUDA-detecting cluster setup: ```yaml theme={null} resources: accelerators: H200:8 setup: | set -euo pipefail INSTALL_MULTINODE=true bash scripts/setup.sh run: | set -euo pipefail export NCCL_NET=IB exec .venv/bin/art-monarch skypilot \ --program train.py:main ``` `set -euo pipefail` is required because SkyPilot runs multiline setup under Bash without enabling fail-fast behavior. For source development, `scripts/setup.sh` selects the CUDA-matched root and private trainer locks and builds HybridEP. It does not install system packages. The image or cluster bootstrap must provide the NVIDIA driver and toolkit, native build tools, NCCL network transport, MOFED/RDMA devices, and the required kernel modules. The CPU example only syncs the root `distributed` extra. Setup is cluster provisioning, not service startup. `art-monarch`, trainer actors, and managed vLLM processes never invoke these shell scripts. Source checkouts launch the already-built `vllm_runtime/.venv`; release wheels may materialize their bundled, locked vLLM environment into a content-addressed cache on first use. Any GPU workload spanning hosts must set one explicit NCCL network contract in its `ClusterSpec`, for example `nccl_transport=NcclTransportSpec(net_name="IB")`, and set `NCCL_NET` to that exact registered name on every node. `IB` covers built-in InfiniBand/RoCE; external network plugins use their registered NCCL name. Before model allocation, ART runs a small collective in both the trainer and managed-vLLM environments and requires each rank to report that exact selected module. It never retries with Socket. Deployment qualification remains responsible for all-GPU bandwidth, GPU Direct RDMA, HCA, and GID validation. Cross-host HybridEP uses `NixlTransportSpec()`. If `metadata_store` is omitted, the controller starts a checksum-pinned etcd process, publishes its routable endpoint, health-checks it from every host, and owns its cleanup. An explicitly managed endpoint remains supported. If `ART_VLLM_RUNTIME_BIN` is set, it must point directly to a standard `.venv/bin/art-vllm-runtime-server` executable. ART derives the matching Python, runtime root, environment, and working directory from that path so the preflight cannot certify a different runtime. Arbitrary command wrappers fail closed. For a published release wheel, install the profile matching the host CUDA toolkit. For CUDA 12: ```yaml theme={null} setup: | set -euo pipefail uv venv --python 3.12 --seed .venv .venv/bin/pip install \ --extra-index-url https://download.pytorch.org/whl/cu128 \ "openpipe-art[megatron]==VERSION" ``` For CUDA 13, use `openpipe-art[megatron-cu130]==VERSION` and the PyTorch `cu130` index. These commands install ART, Monarch, and the CUDA-specific NIXL wheel. The example uses the supported image's `uv` installation to provision Python 3.12. The first trainer launch materializes the pinned Megatron environment, builds source-only CUDA components such as CUDA 12 Apex and ART's HybridEP when needed, and reuses the immutable result on later launches. NIXL and its UCX GDA plugin come from the official relocatable wheel; ART bundles only the matching pinned headers needed to build HybridEP. Release wheels use content-addressed managed Megatron and vLLM runtime bundles. Only wheels built with `scripts/build_package.py` contain those bundles. The install profile provides `uv`; first use needs package-index access unless the node-local cache was prewarmed. `examples/multinode/skypilot_training.yaml` runs a real two-host DP2 SFT step from the published wheel. It synchronizes only the user program, not an ART checkout. Point `ART_SHARED_ROOT` at an existing path mounted on every host, then launch: ```fish theme={null} sky launch -c art-multinode-training \ --env ART_SHARED_ROOT=/mnt/shared/art-multinode-release \ examples/multinode/skypilot_training.yaml ``` Launch it from the project root: ```fish theme={null} sky launch -c art-multinode examples/multinode/skypilot.yaml ``` One task rank runs on each allocated node. Every rank owns one Monarch worker subprocess; rank 0 also attaches the host mesh and runs the controller program. Rank-0 program completion or failure closes the lifecycle sockets and releases the peer task ranks. No manual SSH or per-node command is required. Each task invocation owns fresh worker loops and terminates them after the host mesh shuts down. A later `sky exec` starts new loops; ART does not reattach a second controller to completed workers. Ctrl-C disconnects SkyPilot log streaming; it does not stop the remote job. Check the queue and cancel explicitly when needed: ```fish theme={null} sky queue art-multinode sky cancel art-multinode JOB_ID ``` Only after the previous job is terminal, reuse an existing cluster without rerunning setup: ```fish theme={null} sky exec art-multinode examples/multinode/skypilot.yaml ``` `sky exec` synchronizes the workdir before scheduling, so running it while the previous job is live can change files under that job. Use `sky launch` instead when setup, mounts, the image, SkyPilot config, a wheel, `pyproject.toml`, or a lockfile changed. Setting `num_nodes: 1` uses the same controller on one node. For a local process that explicitly wants the same Monarch service APIs, ART can own one loopback worker directly: ```fish theme={null} .venv/bin/art-monarch local \ --program examples/multinode/program.py:main \ --port 0 \ --startup-timeout 90 ``` Port `0` selects a fresh loopback port. `ArtRuntime.start_local(...)` is the equivalent library API. It accepts the same one-host compiled topology used by multi-node code and owns the worker for the runtime lifetime. SkyPilot provides `SKYPILOT_NODE_RANK`, `SKYPILOT_NODE_IPS`, and `SKYPILOT_NUM_NODES`; ART validates and translates them internally. Port `22222` is the Monarch worker port and `22223` is its job-lifecycle port. Pass `--port N` to reserve `N` and `N + 1` instead. These ports must be reachable between allocated nodes but must not be publicly exposed: ART's Monarch runtime uses unauthenticated `trust_all_connections` transport. ## Existing SSH hosts For preallocated machines, start and own all workers from one controller command: ```fish theme={null} .venv/bin/art-monarch ssh \ --host gpu-a=10.0.0.10 \ --host gpu-b=10.0.0.11 \ --python /shared/project/.venv/bin/python \ --program /shared/project/train.py:main ``` Each value is `SSH_TARGET=WORKER_HOST`. Omit `=WORKER_HOST` when the SSH target is also the private address to which Monarch should bind. The controller must have both passwordless SSH access to every `SSH_TARGET` and a direct trusted private or VPN route to every `WORKER_HOST:N`. SSH options such as `ProxyJump` or `--ssh-arg=-F` affect only launch and stop commands; they do not tunnel Monarch traffic. SSH mode uses only worker port `N`, not SkyPilot's lifecycle port `N + 1`. The selected Python executable and source script must exist at the same paths on every host. ART uses non-interactive SSH, verifies that each launch-specific worker PID owns its listener, and monitors each foreground SSH process for the controller lifetime. A pre-existing listener is a hard error rather than a worker to reattach. SIGTERM and SIGHUP trigger bounded remote cleanup before the controller exits. The lower-level `worker` and `controller` subcommands remain available for schedulers or process supervisors that own worker lifecycle themselves. Those supervisors must replace worker loops before a subsequent controller attach.