Model serves¶

CLI overview · Router · Models & recipes

The serves family owns local model-server definitions and lifecycle. A serve is manifest-owned; a recipe is a reusable model-and-engine configuration managed under models recipes.

Each manifest entry also belongs to a user-facing stack. Stacks are the lifecycle ownership boundary for separate workloads such as serving, auxiliary, voice-audio, voice-proxy, and comfyui. The CLI maps a stack to the stable Docker Compose project anvil-<stack>, injects that project when the up command omits it, and refuses to manage an existing container owned by another stack unless serves up --recreate is explicit. If an authored --project-name contradicts stack, manifest loading fails before Docker is called. Entries that omit stack retain the serving default.

Commands¶

Command	Purpose
`serves render`	Render a model serve definition.
`serves up`	Start manifest-owned serves.
`serves up-for`	Resolve a chat alias to its backing serve and start it.
`serves down`	Stop and remove manifest-owned serve containers.
`serves rm`	Remove a manifest-owned serve.
`serves adopt`	Adopt an existing serve into manifest ownership.
`serves switch`	Switch a deployment role to an activation-ready recipe.
`serves promote`	Preflight and promote a staged recipe with rollback.
`serves mode status\|preview\|enter\|leave`	Inspect or transact exclusive TP=2 ownership.
`serves profile list\|preview\|apply`	Select a declared split or exclusive serving profile.
`serves status`	Show bounded serve status.
`serves lint`	Report manifest defects that no other surface makes visible.
`serves rollback-check`	Prove every declared rollback is actually usable.
`serves probe`	Run one engine-aware functional request.
`serves groups`	List serve groups and their members.
`serves logs`	Read bounded serve logs.
`serves multiplex`	Run the single-resident model multiplexer.

Every leaf uses its real parser for exact usage, local arguments, choices, and defaults. The registry adds its canonical summary, global options, safety policy, and direct reference link. For example:

anvil-serving serves up --help
anvil-serving serves switch --help

This page owns the longer workflows and behavioral guidance. Keeping that prose out of the runtime registry avoids a second documentation copy while the generated command index and reference audit still enforce the public paths.

Select manifests and groups¶

Serve lifecycle commands operate on the configured manifest set. Use --group NAME to target every serve with the matching tag; repeat the option for multiple groups. --group all selects every serve in the set.

anvil-serving serves groups
anvil-serving serves status --group ocr
anvil-serving serves status --group all

Groups provide the supported way to turn an optional workload such as OCR on or off without inventing a separate lifecycle script.

Bare serves status polls only entries assigned to an authored group—the operator-supported serving path. Untagged candidates and experiments remain available by explicit name or --group all, but are not contacted merely because they exist in the manifest.

When the manifest declares GPU-role budgets, status also performs bounded label-only discovery of active models recipes load containers. A container that is not represented by a manifest entry is reported as unmanaged-by-manifest; its Docker GPU selection is mapped through the installed topology to the stable role ids. It is never silently omitted from the ownership rows. If selection or discovery cannot be resolved, the overall mode is unresolved rather than split.

serves mode status remains a successful read-only report when the mode is unresolved; consumers should inspect the returned mode and unresolved fields rather than treating the report itself as an execution failure.

Start and stop serves¶

anvil-serving serves up --group ocr --dry-run
anvil-serving serves up --group ocr --confirm
anvil-serving serves down --group ocr --confirm
anvil-serving serves down CANDIDATE --keep-container --confirm
anvil-serving serves groups
anvil-serving serves logs OCR_SERVE_NAME

Only manifest-owned resources are mutated. Destructive leaves require confirmation. down stops gracefully and then removes each selected container by default, preventing stopped experiments and stale container configuration from accumulating in Docker. Use --keep-container when retaining the stopped container and its logs is intentional. --confirm is the only public consent spelling; the removed serves rm --yes and serves adopt --yes forms fail with migration guidance before reaching Docker.

serves up returns only after the selected serve's declared health endpoint is ready. A failed or timed-out readiness check fails the command instead of reporting a successful start.

Start by alias¶

Answering "how do I start what llm.primary needs" means reading the router config's [router.model_routes] for the tier, then the serve manifest for the [[serve]] whose router_tier matches it — the join key exists in the data, but nothing walks it. serves up-for ALIAS does:

anvil-serving serves up-for llm.primary
anvil-serving serves up-for llm.primary --confirm
anvil-serving serves up-for llm.voice --config ./router.toml --json

Without --confirm, it only resolves and prints the chain — alias, tier, serve name, container, port, the exact up argv, and the manifest file it came from — closing the four-file read in one command. --confirm delegates to serves up for the resolved serve; --dry-run forwards to it. An unknown alias exits 2 and lists the configured aliases; a tier with no backing serve in the manifest set exits 1.

A tier can legitimately back more than one serve — a promoted primary and its rollback commonly share router_tier (and a port), since the rollback config routes the same alias to the same tier id. up-for refuses to guess between them: it prints every candidate serve with its groups and exits 1, requiring serves up NAME to pick explicitly. Auto-selecting the wrong one on a shared port is worse than asking — the same "detection beats prevention" judgment rollback-check and lint already make (see Strategy: make divergence loud).

Replica lifecycle limits¶

Replica scheduling and member admission are available, but single-serve lifecycle shortcuts do not manage a replica set. serves up-for refuses a replica alias even in preview. Promotion, its resume/rollback/recovery paths, and mode/profile transactions refuse affected replica tiers with replica_lifecycle_unsupported before lifecycle mutation. They do not choose a member, silently operate a direct-tier replacement, or bypass the guard via --skip-preflight-checks.

For local mode preview|enter|leave and profile preview|apply, pass --config ./router.toml to identify the active router config. Without it, only the selected operator home's default config is used; the current directory and target/rollback profiles are not guessed as the active config. Missing, unreadable, invalid or incomplete active ownership produces replica_lifecycle_configuration_unavailable before runtime probes or mutation. The guard covers the target, all GPU-inference serves and the restore group, including stopped entries, plus the target's declared router/rollback profiles.

Library callers supply an already parsed active_config to cmd_mode or cmd_profile. Explicit local --config is refused on remote mode/profile dispatch; it is not forwarded as a path meaningful on another host. Status, profile listing and operations with no affected routed tier retain their existing behavior. Member quiesce/drain/readmit controls remain separate from serve installation and promotion; see router tier transitions.

Split and exclusive TP=2 modes¶

Ordinary split-mode entries reserve one gpu_role. An exclusive candidate declares exactly two gpu_roles, operating_mode = "dual-gpu-exclusive", and tensor_parallel_size = 2. It cannot be started with ordinary serves up.

anvil-serving serves mode status --manifest ./serves.toml
anvil-serving serves mode preview TP2_SERVE \
  --restore-group split-stack --manifest ./serves.toml --config ./router.toml
anvil-serving serves mode enter TP2_SERVE \
  --restore-group split-stack --manifest ./serves.toml --config ./router.toml \
  --preserve-on-failure --confirm
anvil-serving serves mode leave TP2_SERVE \
  --restore-group split-stack --manifest ./serves.toml --config ./router.toml --confirm

Preview reports both stable GPU roles, the TP size, active workloads to drain and stop, workloads blocked while exclusive, and the explicit rollback group. Entry fails closed on unresolved Docker state, drains routed competitors, stops all GPU inference, rechecks both roles, then starts the exclusive owner. Failure restores the split group. Leave stops the owner before restoring that group. By default, failed entry removes the candidate before rollback. Add --preserve-on-failure when debugging a candidate: Anvil stops the failed target, retains the stopped/exited container and its logs, and then restores the split group. If the target cannot be proven stopped, Anvil removes it so an unhealthy or restarting process cannot retain either GPU during restoration. The serves_mode MCP tool exposes the same structured plan; live remote entry/leave requires its separate human-approval gate. mode enter refuses before any mutation on a lint/rollback-check error; see Preflight gate on promote and mode enter. It also refuses before mutation while an active recipe-loaded container owns a GPU outside the selected manifest, or when recipe-owner discovery is unavailable. Discover and inspect that owner with models recipes running, then unload it or add the intended durable manifest ownership before retrying the mode transaction.

Serving profiles¶

A serving profile is a small operator-owned TOML selector that binds one exclusive TP=2 target to one named split restore group. It does not duplicate model flags, GPU UUIDs, router routes, or lifecycle commands: those remain in the serves manifest and its router profiles. This keeps a profile switch from becoming a second implementation of the same serve.

schema = "anvil-serving/serve-profiles/v1"

[[profile]]
id = "deepseek-0731-tp2"
mode = "dual-gpu-exclusive"
exclusive_target = "tp2-deepseek-v4-flash-0731"
restore_group = "qwen35-thinkingcap-split"
startup_timeout = 1200

[[profile]]
id = "qwen35-thinkingcap-split"
mode = "split"
exclusive_target = "tp2-deepseek-v4-flash-0731"
restore_group = "qwen35-thinkingcap-split"
startup_timeout = 1200

Use the CLI from the same operator session:

anvil-serving serves profile list
anvil-serving serves profile preview qwen35-thinkingcap-split --config ./router.toml
anvil-serving serves profile apply qwen35-thinkingcap-split --config ./router.toml --confirm
anvil-serving serves profile apply deepseek-0731-tp2 --config ./router.toml --confirm

profile apply delegates to the existing guarded mode transaction. Entering the exclusive profile runs the same lint and rollback checks as mode enter; leaving it drains the routed tier, installs its declared rollback router profile, restores the named split group, and readmits only the explicitly mapped tiers. A split profile can only be applied by leaving its declared exclusive owner. If the host is already split, the command refuses rather than assuming which router profile or resident set is active. There is no automatic model fallback or request classifier. startup_timeout is an optional finite positive readiness bound in seconds (default 600); poll_interval optionally sets the positive readiness polling interval (default 5).

For a confirmed apply, the first Ctrl-C is deferred so the in-flight success-or-rollback transaction can finish. Pressing Ctrl-C a second time forces an emergency exit and can leave detached lifecycle work in progress; inspect serves mode status before taking another action.

Lint¶

serves lint is static analysis over the loaded manifest set. It touches no Docker and no network, and it exits non-zero when it finds an error, so it can gate CI or a pre-promotion check.

anvil-serving serves lint

Add --json for the same report structurally, matching the status and group JSON conventions.

Three checks, each added because the defect it finds occurred live while every other command reported success (see Strategy: make divergence loud):

Check	Severity	What it catches
`duplicate-serve-name`	error	Two entries sharing a `name` after container de-dup. Name selection becomes ambiguous and one entry silently wins, so an edit to the losing copy is invisible at runtime.
`missing-registry`	error	A `--registry` path inside an `up` command that does not exist. `serves promote` and `serves mode enter` both refuse to start over this before touching a container (see below).
`worktree-anchored-registry`	warning	A recipe registry resolving inside a linked git worktree, which `git worktree remove` deletes.

load_manifest_set refuses a duplicate name outright, so a shadowed edit can no longer reach a live command. serves lint deliberately loads leniently and still reports the defect — the command you reach for when blocked must not be the one that breaks.

Sharing a container across files is not a defect — that is the supported read-only mirror pattern that load_manifest_set de-dupes deliberately. Only a duplicate name surviving that de-dup is reported.

Rollback check¶

serves rollback-check proves every declared rollback path is actually usable. It touches no container state — it reads the manifest set and promotion plans, validates router configs, and asks Docker whether images are present locally.

anvil-serving serves rollback-check
anvil-serving serves rollback-check --restore-group split-default

It exists because two rollback paths were found broken live on 2026-08-08, each only by accident:

a promotion plan's rollback_router_config referenced a router profile file that did not exist;
the split-restore primary serve's compose image was a nightly tag (vllm/vllm-openai:nightly-...) evicted from Docker Hub, so the documented rollback group could not start.

A rollback that cannot run is a false safety net. Add --json for the same report structurally, matching lint's conventions.

Check	Severity	What it catches
`promotion-topology`	error	A promotion plan whose target/rollback router configs no longer validate against each other (mismatched tiers, ports, or model identity).
`rollback-profile-invalid`	error	A routed exclusive serve's `rollback_router_config` exists but fails to parse/validate as a router config.
`rollback-image-missing`	error	A compose image that a promotion plan's rollback serve, or a `--restore-group` serve, depends on is not present locally (`docker image inspect` fails) — the exact shape of the evicted-nightly-tag incident.
`image-unverifiable`	info	A dependent serve's `up` command has no compose file (for example a `models recipes load` command), so image presence cannot be checked. Not an error — some rollbacks legitimately have no compose image.
`docker-unavailable`	warning	Docker itself could not be reached; image checks were skipped rather than failing the whole report.
`unknown-restore-group`	error	A `--restore-group` that matches no serve. Silently verifying nothing is itself a false safety net.

Only error-severity findings fail the exit code; warning and info are reported but never block.

Preflight gate on promote and mode enter¶

serves promote and serves mode enter both run lint and rollback-check before their first mutation — the same reports described above, printed in the same format (to stderr, so a --json caller sees them in the error envelope). Serve checks cover the whole manifest set for both transactions. The promotion plans checked differ: promote gates on the plans of the manifest it was invoked on — exactly the plans the transaction itself executes — while mode enter (like standalone rollback-check) loads promotion plans from every file in the set. --dry-run still runs the gate, since both checks are read-only. mode enter's rollback-check uses its own --restore-group; promote has none, so its rollback-check runs without one.

The abort decision is scoped, not whole-set: only an error-severity finding relevant to this transaction's own serves — promote's resolved plan's target and rollback (plus the plan name itself, so a promotion-topology error on the plan being promoted still blocks); mode enter's target plus every serve tagged with --restore-group — aborts with exit 3 before a container is touched. An error finding about a serve outside that set (for example a stale missing-registry on an unrelated scaffold entry) prints to stderr as ADVISORY (outside this transaction): and does not block; refusing every command over one unrelated manifest entry is exactly what a not-yet-created registry file legitimately being work in progress would otherwise force. warning/info findings are always advisory and are printed but never block. An unresolvable promote plan name (no match, or more than one) skips the gate entirely and falls through to promote's own "must match exactly one [[promotion]] plan" refusal — there is no plan to scope to, and reporting that as a lint/rollback-check abort would be misleading.

anvil-serving serves promote PROMOTION_PLAN --confirm
anvil-serving serves mode enter TP2_SERVE --restore-group split-stack \
  --manifest ./serves.toml --confirm

Pass --skip-preflight-checks to bypass this lint/rollback gate; doing so prints an unmistakable warning to stderr naming exactly what was skipped. serves mode leave|preview|status do not run this lint/rollback gate — --skip-preflight-checks is rejected (exit 2) on anything but mode enter. The standalone serves lint/serves rollback-check commands are unchanged, still report over the whole manifest set with no scoping, and remain available on their own. The early replica lifecycle guard is separate and cannot be bypassed.

Functional probes¶

Health proves that a process is accepting requests; it does not prove the serve's defining modality works. serves probe provides a bounded, manifest-aware request for the supported purpose engines:

anvil-serving serves probe embeddings
anvil-serving serves probe reranker
anvil-serving serves probe ocr --image ./screen.png --text "Read all visible text."
anvil-serving serves probe comfyui --manifest ./serves.comfyui.toml

Embedding probes require a non-empty vector, reranker probes require one finite score per document, OCR/vision probes require non-empty recognized text, and the ComfyUI probe requires system metadata. Results are bounded JSON and omit image bytes. Chat LLMs retain the stronger eval preflight contract; audio serves use voice benchmark.

On a Windows/WSL machine with the default-off host.toml cache policy enabled, confirmed manifest-owned up waits up to 600 seconds for every selected serve's declared HTTP health, then evaluates one best-effort page-cache reclaim. Ad-hoc serves up --compose is excluded. The dry run discloses the resolved policy, and a readiness timeout or reclaim failure warns without stopping the container or changing the successful lifecycle exit code.

Render and adopt¶

anvil-serving serves render --help
anvil-serving serves adopt --dry-run
anvil-serving serves adopt --confirm

render produces a reviewable serve definition. adopt brings an already-running serve under the same ownership contract; it does not silently claim arbitrary containers. An enabled machine cache policy gives adopt the same bounded health wait and single postcondition as manifest-owned up.

Switch Primary by recipe¶

For the common model-selection path, choose the deployment role and recipe directly:

anvil-serving serves switch primary
anvil-serving serves switch primary Laguna-S-2.1-NVFP4 --dry-run
anvil-serving serves switch primary Laguna-S-2.1-NVFP4 --confirm
anvil-serving serves switch primary gpt-oss-puzzle-88B --confirm

With no MODEL, the command lists the resolved registry path and marks each declared choice ready or blocked after validating its plan and effective Compose service; listing does not prompt for confirmation. switch accepts a full model id or an unambiguous basename as the second positional argument. The older --recipe MODEL spelling remains supported for compatibility. It only accepts recipes with a matching [recipe.activation.ROLE] entry, and verifies that the recipe's managed serve and served-model identity match the referenced promotion plan before any mutation. The existing promotion transaction still owns quiesce, drain, preflight, router update, and automatic rollback. Before apply, switch resolves the effective Compose service, binds it to the recipe's image/model/revision/flags/environment/GPU/port, compares the Compose service hash and live container contract, snapshots all router artifacts into the operation directory, compares the deployed router config and profile with the expected source state, and takes exclusive role and promotion locks. A matching active target is a no-op. Each real switch writes a durable operation journal and fresh gate evidence under the operator config directory instead of overwriting dated findings. A normal registry row is intentionally not enough to alter a live routing tier; add a reviewed activation mapping and promotion plan first. Controller and SSH transport parity remain tracked follow-up work; run this command on the resource owner.

An enabled machine cache policy runs once at this public boundary after the switch's existing health, exact-identity, preflight, and router-readiness gates. Nested up work inside the transaction does not reclaim separately.

Advanced: promote a plan¶

For lower-level plan operation and recovery:

anvil-serving serves promote PROMOTION_PLAN --dry-run
anvil-serving serves promote PROMOTION_PLAN --confirm

PROMOTION_PLAN names a [[promotion]] entry in the selected serves manifest. Promotion stages the candidate, runs preflight, and preserves a rollback path, refusing before any mutation on a lint/rollback-check error; see Preflight gate on promote and mode enter. It is separate from models recipes load, which starts a named local container but never promotes router policy. Recipe-loaded candidates are inspected and removed through models recipes status|logs|unload, not raw Docker or manifest-only serves verbs.

Promotion and an explicitly requested rollback also evaluate the enabled machine cache policy exactly once after their existing readiness gates. Controller-dispatched serve operations inherit the behavior because they execute the same resource-owner CLI; no new MCP tool or schema field is added. See host.toml configuration and ADR-0023.

Derive a plan instead of hand-writing one¶

anvil-serving serves promote --derive heavy old-heavy \
  --router-config router-promoted.toml \
  --rollback-router-config router-rollback.toml

--derive TARGET ROLLBACK generates a complete [[promotion]] block from two already-declared [[serve]] entries and their promoted/rollback router configs, instead of hand-writing one. affected_tiers is the sorted set of tier ids in the PROMOTED router config whose model matches TARGET's served_name; a router config where no tier matches is refused rather than silently emitting a plan that promotes nothing. The six numeric fields load_promotions would otherwise apply as defaults (drain_timeout, needle_ctx, tool_batch, startup_timeout, rollback_startup_timeout, poll_interval) are emitted explicitly, so the block is already complete:

[[promotion]]
name = "heavy-promotion"
target = "heavy"
rollback = "old-heavy"
affected_tiers = ["primary-local"]
router_config = "/abs/path/router-promoted.toml"
rollback_router_config = "/abs/path/router-rollback.toml"
drain_timeout = 120
needle_ctx = 32768
tool_batch = 20
startup_timeout = 600
rollback_startup_timeout = 600
poll_interval = 5

Before printing anything, the derived plan is run through the same _validate_promotion_topology check promote itself uses; a derivation the validator would reject is refused up front with the validator's message ("derived promotion plan refused: ...") instead of ever being emitted. Every refusal goes to stderr -- stdout carries exclusively the TOML block, emitted as UTF-8 (TOML's required encoding), so > plan-fragment.toml redirects are always safe, including on Windows consoles with a non-UTF-8 code page. A config path containing a literal {dir} component is refused: the manifest loader substitutes that token with the manifest directory on reload, so an emitted block would resolve to the wrong path. --derive is entirely read-only and does not accept --rollback, --resume, --dry-run, or --skip-preflight-checks. Pass --out PATH to write the block to a file instead of stdout; --out never overwrites an existing file -- it refuses and leaves the file untouched.

Multiplexing¶

anvil-serving serves multiplex --help

The multiplexer coordinates a single-resident model workload. Use it only where the deployment topology assigns that model workload to the current host.