Models & recipes¶

CLI overview · Model serves · Evaluation & benchmarks

The models family manages four related resources: the local model catalog, downloaded artifacts, reusable serve recipes, and cache storage. Use recipes to move from a known working engine configuration to a candidate container; use serves switch when the candidate is ready to replace a deployed role.

Switch Primary to another model¶

You switch the heavy role to a recorded recipe; you do not edit the active Compose service by hand.

anvil-serving models recipes list
anvil-serving models recipes show MODEL
anvil-serving serves switch primary MODEL --dry-run
anvil-serving serves switch primary MODEL --confirm

list shows which recipes activate heavy. show resolves the exact model id or unique basename and prints the engine configuration plus the same switch-preview command. The dry run validates the activation plan without changing the running service; --confirm applies the reviewed switch.

If the model does not have a compatible recorded recipe yet, create or update an operator recipe first. A recipe is the complete known-working model-and-engine configuration, so changing only its model field is not treated as a safe model swap.

Choose a workflow¶

Goal	Start here	Then
See models already on this host	`models sync --dry-run`	Apply with `--confirm`, then inspect `model-library/INDEX.md`.
Download a Hugging Face model	`models pull REPO --dry-run`	Apply with `--confirm`.
Find a known working serve configuration	`models recipes list`	Inspect it with `models recipes show MODEL`.
Discover recipe-loaded containers	`models recipes running`	Select an unambiguous model or exact container for status, logs, or unload.
Replace the deployed Primary recipe	`models recipes list`	Choose a row that activates `primary`, inspect it, then preview `serves switch primary MODEL --dry-run`.
Add or revise an operator recipe	`models recipes create\|update ... --dry-run`	Apply with `--confirm`; retain the numbered backup.
Start a candidate without changing routing	`models recipes load MODEL --container NAME --dry-run`	Apply, run `eval preflight`, then review a `serves switch`.
Remove one exact cached revision	`models cache remove OWNER/REPO --revision COMMIT --dry-run`	Apply with `--confirm` after reviewing the snapshot and reclaimable blobs.
Reclaim cache space broadly	`models cache prune --dry-run`	Add `--execute --confirm` only after reviewing the protected mixture.

Commands¶

Command	Purpose
`models sync`	Synchronize the model catalog.
`models pull`	Pull a model artifact.
`models score`	Rank models from benchmark evidence.
`models recipes list`	List recorded serve recipes.
`models recipes show`	Show one recipe.
`models recipes running`	Discover label-owned recipe containers without registry dependence.
`models recipes create`	Add one recipe to an operator registry.
`models recipes update`	Replace one selected recipe.
`models recipes delete`	Delete one selected recipe.
`models recipes load`	Start a named local container from one recipe.
`models recipes status`	Inspect one exact recipe-loaded candidate container.
`models recipes logs`	Read bounded logs from one exact recipe-loaded candidate container.
`models recipes unload`	Remove one exact recipe-loaded candidate container.
`models cache inventory`	Record a read-only model-cache and Docker storage inventory.
`models cache remove`	Plan or remove one exact cached repository revision.
`models cache prune`	Plan or prune model-cache storage.

Catalog sync¶

sync scans local Hugging Face caches and plain model directories, then writes structured cards/*.json summaries plus a human INDEX.md. Preview resolves the same output and source roots but performs no scan and creates no directory.

anvil-serving models sync --out ./model-library --dry-run
anvil-serving models sync --out ./model-library --confirm

Source precedence is command flags, configured model roots, then platform discovery. Use the platform path separator for multiple --hf-roots or --model-dirs values (: on Linux/macOS, ; on Windows). Apply builds a complete staged catalog, moves any prior catalog to a numbered .anvil.bak.N directory, and installs the replacement. Removed source models therefore do not survive as stale cards. The target must be new, empty, or an existing catalog containing both cards/ and INDEX.md; sync refuses files, links, general directories, the checkout, the current directory, and the user home. One output-specific lock covers the scan and replacement. A worker error or incomplete staged catalog leaves the active catalog untouched.

Artifact pull¶

pull downloads a Hugging Face repository into a named Docker volume. It never places a token value on the command line: --token-env names the source variable, and --token-file is a fallback dotenv file.

anvil-serving models pull openai/gpt-oss-120b --dry-run
anvil-serving models pull openai/gpt-oss-120b --confirm

The named-volume default avoids slow host bind mounts on Windows/WSL2 and remains valid on Linux and macOS Docker hosts. Preview resolves repository filters, token mode (including the environment-variable name and expanded dotenv path, never a token value), preconditions, ordered Docker actions, resumable recovery, and the fact that downloaded bytes have no automatic rollback. It also discloses the machine's automatic WSL cache-reclaim policy. When that policy is enabled, a successful confirmed pull captures operation cache growth and evaluates the best-effort page-cache-only postcondition once. A skip or failure warns without changing the successful download's exit code.

A confirmed pull first queries repository metadata (or uses --expected-bytes), inspects the target snapshot inside the named volume, and fails before downloading unless free bytes cover the missing artifact bytes plus --headroom-gib. The download keeps native hf download progress and resumability. Success is reported only after the exact requested snapshot exists with no incomplete files or broken links.

Recipes¶

A serve recipe records a working model-and-engine configuration independently from a running container. The shipped registry is useful for discovery. Mutations require an explicit operator-owned registry path, which keeps packaged defaults immutable and reviewable.

Read-registry precedence is --registry, $ANVIL_SERVING_HOME/serve-recipes.toml (or ~/.anvil-serving/serve-recipes.toml), a source checkout's immutable ./configs/serve-recipes.toml, then the packaged registry. The latter two are portable product catalogs for discovery, not operator promotion state. Create, update, and delete require an explicit operator-owned registry path; normally pass $ANVIL_SERVING_HOME/serve-recipes.toml. They never mutate the public checkout or packaged catalog implicitly. A selector may be either a recipe's exact model identifier or its unique basename.

Discover recipes¶

anvil-serving models recipes list
anvil-serving models recipes list --registry configs/serve-recipes.toml
anvil-serving models recipes show MODEL
anvil-serving models recipes show MODEL --registry configs/serve-recipes.toml

Start with list; its activates column identifies recipes that can replace a deployed role such as heavy. Use show before a mutation or load to verify the resolved engine, quantization, context, runtime settings, activation plan, direction, Compose service, and exact serves switch ... --dry-run preview command.

Create, update, or delete a recipe¶

The input file must contain exactly one [[recipe]] block.

[[recipe]]
model = "org/model"
status = "unverified"

[recipe.serve]
engine = "vllm"
image = "vllm/vllm-openai:nightly"
port = 30123
flags = ["--served-model-name org/model"]

Start with the fields above, then use models recipes show on a similar shipped recipe for engine-specific GPU, environment, volume, context, and quantization fields. The full registry schema is represented by configs/serve-recipes.toml.

Most vLLM images accept the model as a positional argument through their image entrypoint. For an image with a different API-server entrypoint, set entrypoint to its argv and set model_flag to the single option that introduces the model id:

[recipe.serve]
image = "nvcr.io/nvidia/vllm:26.06-py3"
entrypoint = ["python3", "-m", "vllm.entrypoints.openai.api_server"]
model_flag = "--model"

entrypoint[0] becomes Docker's --entrypoint; the remaining entries are passed after the image. Without these optional fields, recipe loading keeps the existing positional-model behavior.

For an environment-owned launcher that selects the model itself, set model_env instead of model_flag. Anvil injects that variable from the recipe's exact model identity and passes no positional model argument to the launcher. A declared env entry cannot override it.

[recipe.serve]
image = "vendor/runtime@sha256:..."
entrypoint = ["/usr/local/bin/serve-model.sh"]
model_env = "MODEL"

Persistent engine/JIT data uses auxiliary named volumes, never host bind mounts:

[recipe.serve]
named_volumes = [
  "candidate-jit:/cache",
  "candidate-reference:/opt/reference:ro",
]

Each entry is NAME:/absolute/container/path[:ro]. Sources must be Docker volume names; targets must be normalized absolute POSIX paths; repeated sources or targets and attempts to shadow /root/.cache/huggingface fail closed. The model cache remains owned by recipe.download.volume.

anvil-serving models recipes create --recipe-file ./candidate-recipe.toml --registry ./serve-recipes.local.toml --dry-run
anvil-serving models recipes create --recipe-file ./candidate-recipe.toml --registry ./serve-recipes.local.toml --confirm

anvil-serving models recipes update MODEL --recipe-file ./candidate-recipe.toml --registry ./serve-recipes.local.toml --dry-run
anvil-serving models recipes update MODEL --recipe-file ./candidate-recipe.toml --registry ./serve-recipes.local.toml --confirm
anvil-serving models recipes delete MODEL --registry ./serve-recipes.local.toml --dry-run
anvil-serving models recipes delete MODEL --registry ./serve-recipes.local.toml --confirm

Their previews include the resolved registry and source digests, complete proposed recipe TOML, ordered write actions, deferred gate, and manual recovery path. Create, update, and delete use atomic writes and numbered backups. A mutation never overwrites the packaged registry implicitly.

Load a recipe¶

anvil-serving models recipes load MODEL --container my-candidate --registry ./serve-recipes.local.toml --dry-run
anvil-serving models recipes load MODEL --container my-candidate --registry ./serve-recipes.local.toml --confirm

load starts a new, explicitly named Docker container bound to loopback. It does not change router policy or promote the candidate. Validate it with eval preflight, then use serves switch only after human review. The preview's cleanup command is conditional: use it only for a container successfully created by that load, never for a name that existed beforehand.

For an unqualified loader, declare explicit host bounds in [recipe.serve]:

memory_limit_mib = 61440
memory_swap_limit_mib = 61440
host_memory_reserve_mib = 16384

The first value caps container RAM; the second caps RAM plus swap, so equal values forbid additional swap. All three fields are required together. Values must be positive integer MiB; the reserve must be at least 1024 MiB. Choose bounds for the actual host: the example is not a model-fit claim. A bounded load requires local Linux Docker with cgroup v2 memory/swap enforcement and sufficient available RAM and free swap at admission. The host reserve is an admission check, not a reservation against unrelated workloads. Existing recipes without these fields retain their previous behavior.

models recipes status and running --json include host_memory: Docker limits, effective cgroup RAM/swap limits, current/peak usage, OOM events and exit state. Confirm effective limits before trusting a new loader. Counters unavailable after container exit are null; Docker OOM/exit state remains available. These limits do not cap GPU VRAM or prove that the model fits. Keep restoration separate.

When machine-level cache reclaim is enabled, the preview also declares that load will wait up to 600 seconds for the recipe's HTTP health after the container starts. Only then does it evaluate the cache threshold, fixed 1 GiB growth gate, and settled-I/O gate. A readiness timeout skips reclaim and leaves the container running; it does not change the successful load's exit code. Configure the default-off policy in host.toml.

Operate a loaded recipe¶

load labels the candidate with its exact recipe model, revision, canonical recipe digest, source-registry digest, and native-offload classification. Legacy containers that predate the digest labels remain discoverable from the required Anvil ownership and model labels. Status, logs, and unload recheck the full immutable Docker container ID before acting; logs and removal use that ID, not the reusable name. A mistyped name, reused name, or same-name replacement therefore fails closed instead of silently targeting another container.

anvil-serving models recipes running
anvil-serving models recipes running --json
anvil-serving models recipes status MODEL
anvil-serving models recipes status MODEL --container my-candidate --registry ./serve-recipes.local.toml
anvil-serving models recipes logs MODEL --container my-candidate --registry ./serve-recipes.local.toml --tail 200
anvil-serving models recipes logs MODEL --container my-candidate --tail 500 --contains ERROR --contains "KV cache"
anvil-serving models recipes unload MODEL --container my-candidate --registry ./serve-recipes.local.toml --dry-run
anvil-serving models recipes unload MODEL --container my-candidate --registry ./serve-recipes.local.toml --confirm

running asks Docker only for containers carrying the exact io.anvil-serving.managed-by=models-recipes label, then emits a bounded typed inventory: container name and immutable ID, model/revision, available recipe and registry digests, image digest, served identity, bound port, GPU selection, state, and health. It does not return environment values, raw launch commands, credentials, or endpoint addresses. Exited retained candidates are included. Missing or malformed identities and non-Anvil containers are excluded.

When one discovered container matches, status, logs, and unload no longer require its name to be recovered out of band or its recipe to remain in the currently selected registry. Two matching containers fail closed and require an exact --container. An unload without a current recipe treats a missing native-offload label conservatively and invokes the existing twice-verified, exact-path shared-memory cleanup guard after removal.

--contains is a case-insensitive literal filter and may be repeated; a line is emitted when it contains any selected literal. Filtering happens after the exact recipe/container ownership check and preserves stdout/stderr separation. Use it to inspect actionable startup signatures without returning every model load progress line. Remove the filter when the complete bounded tail is needed.

Use these commands for isolated benchmark candidates. Use serves status, serves logs, and serves down for manifest-owned deployments. Do not use raw Docker as the normal candidate lifecycle path.

Model scoring¶

anvil-serving models score --help

Scoring ranks models from retained benchmark evidence. It does not auto-promote a recipe or modify router policy.

Cache remove¶

Use the exact-removal verb when one known repository revision—not a policy selected set—is the target:

anvil-serving models cache remove OWNER/REPO --revision COMMIT --dry-run
anvil-serving models cache remove OWNER/REPO --revision COMMIT --confirm

The selector requires both an exact OWNER/REPO and revision. Preview reports the matching snapshot and bytes that become unreferenced; apply removes only that snapshot, collects only blobs no remaining snapshot references, and verifies the target snapshot is absent. It does not approximate repository identity with a substring or wildcard.

Native Hugging Face caches have an inspection-only removal plan. It takes an explicit local cache root, accepts only a 40-character lowercase commit revision, and never contacts Docker, the network, or credential sources:

anvil-serving models cache remove OWNER/REPO --revision 40_HEX_COMMIT \
  --cache-dir /operator/model-cache/hub --dry-run

The plan separates exclusive from shared blobs and reports an inode-deduplicated physical-byte reclaim estimate (st_blocks * 512) and refs that point to the selected snapshot. APFS copy-on-write or clone accounting can make eventual disk release lower. Native deletion is intentionally unsupported: --confirm is refused until the product has a native ownership and deletion-safety implementation. The scanner refuses symlinked selected-path ancestry and does not follow metadata links outside the selected repository.

Cache inventory¶

Capture a read-only, machine-readable inventory before and after storage work:

anvil-serving models cache inventory
anvil-serving models cache inventory --output ./cache-inventory.json
anvil-serving models cache inventory --cache-dir /operator/model-cache/hub

The model-cache-inventory/v1 result includes filesystem capacity, used, and available bytes; cached repositories, revisions, snapshots, logical and incomplete bytes, and modification timestamps; plus Docker image, container, volume, and build-cache accounting. --volume and --image select the named Hugging Face cache volume and inspection image. The volume is mounted read-only, and --output uses an atomic replacement after requiring an existing parent directory.

Pass --cache-dir either the Hugging Face home that contains hub/, or a direct hub-cache directory such as ~/.cache/huggingface/hub or a custom HF_HUB_CACHE path containing models--* entries. A directory that contains both hub/ and direct models--* entries is refused as ambiguous. This is filesystem-only and reports each standard repository's exact refs and snapshot revisions, snapshot logical bytes, inode-deduplicated logical and allocated blob bytes, local-link integrity, upstream artifact completeness as unverified, and unsafe metadata paths, alongside disk capacity. It makes no Docker, network, or credential access. Repositories downloaded by exact revision may correctly have no refs/ directory. Nonstandard layouts retain unknown fields; symlinked roots and unsafe metadata are never followed.

Modification, creation, and Docker last-used timestamps are observations for inventory and cleanup planning. They are not proof that a model was actually served or benchmarked. Use retained benchmark evidence and protected runtime state for qualification and deletion decisions. The model_cache_inventory MCP tool currently covers managed Docker-volume inventory; native cache inspection is CLI-only until an ownership-aware controller surface exists.

Cache prune¶

Always inspect the plan before deleting cached artifacts:

anvil-serving models cache prune --dry-run
anvil-serving models cache prune --mixture MODEL --execute --confirm

The command preserves the ownership and reservation rules represented by the current serve configuration. A metadata-only hardware caveat is never enough to label a model safe to delete: default deletion requires explicit current-host dead_everywhere evidence. --include-servable deliberately widens the deletion set; combining it with an empty protected mixture also requires --allow-empty-mixture. The removed --yes consent spelling is rejected with guidance to use the shared --confirm flag.

For a structured read-only plan through the agent/controller surface, use the cache_prune_plan MCP tool. CLI --json is the standard result envelope, not a second handler-specific JSON format. Safety refusals and undeleted candidates are preserved in that envelope's error message. Preview lists the resolved scan roots, ordered apply actions, rescan drift, and the irreversible/no-automatic-rollback boundary.

Shipped recipe registry: configs/serve-recipes.toml
Model serves
Evaluation & benchmarks
Benchmarks