Local-model source guide¶
Prepared: 2026-10-13 from the completed source map. This is a curated reading directory, not a benchmark result, deployment recommendation, or compatibility claim. Links identify public material; a source's claim remains bounded by its model revision, quantization, engine, hardware, context, concurrency, prompting, and task protocol.
The curated starting set below has a complete public-safe appendix: source directory appendix and community addendum. They preserve public source links and their evidence limits while omitting private community material.
How to use this guide¶
Use this order for a candidate: official model card and exact revision → exact quant artifact → runtime/release support → a hardware-relevant firsthand report → a locally controlled task and resource measurement. Keep model quality, runtime correctness, throughput, and capacity as separate questions.
Evidence labels used here:
- Primary maintainer: source code, release, model card, or official docs.
- Benchmark owner: methodology or task suite; useful for protocol, not necessarily a local-quant score.
- Firsthand report: a practitioner recorded an experiment; check topology and controls before reuse.
- Lead: a useful discovery pointer whose result needs primary-source or local verification.
Start here: primary model and evaluation authorities¶
| Source | Usefulness | Evidence and boundary |
|---|---|---|
| Qwen models · DeepSeek models · Z.ai GLM · Xiaomi MiMo · MiniMax | Establish base lineage, license, artifacts, templates, and revisions. | Primary maintainer. A mutable card or organization page is not an immutable artifact pin or local fit result. |
| Artificial Analysis and its methodology | Broad external capability priors and a stated aggregation method. | Benchmark methodology. Hosted/model-version scores do not measure a local quant or serving recipe. |
| SWE-bench | Repository issue-repair task family and grader ecosystem. | Benchmark owner. Record split, agent, tool policy, budget, and verifier. |
| Aider leaderboards | Edit-format and multi-language coding signal. | Benchmark owner. Aider's 225-task score is not terminal-agent or repository-navigation evidence. |
| LiveCodeBench | Dated code-generation and execution tasks. | Benchmark owner. Keep the dataset window and generation protocol with any result. |
| Terminal-Bench | Terminal-agent workflows where tools, sandbox, and time limit matter. | Benchmark owner. Compare complete systems, not bare model names. |
Quantization publishers and artifact methods¶
| Source | Usefulness | Evidence and boundary |
|---|---|---|
| NVIDIA Model Optimizer | Vendor conversion/calibration contracts and model-format support. | Primary maintainer. Support must be checked against the exact architecture and runtime. |
| vLLM llm-compressor | Maintained vLLM compression/export path. | Primary maintainer. A supported exporter does not prove serving quality or capacity. |
| Local Inference Lab GLM QAD FP8 branch | Mixed attention/expert precision and CSF artifact changes. | Artifact publisher. The card's lack of GPU/quality validation remains a material gap. |
| Unsloth | Current conversion and GGUF distribution ecosystem. | Maintainer. Treat model-specific benchmark claims independently. |
| ISTA-DASLab GSQ-RCO | A distinct Qwen quantization method and tradeoff lead. | Artifact publisher. Expert-pruned derivatives are a different model arm. |
| Intel AutoRound · HQQ · AWQ | Quantization methodology and format context. | Method sources. They are not a current recipe or a workload result. |
| bartowski GGUF · Ubergarm GGUF | GGUF artifact options and metadata. | Artifact distributors. Popularity and file size do not establish quality. |
Runtime and kernel sources for RTX PRO 6000-class work¶
Hardware-relevant reports must distinguish SM120 RTX PRO 6000 from other Blackwell variants. A generic “Blackwell” claim can describe a different compute capability, memory system, or power envelope.
| Source | Usefulness | Evidence and boundary |
|---|---|---|
| NVIDIA CUDA GPU table · compute-capability guide | Verify architecture and platform constraints. | Primary maintainer. Beginning point for compatibility, not an engine validation. |
| vLLM releases · SGLang releases | Model integration, parser, cache, and serving changes. | Primary maintainers. Pin the released image/commit; do not infer support from main branch. |
| FlashInfer releases · PR 6294 | SM12x MoE, attention, cache, and correctness changes. | Maintainer source. A merged or open fix does not prove a packaged runtime selects that path. |
| llama.cpp · ik_llama.cpp | GGUF, hybrid placement, CUDA support, and speculative paths. | Maintainer/fork sources. Pin the build and artifact; forks and upstream are separate deployment states. |
| ExLlamaV3 · TabbyAPI | EXL3 format, placement, cache behavior, and serving integration. | Maintainer/project sources. Rolling release does not establish production parity. |
| CUTLASS Blackwell guide · FlashAttention · TileLang | Kernel layout, attention and DSL research when profiling identifies a bottleneck. | Kernel sources. Microbenchmarks must not be relabeled as model throughput or task quality. |
| TensorRT-LLM quantization matrix · DeepGEMM | Vendor support boundaries and model-kernel dependencies. | Maintainer sources. Match exact quant, architecture, and kernel dependency. |
Firsthand hardware and real-work reports¶
| Source / author | What it contributes | Recency and host-relevance limit |
|---|---|---|
| gisenberg/local-model-eval | Repository repair rates, empty-patch failures, serving controls, and compact-Qwen comparison controls. | Firsthand. Older than this source pass but high value because it records failures. Reproduce power, runtime, context, concurrency, and task protocol. |
| ambientlight RTX PRO 6000 bench | Larger SWE and sustained-context evidence. | Firsthand adjacent topology. Multi-GPU results do not establish one-card behavior. |
| lengtsp Qwen3.8 benchmark | One-card latency and resource observations. | Firsthand. Single-user historical worklog; no transferable repository-quality claim. |
| lukaLLM DSpark worklog | Multi-turn coding records, draft placement, and memory accounting. | Firsthand. Different CPU/RAM and a limited task-quality screen. |
| SSHdotCodes MiMo runtime + ProCreations artifact | Concrete offload implementation and memory assumptions. | Same publication chain. Runtime, RAM envelope, and tool behavior still need independent checks. |
| ormandj SM120 GLM runtime + quant artifact | Reproducible mixed-precision GLM profiles and artifact integrity. | Maintainer/practitioner. Multi-GPU and custom runtime; do not attribute a result to quant alone. |
| turboderp DeepSeek EXL3 | Artifact and EXL3 implementation context. | Maintainer. Artifact support is not an independent task-quality ranking. |
| SharkWipf Swift EXL3 | Max-Q-adjacent PLE-in-RAM and precision tradeoff lead. | Author report. Swift changes post-training; numbers do not establish repository quality. |
| Strata report + runtime | Clear resource accounting and runtime confounds for Qwen Flash Next. | Firsthand. Drafting and batching differ; limited quality work does not transfer. The report publisher and runtime project are distinct. |
| Ai4Users Qwen debugging | Practical thinking/template failure diagnosis. | Firsthand lead. Different runtime and hardware; isolate the mechanism locally. |
Current bounded runtime and artifact leads¶
| Source | Usefulness | Evidence and boundary |
|---|---|---|
| brandonmusic GLM tr3 4bpw + runtime publisher | Incumbent-quant/runtime publication chain; the current card r19/v84 identifies the same OCI family as the incumbent. | Publisher. External 600 W/overclock DFlash figures do not transfer to a 300 W Max-Q APC/no-spec configuration. No new-weights claim is made. |
| NInfer upstream, ninfer-all, ninfer6000, and WaveCut Q2_0 v3 | Source and artifact leads for Flash-Next placement experiments. | Maintainer/fork/artifact leads. Both compared configs record 512 experts and 10 active; pruning is not established. No matched RAM bound or native TP2 evidence; two-device PP is not TP. |
| Ignis + Trellis a25 artifact | Host-streamed Flash-Next/Trellis serving lead. | Maintainer/artifact lead. One-GPU streamed fit and quality remain unmeasured; no SWE equivalence. The known image metadata supports a later immutable-pin check, not a result. |
X, Reddit, and Discord: discovery with guardrails¶
- X: follow Andreas G for task-level reporting, Stephen J Ronan for source-linked kernel correctness investigations, Ash Hart / TensorFold for model/platform qualification matrices, and Elliot Arledge for kernel evaluation. These are public leads; each report needs its linked artifact or code before influencing a recipe.
- Reddit: r/LocalLLaMA and r/LocalLLM are useful for finding failure reports and reproducibility discussions. Start with the DSpark one-card discussion and Strata/llama.cpp comparison. Votes, screenshots, and token-volume anecdotes are not task-success evidence.
- Discord: public community discussions can surface newly published artifacts, precision experiments, and kernel regressions. Treat them as leads only unless there is a public artifact, source patch, or complete reproducible record. This guide deliberately omits message quotations, member/account details, and access-specific thread links.
Compact operating checklist¶
For each source-derived hypothesis, capture: exact model revision; quant and PLE/KV precision; engine/image/build; GPU type/count/power; host-memory placement; context/concurrency; template/reasoning policy; workload, sample count, failures, and end-to-end time. Do not call two reports independent if one repeats the other's card, runtime README, or figures.
A model card proves identity. A runtime release proves a source change. A firsthand benchmark proves one bounded configuration. Only a controlled local run can establish a decision for another environment.
Local campaign follow-through¶
Start with one exact artifact/runtime pair and its resource envelope, then one contract probe and an independent quality gate. This campaign's local outcomes supersede any pre-local absence-of-result note: Ignis R4 finished 2/5 overall, two graded/resolved and three empty. Neither that record nor the public source collection establishes a universal winner. The retained official-base GLM restoration is a continuity decision with a fresh 2/5 baseline and explicit client limits.