Local AI
Model
Evaluation

Could a replacement
improve the real application?

The question was not whether another model could start or answer a direct API prompt. It was whether a replacement could deliver supported, useful answers within the assistant’s retrieval, citation and timeout rules, under single-user and concurrent demand.

The assistant shared an ASUS GX10 with NVIDIA GB10 unified-memory hardware with other services. Existing workloads and swap use made checkpoint size alone an inadequate test of safe fit. Preserving the working interface and production configuration was part of the evaluation design.

My role and the historical trial recipes

I defined requirements, selected the comparison scope, directed AI-assisted test development and reviewed the evidence. The production recipe was Qwen3.6 35B NVFP4 through vLLM. Contenders were Qwen3.8 27B NVFP4, Ornith 1.5 35B A3B NVFP4, Nemotron 3.5 Lightning 30B A3B NVFP4 and Llama 3.1 8B Instruct. These names identify historical configurations, not claims about current availability or recommendations for another deployment.

Use actual questions.
Verify the application route.

Ten exact questions appearing in two historical user-question exports formed a small fixed test set. Prior answers were reference material, not a correctness key. Fresh answers from the existing backend established the baseline; candidate answers were reviewed against retrieved source evidence.

An early isolated copy used an older application entrypoint. Its preliminary measurements were discarded. The corrected copy came from the actual live answer service, changing only candidate routing. Candidate endpoints used the alias expected by that service, while records retained the actual checkpoints and runtime recipes.

Retrieval followed the application’s retrieval paths, but byte-identical retrieval state across every run was not established. Small answer differences therefore cannot be attributed tightly to the model alone. This was an application-specific comparison, not a broad statistical benchmark.

Visible text was not
the acceptance criterion.

Review considered relevance, support in retrieved evidence, citations and acknowledgment of missing information. HTTP success and visible text were recorded separately from correctness. Only contenders passing the initial answer gate proceeded to load tests at one, five and ten simultaneous requests.

Existing Qwen3.6: ten of ten visible answers in each layer. At ten simultaneous users, the earlier layer completed ten of ten requests; the newer layer completed five of ten. Retaining the backend did not mean accepting unlimited capacity.

Qwen3.8, Ornith and Nemotron: each contender’s first two newer-layer requests timed out at about 46 seconds. Those initial-gate deadline failures did not establish an answer-quality or load ranking, and were not evidence of intrinsically poor models.

Llama 3.1 8B: ten of ten visible answers in each layer, but review found unsupported procedures and missing source links. One answer borrowed an isolation procedure from unrelated equipment; another recast shaft and bearing work as motor replacement. These were safety issues, not stylistic differences. The existing backend acknowledged the missing procedure.

Why the timings are not an intrinsic model ranking

Candidate recipes permitted one active vLLM sequence; production Qwen permitted four. Concurrent results therefore compared deployment recipes, not isolated architectural performance. Llama was faster in the earlier-layer single-user case, but slower in the recorded ten-user median and newer-layer single-user case. That mixed result did not outweigh unsupported technical advice. First-token latency was unavailable because answer calls were not streamed.

Bound the experiment.
Exercise the way back.

Candidate containers used pinned checkpoints, isolated loopback endpoints, context and memory bounds, and model-specific runtime settings. An initial coexistence attempt failed at startup with a GPU memory-allocation error. It produced no valid answer or concurrency results and was not repeated during production hours.

Subsequent testing used an approved after-hours window, one contender at a time, with an independent restoration service and timer. The real stop-and-restore path was checked. Production recovery required visible answers through both the earlier and newer application layers, not merely a healthy model endpoint.

Production was restored and checked through both answer paths. Approved closeout removed trial containers and staged checkpoints and retired the temporary restoration timer. Storage cleanup was operational housekeeping, not evidence of improved model efficiency.

Keep the working backend.
Keep the limits visible.

The useful outcome was a defensible deployment decision: retain the existing backend and investigate its remaining load limits. Successful startup, faster output in one scenario and plausible prose were insufficient substitutes for grounded answers and recoverable operation.

Evidence is limited to the documented October 7 comparison and October 8 closeout findings. This page reports those records, not a fresh evaluation of the hardware or application. The conceptual illustration contains no private system details. No universal model ranking or quantified business return is claimed.