ConfigDesk
A configuration assistant judged on how well it refuses, rather than on how fluent it sounds.
In development
Why it exists
The rules that decide whether an infrastructure configuration is valid are real, knowable and scattered: published product specifications, bill-of-materials logic, power budgets, slot counts, thermal limits, and the memory of a handful of specialists.
No individual seller holds all of it. So invalid designs get built on paper, quoted, and caught late at factory release, which is the most expensive place to catch anything.
The bet I am testing
The instinct with a language model here is to make it fluent and helpful. That instinct is wrong for this problem.
A configurator that confidently produces an invalid design costs more than one that refuses. The value is in the refusals it gets right, and in whether each refusal comes with a reason a seller can take back to a customer. That is why I plan to measure it on refusal precision.
The shape it would take
Retrieve the relevant rules, then check every answer against them with a deterministic validator, so the language model can suggest but only the rules decide. Explain the result in the customer’s language, and route it to a person for approval before anything reaches release.
The important boundary is between the model and the validator. The model may suggest and phrase; it may not decide legality. That keeps the failure mode "unhelpful" rather than "confidently wrong".
How it would be judged
Refusal precision: of the designs it refused, how many were genuinely invalid. And its opposite, which is worse: of the designs it accepted, how many should have been refused. Then explanation quality, meaning whether a person can act on the reason given, and coverage. Fluency is not part of the score.
A boundary worth stating
ConfigDesk is built and described entirely from public specifications. No employer’s internal catalog, guardrails, roadmap or customer detail is used in it, and none appears anywhere on this site.
Where it is
Three phases- Phase 1 · in progress
Rule surface and evaluation design
A rule set assembled from public specifications, and a written evaluation built around refusal precision.
The evaluation design is being written before anything that could be scored by it exists.
Done when: The evaluation can be run by someone else, and would fail a system that is fluent and wrong.
- Phase 2 · not started
Retrieval and validator
A deterministic validator first, retrieval on top of it, explanation last.
Not started.
Done when: The validator refuses an invalid configuration for a reason a seller could repeat to a customer, without the language model deciding legality.
- Phase 3 · not started
Evaluate and publish
Evaluation results including the failure modes, and the repository opened.
Not started.
Done when: Refusal precision is reported honestly alongside the cases it got wrong, rather than a demo that only shows successes.
Next: Finish the evaluation design before building anything it could score.
kushagragoyal.com/lab/configdesk