DeskPilot

A support queue is mostly repetition. The same twenty questions arrive in a thousand phrasings, each needs reading, categorising, tagging with an account and an order, and answering in roughly the same words as last time. It is work that a model is genuinely good at and a person is genuinely wasted on.

The trap is that calling a model is the easy 10%. An LLM is a dependency that is slow, priced per token, occasionally rate-limited, and free to return something shaped nothing like what you asked for. Treat it like a normal service call and the first bad day takes the whole queue down.

The MVP question is not “can the model do this”. It is “what happens on the request where it cannot”.

This is the decision everything else rests on. If the model is a layer, every future change to it is a change to the business logic. If it is a dependency behind an interface, swapping Sonnet for Haiku on a cost-sensitive path is a configuration edit, and testing the pipeline needs no network at all.

The port is Microsoft.Extensions.AI's IChatClient rather than a hand-rolled interface. It is first-party, it already models chat, tool calling and streaming, and its middleware pipeline is where caching and telemetry belong — so those two concerns never leak into the handlers.

Note what surrounds the model call: three deterministic steps before it and two after. That ratio is the whole design. The model is one stage in a pipeline, not the pipeline.

A triage result has to be a record: category, priority, extracted entities, a draft body, a confidence figure. Asking for JSON in a prompt and parsing the reply works until it does not, and it fails at 3am on the ticket that mattered.

1. 1Declare a tool, not a formatThe response schema is a tool definition the model is asked to call. Constraining generation to a schema is a mechanism, where 'please reply in JSON' is a request.
2. 2Validate the result anywayTool arguments are deserialised into a C# record and validated like any inbound payload. A schema-shaped response is not automatically a sensible one.
3. 3Retry once, then escalateA malformed or invalid result is retried a single time with the validation error fed back. A second failure routes the ticket to a human untriaged — which is the correct outcome, not a degraded one.

Every draft lands in an agent's queue as a suggestion. An agent accepts, edits, or discards it, and that decision is recorded against the ticket. Nothing a model wrote reaches a customer unread.

• Accept, edit and discard rates are the product metric — far more useful than any offline benchmark, because they are measured on real tickets by the people who own the outcome.
• Edits are stored as diffs against the draft, which makes them a free, continuously growing evaluation set.
• Confidence below the threshold skips drafting entirely and only routes. A bad suggestion costs an agent more time than no suggestion.

Edit rate is the honest accuracy metric. Everything else is a proxy for it.

Per-token pricing makes throughput a budget question, which is unusual for a backend service and easy to discover too late. Tokens are metered on the way in, not reconciled from an invoice at the end of the month.

Tiering is the one that mattered most. Classification is a small, well-bounded judgement that a fast cheap model does nearly as well; drafting is where quality is visible to a customer. Splitting them cut cost per ticket by roughly two thirds against a single-model design, with no measurable change in edit rate.

Model APIs rate-limit and occasionally return overload errors, and a retry storm against a rate limit makes things worse rather than better. This is a solved problem, and in 2026 it is solved by a first-party package.

• Retries respect Retry-After rather than backing off on a guess.
• 429 and 529 are retried; a 400 from a bad tool schema is not — retrying a deterministic rejection is pure waste.
• An open circuit degrades the feature, not the service: tickets keep being accepted and are queued untriaged.
• Attempt timeout is separate from total timeout, so one slow generation cannot consume the whole request budget.

A prompt is behaviour. Editing one in place with no version and no evaluation is deploying an untested change to the most user-visible part of the system.

• Prompts live in source, not in a database, so they move through review and release with the code that depends on them.
• Each has a version stamped onto every result it produced, which makes “when did quality drop” answerable from the data.
• A change has to clear the evaluation suite before merge. Suggestions and reasoning are not enough.
• The model id is configuration, and it is recorded per result alongside the prompt version — a provider-side model update is otherwise indistinguishable from a regression you caused.

Unit tests cannot assert on generated text, so the temptation is to test nothing and ship on impressions. Instead there are two distinct suites, and only one of them talks to a model.

Splitting them is what keeps CI honest. The deterministic suite is fast, free and blocks merges; the model suite is slow, costs money, and gates releases. Merging the two gives you a test run that is both expensive and flaky, which is how teams end up deleting their eval suite.

Generation takes seconds. A spinner for four seconds reads as broken, and the same four seconds with tokens arriving reads as working — so drafts stream to the agent as they are produced.

• Draft tokens stream over SignalR to the agent viewing that ticket.
• The shared queue updates live as tickets are triaged, so two agents do not pick up the same one.
• A reconnecting client resyncs from the read model rather than replaying a stream it partly missed.
• Client-facing only. Internal work moves over Wolverine, where there is durability and retry.

Every dependency here earns its place, and two of the choices are specifically 2026 choices rather than what the same list would have said a year earlier.

Standard telemetry answers whether the service is up. With a model in the request path there are three more questions it does not touch: what is this costing, how good is it right now, and is the provider having a bad day.

• Every model call is a span carrying model id, prompt version, input and output tokens, cost, cache hit and confidence.
• Cost per ticket and edit rate are dashboards, not queries someone remembers to run.
• Latency is tracked at p50 and p99 separately for classification and drafting — one is fast and predictable, the other is neither.
• Ticket bodies are never logged. The span carries a hash and a token count, and that is enough to debug with.