The assistant was not underperforming. Its architecture was.

Context

An AI assistant embedded in a multi-tenant SaaS product, already shipped and already in use.

The problem

Answer quality was poor and credit consumption per request was high enough to matter commercially. The obvious readings — a weak prompt, the wrong model — had both been tried.

Why it was difficult

An underperforming AI feature looks like a prompt problem, and prompt problems are cheap to test, so that is where the effort goes first. The actual cause was structural: the application was making one large call and asking the model to do everything inside it, which is both expensive and unreliable, and no amount of prompt editing changes that shape.

Diagnosis

I traced the cost and the quality failures to the same root: the assistant had no way to take discrete actions, so every request carried the entire problem at once. That is an architecture question, not a prompting one.

What changed

• Rebuilt the assistant around a native tool-calling loop rather than a single monolithic call
• Defined 16 tools with Zod schemas, so the model selects a narrow action instead of improvising a broad one
• Made execution streaming and multi-step, so intermediate results are visible rather than buffered
• Added intent approval, so the user confirms consequential actions before they run
• Truncated tool results before they re-enter context, which is where most of the wasted spend was

Technical notes

• Tool-result truncation was the single largest cost lever. Unbounded results re-entering context on every step compound quickly across a multi-step loop.
• Vision input is supported in the same loop rather than as a separate path.
• The surrounding system supports both OpenAI and Anthropic models; the loop is not tied to one provider.

Business impact

Cutting credits per request by roughly 80% made the feature materially cheaper to operate, which is what took it off the list of things to reconsider. The structural change is the part that lasts: discrete tools with approval are auditable in a way a single opaque call is not.

What this demonstrates

Architecture and cost remediation for an AI feature that already exists. Not generic AI consulting — the work was diagnosing why a shipped feature underperformed and changing its shape, which is a different job from adding one.