AI Systems / POV

Where Could Specialised Models Fit in an Agent Workflow?

Smaller, specialised AI models may improve specific steps in an agent workflow. Here is how I would evaluate their accuracy, cost, integration and long-term maintenance.

On this page
  1. Should a specialised model handle only certain steps?
  2. How would it fit into the existing harness?
  3. Do the economics work?
  4. Does every client need its own model?
  5. What happens when the workflow changes?
  6. Are we moving back to specialised models?

A smaller, specialised AI model may outperform a much larger general-purpose model on a particular task. That does not mean it should run an entire agent workflow. Does it improve the complete workflow, once we include accuracy, cost, integration and maintenance?

I recently spoke with a technology company founder in Silicon Valley to assess their products and explore whether there were opportunities for us to work together. The company trains AI models using reinforcement learning for agentic workflows.

Their argument, as I understood it, is that we may not need trillion-parameter models for every specialised task. A much smaller open-source model, with only a small fraction of its weights updated during training, could perform significantly better on a specific task than a much larger general-purpose model.

The founder described an example in which a model with a few billion parameters scored in the 90s on a task where a trillion-plus-parameter model scored in the 70s on the same data. The conversation made me think about using such a model in a real system.

Should a specialised model handle only certain steps?

In an agentic workflow, we do not call a model once and expect it to complete everything. One step may classify a request, another may retrieve information, and another may reason over the context or generate and review code. These steps may also interact with tools, databases and APIs.

The model matters, but the harness matters too. The result also depends on what context we provide, how we call tools, how we check outputs and what happens when a step fails. I look at the business implication of that layer in If Agent Skills Are Easy to Copy, Where Is the Moat?.

If a model has been trained for one step, should we use it only there and continue using a larger model elsewhere? The founder’s explanation was that it should improve on its trained tasks while retaining its previous behaviour on other tasks.

I would want to test that. Even if a smaller model improves substantially on one task, does it retain the capabilities needed for the rest of the workflow? I would evaluate the complete workflow, rather than only the step the model was trained for. We might get a better result by using specialised models for a few steps and a general-purpose model for the others.

How would it fit into the existing harness?

I work with coding environments and harnesses such as Claude Code, Codex and Cursor, along with open-source alternatives. If we introduce a specialised model, how easily can we use it? What would we need to change, and which capabilities would we lose or have to implement ourselves?

The founder mentioned that the company is also working on its own harness. That leaves me with another question: are we evaluating a model that fits into our existing architecture, or a complete stack that includes the model, training pipeline, inference infrastructure and harness? The answer affects both integration effort and the value we can attribute to the model itself.

Do the economics work?

The cost argument is the part I find most promising. The founder claimed that the smaller model could outperform a much larger one while delivering an approximately 5–10x economic advantage for the relevant task. The company also supports self-hosting, which could help enterprises that need to keep data within a controlled environment.

Those claims need to be tested against a specific workflow. Self-hosting may address a real requirement, but it does not automatically make a solution cheaper. Costs depend on request volume, quality requirements, infrastructure and the work needed to operate the model.

I would compare the cost per successful request across the whole workflow, at the quality we need.

Does every client need its own model?

If a model is trained using a particular client’s data and deployed only for that client’s workflow, we are effectively creating a specialised model for that environment.

That could make sense at high volume. If an enterprise processes hundreds of thousands or millions of similar requests each day, a small saving per successful request could add up. At lower volume, we still need to pay for training, evaluation, deployment and maintenance.

I would therefore look beyond request volume. How stable is the task? How often do its requirements change? How much does the model improve accuracy, and what does it cost to maintain that improvement?

What happens when the workflow changes?

I have run into a related problem in my own training work: forgetting.

I have trained more than 800 model variations for insurance-specific workflows, experimenting with different parameters, layers and training configurations. The models could perform very well on the data and tasks for which they were trained.

The challenge came when we wanted to introduce new information without losing previously learned capabilities. This is often called catastrophic forgetting. In my experiments, I could not overcome it to a satisfactory degree with the data and compute available to me.

After training, we still need to maintain the model as policies change, new cases appear and we discover exceptions. If we create a separate model for every client and workflow, we need a reliable way to update each one and verify that a new version is better overall.

Are we moving back to specialised models?

General-purpose models changed how we build AI applications. One model could write, summarise, classify, reason over information and help generate code across many use cases. Now, smaller models trained for specific tasks may offer a useful alternative where the task is predictable and the volume justifies the investment.

An agentic workflow can use both. Specialised models could handle selected, high-volume steps, while general-purpose models handle work that requires broader reasoning or encounters unfamiliar situations. The harness could choose between them based on the task, its risk and the quality required.

A harness routes predictable, high-volume steps to a specialised small model and broader or unfamiliar work to a general-purpose model, with risky or unfamiliar cases escalating.
An illustrative architecture, not a tested design: the harness chooses a model per step.

That is the architecture I would explore. Whether it is better depends on the results of a full-workflow evaluation, including how well the models perform as the workflow changes.