Fine-tuned models are winning on cost, latency, and quality at the same time

The production numbers are harder to dismiss than the theoretical argument. Kyle Corbitt puts the latency case plainly: fine-tuned models can reach roughly 30% of frontier model latency while delivering similar or higher quality, and cost per token improves by at least an order of magnitude, often more. Those are not edge-case results. They are what teams are reporting when they do the work of moving off frontier defaults.

Decagon is one of the cleaner examples. Jesse Zhang reports that 90% of Decagon’s workflow now runs on open-source models, and the results are what he describes as all three simultaneously: better on the task, cheaper, and faster. Abridge tells a similar story. Shiv Rao says roughly 40% of its model outputs already come from in-house models, with 60% expected next month after distilling and fine-tuning an open-source replacement for a frontier model. Speechify took the cost argument further still. Cliff Weitzman describes reaching single-digit dollars per million characters at a time when other models run anywhere from $30 to $100 for the same volume, a gap of roughly two orders of magnitude.

The economics behind these deployments have a clean explanation. Marc Andreessen puts the cost of distillation at approximately 2% of the original pre-training cost of a model. That figure reframes the build-versus-API question entirely. The capital barrier to standing up a task-specific model is far lower than most product teams have assumed.

We can typically get you know, latency down to about 30% of what you get from using a frontier model with again similar or usually higher quality than what you were getting from the frontier model. Kyle Corbitt

The pattern holds at the edge of the stack, not only in cloud inference. Joseph Nelson describes nano and pico vision models reaching 180-plus frames per second on a Jetson Nano with 4 GB of RAM. Swyx adds a useful calibration point at the other end of the size range: for small classification tasks, a fine-tuned 1-billion parameter model recovers approximately 95% of frontier model performance. Swyx’s team routes field-of-study classification queries to a self-hosted 800-million parameter model specifically because that task does not need a one-second frontier API call. Matei Zaharia describes open-source self-training pipelines where the same model generates training environments and trains itself, and beats frontier models at specific tasks.

Not everyone experiences the switch as frictionless. Yasser Elsaid pushes back on the popular claim that model switching costs are zero, noting that fine-tuning how a product behaves around a specific model can take three to four months. That is a real constraint, and it belongs in any honest accounting of the migration calculus. Tulsee Doshi adds another angle: users at scale will not tolerate latency even when a slower model offers substantially better quality. The implication cuts both ways. It is an argument against slower frontier models in production, and it is a reminder that the quality ceiling matters less than the latency floor once a product is live.

Andrew Lee offers a concrete illustration of how these tradeoffs play out at the decision layer. Tasklet chose not to ship what Lee referred to as “47” as a default recommended model, noting the cost increase relative to the benefit for iterative knowledge work. It will ship as an advanced user option. That is a deliberate product decision to route the common case away from the most capable model available.

Brendan Foody draws the trajectory out to its logical end, predicting that the majority of inference in five years will run on open-source, custom fine-tuned, or distilled models rather than frontier models. That forecast is grounded in the economics rather than in enthusiasm. Tuhin Srivastava, observing the customer base at Baseten, notes that no customer is running vanilla open-source weights without modifications. The starting point is already customization, not off-the-shelf frontier access. The frontier model remains useful, particularly as a data source: Ahmad Awais describes a common workflow in which practitioners use a high-quality frontier model to generate a taste file for a project and then switch to cheaper models for all subsequent development. Frontier models are becoming training infrastructure as much as they are runtime infrastructure. The teams that figure out where the line falls between those two roles will carry a structural cost advantage that compounds over time.